Next word prediction for reading

2026-09-13

Inspired by how language models are trained, I've started playing with pausing at the end of a line to try to guess the next word. This reading game is almost incomparable to model training in the sense that a model outputs a weighting for every token in its vocabulary, yet I feel that the game helps me take in information in a more complete way. If you learn a language as an adult, you have experienced the disconnect between comprehension and generation ability—you can know a word presented to you, but it's not available to you when you go to speak or write. My feeling is that trying to predict the word beyond the line break helps to make vocabulary and topics more available to me when I talk or write. When I guess the next word and then read on, my guess is almost always incorrect. Comparing the word to my guess doesn't seem to help me learn anything, I don't think; instead, it's just a game that reframes the process of reading. The exercise is interesting in that it makes the next-token prediction task for training language models even more fascinating. It's a beautiful, almost magical task. It is so simple to construct; it is not so impossible that models do not learn; and it leads to general rather than narrow learning.