I hope you don’t mind if I quote an extended bit from that post about recurrent neural nets for generating text by character, because it gives me chills:
First, it’s fun to look at how the sampled text evolves while the model trains. For example, I trained an LSTM of Leo Tolstoy’s War and Peace and then generated samples every 100 iterations of training. At iteration 100 the model samples random jumbles:
tyntd-iafhatawiaoihrdemot lytdws e ,tfti, astai f ogoh eoase rrranbyne 'nhthnee e
plia tklrgd t o idoe ns,smtt h ne etie h,hregtrs nigtike,aoaenns lngHowever, notice that at least it is starting to get an idea about words separated by spaces. Except sometimes it inserts two spaces. It also doesn’t know that comma is amost always followed by a space. At 300 iterations we see that the model starts to get an idea about quotes and periods:
"Tmont thithey" fomesscerliund
Keushey. Thom here sheulke, anmerenith ol sivh I lalterthend Bleipile shuwy fil on aseterlome coaniogennc Phe lism thond hon at. MeiDimorotion in ther thize."The words are now also separated with spaces and the model starts to get the idea about periods at the end of a sentence. At iteration 500:
we counter. He stutn co des. His stanted out one ofler that concossions and was
to gearang reay Jotrets and with fre colt otf paitt thin wall. Which das stimnthe model has now learned to spell the shortest and most common words such as “we”, “He”, “His”, “Which”, “and”, etc. At iteration 700 we’re starting to see more and more English-like text emerge:
Aftair fall unsuch that the hall for Prince Velzonski's that me of
her hearly, and behs to so arwage fiving were to it beloge, pavu say falling misfort how, and Gogition is so overelical and ofter.At iteration 1200 we’re now seeing use of quotations and question/exclamation marks. Longer words have now been learned as well:
"Kite vouch!" he repeated by her
door. "But I would be done and quarts, feeling, then, son is people...."Until at last we start to get properly spelled words, quotations, names, and so on by about iteration 2000:
"Why do what that day," replied Natasha, and wishing to himself the fact the
princess, Princess Mary was easier, fed in had oftened him. Pierre aking his soul came to the packs and drove up his father-in-law women.The picture that emerges is that the model first discovers the general word-space structure and then rapidly starts to learn the words; First starting with the short words and then eventually the longer ones. Topics and themes that span multiple words (and in general longer-term dependencies) start to emerge only much later.
Aftair fall unsuch that the hall for Prince Velzonski’s that me of her hearly, and behs to so arwage fiving were to it beloge, pavu say falling misfort how, and Gogition is so overelical and ofter.
i’d say early-middle-english. 1200 to 1300 maybe.
Yes, that struck me too, and even earlier: ’“Tmont thithey” fomesscerliund Keushey. Thom here sheulke, anmerenith ol sivh I lalterthend Bleipile shuwy fil on aseterlome coaniogennc Phe lism thond hon at. MeiDimorotion in ther thize.“’ …struck me identifiably as something from the British Isles; you could tell me it was "West Cornish” or something and I’d totally believe you. @severnayazemlya is this just a function of the likelihood of certain letters following each other, or is there more to it?
What would happen if you iterated it a billion times? Would it eventually level off? Would it just give you War and Peace exactly as written? Or would it start understanding even higher level concepts like characters and themes?
Also, my new headcanon is that Northern Caves was the output of a neural net trained on the other Chesscourt books.