Sentences are missing for archived texts, which messes up references. about lute

[jzohrab]

Yes, this will make the DB large. A few options:

• store the processed texts (with zero-width-space word delimiters) on disk in the file system, and read from there. Don't know what speed implications this would have. Could store an index in the db (wordrefs -> WoID, TxID), that table would have lots of records (same as book wordcount) but would be fine.
• have a separate "library" db just with the processed texts? Really, if I store things on disk, who cares whether it's in separate files, or in a single sqlite db? It's the same thing in the end. So the dbs would be: library with all texts, and lute, just with words and various other things for current active texts.
• have some clever ??? way to remove sentences that don't add any value (e.g., sentences where all words are well known, or sentences whose words are already contained by other more recent sentences).

[jzohrab]

Not fixing this -- it's a bug due to old code, life moves on, and things get better. We hope. Anyway, the way that text is stored/parsed needs to be revamped a bit, I think, as the db currently contains stuff that it really doesn't need (eg. texts.TxText seems redundant, when we have the Sentences table).