Need help with understanding JWPUB format about jwapi

[MrCyjaneK]

I think Content is encrypted with AES, maybe AES-256

This is the algorithm:

1. Determine the publication card hash
   
   1. Query the SQLite `Publication` table
   2. Create a list with the `MepsLanguageIndex`, `Symbol`, `Year` fields
   3. If the `IssueTagNumber` field is not zero, add it to the end of the list
   4. Join the list with underscores to one string, for example for w_S_202206.jwpub, this would be `1_w22_2022_20220600`
   5. Calculate the SHA 256 hash of that string
   6. Calculate the bitwise XOR with `11cbb5587e32846d4c26790c633da289f66fe5842a3a585ce1bc3a294af5ada7`
      [CyberChef example 1](https://gchq.github.io/CyberChef/#recipe=SHA2('256',64,160)From_Hex('Auto')XOR(%7B'option':'Hex','string':'11cbb5587e32846d4c26790c633da289f66fe5842a3a585ce1bc3a294af5ada7'%7D,'Standard',false)To_Hex('None',0)&input=MV93MjJfMjAyMl8yMDIyMDYwMA)

2. Decrypt the text
   
   1. Query a row from the `Document`, `BibleChapter` or `BibleVerse` table
   2. Read the encoded `Content` field
   3. Run AES-128-CBC, use the first 16 bytes of the hash as AES Key, and the last 16 bytes as Initialization Vector (IV)
   4. Run Zlib Inflate
      [CyberChef example 2](https://gchq.github.io/CyberChef/#recipe=AES_Decrypt(%7B'option':'Hex','string':'909fd5b41ddd8a75ac39c69604828a7d'%7D,%7B'option':'Hex','string':'3bc2c616d0ca2cff6dc4c0d7263a2327'%7D,'CBC','Hex','Raw',%7B'option':'Hex','string':''%7D,%7B'option':'Hex','string':''%7D)Zlib_Inflate(0,0,'Adaptive',false,false)&input=ZjZhZmMwMTEzZmRiMzY4MDE4ZmEzYmEwZDUwNjJlZWFmNGJlNzVhY2Q2NDJkNzM0YTQ2N2M2OTNjODIyMWM2NzM4ODMwYzE0NDQwMjVkZWQ2ZTZmNGZiNjBjZjgyYzcwYWUyYTY5M2EyYTg3NjQ5NWVlMGMxZTU5MDE1NDcyODM0OTMyMGI1OWY2NDAwNzRjMDgzMzYxN2RhYzI5N2ZjZmI1NTY4ODhhMDgzOTAyZGZhYTgyZDNmNmY1MjZhMjI2NTJkZTlhYmEzZjU4NjRkM2RlOTQzMGU2N2Q4NmViNzQwYmUzM2MyNmFjMWVjMmZmNDg0MGE1ODRkYjdhMmMyM2I3Nzc5Y2FlZDNhZGZhZjQ5MzJhZWJhMjM4MDUxNTgzNjRkMjBhNGIxMGU3Njg3YjU4NzBjNDMzZWZmOTk2ZmJjMGI5NGRkYzRkMTI0YjA0NjU2NTUxMjVkZGNmNzI4ZjNlYTgxMDk5YjZkNGM1YjM5NTczM2U5OTZjNjE0YzgyMTA4MTZkNDc3MzViMjI4YzU0MWJiNGJhMTBkMDBlODVkZmVlODlmN2YzOTQ0Y2NmOTk2YWJmMDJkY2QyZTdkMzJlOTU0MGY3ZTVkMDUzMjE2OWFjZTM5ODk5MWI2YzMzYTlmZDdhYmUzMjMyYTU4ODIwYWZhNDFmOTdmMTU2YWI4YTExMzEzMjI5MjhkMDNhOWM4OTE4NGEzOTE5YTQ3NGM2YjM1MWM4MDg2NDkwYWM0ZDYyYTI3MDU2YWMzN2M2NTFmZjYxYmExMmJmZDRhNzk0YmQxNjNlNmM3MjRhYmY2ZjMxMTc1OTQyMDE4OWM3NjNkZGJjMGQxMzQ2Y2Q1ZDUyYjJiNzExYTg1MGY0NWI3MTk5NDIzNDIwNDRhZDVhYjMwMmY1ZGQzMWJjZjlhMjcwZjM4YTlmNmJlMWFjZDlkYzMxYTczYjE1YzBlNDcwNmYyYThjNDE3NmFiNzg1ODI4NDMyMjUwNzM0MzIxMzY1MWRhMTk5ODg5NTMyYzJiOTQyYTkwOTRjMDVlOWY2OGRjM2ExNmU0NTUzNTE4MTA4MzQzODk2YmRhZTA2ZTVmYjIxZDYzMzRjYjk4NmVmMzlkN2Q1MzQ3YTQwNWU3YzkzMGYzYTVlN2U1ZWM1MWY5MDk4OTNkYjg1ZGZlZjUzYTM3ODBhZWQ3OGNkNjRjM2EzMTdjNWM5MDkwZjYyYjNhMGVkMjZiODBlYWMxNzMxMmEzMDkxMzA0MTIxYmM1Mzc4ZTU3MGQ5MjVjMjcwMmYwNmY3ZWJlNTYwNGRmNDRmMjQ0YjA5MjYxODkyYTE0OTIyYzkxMTBhZTIyY2FkMzg1NmM3MDQ1NDlmMmYwNTVmNGI1Mjg0MDQwYzk5Y2I4ZGY2Y2Q4N2Q5NjA1MmIzYTJjY2IxYmE1ZWMxNTMyOWY2ZThlNmM2NjhjMjkyMjViMWMxN2YxYTVhN2MzZmQ0YmVmODRiMzYyYjJkOTg3NGUyMTIyNDUzYzUyMDA3OTFkOTFhYmZjMzU0YzkxMWJlMTY4NmFmNmIzYTJmMjBjZTc2MzBlZjRiMzJjYWExYzcyZTA2NzhlNDcwMTk2NTYzZTZlNTgxZTNjYjAwOTRlOGYyMWIyZTUxZWZjYTNlNDdkZDExN2YzNGFmZWJhNmI2ODJjY2FkZjhkM2Q2ZjE5MDVlNzIxN2JkZTVjMTU3ZThiMmExOWYyYWFiZGEwZTM3OGZkODA3MjgwNGVmNWJhN2ZjMTQzOTg1NmFjNDVkYjY1MDY1OTBkMDI0Zjc5YjY0Njk1ZWJiYTYyN2U2YTdmOTkzYzZlMmY3NDdhZGQ0MmYyOTQyMGVjMzc5NmJjNWU5M2QzNzAwNTg0NzM4ZDI2YmE3ODUzZTVlNjgzMmU0YzQ5NDM1MGI5MThjMTlmZTI1MmY1)

Copying the notes just in here - I may revisit libjw soon

[darioragusa]

@MrCyjaneK I know you can edit the other formats without problems but the most of us use the JW Library app. I download a jwpub knowing that it comes from jw.org or the app and I trust the content. It's not a random txt file sent by a random guy opened with Word or Adobe Reader which may or may not contain the correct informations. An example: if I send to my grandma an EPUB she my be not able to open it but, if I send a jwpub she taps the file, a trusted app she always use show up and for her it's all ok: a normal article with the reliable content that is supposed to be there. A jwpub can still be edited but it's not a thing that anyone with basic knowledge of Word can do: less editors -> less edited files. Perhaps I'm totally wrong but those are my two cents.

[MrCyjaneK]

Felt cute, might delete later

[livrasand]

Perfect, thank you very much, I'll give your credit (although I don't know exactly how, maybe I'll mention your nickname), I'll notify you of Betel's response...

[orangethewell]

Sorry for the long wait, I have been working a lot these days and almost didn't have time to work with open witness library, but today I made a new commit where you can add publications within the program and see them separated on categories. I noted that some pubs like Index, Watchtower study magazines and the Bible still don't work well with my implementation, but when I have a free time, I will work better on that.

[MrCyjaneK]

I'll check the network thing tonight.. I'll download a publication and just watch for the traffic in burp suite, that should clarify if the requests are sent there or not.

[darioragusa]

Security? If anyone could get a publication and easily edit it the risk of spreading misleading information would be very high.

[gabriel-elesbao]

I'd like to be able to modify jwpub files so i can read other books during the meeting in the app, and no one would be the wiser. I've been sitting through close to 40 years of meetings, assemblies, KMS, etc. and I'm so sick of the same repetitive junk that I'd like to be able to read something interesting without the church lady sitting watching my every move realizing. Sounds like watchtower has prevented this from being a plausible way of making meetings bearable. Not surprised.

if you go to 40 years and are dissatisfied, simple, ask to leave, no one is obliged

[orangethewell]

Some news from my rusty implementation, lol:

• I got a pub menu selector with thumbnails working after a long time fixing problems using an ui library that I'm used to;
• For some reason, my decoding stopped working; (???)

anyways, I got energy again to this project after my ui library working good again with me, any news I got, I'll post there

[MrCyjaneK]

Good job @orangethewell. Hit us with results once ready ;)

[livrasand]

anyone know where is the comment with someone pointed every category found on jwpub files? Like books (bk), watchtowers (wt), etc.

Yes, in fact I have it and I published it a long time ago. I'll just leave it here in case it helps someone...

Publication Category	Publication Category Symbol	Traducción
Bibles	bi	Biblias
Insight	it	Perspicacia
Index	dx	Índice
Watchtower	w	Atalaya
Awake	g	Despertad
Kingdom Ministry	km	Nuestro Ministerio del Reino
Books	bk	Libros
YearBooks	yb	Anuarios
Brochures	brch	Folletos
Tracts	trct	Tratados
Program	pgm	Programas
Meeting WorkBook	mwb	Guía de Actividades
Manual/Guidelines	manual	Pautas y Manuales
Talk	talk	Bosquejos/Discursos
Letter	letter	Cartas
Web	web	Artículo de JW.ORG

[MrCyjaneK]

personally I would love it if you could develop an application on your own, I and more than a thousand people would use it, a light, fast and open source app, it could be developed in Go, and maybe use a Fyne interface or something similar. Many devices no longer work with the most recent version of JW Library, imagine how many users will benefit from your app, an app just to read JWPUB's and maybe play videos and images (JWPUB Reader or JWPUB Library, just a crazy name), I think it would not be of your valuable time wasted. You would make it work much better and many low-income brothers continue to use their same old devices.

Well I can't do that fully but if I'd get some help (eg the frontend job done) - I most likely can provide you with with help in doing so - I can even create a simple project in flutter if I get some free time.

You know how the JW Library works, use a web browser to display the content, and you have the keys to decrypt the JWPUB's. Believe me it wouldn't be a waste of time. You can get the official JW Library catalog with the API, and thus have its publications in your app. The languages you can get from JW.ORG too.

Yeah - hence in this repository we have a somewhat working app

I encourage you to make it happen friend. Do you remember the letter I sent to Bethel with my circuit overseer's account? No response either, LOL.

insert joke about ransomware groups having better support than them here

[MrCyjaneK]

oh the builds expired long time ago because I've switched CI. I have somewhere source for a flutter version of jw library.. but no jwpub magic there yet... sadly.

[orangethewell]

@livrasand As I stated before on this discussion, open-witness-library is progressing well. Not quite well as JWApi has already got but its working with some tricky config. Since I do it on my free time, I could not get some of the most important features working, but I'm still active with it and soon will pull a commit with a nice way to read the pubs. (Since you have to hard add it on local data directory for now)

[MrCyjaneK]

@orangethewell is jwpub decryption working for new publications for you?

[orangethewell]

@MrCyjaneK Uhh I think it's still working. For the most pubs that I tested, none of them had any problems (lff too). But maybe I should test it again soon

EDIT: Yeah, its working, tested CA-brpgm24 (Circuit Assembly program for 2023-2024)

[MrCyjaneK]

I will see how I can make it possible for it to be totally free, and at the same time pay for the server and the domain.

In fact, that is the only premise, paying those expenses

1. you don't need domain
2. you don't need the server.

Either you are being dishonest with us or have developed something that creates more problems than it fixes. We don't need yet another JW Library app that have vendor lock in - just moved to some other community based server that is paid. So I'd rather use official app than your service that depends on jw services under the hood (probably).

That being said, @livrasand, - what exactly are you doing on the backend that can't be done on the frontend of your app?

and if you trully need hosting I can provide you with one as long as you open source the entire app (including server) (and translate readme to english) (and make it free to use for all).

[MrCyjaneK]

So my current plan is to:

Make JWPUB format more usable, I'm using FOSSJWPUB as the draft name (.fossjwpub extension), which is adding as few changes as possible but makes it much easier to use.

• simple sqlite database, no double unzipping to get into the contents of the file
• no encryption - all fields are decrypted
• no compression - all fields are uncompressed
• embed all resources into the sqlite database in a new table (_FOSSJWPUB_assets), so no trickery will be required when working with the files.
  Lastly, compress the entire sqlite database, this will allow to benefit from better compression (as can be seen in example) and much less work to just get to the publication content.

As can be seen the new format is a bit heavier without compression (while jwpub is literally a .zip file, with .zip file inside with zlib content in the database, so I wouldn't consider that uncompressed...), but compresses to a much smaller (~19.8% size reduction, without losing any content) document.

➜  libjw ls nwtsty_E.* -lah
-rw-r--r-- 1 user user 160M Nov 16 18:27 nwtsty_E.fossjwpub
-rw-r--r-- 1 user user  93M Nov 16 18:27 nwtsty_E.fossjwpub.xz
-rw-r--r-- 1 user user 116M Nov 15 15:45 nwtsty_E.jwpub
-rw-r--r-- 1 user user 106M Nov 15 15:45 nwtsty_E.jwpub.xz

I still do not intend to support jwpub/fossjwpub as it is in any reading app, but making something similar to jwpub that can be easily worked on (It's just SQL) seems like a pretty obvious task.

My plan is to create a script to convert all the publications into a fully-featured markdown documents, which later I'll use together with a thing like hugo (especially hugo-book or https://github.com/weitblick/epub) to put the code into something actually readable by 3rd party software (as I'm really pissed off by not providing any useful way of using the content outside of app/website).

And 2nd task, that I may or may not finish (depending entirely on you guys, if somebody want to use the code chances are that I'll be much more motivated to actually do it) is to create a cross-platform library app (most likely using fyne.io toolkit), that will use generated earlier markdown documents to display the content.

Also few changes from my side are comming to the main website.. but that's not part of the issue.

[MrCyjaneK]

Everything is working as intended on my end, but I honestly didn't expect this much changes to happen to the catalog.db, I was hoping that a little php script will be able to keep everything up-to-date but it looks like a no-go because it is simply too slow to process all the data (before it finishes updating the MySQL table we already have a new revision available).

I'm rewriting a bit of the code in Go, so it will be a bit more multi-threaded and (I hope) will be able to handle everything faster.

edit: This is a cool learning experience for me, I've never worked with databases that big in terms of schema, and I learn a lot about performance.

[orangethewell]

I've been using a lot ChatGPT to translate publication styles into CSS, there are two ways common to handle these styles:

• Compile in runtime with some tailwind like code compiler
• Just write everything on hand in a single stylesheet.css

After all, handling these publication styles is a big mess, I tried importing JW.ORG styles and switching prefixes, but it seems the result didn't get very well as should be.

I've been kinda busy these days, I haven't touched my implementation for weeks or months. But I had a idea like you said, implementing a local server for jwpub archives and sharing it over LAN, but not planned as a main feature, more like a plugin.

[orangethewell]

Hi! I had this idea (scrapping jwpub files) somedays ago and was searching for anything about these JW Library files. Appearly, these files have some linking directly with jw.org, but even then, I don't got anything about how this linking works. By the way, I was thinking that bytecode should be a id from words table too, but I also don't think this is a directly id, maybe have some instructions that JW made for it.

You're a Jehovah Witness?

[MrCyjaneK]

I even sent a couple of emails with a request for documentation, but got a response that said that they are unable to answer my question from this email address. So my idea was to call the number from https://www.jw.org/en/jehovahs-witnesses/contact/united-states/, but recently I didn't had much time, so I didn't do that.

And yes, I am

[MrCyjaneK]

I've put a lot of time into understanding this format, but still no results worth showing.

It's sad that most of the new publications are PDF/JWPUB only, PDF just doesn't scale well, and JWPUB is ugh.,

I still have an idea - scraping wol.jw.org but I'm against sending hundreds/thousands of request (every image, article, quote, source) to get one publication.

[orangethewell]

Haha, I don't think they would simply give us their code, sadly. Anyways, scrapping wol.jw.org would actually works, but it's the worst idea considering that we should add a lot more code and change almost everything. (Considering too this will make the project a lot more heavy for low-end systems)

After all, all we can do is trial and error. We have at least a hint, that files works like a Epub, with XML files inside it, the difference it's hard modified and for some reason there's binary code that isn't a match with a list of words.

I will try doing something with my knowledge with Python, I don't know that I will help in something, but at least I will try for fun. I really like the fact to use JW Library in PC, and it's sad that Watchtower don't have ported it to some Linux distro. I don't think it will go for a long time, maybe some day they release a version for a famous Linux distro.

A final question... I saw with your project that you use the app API from JW but, this is allowed? Isn't a violation from some of the App's terms of use?

[orangethewell]

Hello! So, I made some experiencies with the JWPUB file to know how it works and I think I got some hot things working! First of all, content is directly related with the page and don't accept something new in (maybe because content have a fixed size bytes and I inserted more than that? I don't know). Furthermore, the Words table don't work the way we thought, I changed a word in this table and all I got is the way I find it on the book, now I need to search by "subjecters" instead "subject", and after all, the word in the documents keep the same.

So, after all, I got a "How to Remain in God's Love" Book with the subjects section with title "Edited Subjects" and a blank "Letter from the Governing Body".

EDIT: I read the documentation that you gave, maybe the begin and end can be the initial byte and the final byte to be converted, but there's the question: Converted in what if it's not an index from words table?

[MrCyjaneK]

Wow! That's great! I lost so much time with the Words table.. So you are saying that Contents is directly related to the content? Not just reference the Words?

Have you seen things below the sentence Huh It's quite short. in the docs? https://raw.githubusercontent.com/MrCyjaneK/jwapi/master/docs/jwpub/index.md

You also need to have there:

• Some heading/subheadings/fonts etc...
• Images (probably by ID)
• Links to other publications
• And the content itself

What is the news from God? translates to:

Decimal	1246 616 1131 758 474 499
Hex	4de 268 46b 2f6 1da 1f3

Which is quite short, so my guess was that it use Words table. Maybe it store rendered publications somewhere in cache, that's why changing the table didn't change the content?

Or another scenario the Contents is compressed in some way..

[orangethewell]

Okayyy I think I got a problem with the customized JWPUB and I don't know what exactly was charging it.

I saw what was in the jwpub converting doc before and yeah, it could be it but... There's something strange with it and I don't know what exactly happened.

I changed a lot of things in the original db because I thought I was compacting it with a new jwpub file with my code but no, and when I fixed that, I had changed a lot in the DB and I think I got a corrupted publication (Or modified so long that it's don't load anything). Remembering that I changed just one content column. But this is really strange, I didn't saw that yesterday but even then it's strange how it's going.

After all, there's a lot of things working behind the jwpub specifications, there's even a schema specification for publication view and, with words table, there's some strange tables that's is like a pre compiled search. I'm really thinking about what some of a reading program forum responded to a request to create a support for the JW files, they said these files have requests for the JW API. I don't trust in everything, but this really was stuck in my mind, but even then doesn't make any sense, why a 100mb or + will need from JW? And if it's, how the pioneers book are distributed?

[MrCyjaneK]

So first of all, I had some problems with android studio, then it was just late and I forgot to reply.
After downloading publications there were no requests (execept for few images, that were unrelated to the publication)

[MrCyjaneK]

Haha, I don't think they would simply give us their code, sadly. Anyways, scrapping wol.jw.org would actually works, but it's the worst idea considering that we should add a lot more code and change almost everything. (Considering too this will make the project a lot more heavy for low-end systems)

Yea.. but if we fail that's the only option.

After all, all we can do is trial and error. We have at least a hint, that files works like a Epub, with XML files inside it, the difference it's hard modified and for some reason there's binary code that isn't a match with a list of words.

Not really - it can be converted on the go, and then just kept in some html format.

I will try doing something with my knowledge with Python, I don't know that I will help in something, but at least I will try for fun. I really like the fact to use JW Library in PC, and it's sad that Watchtower don't have ported it to some Linux distro. I don't think it will go for a long time, maybe some day they release a version for a famous Linux distro.

That's sad :( I wish that there would be a decent watchtower library app made with gtk ;p

A final question... I saw with your project that you use the app API from JW but, this is allowed? Isn't a violation from some of the App's terms of use?

Since I don't reupload the content, it is legal, but I'm not a lawyer

https://www.jw.org/en/terms-of-use/

and even if it's against the terms.. sigh. I'm not switching back to android, so I'll continue to develop this app.

(sorry for late reply.. I missed this comment)

[orangethewell]

No no! It's okay, brother! I too don't have so much time for searching more these days, after all, I'm still have 15 years old and have some homework to do here for school. ^^

But I will still following the project flow, if I can get something new here, I make a new response on this issue.

And if you can't got any new thing from the JWPUB convertion, you still have a more easy task to do, like the video player :) (I really like the way the PC JW Library app can be easily "hacked" to have a new video on, lol)

[MrCyjaneK]

After spending hours on this thing, I'll not continue to reverse engineer the JWPUB format, until somebody do that.. for me.

For now I'll try to move to flooding wol.jw.org apis and getting the publications page by page (thanks for abandoning epub btw).

[MrCyjaneK]

Ignore what this weirdo said.

It's here: https://github.com/Miaosi001/JW-Library-macOS/blob/main/JWLibrary/SubViews/PubbView.swift

[MrCyjaneK]

https://github.com/Miaosi001/JW-Library-macOS/blob/main/JWLibrary/Robe/AppuntiDel2019.png

[mjacobus]

@MrCyjaneK did you figure out how to read Document.Content?

[MrCyjaneK]

@mjacobus darioragusa/JW-Library-macOS#1 (comment)

[MrCyjaneK]

I'm not working on this app anymore, spending time on open source alternative to something that is clearly using DRM when it shouldn't (can somebody give me one single reason for which it is worth to encrypt such content when it is freely available?)
Also I don't feel like playing some sort of cat and mouse thing when somebody can just change the way api sends publications and cut support for earlier versions.

And the elephant in the room. WHY isn't the app open source in the first place?

Until somebody gives me answers to that questions I'm not going to work on this project. wol.jw.org is enough for me.

</project>

[MrCyjaneK]

@darioragusa As they can do with .epub, .mobi, and .pdf.

Also there is a tool for that used widely in the internet, you can sign things with PGP that would allow 3rd party apps to be developed and would cause less risk (currently we can edit the publications - drm is defectivebydesign.org).

[MrCyjaneK]

less editors -> less edited files. Perhaps I'm totally wrong but those are my two cents.

The thing is current method allows editing, and signing would make it impossible while allowing moders like us to easily read the content

[darioragusa]

I don't know much about signing files, but I guess that the app should have a key and using this key with (something, idk) they should get a value. It's like checking the hash? If a bit changes the value is different?

[MrCyjaneK]

It's like checking if the content was modified, the content can be signed to verify that it was created by somebody and after modding it the signature will not match. It's like encrypting but you can see the content and can't modify it.

[darioragusa]

Ok, but this way they shouldn't save the signatures for every version of every article in every publication in every language?

[MrCyjaneK]

pgp signatures do not add a lot of extra size to publication so I don't consider this a problem. (hence you could sign a sha512sum of publication and get similar result) + you can sign them as they are served to download.

[darioragusa]

If the signature is stored with the publication what stops me to change it?

[MrCyjaneK]

You can change it - you can even sign it with your key but it will be invalid

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

This is a message I have decided to sign, try to mess with it and it will no longer be signed.
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCAAdFiEE0gTaRRUXZfyrr8PQPD6SA9PleeEFAmO36L8ACgkQPD6SA9Pl
eeERiAv+MXm2VjIZMvOgXwKT5bDmwMpfK8liOdT/IhoFvNsTwMiWUQRHzp12OJtz
U+V26gq6lmBJsKsyij6AAvefy048mAzGnAMRR5c9uqkYs2R66jqUIRNERCE2XKdu
uiJAhmpMqNughA0/h19/As1xCrepZpo+W1SEE8yEPZp13eZ0gylmS0pBqXR5QcHB
JNAIMV84xOAntQNe2dzs6lBhhWdF3EvE5L50so2EiXGulr5mIdPwIkaCUSQIZYRd
2aWLwcA4j8ZN/UfY6YbCyhSyH5Fm4WXZ17tsPSuOqBE7QhW100gPiQjPDGc5ZUwN
SjvRIxrvCZ9rPg/PQnAOIgxALilBW3y6Jaq73XTBFaOArkOxmWh8rFhL7OkMdyW7
ewpAVjU90ChYEJ17BZpM+cSYIYcRwsYdtNQcQVl1fViBFlBFY1PEm4mvbbHK4GLQ
aRBnSsTbabNQQLij3hk/Wc9RLEe49pk/tmeDlqrtF5ELbFWRBtM0R63H3qfXkEL7
vFqGsJB4
=gBZ4
-----END PGP SIGNATURE-----

[MrCyjaneK]

[darioragusa]

Rephrasing: what stops me to change it and sign it again with their key? Like how we got this value 11cbb5587e32846d4c26790c633da289f66fe5842a3a585ce1bc3a294af5ada7, someone will leak the key used to sign.

[MrCyjaneK]

The fact that public key used to verify is available to as and private key is being kept secret offline and used to sign publications, you can't sign something with public key

[darioragusa]

Oh, thanks, got it. The only thing I can think of is that I don't know how feasible it is to keep it offline, but yes, in theory it would be better.

[MrCyjaneK]

It works just fine offline - and it's commonly used in all package managers on linux that I have used, and to provide live iso's of systems.

anyway I just wish for a simple docs of that format.. or not abandoning good old .epub at least...

[orangethewell]

I've almost got a fully working reader for JWPUB format, or at least what you guys had got. There are just two issues that stop me from publishing it:
1 - Legal issues with jw.org terms of service (After all, using the key they got is basically reverse engineering and after recent updates of terms of service of JW Library, they said that we can't use part of app code)
Note about this: There's a jw developer support mail, but I couldn't get any response from them
2 - I just didn't finished the project at all, but the base code for decoding the content is working and it's included on my project made in Rust

Even then, my use would be more personal, so until I post anything on internet, there's no legal issues after me. Or, maybe I can post the production program, without infringing any legal rights, it's just a reader, don't make any access to the official site API after all

[MrCyjaneK]

@orangethewell ianal and this is not a legal advice BUT go ahead and publish it if you want and feel free to link it here.

I'm not working on my project and not using official apps for reasons stated above #1 (comment), but if some open source alternative would appear I'd be very happy to use it.

[Worthington412]

I'd like to be able to modify jwpub files so i can read other books during the meeting in the app, and no one would be the wiser. I've been sitting through close to 40 years of meetings, assemblies, KMS, etc. and I'm so sick of the same repetitive junk that I'd like to be able to read something interesting without the church lady sitting watching my every move realizing. Sounds like watchtower has prevented this from being a plausible way of making meetings bearable. Not surprised.

[orangethewell]

I published Open Witness Library on a public repo. Isn't 100% working, but the base thing is working with images and a document summary. You can track the project development here. :)

OBS: Sorry, by a mistake I made, it's only reading PT-BR pubs ("T"-tagged jwpubs), I will send a commit to fix it fast as possible.

[MrCyjaneK]

@orangethewell that is awesome! I'll check it out when I'll have some time

[livrasand]

@MrCyjaneK Do you think PGP works? where could it be embedded? in the manifest.json? I like PGP

[MrCyjaneK]

It could be embedded in app. Hence I could even implement it if I got access to app source and the backend.
But no chances for that I guess.

[livrasand]

I am currently in communication with Bethel, in the company of my circuit overseer, I would like to comment on what you have said, if you want and are willing, I can give you the credit, but could you give me more details on how you would use the PGP?

If you don't feel like it, or don't want to, don't worry, I understand.

[MrCyjaneK]

@livrasand

1. Get rid of the pointless DRM - encryption that is currently in the JWPUB format
2. Sign the files using PGP key (private key on server that signs all publications)
3. Embed that publickey in JW Library app
4. Verify upon download if JWPUB downloaded matches the signature

Also, if you are that far (I tried contacting someone responsible for the app for 2 years to say that there was a RCE issue in the JW Library app and failed to reach anybody who would care about it), consider asking then about opensourcing the app. There is nothing to lose since we don't make profit on the app and I'd really like to patch some features and add some modifications to the app functionality. Keep me updated on how it goes.

[orangethewell]

anyone know where is the comment with someone pointed every category found on jwpub files? Like books (bk), watchtowers (wt), etc.

[arthurwweber]

After doing some digging into this matter there are a few observations I wish to make:

• the Contents field in the publications is the HTML markup of the given document (article) or Bible verse, encrypted using AES (i.e. symmetrical encryption using a key and an initial vector) and then compressed using the standard Deflate algorithm. Early versions of JW Library were not as obfuscated as the current ones, and you could easily generate the key and the IV based off a constant byte array defined in the code, the SHA256 signature of "{meps_language_id}{pub_symbol}{year}" and some XOR operations between pairs of bytes. Later versions of the MEPS platform changed the initial byte array and new JWPUB files were generated to reflect the new encryption parameters. If you try decompiling the latest version of JW Library, all identifiers are obfuscated and it's much more challenging to parse through the code.
• Publications are not signed. The only thing where PKI-certified communication takes place is when the app downloads or streams media from JW.org and its CDN. In addition, the app itself is signed. All the publications have for the app to check their integrity is the SHA256 hash of the payload in the ZIP container, which is stored in manifest.json.
• In conclusion, technically there is the possibility to create custom JWPub files and successfully open them in JW Library.

[MrCyjaneK]

Friendly reminder that security by obscurity is not security.

Not to mention that JW Library as an app is just a terrible piece of code that tends to run slow and register many handlers for different kind of files.
I'd honestly develop a app on my own but with the attitude from the JW Dev team I feel that it will be a waste of my precious time. Instead I'm just using wol.jw.org on my phone (since the official app won't run, and I can't create my own).

And as I've said in some other issue - there is no point in me spending time on reverse engeenering this abomination with jwpub extension.

Being hostile towards other developers is a terrible thing to do, and it honestly feels wrong to encrypt JWPUB, why would you force people to use your own baddly.

If you try decompiling the latest version of JW Library, all identifiers are obfuscated and it's much more challenging to parse through the code.

For that exact reason I'm considering JW Library app a malware (hence it malfunctions frequently), why would you make it so hard for a group of a few people who just want to read the publications in their own way? What is in there that you want to hide? Why don't you simply provide an up-to-date .epub on the site?

All the publications have for the app to check their integrity is the SHA256 hash of the payload in the ZIP container, which is stored in manifest.json.

So basically they store information about if the file is correct inside of that file? Lol.

[MrCyjaneK]

also it's been more then 2 years since I've tried to report a security issue in JW Library. So yea I ain't using that.

Not to mention that latest bible is available only through the jwpub format - together with other books.

I just hate the decisions that developers in the app dev team make. Like why not use widevine and play integrity in the app? And web integrity api in the website? After that it will be top 1 most unaccessible website for people who make it better.

So one last thing - if you don't deliver a good product (which JW Library app is not) then at least don't focus on making it harder for people to fix what you delivered broken.

disgusting.

[livrasand]

@MrCyjaneK, personally I would love it if you could develop an application on your own, I and more than a thousand people would use it, a light, fast and open source app, it could be developed in Go, and maybe use a Fyne interface or something similar. Many devices no longer work with the most recent version of JW Library, imagine how many users will benefit from your app, an app just to read JWPUB's and maybe play videos and images (JWPUB Reader or JWPUB Library, just a crazy name), I think it would not be of your valuable time wasted. You would make it work much better and many low-income brothers continue to use their same old devices.

You know how the JW Library works, use a web browser to display the content, and you have the keys to decrypt the JWPUB's. Believe me it wouldn't be a waste of time. You can get the official JW Library catalog with the API, and thus have its publications in your app. The languages you can get from JW.ORG too.

I would help make that project a reality. Many, believe me, many would use it as an alternative, even some, came to prefer it.

I encourage you to make it happen friend. Do you remember the letter I sent to Bethel with my circuit overseer's account? No response either, LOL.

[arthurwweber]

Friendly reminder that security by obscurity is not security.

True, however this practice was not invented by the MEPS Programming department at WHQ. Obfuscation of code and/or content has always been a go-to step when one wanted to give "muggles" a hard time deciphering stuff.

register many handlers for different kind of files.

Using customized URI handlers is also an expected approach. Let's not forget that all these different formats for the publications (high resolution PostScript/PDF for print, low resolution PDF for web publishing, HTML on JW.org, HTML on WOL, ePub, JWPub, WTLIB Pub) are being produced from the same source files by the different modules of the MEPS suite (which is why P no longer stands for Phototypesetting, but Publishing). In order for the hyperlinks to work properly inside the app, those link will have to use an internal URI schema.

Being hostile towards other developers is a terrible thing to do, and it honestly feels wrong to encrypt JWPUB, why would you force people to use your own badly.

This is also sort of expected. Remember that JWPub is a proprietary format designed by the MEPS Programming department (under the direction of the Writing Committee of the Governing Body). Historically brothers at Bethel are not allowed to ask for outside input, and the Governing Body has repeatedly been very reluctant toward random unsolicited feedback from the field.

why would you make it so hard for a group of a few people who just want to read the publications in their own way?

Because they want to keep things official. Because they want to enforce their right of ownership. And they want to prevent unauthorized individuals to abuse of the content or otherwise tamper with the official way of doing things.

What is in there that you want to hide? Why don't you simply provide an up-to-date .epub on the site?

They have nothing to hide per se. The Study Bible is not a complete publication, so generating an .epub is probably not an option. It's been a pattern with publications that are work in progress to be released incrementally as JWPub's (e.g. the New World Translation in languages where books are partially translated, the Insight volumes in Romanian)

So basically they store information about if the file is correct inside of that file? Lol.

That isn't new either. You are surely aware that ZIP files for example also have a CRC field that unpacking checks to make sure that the integrity of the file has been preserved over time and successive transfer operations.

[MrCyjaneK]

@arthurwweber

Using customized URI handlers is also an expected approach. Let's not forget that all these different formats for the publications (high resolution PostScript/PDF for print, low resolution PDF for web publishing, HTML on JW.org, HTML on WOL, ePub, JWPub, WTLIB Pub) are being produced from the same source files by the different modules of the MEPS suite (which is why P no longer stands for Phototypesetting, but Publishing). In order for the hyperlinks to work properly inside the app, those link will have to use an internal URI schema.

so enlighten me why text/vcard is being used by JW Library to hook (actually it is */* if I'm still correct). Also - I didn't use the app in 3 years or so, but afaik opening links in jw.org just opens web browser not web browser in app + you can't simply open pdf in the library. Have you used JW Library app?
Honesty - I couldn't findy any extension that would actually do something else than start the application except for the backup file. So you clearly implemented that incorrectly - because even if PDF opening was intended feature of JW Library app then it simply doesn't work (and I'm getting tired of helping people to unset JW Library as their default app.

That isn't new either. You are surely aware that ZIP files for example also have a CRC field that unpacking checks to make sure that the integrity of the file has been preserved over time and successive transfer operations.

Oh of course I'm aware of this - just not seeing the point of that - I'd rather recommend some kind of signing - as used in .apk files.

They have nothing to hide per se. The Study Bible is not a complete publication, so generating an .epub is probably not an option. It's been a pattern with publications that are work in progress to be released incrementally as JWPub's (e.g. the New World Translation in languages where books are partially translated, the Insight volumes in Romanian)

So in fact people without official glowing in the dark JW Library app can't read them.

This is also sort of expected. Remember that JWPub is a proprietary format designed by the MEPS Programming department (under the direction of the Writing Committee of the Governing Body).

I understand that jwpub offers some unique features but as far as I know there is no publication that exists in JW Library that wouldn't be available under wol.jw.org - which means that all jwpub content can be displayed as html (and it is either webview or richtextview of some kind in the app I guess) WHICH MEANS that you can deliver .epub with the exact same feature set, and ship a manifest.json in the epub file itself - without losing any metadata.

I get that they have decided to reinvent a wheel (for sure without looking at any existing wheels that already work and are widely used (some even support DRM!)), but the wheen they have designed is just a terrible piece of code - have you tried opening a publication on a device older than 5 years? (In comparision my library that is in this repository worked flawlessly on a number of phones without any lag - despite being my learning project with many bad decisions made on the way.)

Historically brothers at Bethel are not allowed to ask for outside input, and the Governing Body has repeatedly been very reluctant toward random unsolicited feedback from the field.

Oh I know that. My security bug report didn't receive any attention since well.. 2020? This is violating google play policy afaik. If you provide a contact email it must be a contact email, not a echo service (which is good, I use it to test my email servers).

Also, I'd prefer to not discuss religious topics in here - let's just focus on the abomination that was created in the MEPS and is available to download freely on the internet.

[livrasand]

I tried to try JWapi, but I ran into several problems, your website gives me problems with SSL, I access it still and the page stays in an infinite load, until it gives as error 504. I tried to download the binary version Lorca for Windows or for Kali and neither. Do you need help with your SSL? Maybe you could try a free one, for example, ZeroSSL.com.

I'll try to work on an app in Go that works as a JWPUB's reader, maybe I'll be successful. For iOS and Android (old devices will surely appreciate it).

I'm no Go expert, but I've replicated some things and tried others, and so far they've worked for me. If you don't mind, maybe he'll ask you for help in your spare time. I'll also try using JWapi again, and maybe replicate what you already do.

[livrasand]

Oh excellent, do you need help with something? I would love to support.

[MrCyjaneK]

I (personally) think that the best thing to do right now is to just decrypt the jwpub, parse it into something more usable (I'd personally go for epub) and then fork some already existing epub reader to give it look and feel of JW Library app

[MrCyjaneK]

Also did any of you try to work with the watchtower library? It contains all the data but.. Well it's a mix of xml and binary blobs that binwalk doesn't recognize at all

[orangethewell]

I (personally) think that the best thing to do right now is to just decrypt the jwpub, parse it into something more usable (I'd personally go for epub) and then fork some already existing epub reader to give it look and feel of JW Library app

It isn't my objective, but for sure I want to make a way of people make it possible. Like creating a plugin system for Open Witness Library that people can add their own plugins for do stuff they need (Assignments, turn the program into a home server, exporting stuff, etc.)

Real yeah, it would be kinda cool be able to export publication content in any type!

[livrasand]

Maybe something like this:

https://www.npmjs.com/package/jw-epub-parser
https://github.com/sws2apps/jw-epub-parser

You can take the idea of working to do it with the JWPUB. A few days ago I found an open door at JW.ORG. Which gave me access to a training website for Betelitas or for members who will work with the branch (I have also reported it and still no response).

There I found how MEPS works, I share what I found, maybe it will help you.

The first course, is a Introduction
This course provides you with a brief introduction of Digital Publishing. The goal of this course is to ensure you have a complete picture of the importance of your role and how it affects the publications.

Unit 1: Introduction to MEPS
The work as a Digital Publisher requires some knowledge of MEPS. MEPS contains a lot of features, but in this lesson only the things that are necessary for a Digital Publisher to know will be discussed.

Unit 2: Format a Publication
Before a publication is added to a research library it must be formatted. Formatting involves adding necessary information to a MEPS document to control its appearance and how the information is presented to readers.

Unit 3: Create a Research Library
In this unit, learn to build your own Bethel Family (BF) library, and index a publication in it. Improve the appearance of the publication, and ensure that the library is free from errors. Once a BF library for a new language has been set up, WTS will help you continue to add new publications to it and create digital publications from it. The exercises in each lesson must be finished in order to continue the exercises in the next lesson.

Unit 4: Using WTS to Produce Digital Publications
Now that you have some basic knowledge about MEPS and a working Research Library, you are ready to produce digital publications. Digital publications are produced after the publication is sent for printing. The digital publications you can read on jw.org are called WPUBs. The digital publications that can be downloaded from jw.org are EPUB files. EPUBs can be read on tablets and other electronic readers. Finally, there are JWPUB files that can be read on the JW Library app.

The production of digital publications could be done manually using the tools that are available in the MEPS Research Suite. However, this involves many manual steps, and errors could be introduced easily if a step is omitted. To make it easier and to improve the quality, the production is now integrated in the WTS processes.

In this lesson we will follow the WTS "Trans-Comp-Audio Prep-Research Format" (TCAR) and "Audio Record-Index-Digital Publishing (Print)" (AIDP) processes. Since documentation already exists for these processes, we will refer to them. Every activity in this process will not be discussed since many of them are very simple. Open the process page and under the heading "General" chose one of these processes. There you will be able to find information about each activity in the process.

Unit 5: Maintaining Research Libraries
Research libraries need to be keept up to date. Every month new publications are released and they need to be added to the research libraries. In this unit we will discuss what libraries we are working with and how to make the updates available to the Bethel family.

I haven't been able to get into any so far, maybe you @MrCyjaneK could make it.

[arthurwweber]

@orangethewell Can you please update me on the decryption algorithm since it's working for you?

[orangethewell]

Yeah, here: darioragusa/JW-Library-macOS#1 (comment)

[MrCyjaneK]

@livrasand no. You made it paid. + I don't understand a single thing from readme (except for currency indicator).

[orangethewell]

@livrasand I can't accept it too, even if it's open source. We learn so much of not to make commerce inside congregation and I can't see it in a nice way.

Even though I have a lot of work with the Open Witness Library, I don't plan making any profit with that, it will be free and open source, since part of the code isn't mine.

[MrCyjaneK]

as I've already said - the way to go (imo) is to prepare a script to convert JWPUB into epub and use some already existing epub reader to give it nice looking skin. Or hence even roll our own reader since epub is html in a zip.

[livrasand]

I thank you very much friends for your comments, I will take them into account, I will see how I can make it possible for it to be totally free, and at the same time pay for the server and the domain. In fact, that is the only premise, paying those expenses, out there would be totally free. I need to update the website, but KHA itself is free to some extent, and I plan to make it completely free.

Thanks again, and I will put your comments into practice.

[livrasand]

Clear @MrCyjaneK! That's how it will be, I'll make the changes in the next few days. Thank you very much 👍🏻 the app does not use a server or domain, my website does, goattendant.com.

[MrCyjaneK]

https://pages.github.com/ @livrasand this is where you can host the site. Also does your app use JWPUB?

[livrasand]

Thank you very much friend, no, it does not use JWPUB, for the meeting assignment programs I will use EPUB, and the GETPUBMEDIALINKS API from JW.ORG.

[MrCyjaneK]

epub is not available in some languages for some of the publications

[orangethewell]

Anyone could discover what type of css library does the publications use? I thought it was Tailwind, but it seems have a slightly different class signature

[livrasand]

@orangethewell Yes I have the CSS. You can see it here:

https://github.com/livrasand/livrasand.github.io/tree/main/JW-Library-Visualizer-API

I extracted the CSS, but JW Library uses a JSX file written in React, inside is the CSS. Contains all JW Library styles. I am currently working on ReviwDocs, an application like Word that will use that CSS to give styles and make it easier and faster to create JWPUB. Maybe you can use the function or the algorithm for your project (which is great).

[orangethewell]

@yuniermv you don't save data on .jwpub files, in fact, you save it within your app database, or a outside file to recover the notes when you reload the publication

[arthurwweber]

Yeah. I know. I was referring to how I can encrypt the resulting html from the BLOB once I have edited the html document.

You first recompress the HTML using ZLib Deflate and then encrypt it back using AES-128-CBC with the same parameters, you update the blob using the UPDATE SQL command in the SQLite database. Then all you have to do is rewrite the JWPUB file: you update the contents file with the modified .db file and you recompute its hash that you have to store in the manifest.json file.

[MrCyjaneK]

Hey! Does any of you guys maybe have some old catalog.db lying around? I kind of want to check for something. Some publications seems missing.

[MrCyjaneK]

I think that I've hit a roadblock.

What does one do when Extract references a publication that isn't available in the Extract table itself?

For example

SELECT `Content` FROM `Extract` WHERE `Link`='p/E:502014236/'

Would result in an extract that is fine but itself contains reference to jwpub://p/E:1102012654/ which cannot be found in the Extract table.

What do you think should be the behavior of the publication? Should it "grab" the publication from external source by the MipsDocumentId no matter what, to ensure that once somebody downloads the book it will work, or should it just contain a link to download the other publication?

Also, this shows entirely different issue that occurs: We cannot for sure know where to look for the publication. Afaik there is no endpoint to get publication by MepsDocumentId.. Which indicates that we need to be able to search for it in all publications...

Quick google search for "1102012654" resulted in some wol.jw.org results - but the result I got was in Polish (and the MepsDocumentId is in English (not to mention the fact that the article is not the one mentioned in the Extract))

For future reference, link: https://wol.jw.org/pl/wol/d/r12/lp-p/1102012654

Even though I wanted to avoid doing what I'm about to do I think that the only way to go is to combine all JWPUBs into one, and then build an api/tools on top of that.

If what I know is at least somewhat correct, all JWPUBs available to download weight a little under 0.5TB, ~434.719 GiB, to be exact - according to my early calculations..

[arthurwweber]

MepsDocumentId is the unique identifier for an article. If the article is downloadable on its own, you can use GetPubMediaLinks to identify it (https://b.jw-cdn.org/apis/pub-media/GETPUBMEDIALINKS?output=json&docid=1102012654&langwritten=E returns hl, section 12). Otherwise you might have to index the TOC of each publication into a table where you can check on the fly whether you have that certain publication.

[MrCyjaneK]

@arthurwweber Thanks for sharing the link, it is sowing correct publication, so I will use it for sure.

[MrCyjaneK]

There is also one more issue, that I'd like to fix.. Is there any way to convert MepsLanguageId to the langcode?

I have this file: https://github.com/MrCyjaneK/jwapi/blob/master/libjw/mepsmap.go that I generate using a script, but something tells me that there must be at least slightly better solution to this.

[arthurwweber]

There is a Languages table in mepsunit.db, that's where you find the respective language mnemonic for each MepsLanguageId.

[MrCyjaneK]

I found it in /data/data/org.jw.jwlibrary.mobile/databases/mepsunit.db. Can't open the app to watch for traffic atm, thanks

[MrCyjaneK]

New discovery:

• Hyperlink table seems to be present in some of the publications (one can be observed in CA-copgm24)
• Heading table (rsg19)

No more 'surprises', 8,092,301 rows (7.3 GiB) for all English and Polish JWPUBs smashed into one database. I'm starting a full import now.

Another discoveries, that broken the program :| :

• BibleChapter.HasQuotedScriptures can be seen in bi12_KZ
• Multimedia.TrackNumber - g_G_201502
• VideoMapParagraphLocation- g_G_201502
• VideoMap - mwb_E_201603
• MultimediaExtract - mwb_PR_201604
• Around row 157,666,673 - Index for table 'SearchIndexDocument' is corrupt; try to repair it - #1030 - Got error 194 "Tablespace is missing for a table" from storage engine InnoDB
  
  Restart solved this for me but ugh

[MrCyjaneK]

After resolving schema issues, and fixing my database I'm doing a last non-final import of the data (314m rows in and no critical issues spotted).

contents.db is just a pain to work with, tables/colums come and go as they like, in one publication you can have few versions of database for different languages. Most notably CaptionContent table seems to be added randomly to publications:

Example log from export

12:54:49 main.go:155: Worker 5 processing: jy_RMV.jwpub
12:54:49 main.go:155: Worker 9 processing: jy_RR.jwpub
12:54:50 main.go:155: Worker 9 processing: jy_RU.jwpub
12:54:50 main.go:155: Worker 5 processing: jy_SA.jwpub
12:54:50 decrypt.go:301: no such column: CaptionContent
12:54:52 main.go:155: Worker 10 processing: jy_SB.jwpub
12:54:52 main.go:155: Worker 4 processing: jy_SBO.jwpub
12:54:54 main.go:155: Worker 2 processing: jy_SC.jwpub
12:54:54 main.go:155: Worker 12 processing: jy_SE.jwpub
12:54:54 decrypt.go:301: no such column: CaptionContent
12:54:54 decrypt.go:301: no such column: CaptionContent
12:54:55 main.go:155: Worker 11 processing: jy_SEN.jwpub
12:54:55 main.go:155: Worker 8 processing: jy_SG.jwpub
12:54:56 main.go:155: Worker 7 processing: jy_SH.jwpub

[MrCyjaneK]

5 days of interrupted work and I've imported most of JWPUBs that were easy to obtain (curl + jq in a for loop). I have managed to downlaod 414.89 GiB of them, and discovered few interesting things on my way

• content.db is pure evil, seriously, how did one end up this situation. All publications seem to be last edited around 2019-11-01 13:16:17, yet the schema is ugh.
• Word and SearchIndexDocument... take up about 125GiB of my database (accounting for more than 1/3rd of entire DB)
• Assets are pretty much the same in many publications, 600k unique over 3,3m all assets.
• Speking of Assets (this is the entire table) I have no idea what is it.
  
• I've managed to rebuild lff_E (that is import and export back to jwpub), indexes are already proper in the mysql so the whole process took a fraction of second + zipping.
• Also, if someone want to experiment with it - it is really fast as long as you limit yourself to one / two languages (6/12GB). After that DB is much slower (and I mean it), mostly due to indexing that happens along the way.
• Award for biggest table (in terms of rows) goes to: Word ~500m
• Award for biggest table (in terms of size) goes to: Word 61GiB
• Award for slowest table in terms of INSERT goes to: Word / SearchIndexDocument I can't really decide.
• Award for the most confusing and smallest tables goes to:
  

What's next on the roadmap for me?

• Putting bunch of Go code into a working API, and publishing DB access to the internet because why not.
• Modding the nwtsty to contain more references (basically loop over every single piece of publication, and put a link in footnotes if given verse was mentioned there), because it will be fun
• Add eruda.js to publications, to mess around with renderer/richtext/webview - I'm unsure what we have there in latest version
• Create a custom JW Library server (just a few endpoints, and I already have the data in a pretty easy to work with format)
• If the thing that displays content is a webview I'd like to play a game of flappy bird there.
• Make the db auto update to match latest catalog.db
• and after I finish having fun, move to creating open source library clone using Go fyne toolkit. I'll most likely use FOSSJWPUB format I've invented along the way which is basically content.db but with some additions to make life easier (and I kind of don't want to mess around with

aaand obviously, I'll make some kind of blogpost too with some fun technical details that I've skipped.

[GuyMicciche]

I have a database extracted from the jwpub file. Like for example, nwtsty.db. Some fields have BLOB and I can't see the data using sql. How can I see the data?

[MrCyjaneK]

@GuyMicciche #1 (comment) ChatGPT is actually pretty good in converting that message into code (worked for golang)

[MrCyjaneK]

what are you working on btw?

[livrasand]

If you need help knowing how a JWPUB works or the styles you can use, I recommend you take a look at the Reviw wiki

[GuyMicciche]

what are you working on btw?

I have a python script that I already got headway with. Give a link to the db file, and it generates the correct hashes. But right now I'm having trouble getting the data from "Content" blob. should I share the script here?