Chrome demo frequently terminates while loading about web-llm

[CharlieFRuan]

@neubig @kudaibergenu Just updated the npm to 0.2.24: #323

This should fix it. Let us know if issues persist. Thank you!

[neubig]

Thanks for the fast response @CharlieFRuan !
I upgraded to the latest version and it does seem to not be hanging anymore, so I'll close the issue.

[kudaibergenu]

I am experiencing the same problem. So far, I understood the problem starts once it tries to pull the models from the cache.

[CharlieFRuan]

Apologies for the inconvenience, I'll look into this issue today. Meanwhile, clearing chrome's cache might alleviate the problem.

[mrsheep76]

I'm still experiencing this issue even after updating to the latest version (0.2.24).

Would you be open to further investigation? Some notes from my own experience:

• It happens 90%+ of the time
• The progress bar usually gets about 70-80% finished before the crash
• Clearing the Chrome cache temporarily fixes the problem, but it begins immediately again on the next reload from cache
• I reproduced it at least once from the "get-started" example (not Chrome extension), but it seems to happen somewhat less often than in the Chrome extension.

Happy to help chase this down - thanks!

[CharlieFRuan]

Hi @mrsheep76 what is the error that you are seeing? Is it also exit(1)? Would be great if you could share a screenshot of the console/UI

[mrsheep76]

Hi @CharlieFRuan - yes, same error as described at the top of this bug - exit(1). I made sure to rebuild the plug-in (npm run build) with the new version of the npm with your changes. Here is a Console screenshot, which I took from inspecting the plug-in popup:

And here is the UI after it crashed:

Let me know if there's anything else I can do to gather information - and thanks for investigating!

[neubig]

And actually I'm still encountering crashes as well, although it seems less frequent than before.

[CharlieFRuan]

@mrsheep76 @neubig Thanks for the info, I'll look into it today!

[CharlieFRuan]

@mrsheep76 @neubig I was not able to replicate the error (either via simple chat or the chrome extension example). But I published 0.2.25, with the hope of fixing this (assuming the issue is due to parallelly downloading too many shards at the same time). For more description, see #328. Let me know, thanks!

[mrsheep76]

@CharlieFRuan Thanks for the attempted fix! Unfortunately I'm still getting crashes 90%+ of the time when loading from cache -- same frequency as before from what I can tell. It does occasionally work, so I wonder if there's a race condition somewhere?

Is there any way for me to get a more informative error message and pass it to you, since you cannot reproduce it yourself? The exit(1) is obviously pretty generic...

[CharlieFRuan]

Hmm I see; before we further dig into this, could you perhaps try a smaller model, say Gemma 2B (assuming that you are using Mistral 7B q4f16, the default one in the chrome extension example). Not sure how much RAM you have, but it could be that the model is too large? See discussion in #324 for specifics.

[CharlieFRuan]

And I just noticed that it states Loading model from cache in the log, rather than Fetching param cache. This means that all shards of the model parameters are already in the browser cache. Perhaps you can try to clear the cache, and let it fetch the params again. The Cached images and files in chrome should do it:

[CharlieFRuan]

I also further bumped to 0.2.26, which adds an additional try-catch, attempting to provide more information to the exit(1).

[neubig]

Thanks! I tried 4-5 times now, including after clearing cache and every time worked! So maybe this is fixed with the new version. Closing for now, thanks @CharlieFRuan !

[CharlieFRuan]

@neubig Thanks for confirming; feel free to reopen any time!

[mrsheep76]

Good news - it is now working for me as well!

I had a temporary glitch with my build environment during my last test run, so I suspect one of the earlier fixes actually did the trick (perhaps limiting the number of concurrent threads)?

In any case, it is working consistently now. Thank you for all the help!

[CharlieFRuan]

@mrsheep76 Thanks, that is great to hear! Feel free to open new issues if there are other problems