Comments (7)
Ah, that makes a lot of sense. Closing this one. Thank you!
from transformers.js.
Okay yes, it looks like you are creating multiple versions of the model, leading to an out of memory error. The current Caching API caches the model, but does not force you to only create one instance of the model. If you move your
const pipe = await pipeline("text2text-generation", "flan-t5-base");
to the same level as env.localURL
, that should solve your problem.
Alternatively, you can follow what I did in my worker example (https://github.com/xenova/transformers.js/blob/main/assets/js/worker.js), I use a singleton class to ensure that only one pipeline is created.
Something like this should work:
import { pipeline, env } from "@xenova/transformers";
env.remoteModels = false;
env.localURL = "/models";
const pipe = await pipeline("text2text-generation", "flan-t5-base");
onmessage = async (event: MessageEvent) => {
const completion: string[] = await pipe(
"Summarize: The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct."
);
console.log(completion);
return completion[0];
};
from transformers.js.
Could you provide the text you used? It may be too long for the context window (and I am not doing truncation of inputs at the moment).
Edit: Nevermind, the error looks to be when you create the session. The error looks to be coming from onnxruntime-web. Could you perhaps provide more steps to see if I can reproduce it on my side?
Or, could you provide the full code you are using? It's quite difficult to determine the root cause from the above error message. The only thing I can think of right now is you are loading the model multiple times, leading to an out of memory error?
from transformers.js.
Here is my minimal reproducible example within a typescript web worker:
import { pipeline, env } from "@xenova/transformers";
env.remoteModels = false;
env.localURL = "/models";
onmessage = async (event: MessageEvent) => {
const pipe = await pipeline("text2text-generation", "flan-t5-base");
const completion: string[] = await pipe(
"Summarize: The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct."
);
console.log(completion);
return completion[0];
};
Here are my logs after running 5 times in a row:
['The Eiffel Tower is the tallest structure in France, and the tallest structure in the world.']
['The Eiffel Tower is the tallest structure in France, and the tallest structure in the world.']
['The Eiffel Tower is the tallest structure in France, and the tallest structure in the world.']
['The Eiffel Tower is the tallest structure in France, and the tallest structure in the world.']
Uncaught (in promise) RangeError: offset is out of bounds
at Uint8Array.set (<anonymous>)
at e.createSessionAllocate (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:450663)
at Object.e.createSession (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:451415)
at e.createSession (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:443678)
at e.OnnxruntimeWebAssemblySessionHandler.loadModel (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:446572)
at Object.createSessionHandler (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:156408)
at Function.create (webpack-internal:///../../node_modules/onnxruntime-common/dist/lib/inference-session-impl.js:176:39)
at async constructSession (webpack-internal:///../../node_modules/@xenova/transformers/src/models.js:20:19)
at async Promise.all (index 2)
at async Function.from_pretrained (webpack-internal:///../../node_modules/@xenova/transformers/src/models.js:102:75)
e.createSessionAllocate @ ort-web.min.js?d8cf:6
e.createSession @ ort-web.min.js?d8cf:6
e.createSession @ ort-web.min.js?d8cf:6
loadModel @ ort-web.min.js?d8cf:6
createSessionHandler @ ort-web.min.js?d8cf:6
create @ inference-session-impl.js?f23d:170
Promise.then (async)
asyncGeneratorStep @ index.ts?9da8:168
_next @ index.ts?9da8:168
eval @ index.ts?9da8:168
eval @ index.ts?9da8:168
eval @ index.ts?9da8:173
from transformers.js.
@xenova the above solution worked great.
However, with this same pipeline, I seem to be seeing this error come up fairly often:
TypeError: Cannot convert undefined to a BigInt
at BigInt (<anonymous>)
at eval (models.js?a626:289:1)
at Array.map (<anonymous>)
at Function.toI64Tensor (models.js?a626:289:1)
at Function.prepare_inputs (models.js?a626:303:1)
at Function.forward (models.js?a626:557:1)
at Function.runBeam (models.js?a626:544:1)
at Function.generate_single (models.js?a626:366:1)
at eval (models.js?a626:326:1)
at Array.map (<anonymous>)
Any ideas? I don't think it has to do with context window size, since my input is fairly small and has worked on bigger.
from transformers.js.
This occurs when the tokenizer produces undefined tokens (most likely due to unknown characters). Could you send the input you used? Also, could you open a new issue with that? Since it is probably a real bug.
from transformers.js.
Yes I'll open a new issue, thanks.
from transformers.js.
Related Issues (20)
- Options for the "translation" pipeline when using Xenova/t5-small
- Have considered using wasm technology to implement this library? HOT 1
- Example not working on Chrome/Arc v.124(M1 Mac) HOT 5
- Can you use all transformers models with transformers.js?
- Request to support suno-ai/bark model HOT 2
- Something went wrong during model construction (most likely a missing operation). Using `wasm` as a fallback. HOT 4
- v3: Issue loading T5-Small with webgpu backend HOT 1
- WebAssembly Compilation Error Due to CSP Restrictions HOT 5
- [v3] webgpu-chat demo fails with `no available backend found` HOT 1
- How to point to a specific model path in order to use compressed models? (brotli) HOT 1
- Method to send `HF_TOKEN` from the client side
- Pipeline execution time with 'image-classification' pipeline HOT 1
- Mistral Tokenizer V2
- Allow overriding model file name to support custom implementations of various compressions (brotli, gzip, etc)
- Progress callback for Moondream?
- NanoLlava is repeating itself a lot HOT 1
- Shouldn't this work? #v3 HOT 3
- Using AutoModel, AutoTokenizer with distilbert models HOT 1
- Support stopping criteria for sequence generation HOT 1
- WebGPU and WASM Backends Unavailable within Service Worker (V3 issue) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transformers.js.