[BUG] Quickadd can't process large context for gpt-4-1106-preview about quickadd

[chhoumann]

I've actually built Chunked Prompts to overcome context size limits.

I found that I more often needed large amounts of text processed, rather than specific lookup of chunks of text. E.g. a vector store wouldn't solve the issue of summarizing an entire book, only the problem of answering a question based on the text in a book.

And since many vector stores can be used through simple HTTP APIs, it would be trivial to create a user script to send one's data to it, and then make a lookup every time one wanted to prompt the LLM using some text from the store.

[chhoumann]

Hey @Triquetra - HTTP 429 Too Many Requests response status code indicates that there has been sent too many requests in a given amount of time.

In other words, it seems you reached the rate limit. I was able to fire a good few requests with e.g. 21k, 9k, and 12k tokens, and they all succeeded. But after a bit, I too got rate-limited and got the 429 error.

[chhoumann]

OpenAI has set a rate limit of 40k tokens per minute, so even though the model supports much more than that, it isn't really feasible to send more than 40k tokens in a single prompt.

https://platform.openai.com/account/limits

This is for my usage tier (3), at least. You can check whether this is the same for you.

[Triquetra]

Have you considered implementing vector store embeddings and completions to overcome these context limits?