Constrained beam search in chat about transformers

[esaks1]

I have not tested it enough to trust that it is a safe solution but what I ended up doing to fix this issue was to go to beam_search.py and slice the token_ids list in all calls to reset(), so that it does not include the prompt.

[gante]

Hi @esaks1 👋 Thank you for opening this issue!

Constrained generation is indeed not being actively maintained due to lack of bandwidth -- it deserves a proper dive at some point in the future.

Meanwhile, I'd like to suggest the following workarounds:

1. If your constraints are structured, i.e. you know the shape of your desired LLM output, have a look at this library: https://github.com/outlines-dev/outlines
2. If you simply want to fit certain words in the answer, you may consider passing sequence_bias to make those words more likely. If they become too repetitive with sequence_bias, you should be able to avoid repetitions with repetition_penalty (full docs)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.