Problem of oaieval.py about evals

[aaronsmithtv]

The behaviour is explained in custom-eval.md:

If you notice evals has cached your data and you need to clear that cache, you can do so with rm -rf /tmp/filecache.

This will be a difficult spot for people looking through custom-eval I think. I've also noticed that the flag --no-cache exists in the list of oaieval options. I'm not sure if this is intended to prevent .jsonl files from being cached (which would be a nice feature), but as far as I can tell the code associated with the arg isn't doing anything:

evals/evals/cli/oaieval.py

Lines 173 to 175 in 19bfdba

	api_extra_options = {}
	if not args.cache:
	api_extra_options["cache_level"] = 0

Though of course @andrew-openai will know a lot more than I do!

[samclane]

I'm currently having the same problem. I'm unsure of how to clear the cache to get it to "see" my new jsonl file. I've tried to change the name of the test to no avail

EDIT: After running with --debug, I found it was loading a pkl file from my /tmp folder. After deleting it I got it to run all my samples.

[JimmyfaQwQ]

Thanks, it helps.

The behaviour is explained in custom-eval.md:

If you notice evals has cached your data and you need to clear that cache, you can do so with rm -rf /tmp/filecache.

This will be a difficult spot for people looking through custom-eval I think. I've also noticed that the flag --no-cache exists in the list of oaieval options. I'm not sure if this is intended to prevent .jsonl files from being cached (which would be a nice feature), but as far as I can tell the code associated with the arg isn't doing anything:

evals/evals/cli/oaieval.py

Lines 173 to 175 in 19bfdba

	api_extra_options = {}
	if not args.cache:
	api_extra_options["cache_level"] = 0

Though of course @andrew-openai will know a lot more than I do!