Add retry on 504 Gateway Time-out and "connection closed before message completed" about async-openai

[Christof23]

Hi @64bit, thanks for your response.

I take your point on the 504 response; however I've seen this response while other concurrent requests have gone through successfully. I'm not sure how OpenAI are configuring their infrastructure, but it seems that a 504 with their API doesn't necessarily mean the API is unavailable.

For the reqwest client, I think I tried the fix with the client configuration, but I'll double check.

[Christof23]

PR for the 504: #199

[64bit]

Hi @Christof23 ,

Thank you for the issue and PR.

What rationale is there to retry when API is down - and could potentially be down for hours until incident is resolved?

[64bit]

Similarly, I sometimes receive Error: IncompleteMessage: connection closed before message completed for some requests which are not retried. I believe this is a hyper issue: hyperium/hyper#2136

Based on the comment, does configuring same parameter for custom reqwestclient fix the issue?

[64bit]

Right thing to do here is contact OpenAI and report that their API is failing. Introducing retries is propagating that failure to everyone - and making library in-deterministic.

That said, one thing that could be done in async-openai is propgate the payload causing deserialization error so that library users can decide what right action to perform.

So OpenAIError::JSONDeserialize(serde_json::Error), becomes OpenAIError::JSONDeserialize(serde_json::Error, String), where the String is payload that OpenAI responded with.