When batches expire, are we dealing with it correctly?

[RyanMarten]

Should at least use the responses that have been completed.

Ofc the best thing would be to resubmit only requests that weren't completed.

Instead of resubmitting the whole batch that expired.

But the issue with this is now there isn't a 1 to 1 connection between requests --> batch --> responses. Keeping it simple we can just submit a smaller batch which is the remaining amount for that original batch.

[RyanMarten]

Right now we are not downloading the output or error files:
https://github.com/bespokelabsai/curator/blob/dev/src/bespokelabs/curator/request_processor/openai_batch_request_processor.py#L946-L977

[RyanMarten]

And does not create a response fie:
https://github.com/bespokelabsai/curator/blob/dev/src/bespokelabs/curator/request_processor/openai_batch_request_processor.py#L1004-L1007

[RyanMarten]

(1) The first thing we should do just allow for brute-force retry (aka run the same program again, Curator treats it as a completely failed batch and resubmits).

(2) Then we should add the more fine-grained solution

[RyanMarten]

On (1) add an if statement when we check which batches are already submitted and don't mark as submitted if failed or expired or cancelled

For the quickest fix, can just add to _submitted.jsonl.

But maybe should write to _failed.jsonl and _expired.jsonl and should rewrite _submitted only with those not failed or expired??

can write all these files before each sleep

https://github.com/bespokelabsai/curator/blob/dev/src/bespokelabs/curator/request_processor/openai_batch_request_processor.py#L721-L755

[RyanMarten]

We also want to resubmit requests that don't have a valid response format #86

(although very expensive to do if we resubmit the whole batch multiple times, so just submit failed requests)

[RyanMarten]

Responses in the file will look like this

https://platform.openai.com/docs/guides/batch/getting-started?lang=node#batch-expiration

{"id": "batch_req_123", "custom_id": "request-3", "response": null, "error": {"code": "batch_expired", "message": "This request could not be executed before the completion window expired."}}
{"id": "batch_req_123", "custom_id": "request-7", "response": null, "error": {"code": "batch_expired", "message": "This request could not be executed before the completion window expired."}}

[RyanMarten]

When we get failed requests (or requests with finish_reason of length or content filter, resubmit these in batch).
for the finish_reason we can use the litellm map. Started PR for improving that function.

• Update map_finish_reason BerriAI/litellm#7264