[Bug]: Auth issues trying to run replicate model

[geekodour]

What happened?

I have a private replicate model.

I've set the replicate env var, REPLICATE_API_KEY=<key found in the model page>

        model_version = "c1bc50532...29881831e55c"
        model_name = "replicate/abc/xyz"
        full_model_name = ":".join([model_name, model_version])
        response = completion(
            model=full_model_name,
            messages = [{ "content": prompt, "role": "user"}]
        )

But when I run it I get the following auth issue:

LiteLLM completion() model= abc/xyz:c1bc50532a26...81831e55c; provider = replicate
HTTP Request: GET https://huggingface.co/abc/xyz:c1bc50532a2f2...1831e55c/raw/main/tokenizer_config.json "HTTP/1.1 401 Unauthorized"
HTTP Request: POST https://api.replicate.com/v1/models/c1bc50532a2..7c8d29881831e55c/predictions "HTTP/1.1 404 Not Found"

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

2024-12-20T06:42:23.713006Z [error    ] Error generating lyrics: litellm.APIError: ReplicateException - Client error '404 Not Found' for url 'https://api.replicate.com/v1/models/c1bc50532a265a8f27a72f4a...81831e55c/predictions'

Couple of issues, unsure why it's trying to lookup Hugginface? second, the prediction endpoint does not have the repo/model in the url, it just has models/ so the 404 makes sense, it should be org/model_name/version or something like that.

Is the replicate setup of private replicate model broken for litellm?

Relevant log output

LiteLLM completion() model= abc/xyz:c1bc50532a26...81831e55c; provider = replicate
HTTP Request: GET https://huggingface.co/abc/xyz:c1bc50532a2f2...1831e55c/raw/main/tokenizer_config.json "HTTP/1.1 401 Unauthorized"
HTTP Request: POST https://api.replicate.com/v1/models/c1bc50532a2..7c8d29881831e55c/predictions "HTTP/1.1 404 Not Found"

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

2024-12-20T06:42:23.713006Z [error    ] Error generating lyrics: litellm.APIError: ReplicateException - Client error '404 Not Found' for url 'https://api.replicate.com/v1/models/c1bc50532a265a8f27a72f4a...81831e55c/predictions'

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

^1.55.7

Twitter / LinkedIn details

No response

[geekodour]

I am not sure if the api endpoint litellm is trying to hit for replicate even exists: https://replicate.com/docs/topics/predictions/create-a-prediction

[geekodour]

Inability to debug: #6214

Seems like it's also deprecated?

litellm/litellm/_logging.py

Lines 10 to 12 in 888b3a2

	logging.warning(
	"`litellm.set_verbose` is deprecated. Please set `os.environ['LITELLM_LOG'] = 'DEBUG'` for debug logs."
	)

but documented otherwise
also https://docs.litellm.ai/docs/debugging/local_debugging

[krrishdholakia]

Hey @geekodour the endpoint litellm calls does work for their public models -

litellm/tests/local_testing/test_completion.py

Line 3140 in 2c36f25

async def test_completion_replicate_llama3(sync_mode):

unsure why it's trying to lookup Hugginface?

it's trying to pull the chat template for hf, to be able to call the model in the correct format

the prediction endpoint does not have the repo/model in the url, it just has models/ so the 404 makes sense

can you point me to the relevant doc to see how a private model should be called

and a working curl for you for replicate