apiclawnotes

Using ellmer in R With a Custom OpenAI-Compatible API: chat_openai_compatible, credentials and the Double "Bearer" Trap

Published on

I'm Aman Kumar. Most of the "how do I point my client at a non-OpenAI endpoint" questions I get are about Python or TypeScript, but R users ask too, and these days the answer in R is usually ellmer, the tidyverse package for talking to LLMs. People have a company gateway, a vLLM box, or a third-party OpenAI-compatible service, and they want chat$chat("...") to go there instead of api.openai.com.

Rather than guess from the docs, I installed ellmer 0.5.0 (with httr2 1.3.0) on R 4.5.0 and pointed it at a small local server that logs every request: path, Authorization header and JSON body. Everything below is what actually showed up in that log.

The short version

library(ellmer)

chat <- chat_openai_compatible(
  base_url    = "https://apiclaw.biz/v1",          # your server, including /v1
  credentials = function() Sys.getenv("MY_GATEWAY_KEY"),
  model       = "YOUR_MODEL_ID"
)

chat$chat("Summarise the mtcars dataset in two sentences.")

Put MY_GATEWAY_KEY=... in ~/.Renviron (run usethis::edit_r_environ() if you have usethis) and restart R. Replace YOUR_MODEL_ID with an ID your server actually lists. That's the whole setup. The rest of this post is about the four or five ways it quietly goes wrong.

Trap 1: chat_openai() with base_url sends /v1/responses

The obvious first attempt is chat_openai(base_url = "https://my-server/v1", ...). In 0.5.0 that does not call Chat Completions. My log showed:

POST /v1/responses
{"input": [...], "model": "model-a", "stream": true, "store": false, "service_tier": "auto"}

The body has input instead of messages, plus store and service_tier. A server that only implements /v1/chat/completions answers 404, and ellmer shows you HTTP 404 Not Found with whatever message the server returned. If your provider does implement the Responses API, fine. Most OpenAI-compatible gateways and self-hosted servers implement Chat Completions first, and chat_openai_compatible() is the function that targets it. Every request it made in my tests went to POST {base_url}/chat/completions with a normal messages array.

A related surprise: chat_openai() has base_url = "https://api.openai.com/v1" hard-coded as its default, so setting the OPENAI_BASE_URL environment variable did nothing for it. My test request went straight to OpenAI and came back with OpenAI's own "Incorrect API key provided" 401. chat_openai_compatible() does read OPENAI_BASE_URL, but only when you pass base_url = NULL or "" explicitly. If you omit base_url entirely it stops with "base_url is required for OpenAI-compatible APIs."

Trap 2: the deprecated api_key argument sends "Bearer Bearer"

This is the one that cost me the most head-scratching. Plenty of blog posts and older examples pass the key directly:

chat_openai_compatible(base_url = B, api_key = "sk-...", model = "...")  # don't

In 0.5.0 that prints a deprecation warning ("The api_key argument ... is deprecated as of ellmer 0.4.0. Please use the credentials argument instead"), and then the request went out with this header:

Authorization: Bearer Bearer sk-...

The server rightly returned 401. Reading the source, the deprecated-argument shim wraps your key as "Bearer <key>" for this function, and the request builder adds Bearer again. chat_openai() with api_key produced a single, correct Bearer in the same test, which is why the bug is easy to miss if you switch between the two functions.

The fix is to use credentials, which takes a function (not a string; a string is rejected with "credentials must be a function or NULL"). Return the bare key and ellmer adds Bearer once:

credentials = function() Sys.getenv("MY_GATEWAY_KEY")

Don't return "Bearer sk-..." from that function either; that also produced a double prefix and a 401. A function is nicer anyway: the key is read when the request is made, so rotating the env var doesn't require rebuilding the chat object.

Trap 3: no credentials means your OpenAI key goes to the custom host

If you pass neither credentials nor api_key, chat_openai_compatible() falls back to the OPENAI_API_KEY environment variable. With it unset, you get "Can't find env var OPENAI_API_KEY" before any request is made. With it set, the request succeeded against my mock, which means your real OpenAI key was sent to a server that is not OpenAI. That's harmless against localhost and a bad habit against anything else. Always pass credentials explicitly when the host isn't OpenAI.

Trap 4: params(max_tokens) becomes max_completion_tokens

ellmer has a provider-neutral params() helper. I passed params(temperature = 0.2, max_tokens = 300, seed = 1, top_k = 5) and the request body contained:

{"model": "model-a", "temperature": 0.2, "top_k": 5, "seed": 1, "max_completion_tokens": 300, ...}

Two things to notice. max_tokens was renamed to max_completion_tokens, the newer OpenAI name. Servers that only understand the older max_tokens will ignore it, and your length cap silently stops working. And top_k was passed through even though OpenAI itself doesn't take it; whether your server accepts or rejects unknown fields is up to the server.

If your server wants max_tokens, send it raw with api_args, which is merged into the body as-is:

chat_openai_compatible(..., api_args = list(max_tokens = 200))

That produced "max_tokens": 200 in the log, with nothing renamed. api_headers = c("X-Team" = "analytics") added the header to every request, which is handy for gateways that tag usage by header.

Paths, slashes and model IDs

  • base_url must include /v1 (or whatever prefix your server uses). With base_url = "http://127.0.0.1:8811" ellmer posted to /chat/completions and got a 404.
  • A trailing slash (.../v1/) was fine; the path still came out as /v1/chat/completions.
  • model is required. Without it you get "model is required." and no request.
  • The model string is sent exactly as written, so use the ID your provider lists, not a display name.

models_openai(base_url = ..., credentials = ...) calls GET {base_url}/models, but it expects every entry to include a numeric created field. My mock initially returned only id, object and owned_by, and ellmer failed with the unhelpful "values must be length 1, but FUN(X[[1]]) result is length 0". Once I added created, it returned a tidy data frame. If you hit that error, fetch /v1/models with curl and look at the shape.

Streaming, token counts and cost

Calling chat$chat() in a script streamed by default in my runs: the body had "stream": true and "stream_options": {"include_usage": true}. Passing echo = "none" switched it to "stream": false, and so did chat_structured() and parallel_chat().

chat$get_tokens() reads usage from the final stream chunk. When my mock streamed text but skipped the usage chunk, ellmer reported 0 input and 0 output tokens, with no warning. If your server doesn't honour stream_options, your token accounting in R will be zeros, not NA. chat$get_cost() returned NA for my made-up model, because ellmer only knows prices for models in its built-in table.

Also remember that a chat object is a conversation. My second chat$chat("again") resent the whole history (user, assistant, user). For a loop over 500 rows, create a fresh chat per row or use parallel_chat(), otherwise every call carries every previous turn.

Structured output and tools

chat$chat_structured("Asha is 31", type = type_object(name = type_string(), age = type_integer())) sent a response_format of type json_schema with "strict": true and "additionalProperties": false. My mock returned JSON and ellmer gave back a proper R list. A server that doesn't support json_schema response formats will either reject that or ignore it, so test this one specifically against your provider.

Tool calling worked end to end: after chat$register_tool(tool(get_weather, ...)), the first request carried the tool definition, my mock returned a tool_calls response, ellmer ran the R function, and the second request contained user, assistant, tool messages.

Retries and timeouts

ellmer retries through httr2. Against a server that always answered 429 with Retry-After: 1, I saw 3 attempts and an error after about 3 seconds. A 503 was also retried 3 times (about 6 seconds total with the backoff). A plain 500 was not retried at all: one request, then the error.

Two options control this. options(ellmer_max_tries = 1) dropped it to a single attempt. options(ellmer_timeout_s = 3) against a deliberately slow server gave "Timeout was reached" after three timed-out attempts with 4-second backoffs between them, about 15 seconds of wall time. If you run ellmer inside a Shiny app or a scheduled job, set both deliberately rather than inheriting the defaults.

A checklist

  1. Use chat_openai_compatible(), not chat_openai(base_url = ...), unless your server implements /v1/responses.
  2. Include /v1 in base_url.
  3. Pass credentials = function() Sys.getenv("...") returning the bare key. Avoid api_key in 0.5.0.
  4. Use your provider's exact model ID.
  5. Use api_args = list(max_tokens = ...) if your server ignores max_completion_tokens.
  6. Check get_tokens() after a streamed call; zeros mean no usage chunk.
  7. Set ellmer_timeout_s and ellmer_max_tries for anything unattended.

Disclosure: I build APIClaw, an OpenAI-compatible gateway whose base URL appears in the first example. Nothing above is specific to it; the same behaviour applies to any server that speaks Chat Completions.