Pointing Chainlit at a Custom OpenAI-Compatible API: No ChatOpenAI Wrapper, /v1 You Must Keep and History the SDK Will Not Manage

Chainlit is a Python chat UI. You write @cl.on_message, it runs a Starlette app, and the browser gets a polished chat screen. What it does not ship is its own LLM client. There is no cl.ChatOpenAI and no cl.OpenAI. The documented path for an OpenAI-compatible backend is the official openai package's AsyncOpenAI, plus an optional cl.instrument_openai() call so generations show up as steps in the UI.

I installed Chainlit 2.12.0 and openai 3.35.1 on Python 3.13.5, pointed AsyncOpenAI at a local mock of /v1/chat/completions that logged method, path, selected headers and the full JSON body, and exercised the patterns the Chainlit docs use for custom base URLs, streaming and session history. The examples point at APIClaw, an OpenAI-compatible gateway I build, so weigh my opinions accordingly. Its base URL is https://apiclaw.biz/v1; replace YOUR_MODEL_ID with an exact ID from your provider's model list.

Install and a minimal app

pip install chainlit openai

from openai import AsyncOpenAI
import chainlit as cl

client = AsyncOpenAI(
    base_url="https://apiclaw.biz/v1",
    api_key="YOUR_API_KEY",
)
cl.instrument_openai()

settings = {
    "model": "YOUR_MODEL_ID",
    "temperature": 0,
}

@cl.on_message
async def on_message(message: cl.Message):
    response = await client.chat.completions.create(
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": message.content},
        ],
        **settings,
    )
    await cl.Message(content=response.choices[0].message.content).send()

cl.instrument_openai() wraps Completions.create and AsyncCompletions.create through LiteralAI so each call becomes an llm step in the Chainlit UI. It does not change the HTTP request. Against the mock, instrumented and non-instrumented clients produced the same path, headers and body.

You can also leave the constructor empty and set OPENAI_BASE_URL and OPENAI_API_KEY in the environment (or a .env that Chainlit loads). A bare AsyncOpenAI() picked both up and posted to /v1/chat/completions with Authorization: Bearer … exactly as expected.

What happens if you omit /v1

The openai SDK appends chat/completions to whatever base_url you give it. It does not silently insert /v1.

My mock accepted both paths, so the no-/v1 case returned 200 and looked fine. A gateway that only serves the OpenAI-shaped /v1/... tree will 404 on /chat/completions. For APIClaw that means the working value is https://apiclaw.biz/v1, not https://apiclaw.biz. Trailing slash on /v1/ was harmless.

Streaming versus non-streaming

Chainlit's streaming docs use stream=True on the create call and msg.stream_token(...) on a cl.Message. Against the mock:

• With stream=True, the body included "stream": true. Chunks arrived as SSE; assembling delta.content rebuilt the full reply.
• With stream=False passed explicitly, the body included "stream": false.
• With neither stream nor temperature set, the body was only model and messages. The SDK does not invent a default temperature or a stream: false field unless you ask.

So if your gateway treats a missing stream differently from stream: false, omit the key rather than setting it, or set it deliberately. Temperature 0 and 0.7 both appeared in the body exactly as passed; nothing was stripped.

A streaming handler that also keeps history looks like this:

@cl.on_chat_start
async def start():
    cl.user_session.set(
        "message_history",
        [{"role": "system", "content": "You are a helpful assistant."}],
    )

@cl.on_message
async def main(message: cl.Message):
    history = cl.user_session.get("message_history")
    history.append({"role": "user", "content": message.content})

    msg = cl.Message(content="")
    stream = await client.chat.completions.create(
        model="YOUR_MODEL_ID",
        messages=history,
        stream=True,
        temperature=0.7,
    )
    async for part in stream:
        if token := part.choices[0].delta.content or "":
            await msg.stream_token(token)

    history.append({"role": "assistant", "content": msg.content})
    await msg.update()

History and session: you own the array

Chainlit does not automatically resend prior turns to the model. cl.user_session is an in-memory dict keyed by the websocket session id. The default user_session_timeout in Chainlit 2.12.0 is 1296000 seconds (15 days). Whatever you set under a key is what you get back; the LLM never sees it unless you put it in messages=.

I replayed the docs' pattern for three user turns against the mock. Message counts on the wire were 2, then 4, then 6:

1. [system, user]
2. [system, user, assistant, user]
3. [system, user, assistant, user, assistant, user]

Every prior assistant reply was resent in full. That is what you want for a coherent chat, and also how context windows fill up. If you need trimming, summarization or a sliding window, you write it yourself before the create call. Restarting the Chainlit process clears in-memory sessions; persisting across restarts needs Chainlit data layer configuration, which is separate from the OpenAI client.

Retries on 429, 503 and 401

Retries are entirely the openai SDK's job. Chainlit does not add its own. Fresh AsyncOpenAI clients default to max_retries=2, which means one initial attempt plus two retries (three HTTP posts max).

Against the mock with forced errors:

So rate limits and server errors back off and retry; a bad API key fails once. If you want no automatic retries in a chat UI (so the user sees the error instead of waiting), construct the client with max_retries=0. A wrong key with max_retries=0 raised AuthenticationError with the mock's invalid_api_key payload on the first post.

Surprising defaults worth knowing

• No built-in model list or provider config. Model id, base URL and key are whatever you pass to AsyncOpenAI / create. Typo the model and the gateway answers; Chainlit will not catch it.
• System prompts are opt-in. If your on_message only sends the latest user text, that is all the model gets. The Spanish-system example in the Chainlit OpenAI docs hard-codes the system message on every call; the streaming example stores it once in user_session.
• Sync OpenAI works but the docs and UI assume async. A sync client posted with User-Agent: OpenAI/Python 3.35.1 and x-stainless-async: false. Prefer AsyncOpenAI inside @cl.on_message so you do not block the event loop.
• Instrumentation is optional and UI-only. Skip cl.instrument_openai() and the HTTP traffic is unchanged; you just lose the generation step in the playground.
• Do not stack integrations. The docs warn that enabling OpenAI instrumentation together with LangChain or LlamaIndex instrumentation double-records the same generation.

A practical checklist

1. pip install chainlit openai — I used 2.12.0 and 3.35.1.
2. Set base_url to a value that already includes /v1 for gateways that expect it.
3. Prefer env vars or constructor args; both work. Keep the key out of the repo.
4. Manage message_history in cl.user_session yourself; expect the request to grow by two messages per turn.
5. Use stream=True with stream_token for the typing effect; omit stream if you want the SDK's minimal body.
6. Leave max_retries at 2 for flaky networks, or set 0 if you want hard failures in the UI.
7. Call cl.instrument_openai() once after creating the client if you want steps in the UI — and not if you already instrument LangChain.

That is the whole integration surface. Chainlit owns the chat UI and session bag; the openai client owns the wire format, retries and streaming. Point base_url at any OpenAI-compatible /v1, keep history under control, and the same app code works against a local mock, a self-hosted server or a hosted gateway.