extension-inference

Caffeine Inference

LLM extension for Caffeine AI.

Orchestrator routing notes

Treat “use an LLM / GPT / chatbot / summarise with AI” as a first-class platform feature. The default path is Caffeine Inference: an OpenAI-compatible chat endpoint that Caffeine hosts, authenticates, and bills for the app. The canister gets its credentials from the platform at runtime; nobody pastes an API key, and the app never stores or returns one.

User intent	Capability
Chat / summarise / classify with an LLM in a Caffeine app	caffeineai-inference-client ChatApi.createChatCompletion via this skill
Call api.openai.com with a user-pasted sk-...	extension-openai only

Do not load extension-openai for a normal Caffeine-app LLM. Do not ask the user for an OpenAI API key. Do not add setApiKey endpoints, a key-settings page, or a model picker.

Backend

1. Add caffeineai-inference-client to mops.toml

mops add [email protected]

Requires Mops ≥ 2.13. Minimum version: caffeineai-inference-client ≥ 0.2.0.

2. Config comes from the platform

Config.fromEnv<system>() returns a complete Config — endpoint, bearer, and is_replicated = ?false — from the credentials the platform provisions for the app. There is no key to collect and nothing to configure.

• Call fromEnv<system>() inside the shared method, or in a <system>-parameterised helper, on every request. A module-level let config = fromEnv will not compile, and a cached Config can go stale when the platform rotates credentials on a running canister.
• It traps when the app has no inference credentials. Let that error reach the caller; hiding it makes a failed AI call look like a working feature:
  • Never pre-check CAFFEINE_INFERENCE_API_KEY (or any other environment variable) and never guard fromEnv with a check that skips the call.
  • Never turn a failed call into an empty success, such as an empty string, an empty list, or placeholder text. Trap or return the error so the frontend can show it and offer a retry.
  • Do not add a "configure AI" empty state or a key-input fallback.
• Never log the Config, never copy its auth into actor state, and never return it (or any part of it) from a query / shared function.

3. is_replicated = ?false is REQUIRED

fromEnv already sets this. Do not override it to ?true or null.

1. Security. A replicated outcall sends the bearer from every replica.
2. Billing. Replicated outcalls multiply inference spend by subnet size.
3. Determinism. LLM bodies are sampled; consensus would fail.

4. Canonical layout

import Inference "lib/inference";

actor {
  public shared func chat(prompt : Text) : async Text {
    await* Inference.runChat<system>(prompt);
  };
};

import { fromEnv } "mo:caffeineai-inference-client/Config";
import ChatApi "mo:caffeineai-inference-client/Apis/ChatApi";
import ChatCompletionRequest "mo:caffeineai-inference-client/Models/ChatCompletionRequest";
import ChatCompletionRequestMessageOneOf2 "mo:caffeineai-inference-client/Models/ChatCompletionRequestMessageOneOf2";
import Runtime "mo:core/Runtime";

module {
  public func runChat<system>(prompt : Text) : async* Text {
    let config = fromEnv<system>();
    let userMessage = ChatCompletionRequestMessageOneOf2.JSON.init({
      content = #string(prompt);
      role = #user;
    });
    let req = ChatCompletionRequest.JSON.init({
      messages = [#user(userMessage)];
      model = "router";
    });
    let resp = await* ChatApi.createChatCompletion(config, req);
    if (resp.choices.size() == 0) {
      Runtime.trap("Inference returned no choices");
    };
    resp.choices[0].message.content
      ?? Runtime.trap("Inference returned no text content");
  };
};

5. model = "router" — the platform picks the model

"router" is the only public model id, and it is a routing tier rather than a model name. Caffeine Inference sizes each request to the complexity of the query — a small fast model for simple prompts, a stronger one for hard reasoning — and reports "router" back as the response model, so provider names never reach the app.

• Always send model = "router".
• Do not add a model dropdown, a "use GPT-4" toggle, or a model parameter on the backend endpoint. There is nothing for the user to choose.
• Steer quality with the prompt and with the declared sampling fields (temperature, top_p, max_completion_tokens), not with model selection.

6. Call shapes

• Function form: ChatApi.createChatCompletion(config, req) : async* — use await*.
• Suite form: let api = ChatApi(config); api.createChatCompletion(req) : async.

7. Available API surface — chat completions

[email protected] is generated from public-api-v0.1.0:

Module	Entry point	Route
ChatApi	createChatCompletion	POST /v1/chat/completions
ModelsApi	listModels	GET /v1/models — catalog only; the model is always "router", so an app never needs this

import ChatApi "mo:caffeineai-inference-client/Apis/ChatApi";
import { fromEnv } "mo:caffeineai-inference-client/Config";

Chat completions are the whole product surface. Not available on this host (404, and not in the package): embeddings, images, audio, moderations, files, legacy completions, Assistants, Responses, and raw ic.http_request. If the spec genuinely needs an OpenAI-only API with a pasted sk-..., switch to extension-openai.

8. Cycles

Leave cycles unset (null): each call attaches the fee the IC charges for that request, which is zero on subnets where outcalls are free. ?n attaches exactly n cycles: the IC rejects the call when n is below its fee, and the call traps on a subnet where the canister holds fewer than n.

Streaming (stream = ?true) is unsupported — management-canister HTTP returns the full body. Leave stream = null.

9. Things that will bite you

• Call fromEnv<system>() inside the shared method (or a <system> helper). A module-level let config = fromEnv will not compile.
• Surface failures. No credential pre-checks, no guard around fromEnv, and no empty result in place of an error. See §2.
• model = "router" — not "gpt-4o-mini". See §5.
• User turns are #user(ChatCompletionRequestMessageOneOf2.JSON.init({ content = #string(prompt); role = #user })).
• JSON.init for required fields; layer optionals with record update. Do not hand-list every null.
• resp.choices[0].message.content is ?Text. Check choices.size() first.
• One chat call is one HTTP outcall inside an update call: budget seconds, not milliseconds.

Frontend

The app is ready to chat on first load — there is nothing to configure.

1. No API-key UI. No settings page, no password input, no "configured?" indicator, no localStorage. If a spec or mock shows an "AI settings" screen, drop it.
2. No model picker. See §5.
3. Call the backend chat endpoint (chat(prompt)) and render the returned text. There is no frontend LLM SDK — the canister is the client, so the credentials never reach the browser.
4. Show a pending state while the call is in flight (an outcall round-trip takes seconds) and surface a retry on trap.