LLM

Language-model agents: the client, its cache, budgets, retries, providers and structured decisions. How to use them is in Language-model agents.

Language-model agents with runs that replay and costs that stay bounded.

Every call goes through a Client: it builds the request, answers from the ResponseCache when it can, holds the call against the Budget, sends it with retries and a concurrency limit, and caches the reply. Agents mix in LLMAgent and call self.decide(prompt, schema); decide_all asks a whole population at once and returns the decisions in agent order.

Adapters: AnthropicProvider (Messages API), OpenAICompatibleProvider (OpenAI and local servers such as Ollama or llama.cpp) and FakeProvider (tests and teaching). They use the standard library only; no provider SDK is needed.

Example

>>> from modelly import Agent, Model
>>> from modelly.llm import Client, FakeProvider, LLMAgent
>>> class Voter(LLMAgent, Agent):
...     persona = "You are a voter."
>>> model = Model(rng=7)
>>> model.llm = Client(FakeProvider(lambda r: '{"vote": "yes"}'))
>>> Voter(model).decide("Vote?", {"properties": {"vote": {"enum": ["yes", "no"]}}})
{'vote': 'yes'}

See docs/llm_agents.md for reproducibility, budgets, keys and limitations.

Client and agents

class Client(provider: Provider, *, model: str | None = None, cache: ResponseCache | None = None, budget: Budget | None = None, retry: RetryPolicy | None = None, max_concurrency: int = 4, system: str = '', seed: int | None = None, **params: Any)

Language-model calls with a cache, a budget, retries and a concurrency limit.

Parameters:

• provider – The adapter: AnthropicProvider, OpenAICompatibleProvider or FakeProvider.

• model – The model name; the provider’s default when None.

• cache – Where replies are kept; a fresh in-memory cache when None.

• budget – Token and cost limits; no limit when None.

• retry – How failed calls are retried.

• max_concurrency – The most calls in flight at once.

• system – The system prompt used when a call sets none.

• seed – Part of every cache key; agents pass their model’s seed instead.

• **params – Default call parameters (temperature, max_tokens, …).

Example

>>> from modelly.llm import FakeProvider
>>> client = Client(FakeProvider())
>>> client.complete("Hello").cached
False
>>> client.usage.calls
1

Set the provider and the defaults of every call.

property usage: Usage

Calls, cache hits, retries, tokens and cost so far.

next_index(scope: str) → int

The next call counter of scope, for calls not made by an LLMAgent.

request(prompt: str | Sequence[Message], *, scope: str = 'model', call_index: int | None = None, system: str | None = None, seed: int | None = None, schema: dict | None = None, **params: Any) → Request

Build a Request; prompt is a user message or a list of messages.

complete(prompt: str | Sequence[Message], **options: Any) → Response

One call: options are those of request().

complete_many(requests: Iterable[Request]) → list[Response]

Replies to requests, in their order.

Cached replies come first and cost nothing. The calls left are held against the budget in order; up to the first that could pass it, they are sent concurrently, counted and cached, then BudgetExceeded is raised for the rest.

Raises:

• CacheMiss – In replay mode, for the first call the cache does not hold.

• BudgetExceeded – Before the first call that could pass the budget.

• ProviderError – The first call (in order) that failed after its retries.

lookup(request: Request) → Response | None

The cached reply of request; a miss in replay mode raises.

reserve(batch: list[Request], pending: list[int]) → tuple[list[int], BudgetExceeded | None]

The pending calls the budget allows, and the refusal that stopped the rest.

dispatch(batch: list[Request], replies: list[Response | None], allowed: list[int]) → None

Send the allowed calls, then count and cache their replies in order.

send_here(request: Request) → Future

Send request in this thread; its outcome as a finished future.

collect(request: Request, future: Future, replies: list[Response | None], index: int) → ProviderError | None

Count and cache one finished call, or return its error.

class LLMAgent

Mix into an Agent (class Voter(LLMAgent, Agent)) to ask model.llm.

Variables:

• persona (str) – The system prompt of this agent’s calls; the client’s when empty.

• llm_calls (int) – How many calls this agent made (including repairs).

Example

>>> from modelly import Agent, Model
>>> from modelly.llm import Client, FakeProvider
>>> class Voter(LLMAgent, Agent):
...     persona = "You are a careful voter."
>>> model = Model(rng=1)
>>> model.llm = Client(FakeProvider())
>>> Voter(model).ask("Vote?").startswith("fake reply")
True

property llm: Client

The client of this agent’s model.

property llm_scope: str

Who is asking, in cache keys and error messages.

next_call_index() → int

Take this agent’s next call index.

llm_request(prompt: str, *, schema: dict | None = None, **params: Any) → Request

A request from this agent; with a schema, the prompt asks for JSON.

ask(prompt: str, **params: Any) → str

The reply text to prompt.

decide(prompt: str, schema: dict, *, repairs: int = 1, **params: Any) → Any

A decision matching schema, with up to repairs repair calls.

Raises:

DecisionError – When the reply is still invalid after the repairs.

decide_all(agents: Sequence[LLMAgent], prompt: str | Callable[[Any], str], schema: dict, *, repairs: int = 1, **params: Any) → list[Any]

One decision per agent, sent concurrently, returned in the order of agents.

Parameters:

• agents – LLMAgent instances of one model.

• prompt – The same prompt for all, or a function of the agent.

• schema – What each decision must match.

• repairs – Repair rounds for invalid replies.

• **params – Call parameters for every request.

Requests and replies

class Request(provider: str, model: str, messages: tuple[Message, ...], system: str = '', params: tuple[tuple[str, Any], ...] = (), scope: str = 'model', call_index: int = 0, seed: int | None = None, schema: dict | None = None)

Everything that decides a reply, and who asked.

scope and call_index say which caller made the call and how many it made before: an agent’s id and its own call counter. They are part of the cache key, so two agents asking the same question get their own replies, and the key of a call does not depend on which thread sent it.

Example

>>> r = Request("fake", "m", (Message("user", "Hi"),), scope="a1", call_index=0)
>>> len(r.key())
64

property max_tokens: int | None

The max_tokens parameter, the most the reply may use.

identity() → dict[str, Any]

The fields the cache key hashes, as plain JSON values.

key() → str

SHA-256 of the canonical JSON of identity(): the cache key.

prompt_chars() → int

Characters in the system prompt and every message.

excerpt() → str

The start of the last message, for error messages.

follow_up(reply: str, prompt: str, call_index: int) → Request

This conversation, the model’s reply, then a new user prompt.

class Response(text: str, input_tokens: int = 0, output_tokens: int = 0, cached: bool = False, key: str = '')

A reply: its text, the tokens it used, and whether it came from the cache.

property tokens: int

Input plus output tokens.

class Message(role: str, content: str)

One turn of a conversation: who spoke ("user" or "assistant") and what.

Cache

class ResponseCache(path: str | Path | None = None, mode: str = 'read-write')

Replies by cache key, in memory (path=None) or in a SQLite file.

Parameters:

• path – The SQLite file; created when missing. None keeps the cache in memory.

• mode – "read-write", "read-only" or "off".

Open (or create) the store.

classmethod from_file(path: str | Path, mode: str = 'read-only') → ResponseCache

An in-memory cache filled from an exported file (JSON lines or SQLite).

The file itself is never written, so a cache shipped with a study bundle or a package stays as it was.

property reads: bool

Whether lookups are made (every mode but "off").

property writes: bool

Whether new replies are stored ("read-write" only).

get(key: str) → Response | None

The stored reply for key, marked cached, or None.

put(request: Request, response: Response) → None

Store response under the key of request, in one transaction.

insert(rows: list[tuple]) → None

Write rows (in COLUMNS order) atomically.

rows() → Iterator[tuple]

Every stored row, sorted by key.

export(path: str | Path) → Path

Write the cache to path: JSON lines for .jsonl (.jsonl.gz), else SQLite.

The file is written next to its target and renamed over it, so a reader never sees half a file.

merge(path: str | Path) → int

Add every row of an exported file; return how many rows it held.

close() → None

Close the connection.

Budgets and retries

class Budget(max_tokens: int | None = None, max_cost: float | None = None, prices: Mapping[str, ~modelly.llm.budget.Price]=<factory>, usage: Usage = <factory>)

Limits for one run and the usage counted against them.

Parameters:

• max_tokens – The most input plus output tokens the run may send for. None: no limit.

• max_cost – The most the run may cost. Needs a price for every model it calls.

• prices – Model name to Price.

estimate(request: Request) → tuple[int, int]

Worst-case (input, output) tokens of request.

price_of(model: str) → Price | None

The price of model; required when a cost budget is set.

reserve(request: Request) → None

Hold the worst case of request, or raise BudgetExceeded.

refuse_over(tokens: int, cost: float, request: Request) → None

Raise when tokens or cost passes its limit.

release(request: Request) → None

Give back the reservation of request once its reply is counted.

record(request: Request, response: Response, retries: int) → None

Count a reply from the provider.

record_hit() → None

Count a reply served from the cache.

class Price(input_per_million: float, output_per_million: float)

Price per million input tokens and per million output tokens, in any currency.

cost(input_tokens: int, output_tokens: int) → float

The price of one call.

class Usage(calls: int = 0, cache_hits: int = 0, retries: int = 0, input_tokens: int = 0, output_tokens: int = 0, cost: float | None = 0.0)

What a run used so far.

Variables:

• calls (int) – Calls sent to the provider (retries not included).

• cache_hits (int) – Replies served from the cache.

• retries (int) – Extra tries after a 429, a 5xx or a lost connection.

• input_tokens (int) – Prompt tokens of the calls sent.

• output_tokens (int) – Reply tokens of the calls sent.

• cost (float | None) – The cost of the calls sent, or None when a model has no price.

property tokens: int

Input plus output tokens.

class RetryPolicy(max_retries: int = 3, base_delay: float = 1.0, max_delay: float = 30.0, sleeper: ~collections.abc.Callable[[float], None] = <built-in function sleep>)

How many times to retry a failed call and how long to wait in between.

Parameters:

• max_retries – Extra tries after the first; 0 never retries.

• base_delay – Seconds before the first retry; doubled each time after.

• max_delay – The longest wait, also capping a server’s Retry-After.

• sleeper – Called with the seconds to wait; time.sleep by default.

Example

>>> RetryPolicy(base_delay=1.0).delay(2, None)
4.0

sleeper()

sleep(seconds)

Delay execution for a given number of seconds. The argument may be a floating-point number for subsecond precision.

delay(attempt: int, retry_after: float | None) → float

Seconds to wait before retry number attempt (counting from 0).

run(send: Callable[[Request], Response], request: Request) → tuple[Response, int]

send(request), retried on a retryable error; return the reply and the retries.

Providers

class Provider(*, api_key: str | None = None, base_url: str | None = None, timeout: float = 120.0)

The interface of every adapter: a name and complete().

Subclasses set name (part of every cache key), default_model and key_variable, the environment variable holding the API key.

Keep the endpoint; the key stays in memory only.

static default_url() → str

The endpoint used when base_url is None.

api_key(required: bool = True) → str | None

The explicit key, else the environment’s; raise when one is required.

complete(request: Request) → Response

Send request and return the reply.

class AnthropicProvider(*, api_key: str | None = None, base_url: str | None = None, timeout: float = 120.0)

The Anthropic Messages API (POST /v1/messages); key from ANTHROPIC_API_KEY.

Example

>>> AnthropicProvider(api_key="sk-...")
AnthropicProvider(base_url='https://api.anthropic.com')

Keep the endpoint; the key stays in memory only.

static default_url() → str

The endpoint used when base_url is None.

complete(request: Request) → Response

Call the Messages API and join the text blocks of the reply.

class OpenAICompatibleProvider(*, name: str | None = None, **options: Any)

Any POST /chat/completions endpoint: OpenAI, or a local Ollama or llama.cpp.

The key comes from OPENAI_API_KEY and is optional, since local servers need none. name is part of the cache key: give each endpoint its own when they serve different models under one name.

Example

>>> OpenAICompatibleProvider(base_url="http://127.0.0.1:11434/v1", name="ollama")
OpenAICompatibleProvider(base_url='http://127.0.0.1:11434/v1')

Set the endpoint, and a name for its cache keys.

static default_url() → str

The endpoint used when base_url is None.

complete(request: Request) → Response

Call chat completions; the system prompt is the first message.

class FakeProvider(responder: ~collections.abc.Callable[[~modelly.llm.request.Request], str] = <function echo>, failures: ~collections.abc.Iterable[int] = ())

Replies come from responder(request) -> str; tokens are counted from length.

Parameters:

• responder – A module-level function (so a model holding it can be saved).

• failures – HTTP statuses to raise, one per call, before answering normally: failures=[429, 503] fails twice, then replies.

Example

>>> from modelly.llm import Client
>>> client = Client(FakeProvider(lambda request: "yes"))
>>> client.complete("Agree?").text
'yes'

Keep the responder and the failures to play back.

complete(request: Request) → Response

Fail with the next queued status, if any; otherwise call the responder.

Schemas

validate(value: Any, schema: dict, path: str = '$') → Any

Return value when it matches schema; raise SchemaError otherwise.

Parameters:

• value – Parsed JSON.

• schema – A schema using the supported keywords.

• path – Where value sits, for the error message.

parse_json(text: str) → Any

The JSON value in a reply: plain, inside a code fence, or after some prose.

Raises:

SchemaError – When the reply holds no JSON value.

Errors

exception LLMError

Base class of every error raised by modelly.llm.

exception ProviderError(message: str, *, status: int | None = None, retry_after: float | None = None, retryable: bool | None = None)

A provider refused or failed a call.

Variables:

• status – The HTTP status, or None when no response came back.

• retryable – Whether the same call may succeed later (429, 5xx, a lost connection).

• retry_after – Seconds the server asked to wait, when it said so.

Record the status and whether another try may succeed (from it, by default).

exception CacheMiss(scope: str, call_index: int, key: str, excerpt: str)

A replay (mode="read-only") asked for a response the cache does not hold.

Variables:

• scope – Who made the call (an agent’s id, or the scope passed to the client).

• call_index – The caller’s call counter for this call.

• key – The cache key that was looked up.

Name the call, so the user can see which prompt was never recorded.

exception BudgetExceeded

The next call could take the run past its token or cost budget.

Raised before that call is sent, so the money is never spent. Responses already received in the same batch are cached and counted first.

exception SchemaError(path: str, problem: str)

A value does not match a schema; path points at the first mismatch.

Say where the value breaks the schema and how.

exception DecisionError(scope: str, reply: str, reason: str)

A model reply stayed invalid after every repair attempt.

Variables:

• scope – Who asked.

• reply – The last reply received.

Keep the reply so the prompt or the schema can be fixed.