How We Measure
A quick mental model for what actually becomes part of your graph, and how to put your scores to work.
What counts as a node
Two things build your graph: agents (a system prompt) and tools (a description). Everything else — plain LLM calls with no system prompt — isn't a decision point, so it's not counted.
Has a system prompt — a real decision point.
Has a description — same treatment as an agent.
No system prompt — treated as plumbing, not a decision.
Turning scores into improvements
Score + explanation isn't a report card — it's feedback you can act on.
Agent skips a step, scores low, explanation says why — feed that back into your orchestrator (or that agent's instructions) and retry. decision_identifier lets you confirm scores actually improve.
A strategy for building trust over time
Start with observability, then gate the moments that matter.
- New network — observability only (
ALWAYS), no manual gates yet. Watch the scores to see how it performs before touching anything. - Once you spot the risky step — add a manual gate right before it, not before every step.
A bad find or a rough draft costs nothing to redo. An unwanted purchase does — so that's where the gate goes, not earlier.
See the Usage Patterns section for the code behind a gate.
Authentication
API keys authenticate your agent's calls to Trellar. You'll need one before you can call the client library.
- Create a Trellar account (or sign in if you already have one).
- Once you're in the dashboard, go to API Keys and click Create API Key.
- Copy the key right away — for security, you won't be able to see it again.
Installation
trellar is the Python client for Trellar's confidence evaluation API. Install it, then set your API key so the client can authenticate.
Install
pip install "trellar[langchain]"Requires Python 3.9+. The langchain extra is required because agent runs are captured via a LangChain callback handler.
Set your API key
export TRELLAR_API_KEY=your-api-keyOr pass it directly: evaluate_confidence(api_key="...").
Quick Start
Attach the guard to your LangChain / LangGraph run, then call
evaluate_confidence() whenever you want a score — trace and agent context are picked up automatically.
from trellar import get_agent_guard, evaluate_confidence
# agent_name must be a stable, unique name for this agent graph — the
# backend uses it to track the graph's network profile across runs.
guard = get_agent_guard("research-agent")
graph.invoke(inputs, config={"callbacks": [guard]})
# context, trace_id, and agent_name are picked up from the guard
result = evaluate_confidence()
print(result.score) # int, 1-10
print(result.explanation) # str, human-readable reasoning
print(result.decision_identifier) # uuid, for tracing & BISingle LLM calls
Calling a chat model directly with no wrapping graph? Use get_single_call_guard instead — same idea, for one bare llm.invoke().
from trellar import get_single_call_guard, evaluate_confidence
guard = get_single_call_guard("single-llm-call")
llm.invoke(messages, config={"callbacks": [guard]})
result = evaluate_confidence()Agents built with create_react_agent are already compiled graphs, so they work with get_agent_guard as usual — get_single_call_guard is only for a genuinely bare model call.
The response object
Each call to evaluate_confidence() returns a result with the following fields:
The confidence score for this run — 1 is low confidence, 10 is high confidence.
A human-readable summary of the reasoning behind the score — handy for logs, audits, or showing a reviewer why a run was flagged.
A unique identifier for this evaluation. Use it to trace and correlate the decision across your own BI dashboards or third-party systems.
NetworkHaltedError
Not returned as a field, and not something that can trigger on its own. It only fires if you've created a rule with Stop Execution enabled and attached it to this agent's network — if that rule matches, evaluate_confidence() raises NetworkHaltedError instead of returning normally. No rule configured means this never happens. Catch it to halt the run gracefully.
from trellar import evaluate_confidence, NetworkHaltedError
try:
result = evaluate_confidence()
except NetworkHaltedError:
# Trellar determined this agent's network should stop — halt the run
return {**state, "halt": True}Observability Modes
Don't want to remember to call evaluate_confidence() yourself every time? Pass an observability_mode to
get_agent_guard() and it auto-triggers the call for you when the graph's root run finishes.
from trellar import get_agent_guard, ObservabilityMode
guard = get_agent_guard("research-agent", ObservabilityMode.IF_NOT_EVALUATED)
graph.invoke(inputs, config={"callbacks": [guard]})
# evaluate_confidence() has already run automatically if no node called it.Available modes
Never auto-call. Identical to not passing observability_mode at all — call evaluate_confidence() yourself wherever you need a score.
Always call evaluate_confidence() when the run finishes — in addition to any manual calls already made during the run.
Call evaluate_confidence() when the run finishes, but only if it wasn't already called successfully earlier in the run (e.g. from a gate node).
NetworkHaltedError — is caught and logged instead of propagating out of graph.invoke(). A manual call to evaluate_confidence() still raises normally.
observability_call: true in the payload sent to the backend (false for a normal, manually-invoked call), so it can tell automatic pings apart from explicit ones.
Usage Patterns
Call evaluate_confidence() from a graph node (or right after invoke()), at the point in the run you want scored. There are three common patterns:
Gate
Validate before the graph continues — halt or branch based on result.score.
def confidence_gate(state):
result = evaluate_confidence()
if result.score < 7:
return {**state, "halt": True, "reason": result.explanation}
return {**state, "halt": False}Observe
Record a score without restricting the graph — the run continues either way.
def report_confidence(state):
result = evaluate_confidence()
return {**state,
"confidence_score": result.score,
"confidence_explanation": result.explanation}Safety net
Gate on the runs that matter, and let IF_NOT_EVALUATED catch everything else — a score is always on record.
guard = get_agent_guard(
"research-agent",
ObservabilityMode.IF_NOT_EVALUATED,
)
graph.invoke(inputs, config={"callbacks": [guard]})
# a manual gate node may or may not have called
# evaluate_confidence() already — either way, a score
# is now on record for this run.AI Setup Prompts
Don't want to wire this up by hand? Copy one of these prompts into Cursor, Claude Code, or any other coding agent — paste it into a chat in your repo and it will read your code and do the integration for you.
Full integration
Finds your LangChain / LangGraph agent, installs the client, and wires up get_agent_guard() end-to-end.
I want to integrate the "trellar" client into this project so my agent's runs get scored for confidence.
Do the following:
1. Install it: pip install "trellar[langchain]" (needs Python 3.9+; the "langchain" extra is required because runs are captured via a LangChain callback handler). Use my project's existing package manager/lockfile if there is one.
2. Make sure the API key is read from an env var called TRELLAR_API_KEY — add it to my .env / secrets setup if I have one, and ask me for the value instead of hard-coding it.
3. Find where my LangChain/LangGraph graph is built and invoked (look for StateGraph, .compile(), or calls to .invoke()/.stream()/.ainvoke()).
4. Import get_agent_guard and evaluate_confidence from trellar. Create one guard per distinct agent graph with get_agent_guard("a-stable-unique-name-for-that-agent"), and pass it as config={"callbacks": [guard]} (merging with any callbacks I already pass) on every invoke/stream call for that graph.
5. Pick the right observability mode for each guard based on how the graph is used:
- ObservabilityMode.NONE (default) if I want to call evaluate_confidence() myself.
- ObservabilityMode.ALWAYS to score every run automatically when it finishes.
- ObservabilityMode.IF_NOT_EVALUATED to score automatically only when nothing already called evaluate_confidence() during the run.
Ask me which behavior I want if it's not obvious from context.
6. If it makes sense for my graph, add a manual evaluate_confidence() call at a meaningful checkpoint (e.g. before a risky or irreversible action) and branch on result.score. Catch NetworkHaltedError around any manual call, since it can be raised if I've configured a "Stop Execution" rule on that agent's network in the Trellar dashboard.
7. Don't wrap auto-triggered (observability-mode) evaluations in extra error handling — they never raise. Only add try/except around manual evaluate_confidence() calls.
8. Show me a summary of every file you changed before we're done.Add a confidence gate
Already have the guard installed? Drop a scored checkpoint into one specific node or function.
This project already uses the "trellar" client (get_agent_guard is attached to my graph's callbacks). Add a confidence gate at <describe the node/function/step here>.
Do the following:
1. In that node/function, call `result = evaluate_confidence()` — no arguments needed, it picks up the current trace, context, and agent name from the guard automatically.
2. Branch on `result.score` (int, 1-10): if it's below a sensible threshold (default to 7 unless I say otherwise), stop this run from continuing — return/raise/halt in whatever way matches the surrounding code — and surface `result.explanation` in the halt reason/log message.
3. If the score is high enough, let execution continue as normal.
4. Wrap the call in a try/except for NetworkHaltedError (from trellar) too — it means a "Stop Execution" rule I configured in the Trellar dashboard already matched, so handle it the same way as a low score.
5. Keep `result.decision_identifier` (uuid) around in whatever logging/tracing/BI event you emit for this step, so I can correlate it later.
6. Show me the diff for the change before finishing.