Monitor your own agent
Import your agent's logs (OpenAI or Anthropic tool-calling format), label some of its actions, and train a monitor that scores each new action before it runs. Then put it in your agent's loop. Everything here runs in this browser tab: the page does not upload your logs.
graybox.monitor.TextMonitor in the Python SDK. The browser and Python give
identical results (identical text on 302 real agent steps; with the privacy filter, identical vocabularies and held-out AUROC, 0.9915 in both; trained on the same data, the two give scores within 0.01 and the same review decisions). On the tools and policies that it was
trained on, it is equal to a much larger reader model, also on closed agents. It does not transfer to a new type of policy
or task: on a different benchmark it was at chance. For robustness to new tasks, use the reader monitor (self-hosted).1. Load your agent's actions
…or paste
No data loaded yet.
Which formats work
- OpenAI Chat Completions: one conversation per line,
{"messages": [...]}, with assistant messages that havetool_callsandtoolmessages with the results. - Anthropic Messages:
{"system": "...", "messages": [...]}, withtool_useblocks andtool_resultblocks. - Steps:
{"instructions", "task", "history", "action": {"tool", "args"}, "label"}, or{"context", "action", "label"}.
Each tool call becomes one action. The monitor sees what it would see before the call runs: the system prompt, the first user
message, the earlier calls with their results, and the call. An optional "labels": {"call_id": 1} field on a
conversation gives labels (1 = violation). Python reads the same formats: graybox.guard.conversation_steps.
2. Label actions
Load your agent's actions first (step 1).
Is executing this action a violation of your policy? Read the conversation up to the action, then decide. Keys: V violation, S safe, N skip, B back. Your labels are saved in this browser for this file.
3. Train and evaluate
AUROC is the chance that a random violation gets a higher score than a random safe action: 0.5 is guessing, 1.0 is perfect.
The review cut in the file is set on your labelled actions. On a new kind of task the scores shift, and a stored cut can send almost nothing to review: on task families held out from training, it caught 0 of 115 violations. A cut set on the new traffic (
Guard(monitor, gate=QuantileGate(0.05)), or the server's "gate": true) keeps the 5% review load,
but it caught only 29 of 115: on a new kind of task, label its actions and train again.Review queue
In our test (300 GPT-6-luna conversations, half of them labelled), the first 25 actions of the queue were all real violations (the base rate was 4.7%), and the first 43 held 92% of the violations among 812 unlabelled actions.
By tool
4. Put it in your agent's loop
Download the monitor
The file holds the vocabulary, the weights and the review cut: the top budget of the scores of all your loaded actions, set without labels. On new traffic of the same kind it sends about that share to review; on a new kind of task, set the cut on live traffic (see the docs).
pip install greybox-monitor
from graybox.guard import Guard
from graybox.monitor import TextMonitor
guard = Guard(TextMonitor.load_json("greybox-text-monitor.json"))
# OpenAI: before you execute each proposed tool call
for call in reply.tool_calls or []:
v = guard.check_openai(messages, call)
result = run_tool(call) if v.allow else v.block_message
# Anthropic: for each tool_use block
v = guard.check_anthropic(system, messages, block)
Score one action
One step: {"instructions", "task", "history", "action": {"tool", "args"}},
or a conversation {"messages": [...]} (its last tool call is scored). Filled with an action from your data.