Judge, file by file
Every file in Project 3: the three Maven modules (97 Java classes plus prompts and configuration), all 59 test files including the fakes that stand in for Claude and Arena, and every ops, evaluation and documentation file. The project's own tour is project3_explained.html (its section 5 predates ArenaToolExecutor, ArenaReader and AttemptRecord; this atlas includes them).
All code and tests pass without an API key. The live smoke test has never run, and evals/ contains no dataset or results yet. Describe evaluation classes as "built and tested on synthetic fixtures", never as having produced numbers.
1. The shape of the repository
Project 3/ Maven reactor, Boot 4.1.2, anthropic-java 3.10.0 (pinned), Resilience4j 2.4.0
├── pom.xml parent: pinned versions, test.excludedGroups=live, Failsafe, JaCoCo 85%/module
├── judge-core/ 49 classes NO Spring, NO Kafka: agent loop, tools, routing, policy, cost, prompts, settings
│ └── resources/ judge-defaults.yml, prompts/v1/{system,rulebook}.md, prompts/CHANGELOG.md
├── judge-service/ 21 classes Spring Boot app: Kafka listener, Arena client, applier, metrics, error handling
├── judge-eval/ 27 classes CLI app: collect | run | compare, replaying recorded tools through JudgeEngine
├── evals/ README.md, dataset/ and results/ (empty, .gitkeep)
├── ops/ README.md, prometheus-judge.yml, grafana/dashboards/judge.json
├── Dockerfile compose.yaml .env.example .dockerignore .gitignore LICENSE README.md .github/workflows/ci.yml
└── (local only: CLAUDE.md, handoff.md, todo.md, the spec, code_explanation/)
JudgeEngine.judge(flag, tools) in judge-core is the single entry point both apps use: the service passes live Arena tools, the evaluation passes recorded ones.
2. judge-core: root, settings and prompts
Paths relative to judge-core/src/main/java/dev/fairplay/judge/core/.
| File | What it is | Why it is like this | Learn |
|---|---|---|---|
JudgeEngine.java | create(settings, gateway, clock) wires prompts, the loop, the router and the policy; investigate wraps the tools in a fresh MemoizingToolExecutor per flag; judge(flag, tools[, status]); decide(investigation, status) to rerun the policy after a 409 | One decision path for production and evaluation, so the evaluation measures the real thing. | 17 |
JudgeJson.java | The one shared Jackson 3 JsonMapper, ignoring unknown fields | Immutable, thread-safe, tolerant reader. | 04 |
JudgeSettings.java | A record tree (models, limits, routing, caching, policy, anthropic retry/breaker/limiter, pricing) validated in compact constructors: prompt version matches [A-Za-z0-9._-]+ (a folder name, not a path); both models need prices; autoBanMinConfidence ≥ 0.80 (Arena's floor); requirePromptFiles() | Plain Java (no Spring annotations), bound from outside with Spring's Binder; the regex prevents path traversal into other folders. | 03 |
prompt/PromptLibrary.java | Loads prompts/<version>/system.md and rulebook.md once, hashes them (SHA-256), and builds the two system blocks, the rulebook carrying cacheControl only when caching is on | The single cache breakpoint; nothing per-flag in the cached prefix. | 11 |
prompt/PromptVersion.java | version, the two texts and their hashes | Hashes go into every evaluation report header, tying results to exact prompt text. | – |
prompt/UserPromptBuilder.java | The per-flag user message: "Flagged at <ISO time>. Investigate this player and return your verdict." plus the pretty-printed flag | All per-flag data after the cache breakpoint; never a label. | 11 |
resources/judge-defaults.yml | Every shared setting: prompt version (from JUDGE_PROMPT_VERSION), the two models, limits 6/8/1500, routing thresholds, caching, policy 0.90 and a 30-minute window, Anthropic timeout 60 s, retry 4 from 1 s ×2 up to 20 s, breaker 20/10/50%/30 s/3, limiter 120 per minute, and the price table | Imported by both apps, so they judge with identical settings. | 12 |
resources/prompts/v1/system.md | The role, priorities ("Banning an innocent player is the worst outcome"), the step-by-step method and the evidence rule | Behaviour is configuration, versioned like code. | 11 |
resources/prompts/v1/rulebook.md | Decision definitions, game facts (top speed 7.0 with noise to 7.5; human reaction ranges; headshot and accuracy ranges), how to weigh evidence, stricter criteria for PLATINUM and PRO, confidence and review guidance, evidence format, tool use | Domain knowledge the model can't invent; the cached part of the prompt. | 11 |
resources/prompts/CHANGELOG.md | One row per prompt version; only v1 exists | v2–v6 must each come from a real dev report: one change, never edited after evaluation. | 11 |
3. judge-core/agent: the loop, the output and the gateway
| File | What it is | Why it is like this | Learn |
|---|---|---|---|
agent/AgentLoop.java | The hand-written tool loop (at most 6 turns and 8 tool calls); always wraps the backend in GuardedToolExecutor; answers all tool calls of one turn in one message in order; parses on end_turn; any other stop reason is invalid; converts SDK tool input (Jackson 2) to Jackson 3 by writing a string and reading it back | Every turn and token is visible and testable with a scripted gateway. | 11 |
agent/AgentRun.java | A successful attempt: verdict + stats | – | – |
agent/AgentStats.java | Token usage, turns, tool calls, stop reasons, per-tool outcomes; an EMPTY constant | Failed attempts carry it too, because they still cost money. | – |
agent/MessagesGateway.java | A one-method interface: Message create(MessageCreateParams) | The seam that lets tests replace Claude. | 13 |
agent/ResilientMessagesGateway.java | The production gateway: Retry(CircuitBreaker(RateLimiter(call))); logs only request id, model, token counts and stop reason; maps 400 to NonRetryableJudgeException and outages to LlmUnavailableException | Retry outermost; one retry layer. | 12 |
agent/AnthropicResilience.java | Builds the three Resilience4j components named anthropic: jittered exponential backoff that honours Retry-After (capped at 60 s); retries only 429, 5xx and I/O; the breaker records 5xx and I/O and ignores 400/401/403/429; the limiter; logs breaker transitions at WARN; exposes registries for metrics | Failure classification is the heart of resilience. | 12 |
agent/VerdictSchema.java | The verdict JSON schema as a text block, turned into the SDK's OutputConfig through the untyped path (one putAdditionalProperty per field) | The schema is a fixed contract (a documented decision). | 11 |
agent/VerdictParser.java | Strict parsing: exactly one JSON object (FAIL_ON_TRAILING_TOKENS), exactly the six fields, no nulls, correct types, enums upper-cased; then validation | Trailing text or Markdown fences are parse failures, never silently accepted. | 11 |
agent/VerdictValidator.java | Confidence 0..1 and not NaN; 1–6 evidence items of 1–300 characters; explanation 1–600; BAN needs a cheat type; CLEAR needs NONE | Rules the enforced schema can't express. | 11 |
agent/InvalidVerdictException.java | Kind PARSE or VALIDATION, carrying the attempt's stats (withStats) | Counted separately in metrics; triggers escalation. | – |
agent/TurnLimitExceededException.java | Turns or tool calls exhausted, with stats | Escalation reason TURN_LIMIT. | – |
agent/LlmUnavailableException.java | Claude unreachable after retries (429, 5xx, timeout, network, open breaker, limiter timeout, 401/403) | Means "hold the flag", never drop it. | 12 |
agent/NonRetryableJudgeException.java | A malformed flag or a Claude 400 | Means "dead-letter now". | 09 |
4. judge-core/tools: what the model may touch
| File | What it is | Why it is like this | Learn |
|---|---|---|---|
tools/ToolExecutor.java | The interface: execute(toolName, input, flaggedPlayerId) | Live, recorded, guarded, memoising and fake implementations all share it (decorators). | 17 |
tools/ToolResult.java | Content text + an isError flag; ok() and error() | Errors go back to the model so it can recover. | – |
tools/ToolCatalog.java | The four strict tool definitions in a fixed order, each requiring only player_id (the two list tools also take limit), with additionalProperties: false | Fixed order and text keep the cached prefix stable; strict schemas can't hold min/max, so limit's range is enforced by the guard. | 11 |
tools/GuardedToolExecutor.java | The safety decorator: known tool, object input, allowed properties only, the flagged player only, limit 1–20 (default 10); passes a canonical input to the delegate; echoes model text back at most 60 characters | A prompt-injected model can't reach another player or pass odd arguments. | 11 |
tools/MemoizingToolExecutor.java | Caches results per tool name + canonical (key-sorted) JSON input in a ConcurrentHashMap; one instance per flag; exceptions not cached | Sonnet reuses Haiku's reads; Arena is queried once per flag. | 02 |
tools/ArenaReader.java | Read-only Arena port: player, recentMatches, moderationHistory (empty = 404) | Implemented by the service's resilient client and by the evaluation's plain one. | 17 |
tools/ArenaToolExecutor.java | Turns Arena responses into compact tool results: profile (status, country, account age in days, matches, rating), stats and percentiles (or a note when under 5 matches), matches (headshot rate and accuracy to 3 decimals), history (reason cut to 200 characters); reuses the pre-check's player response | Fewer tokens, no irrelevant fields, one Arena read. Lives in core (a documented decision) so collect records with the same code. | 11 |
tools/ArenaUnavailableException.java | Arena unreachable after its retries, or its breaker is open | Aborts the investigation so the flag is retried, never judged on missing data. | 12 |
5. judge-core: routing, policy, cost and the domain model
| File | What it is | Why it is like this | Learn |
|---|---|---|---|
routing/ModelRouter.java | Haiku first; escalate per the rule or on INVALID / TURN_LIMIT; Sonnet in a fresh conversation, final; records every attempt with cost and latency; modes via triageEnabled (baseline) and escalationEnabled (haiku-only) | Cost-aware routing with independent second opinions. | 11 |
routing/EscalationRule.java | First match wins: LOW_CONFIDENCE (certainty max(c, 1−c) < 0.85), HIGH_TIER_BAN, THIN_EVIDENCE (< 2 items) | The symmetric reading is a documented, still-open decision. | 16 |
routing/EscalationReason.java | INVALID, TURN_LIMIT, LOW_CONFIDENCE, HIGH_TIER_BAN, THIN_EVIDENCE | The judge.escalations metric tag. | – |
policy/DecisionPolicy.java | The 7-row table from verdict + status to exactly one ArenaAction, then state adjustments (FLAG only from ACTIVE or REINSTATED, never BAN a banned player); reasons prefixed [Judge v1, model] and cut to 500 characters | A pure function: the model never chooses an HTTP call. | 11 |
cost/CostCalculator.java | (input·p_in + cacheWrite·p_cw + cacheRead·p_cr + output·p_out) / 106, BigDecimal, 6 dp HALF_UP | Measured cost per verdict, exact money arithmetic. | 02 |
cost/PriceTable.java | Per-model prices from settings; an unknown model throws | Settings validation already ensures both models are priced. | – |
model/SuspiciousPlayer.java | The Judge's copy of Telemetry's flag (ignores unknown fields); playerUuid() is a method, not a JSON field | The contract is the JSON, not a shared jar. | 01 |
model/Evidence.java, Rule.java, SkillTier.java | Copies of Telemetry's types; SkillTier.isTopTier() is PLATINUM or PRO | Each service owns its model of the contract. | – |
model/Verdict.java | decision, cheatType, confidence, evidence, explanation, needsHumanReview (null-checked) | Plain data; validity comes from the validator. | – |
model/Decision.java, CheatType.java | BAN/WATCH/CLEAR and AIMBOT/SPEEDHACK/NONE, each with a case-insensitive parse | Models may change capitalisation even under a schema. | – |
model/TokenUsage.java | input, cacheWrite, cacheRead, output; fromSdk (absent cache fields count as 0); plus | Total input = input + cacheRead + cacheWrite. | 11 |
model/Attempt.java, AttemptRecord.java | One model's try (model, outcome FINAL/ESCALATED/INVALID/TURN_LIMIT, stats, cost, latency, reason, escalation, invalid kind) and its flattened JSON form | AttemptRecord is in core because the evaluation reuses it. | – |
model/Investigation.java | The final verdict (or null) plus every attempt; final model, escalated, total cost and latency | Everything the models did, before any action. | – |
model/ArenaAction.java | The chosen action: kind, label, reason, evidence, confidence (or null), review flag | Always sent with source JUDGE. | – |
model/ActionKind.java | BAN, FLAG, CLEAR, NOTE: what is sent to Arena | – | – |
model/JudgeAction.java | BAN, FLAG_FOR_REVIEW, NOTE, CLEAR, SKIPPED: what is recorded | A review request sent as a NOTE (for an already-flagged player) still counts as FLAG_FOR_REVIEW. | – |
model/JudgeOutcome.java | Investigation + the chosen action | – | – |
model/PlayerStatus.java | Arena's five statuses; canBeFlagged() (ACTIVE, REINSTATED); isBanned() (BANNED, APPEALED) | Mirrors Arena's state machine on the client side. | – |
6. judge-service: the running application
Paths relative to judge-service/src/main/java/dev/fairplay/judge/service/.
| File | What it is | Why it is like this | Learn |
|---|---|---|---|
JudgeServiceApplication.java | The Boot entry point | – | – |
config/StartupChecks.java | Refuses to start without ANTHROPIC_API_KEY and the Arena password, naming what's missing and never printing values | Fail fast, never leak secrets. | 05 |
config/JudgeSettingsConfig.java | Binds judge.* onto the Spring-free JudgeSettings with Binder and checks the prompt files exist; enables JudgeProperties | Core stays free of Spring. | 03 |
config/JudgeProperties.java | Service-only settings: Kafka topics and retry interval; the Arena URL, account, timeouts, retry | Validated in constructors. | – |
config/AnthropicConfig.java | One SDK client (maxRetries(0), request timeout, key and optional base URL from the Environment); the resilience stack bound to Micrometer; the gateway; the JudgeEngine | Reading ANTHROPIC_BASE_URL from the Environment lets tests point it at WireMock (a documented decision). | 12 |
config/ArenaConfig.java | The UTC clock bean and the ArenaClient bean with its retry and breaker bound to Micrometer | Takes StartupChecks as a parameter so the check runs first. | 03 |
config/KafkaConfig.java | Declares only its own topics (judge-verdicts: compacted, 6 partitions; the DLT: 7 days); DefaultErrorHandler with FixedBackOff(5 s, unlimited) and non-retryable exceptions to the DLT on the same partition | Never touches Telemetry's topics; never drops a flag during an outage. | 09 |
arena/ArenaClient.java | Implements ArenaReader and sends actions; RestClient on the JDK HTTP client with 2 s connect and 5 s read timeouts; token login and one refresh on 401; retry (3 attempts from 500 ms) around a breaker (20/10/50%, open 15 s) for 5xx and I/O; 409 → ArenaConflictException; other 4xx on an action → non-retryable; persistent 401 or 403 → unavailable (held) | A long-running client deserves a breaker; credentials problems are held, not dropped. | 12 |
arena/ArenaConflictException.java | Arena answered 409 to an action (someone acted in between) | Triggers re-read and re-decide. | – |
arena/PreChecks.java | Free checks from one GET /players/{id}: unknown (missing, not a PLAYER, or an unknown status) or already banned (BANNED or APPEALED) means skip | No tokens spent on pointless flags. | 11 |
arena/ArenaActionApplier.java | Sends the chosen action; on 409 re-reads the status and runs the policy once more; a second 409 (or a vanished player) records a NOTE with no Arena action | Handles the one race the state machine allows, without involving the model. | 12 |
verdict/JudgeService.java | One flag end to end: player read → pre-checks + "recently judged" → JudgeEngine.judge with an ArenaToolExecutor reusing the player → apply → build the record → metrics → one structured log line | Publishing is the listener's job, after Arena succeeded. | 01 |
verdict/JudgeVerdictRecord.java | The judge-verdicts record: ids, flag info, decision, cheat type, confidence, evidence, explanation, review flag, action, Arena action id, skip reason, attempts, total cost and latency, prompt version, decidedAt | The Judge's only state, keyed by flag id on a compacted topic. | 09 |
verdict/VerdictRecords.java | Builds judged and skipped records (no verdict means review requested) | – | – |
verdict/SkipReason.java | ALREADY_BANNED, UNKNOWN_PLAYER, RECENTLY_JUDGED | – | – |
kafka/FlagListener.java | @KafkaListener(id = "judge", autoStartup = "false"): parse (bad → non-retryable, never echoing the text), skip already-judged flags, handle, publish and wait for the ack, remember | Idempotent at-least-once processing. | 09 |
kafka/VerdictPublisher.java | Sends the record keyed by flag id; the future completes on the broker's ack (acks=all) | A re-judged flag replaces its old record after compaction. | – |
kafka/ProcessedFlags.java | In-memory: every flag id seen, and per player the latest BAN or FLAG_FOR_REVIEW verdict (newest wins regardless of replay order) | Only actionable verdicts count, so a later skip can't hide an earlier review flag (a documented decision). | 16 |
kafka/VerdictSnapshotReader.java | Reads the whole verdicts topic as of now (no group, read_committed, to the recorded end offsets, 30 s timeout), skipping unreadable records | The same technique as Telemetry's snapshot reader. | – |
kafka/ProcessedFlagsLoader.java | On ApplicationReadyEvent: replay the verdicts, then start the listener container | A restart never judges a flag twice. | 03 |
metrics/JudgeMetrics.java | All judge.* meters: verdicts by decision and action, latency timer, cost distribution (min 1e-6, max 1.0), attempts, escalations, invalid verdicts, LLM calls by stop reason, tokens by kind, tool calls, skips, duplicates; low-cardinality tags only | The cost histogram needed custom bounds: the default started at 1, putting every sub-cent verdict in one bucket. | 14 |
resources/application.yml | Imports judge-defaults.yml; virtual threads; consumer group judge-agent, read_committed, 1 record per poll, 10-minute poll interval; idempotent producer; ack per record; concurrency 3; port 8083; topics and Arena settings | – | 09 |
7. judge-eval: the evaluation harness
Paths relative to judge-eval/src/main/java/dev/fairplay/judge/eval/. Built and tested on synthetic fixtures; not yet run on real data.
| File | What it is | Why it is like this | Learn |
|---|---|---|---|
EvalApplication.java | A non-web Boot app that exits with the command's code | – | – |
EvalCommandLine.java | ApplicationRunner + ExitCodeGenerator delegating to EvalCli | – | – |
EvalArgs.java | Parses one command word plus --name=value options; unknown options are errors; dotted options (--judge.caching.enabled=false) are left to Spring | A typo like --mdoe is reported, not ignored. | – |
EvalCli.java | The three commands: collect, run (prompt, mode, subset, limit) and compare; secrets and network behind suppliers; exit 0 or 2 | compare needs no key, Arena or Kafka; missing secrets are named before any work. | – |
config/EvalConfig.java | Wiring: the same JudgeSettings binding as the service; one SDK client with maxRetries(0); lazily memoised suppliers for the gateway and collector | Nothing networked is built until a command needs it. | – |
config/EvalProperties.java | eval.*: dataset path, results dir, concurrency 4, collect targets (120 cheaters, 80 non-cheaters, seed 7), Arena account | – | – |
collect/TopicSource.java | An interface for reading a whole topic | A seam so collect is tested without a broker. | 13 |
collect/KafkaTopicSource.java | The real implementation: no group, read_committed, never commits | Can't disturb Telemetry's consumers. | – |
collect/GroundTruth.java | Telemetry's ground-truth record, read only here | The Judge never sees it while judging. | – |
collect/CaseSampler.java | One candidate per player (a cheater's earliest flag at or after the cheat start; any flag for others); proportional split across cheat types (largest remainder); PRO before HONEST for non-cheaters; ids c-001... in shuffled order; stratified 50-case dev subset | Pure and deterministic; ids don't reveal labels; dev keeps the overall ratio. | 11 |
collect/ArenaHttpReader.java | A small Arena reader for collect (login, three GETs, a few retries, one re-login) | Runs once for a minute, so no breaker needed. | – |
collect/DatasetCollector.java | Reads ground truth (latest per key, tombstones) and flags, samples, records the four tools' answers per case (lists at 20 items) with the production ArenaToolExecutor, writes cases.jsonl and a README | Snapshots make runs repeatable and cheap. | 11 |
model/EvalCase.java | caseId, subset, source run, a Label (isCheater, segment, cheatType), the flag, tool snapshots | The label lives only in label. | – |
model/Subset.java, SubsetSelection.java | DEV/HOLDOUT (written lowercase) and ALL/DEV/HOLDOUT for --subset | – | – |
run/EvalMode.java | ROUTED, BASELINE (Sonnet only, no caching), HAIKU_ONLY (never escalate), each deriving settings from the shared defaults | The modes that split cost savings into routing and caching. | 11 |
run/RecordedToolExecutor.java | Answers from snapshots, cutting lists to the requested limit; refuses other players like production | Lives in eval, not core (a documented decision). | – |
run/EvalRunner.java | Replays cases through JudgeEngine on virtual threads (at most concurrency at once, sharing the rate limiter); an unanswerable case becomes an error result; results keep dataset order | Bypasses pre-checks so every case is judged; the player is assumed ACTIVE. | 02 |
run/CaseResult.java | One case's outcome: decision, action, attempts, cost, latency, final outputs and parse or validation failures, or an error | The base for schema validity. | – |
run/DatasetFiles.java | Reads and writes JSONL; SHA-256 of the dataset | The hash ties reports to the exact dataset. | – |
score/Scorer.java | Pure scoring: banned only if the policy produced BAN; precision, recall, false-ban reduction, review rates, escalation rate, schema validity, cost and latency percentiles (nearest rank), confusion table, recall per cheat type, false bans per segment; lists of false bans and missed cheaters | Error cases counted but not scored. | 11 |
score/EvalMetrics.java | All of the above as a record; undefined ratios are null ("n/a") | – | – |
report/RunInfo.java | Prompt version and hashes, models, mode, caching, subset, dataset hash, start time; modeLabel() adds "(no cache)" | Every result is traceable. | – |
report/RunSummary.java | The .summary.json: run info plus overall, dev and holdout metrics | – | – |
report/ReportWriter.java | Writes .jsonl, .summary.json and a Markdown report ending with every false ban and missed cheater with evidence and explanation | That list is what you read to write the next prompt version. | – |
report/ComparisonWriter.java | Builds COMPARISON.md from all summaries: rows per version, mode and subset; the cost saving split into routing and caching; total schema validity | – | – |
report/Format.java | Percent, USD, number and table-cell formatting (pipes and newlines removed) | Model text can't break a Markdown table. | – |
resources/application.yml | Non-web, no banner; imports judge-defaults.yml; eval settings with environment overrides | – | – |
8. Tests: what each class proves
judge-core: 194 tests (97.2% lines); judge-service: 77 unit + 12 integration (96.9%); judge-eval: 105 unit + 1 integration (96.2%). Counts in brackets are test methods.
judge-core tests and shared helpers (a test-jar)
| Class | What it proves or provides |
|---|---|
agent/ScriptedGateway | A fake MessagesGateway that plays back scripted responses and records requests |
agent/TestMessages | Builders for SDK Message objects (tool use, final text, stop reasons, usage) |
agent/StubAnthropicServer | A local HTTP stub of the Messages API, so the real SDK client can be tested |
tools/FakeToolExecutor, TestSettings, MutableClock | Canned tool answers with call recording; default settings; a controllable clock |
SdkSmokeTest (1, tagged live) | One real tool round trip ending in a parseable verdict; confirms the real request shape and prints cache token counts. Never run yet. |
agent/AgentLoopTest (21) | Tool rounds, multiple tools per turn in order, tool errors returned with isError, another player refused whatever executor is passed, backend exceptions abort, invalid stop reasons and unparseable text with costs reported, limits |
agent/ResilientMessagesGatewayTest (16) | Against the real SDK and a stub server: 429 (honouring retry-after), 529, 500, resets and timeouts retried; max attempts become unavailable; 400 non-retryable; 401/403 held as unavailable; the breaker opens, fails fast and closes again |
agent/AnthropicResilienceTest (9) | Retry-After parsing and capping; only transient failures retried; exponential jittered backoff stays below the maximum; one shared breaker named anthropic |
agent/VerdictParserTest (13) | Valid parse; integer confidence; enum case; whitespace; every kind of PARSE failure; an out-of-range value is a VALIDATION failure |
agent/VerdictValidatorTest (8) | Every boundary of confidence, evidence count and length, explanation length; BAN/CLEAR cheat-type rules |
agent/VerdictSchemaTest (3) | The schema matches the contract; every object forbids additional properties; the output config carries it |
tools/ToolExecutorGuardTest (11) | Other players never reach the backend; non-UUID ids count as other players; unknown or null tools; long names cut before echoing; unexpected properties; limit boundaries; the backend sees only canonical input |
tools/MemoizingToolExecutorTest (7) | Same call once; argument order doesn't matter; different limit or tool calls again; error results cached; exceptions not cached; nested keys sorted |
tools/ToolCatalogTest (4) | Four tools with exact names in order; all strict, requiring only player_id; only list tools take a limit; definitions stable across calls |
tools/ArenaToolExecutorTest (14) | Documented fields only; one Arena read for both player tools (and none when the pre-check's response is passed); compact rates; reasons cut; unknown player is an error; unreachable Arena aborts |
routing/ModelRouterTest (24) | When Haiku is final, every escalation reason, symmetric confidence boundaries, Sonnet's fresh conversation with the same tool cache, failures on both sides, modes |
policy/DecisionPolicyTest (17) | Every row (row 2 inclusive at exactly 0.90), state adjustments, and no BAN below the Arena floor is ever produced |
cost/CostCalculatorTest (10) | Worked prices for each kind of token, rounding, cached reads cheaper than uncached, no overflow, unpriced model rejected |
prompt/PromptLibraryTest (7) | v1 loads; cache control only when enabled; block order; no per-flag content in system blocks; identical text every call; missing version fails; the rulebook describes the catalog's tools |
prompt/UserPromptBuilderTest (4) | ISO time first; the flag round-trips; unused evidence fields omitted; no label |
model/SuspiciousPlayerTest (6) | Reads the contract example; unknown fields ignored; missing username and tier allowed; UUID handling; top tiers |
JudgeEngineTest (7) | End-to-end in core: confident ban; reason names the final model; no verdict means review; evaluation assumes ACTIVE; policy rerun after a conflict; tool cache per flag, not per engine |
JudgeSettingsTest (13) | Defaults valid; auto-ban below Arena's floor rejected; every range and section check; prompt version must be a folder name, not a path |
judge-service tests
| Class | What it proves or provides |
|---|---|
support/WireMockContainer | Runs wiremock/wiremock:3.13.2 and programs stubs and fault scenarios over its admin API (no WireMock Java dependency) |
support/KafkaTestSupport, support/Flags | Kafka container access, topic helpers, flag builders and awaitVerdict |
arena/StubArenaServer | A fake Arena for the client unit tests |
JudgeFlowIT (5) | One flag gives one verdict and exactly one JUDGE action in (WireMock) Arena; replaying the flag adds no Arena and no model call; a banned player is skipped with no model call; a malformed flag lands in the DLT; /actuator/prometheus exposes the dashboard's meters |
ProcessedFlagsIT (1) | Earlier verdicts are loaded first and their flags are not judged again |
ResilienceIT (6) | 429, 529, timeout and reset are retried to a verdict; a 400 goes straight to the DLT; a 20-second outage drops no flag and opens the breaker (20 of 20 judged afterwards) |
arena/ArenaClientTest (18) | Bearer token and single login; 404 is empty; paging; 401 refresh once; refused credentials, failed login and closed port are "unavailable"; 5xx and dropped connections retried; the breaker opens and fails fast; action requests |
arena/ArenaActionApplierTest (5) | Accepted action reported; a conflict re-reads and decides once more; a second conflict or a vanished player gives a NOTE without an action |
arena/PreChecksTest (5) | Active judged; missing or non-player unknown; banned and appealed skipped; unknown status treated as unknown |
verdict/JudgeServiceTest (10) | Ban applied and recorded; one Arena read; skips (banned, unknown, recently judged; older judgements don't block); no verdict means review; an unreachable Arena stops the flag for retry; contract field names |
kafka/FlagListenerTest (7) | Judge, publish, remember; duplicates ignored and counted; not remembered when publishing fails; outages rethrown so the offset isn't committed; malformed flags non-retryable; the error message never contains the flag text |
kafka/ProcessedFlagsTest (6) | The 30-minute window per player; notes, clears and skips don't start it; a skip doesn't hide an earlier review flag; newest record wins regardless of replay order |
config/KafkaConfigTest (6) | Verdicts topic compacted with 6 partitions; DLT kept 7 days; no Telemetry topic declared; outages and unexpected errors retried, never recovered; malformed flags to the DLT on the same partition |
config/ConfigWiringTest (9) | Startup with both secrets; each missing secret named without values; SDK client from the environment with or without a base URL; engine, gateway and Arena client wired and bound to metrics; UTC clock |
config/JudgeConfigBindingTest (8) | Shared defaults bind; environment overrides; auto-ban below the floor stops startup; the shipped v1 prompts load; a missing prompt folder is rejected |
metrics/JudgeMetricsTest (3) | An escalated verdict counts every meter; no verdict counts as NONE; skips and duplicates |
judge-eval tests
| Class | What it proves or provides |
|---|---|
Fixtures | Synthetic cases and flags (never real data) |
EvalCliTest (14) | run writes reports and honours subset and limit; the baseline mode appears in file names; a missing key or password is named; bad options reported; compare works without any key; usage on unknown commands; exit codes |
collect/CaseSamplerTest (12) | Early cheater flags excluded; PRO before HONEST; proportional split; warnings when short; same seed, same ids; ids from c-001; dev subset stratified |
collect/DatasetCollectorTest (8) | Writes the dataset and README; the label is only in the label field, never in the flag or snapshots; latest ground truth wins; bad flags skipped |
collect/ArenaHttpReaderTest (9) | Login once, reads, retries, refresh once, error cases |
collect/KafkaTopicSourceIT (1) | Reads every record of a real topic; nothing for a missing topic |
config/EvalConfigTest (8) | Binding; the same judging settings as the service; nothing networked built early; the gateway needs a key |
run/EvalRunnerTest (8) | Every case goes through the production policy; the label never reaches the model; order kept under concurrency; failures become error results; modes use the right models and caching |
run/EvalModeTest (6) | Each mode's settings, parsing |
run/RecordedToolExecutorTest (8) | Snapshots unchanged; lists cut to the limit; another player refused like production |
run/DatasetFilesTest (4) | Round trip with contract names; blank lines; a missing dataset says "collect first"; the hash changes with content |
score/ScorerTest (14) | Precision, recall, false-ban reduction, review and escalation rates, confusion table, undefined ratios, nearest-rank percentiles, schema validity, error cases not scored |
report/ReportWriterTest (7) | File names; the three files; header, tables and every mistake; n/a for undefined; model text can't break tables |
report/ComparisonWriterTest (7) | Rows per version, mode and subset; the cost split into routing and caching; no cost section without a baseline; schema validity summed |
9. Build, ops, evaluation and documentation files
| File | What it is | Learn |
|---|---|---|
pom.xml (parent) | Three modules; pins anthropic-java 3.10.0 and Resilience4j 2.4.0 in dependency management; test.excludedGroups=live (a property, because Surefire ANDs groups with excludedGroups); Failsafe; JaCoCo 85% per module | 14 |
judge-core/pom.xml | Only anthropic-java, Jackson, the three Resilience4j cores, Micrometer core and SLF4J; builds a test-jar of its helpers | 17 |
judge-service/pom.xml | Core (and its test-jar), webmvc, Kafka, Actuator, Prometheus, resilience4j-micrometer, Boot plugin | – |
judge-eval/pom.xml | Core (and its test-jar), Kafka, RestClient starter, Boot plugin | – |
mvnw, mvnw.cmd, .mvn/wrapper/ | The Maven Wrapper: a pinned Maven version, no installation needed | 14 |
Dockerfile | Multi-stage, ARG MODULE=judge-service, cache mount, JRE, non-root, port 8083 (the eval runs on the host) | 14 |
compose.yaml | The service on Telemetry's external network telemetry_default (Kafka at kafka:19092), Arena through host.docker.internal, secrets from .env, 1 GB limit | 14 |
.env.example | ANTHROPIC_API_KEY, optional ANTHROPIC_BASE_URL (tests only), JUDGE_SERVICE_PASSWORD, Arena URL, Kafka bootstrap, prompt version | 05 |
.dockerignore, .gitignore | Keep secrets, build output and the local notes out | – |
.gitattributes | Line endings by file type: LF for mvnw and *.sh, CRLF for *.cmd, so the scripts run on Linux CI, in the container and on Windows whichever machine checked them out | – |
.github/workflows/ci.yml | sh ./mvnw -B verify with no API key (no test needs one); uploads JaCoCo reports | 13 |
judge-service/src/test/resources/wiremock/tool-use-turn.json, final-ban.json, final-clear.json | Canned Messages API responses: a tool-use turn (with cache-read usage) and two final verdicts | 13 |
ops/README.md | How to add the Judge's scrape job and dashboard to Telemetry's Prometheus and Grafana by hand (this project never modifies Telemetry) | – |
ops/prometheus-judge.yml | The scrape job for judge-service:8083 | 14 |
ops/grafana/dashboards/judge.json | 7 panels: verdicts/min by action, average cost per verdict, p95 latency, cache-hit share of input tokens, escalation rate, human-review share, breaker state | 14 |
evals/README.md | Exact steps to produce the dataset (a Telemetry run with loosened thresholds, Judge stopped) and the four measurement runs. States clearly that nothing has been produced yet | 11 |
evals/dataset/.gitkeep, evals/results/.gitkeep | Keep the empty folders in git until real files exist | – |
README.md | Overview, architecture, quick start, how a verdict is made, evaluation, cost, resilience, the results table with labelled imaginary values (only the fault-injection row measured), design decisions, testing, stack, limitations | 15 |
LICENSE | Apache License 2.0 | – |
Fairplay Judge — Implementation Specification.md | The 1,187-line design, including the rulebook text, routing and policy tables, Kafka integration, resilience, cost, evaluation, testing and build order (local only) | 17 |
CLAUDE.md, handoff.md, todo.md, code_explanation/ | Local working notes (decisions, SDK findings, status) and your existing walkthrough | 16 |
10. Code-level probe questions
Why must the prompt version match [A-Za-z0-9._-]+? Deep
The version becomes part of a classpath path, prompts/<version>/system.md, and it can come from an environment variable. Without the check, a value like ../../something would make the loader read files outside the prompts folder (path traversal). JudgeSettingsTest.promptVersionIsAFolderNameNotAPath covers it.
What's the difference between ActionKind and JudgeAction? Intermediate
ActionKind is what is sent to Arena (BAN, FLAG, CLEAR, NOTE). JudgeAction is what the Judge records and counts (BAN, FLAG_FOR_REVIEW, NOTE, CLEAR, SKIPPED). They differ because Arena only allows FLAG from ACTIVE or REINSTATED: a review request for an already-flagged player is sent as a NOTE with requiresHumanReview, but still counts as FLAG_FOR_REVIEW in metrics and evaluation.
Why is the verdict parser so strict about trailing text? Intermediate
If it accepted "here's my verdict: {...}" or JSON inside Markdown fences, it would hide the model drifting from the required format, and the evaluation's schema-validity metric would be wrong. Strict parsing (FAIL_ON_TRAILING_TOKENS, exactly six fields, correct types) turns any drift into a counted PARSE failure that escalates to Sonnet.
Why did the cost histogram need minimumExpectedValue(0.000001)? Curveball
Micrometer's default histogram buckets for a distribution summary start around 1. A verdict costs a fraction of a cent, so every value fell into the first bucket and the p95 was meaningless. Setting the expected range to 1e-6..1.0 USD gives buckets where the data actually is.
How does the evaluation make sure the model can't see the label? Deep
Three layers, each tested: the label lives only in EvalCase.label, never in the flag or snapshots (DatasetCollectorTest); case ids are assigned in shuffled order so c-017 doesn't hint at anything (CaseSamplerTest); and EvalRunnerTest.theLabelNeverReachesTheModel inspects what's actually sent. Ground truth is read only by collect.
A Claude 401 isn't dead-lettered. Why? Deep
A 401 or 403 means credentials or permissions are wrong, which is an operational problem with the service, not with the flag. Dead-lettering would throw away perfectly good flags because of a bad key. So they become LlmUnavailableException: the flag is held and retried every 5 seconds until someone fixes the key, and the breaker and metrics make the problem visible. Only a 400, which is specific to the request, goes to the DLT.
Why does ArenaToolExecutor take the pre-check's player response? Intermediate
The pre-check already did GET /players/{id} to decide whether to judge. That response also contains everything get_player_profile and get_stat_percentiles need. Passing it in means Arena is read once per flag for those three uses, and with the memoising executor, Sonnet's escalation re-reads nothing. JudgeServiceTest.theToolsReuseThePreCheckResponseSoArenaIsReadOnce checks it.
What does EvalArgs do with --judge.caching.enabled=false? Curveball
It skips any option whose name contains a dot and leaves it for Spring, which binds it as a property override. That's how the "routed without caching" run is made without a special flag. Any other unknown non-dotted option is an error, so typos are reported instead of silently ignored.
What would break if FlagListener logged the flag JSON on a parse error? Intermediate
Nothing functional, but it would put player data (ids, usernames, evidence) into logs, which the project's rules forbid. Logs are widely accessible and long-lived. The listener's messages describe the problem ("flag is not valid JSON") without the content, and FlagListenerTest.theErrorMessageNeverContainsTheFlagText enforces it.
Why can't the service's ArenaClient be used by collect? Deep
It lives in judge-service, which is packaged as a Spring Boot fat jar; the eval module can't depend on another app's repackaged jar. That's also why the read port (ArenaReader) and the tool code (ArenaToolExecutor) were moved into judge-core: the eval gets its own small ArenaHttpReader adapter, but records snapshots with exactly the production tool code.