Fairplay Interview Prep
Module 21 · Codebase atlas

Judge, file by file

Every file in Project 3: the three Maven modules (97 Java classes plus prompts and configuration), all 59 test files including the fakes that stand in for Claude and Arena, and every ops, evaluation and documentation file. The project's own tour is project3_explained.html (its section 5 predates ArenaToolExecutor, ArenaReader and AttemptRecord; this atlas includes them).

P3 Judge· reference; about 2 hours to read once
Status reminder

All code and tests pass without an API key. The live smoke test has never run, and evals/ contains no dataset or results yet. Describe evaluation classes as "built and tested on synthetic fixtures", never as having produced numbers.

1. The shape of the repository

Project 3/                               Maven reactor, Boot 4.1.2, anthropic-java 3.10.0 (pinned), Resilience4j 2.4.0
├── pom.xml                              parent: pinned versions, test.excludedGroups=live, Failsafe, JaCoCo 85%/module
├── judge-core/      49 classes          NO Spring, NO Kafka: agent loop, tools, routing, policy, cost, prompts, settings
│   └── resources/  judge-defaults.yml, prompts/v1/{system,rulebook}.md, prompts/CHANGELOG.md
├── judge-service/   21 classes          Spring Boot app: Kafka listener, Arena client, applier, metrics, error handling
├── judge-eval/      27 classes          CLI app: collect | run | compare, replaying recorded tools through JudgeEngine
├── evals/           README.md, dataset/ and results/ (empty, .gitkeep)
├── ops/             README.md, prometheus-judge.yml, grafana/dashboards/judge.json
├── Dockerfile  compose.yaml  .env.example  .dockerignore  .gitignore  LICENSE  README.md  .github/workflows/ci.yml
└── (local only: CLAUDE.md, handoff.md, todo.md, the spec, code_explanation/)

JudgeEngine.judge(flag, tools) in judge-core is the single entry point both apps use: the service passes live Arena tools, the evaluation passes recorded ones.

2. judge-core: root, settings and prompts

Paths relative to judge-core/src/main/java/dev/fairplay/judge/core/.

FileWhat it isWhy it is like thisLearn
JudgeEngine.javacreate(settings, gateway, clock) wires prompts, the loop, the router and the policy; investigate wraps the tools in a fresh MemoizingToolExecutor per flag; judge(flag, tools[, status]); decide(investigation, status) to rerun the policy after a 409One decision path for production and evaluation, so the evaluation measures the real thing.17
JudgeJson.javaThe one shared Jackson 3 JsonMapper, ignoring unknown fieldsImmutable, thread-safe, tolerant reader.04
JudgeSettings.javaA record tree (models, limits, routing, caching, policy, anthropic retry/breaker/limiter, pricing) validated in compact constructors: prompt version matches [A-Za-z0-9._-]+ (a folder name, not a path); both models need prices; autoBanMinConfidence ≥ 0.80 (Arena's floor); requirePromptFiles()Plain Java (no Spring annotations), bound from outside with Spring's Binder; the regex prevents path traversal into other folders.03
prompt/PromptLibrary.javaLoads prompts/<version>/system.md and rulebook.md once, hashes them (SHA-256), and builds the two system blocks, the rulebook carrying cacheControl only when caching is onThe single cache breakpoint; nothing per-flag in the cached prefix.11
prompt/PromptVersion.javaversion, the two texts and their hashesHashes go into every evaluation report header, tying results to exact prompt text.–
prompt/UserPromptBuilder.javaThe per-flag user message: "Flagged at <ISO time>. Investigate this player and return your verdict." plus the pretty-printed flagAll per-flag data after the cache breakpoint; never a label.11
resources/judge-defaults.ymlEvery shared setting: prompt version (from JUDGE_PROMPT_VERSION), the two models, limits 6/8/1500, routing thresholds, caching, policy 0.90 and a 30-minute window, Anthropic timeout 60 s, retry 4 from 1 s ×2 up to 20 s, breaker 20/10/50%/30 s/3, limiter 120 per minute, and the price tableImported by both apps, so they judge with identical settings.12
resources/prompts/v1/system.mdThe role, priorities ("Banning an innocent player is the worst outcome"), the step-by-step method and the evidence ruleBehaviour is configuration, versioned like code.11
resources/prompts/v1/rulebook.mdDecision definitions, game facts (top speed 7.0 with noise to 7.5; human reaction ranges; headshot and accuracy ranges), how to weigh evidence, stricter criteria for PLATINUM and PRO, confidence and review guidance, evidence format, tool useDomain knowledge the model can't invent; the cached part of the prompt.11
resources/prompts/CHANGELOG.mdOne row per prompt version; only v1 existsv2–v6 must each come from a real dev report: one change, never edited after evaluation.11

3. judge-core/agent: the loop, the output and the gateway

FileWhat it isWhy it is like thisLearn
agent/AgentLoop.javaThe hand-written tool loop (at most 6 turns and 8 tool calls); always wraps the backend in GuardedToolExecutor; answers all tool calls of one turn in one message in order; parses on end_turn; any other stop reason is invalid; converts SDK tool input (Jackson 2) to Jackson 3 by writing a string and reading it backEvery turn and token is visible and testable with a scripted gateway.11
agent/AgentRun.javaA successful attempt: verdict + stats––
agent/AgentStats.javaToken usage, turns, tool calls, stop reasons, per-tool outcomes; an EMPTY constantFailed attempts carry it too, because they still cost money.–
agent/MessagesGateway.javaA one-method interface: Message create(MessageCreateParams)The seam that lets tests replace Claude.13
agent/ResilientMessagesGateway.javaThe production gateway: Retry(CircuitBreaker(RateLimiter(call))); logs only request id, model, token counts and stop reason; maps 400 to NonRetryableJudgeException and outages to LlmUnavailableExceptionRetry outermost; one retry layer.12
agent/AnthropicResilience.javaBuilds the three Resilience4j components named anthropic: jittered exponential backoff that honours Retry-After (capped at 60 s); retries only 429, 5xx and I/O; the breaker records 5xx and I/O and ignores 400/401/403/429; the limiter; logs breaker transitions at WARN; exposes registries for metricsFailure classification is the heart of resilience.12
agent/VerdictSchema.javaThe verdict JSON schema as a text block, turned into the SDK's OutputConfig through the untyped path (one putAdditionalProperty per field)The schema is a fixed contract (a documented decision).11
agent/VerdictParser.javaStrict parsing: exactly one JSON object (FAIL_ON_TRAILING_TOKENS), exactly the six fields, no nulls, correct types, enums upper-cased; then validationTrailing text or Markdown fences are parse failures, never silently accepted.11
agent/VerdictValidator.javaConfidence 0..1 and not NaN; 1–6 evidence items of 1–300 characters; explanation 1–600; BAN needs a cheat type; CLEAR needs NONERules the enforced schema can't express.11
agent/InvalidVerdictException.javaKind PARSE or VALIDATION, carrying the attempt's stats (withStats)Counted separately in metrics; triggers escalation.–
agent/TurnLimitExceededException.javaTurns or tool calls exhausted, with statsEscalation reason TURN_LIMIT.–
agent/LlmUnavailableException.javaClaude unreachable after retries (429, 5xx, timeout, network, open breaker, limiter timeout, 401/403)Means "hold the flag", never drop it.12
agent/NonRetryableJudgeException.javaA malformed flag or a Claude 400Means "dead-letter now".09

4. judge-core/tools: what the model may touch

FileWhat it isWhy it is like thisLearn
tools/ToolExecutor.javaThe interface: execute(toolName, input, flaggedPlayerId)Live, recorded, guarded, memoising and fake implementations all share it (decorators).17
tools/ToolResult.javaContent text + an isError flag; ok() and error()Errors go back to the model so it can recover.–
tools/ToolCatalog.javaThe four strict tool definitions in a fixed order, each requiring only player_id (the two list tools also take limit), with additionalProperties: falseFixed order and text keep the cached prefix stable; strict schemas can't hold min/max, so limit's range is enforced by the guard.11
tools/GuardedToolExecutor.javaThe safety decorator: known tool, object input, allowed properties only, the flagged player only, limit 1–20 (default 10); passes a canonical input to the delegate; echoes model text back at most 60 charactersA prompt-injected model can't reach another player or pass odd arguments.11
tools/MemoizingToolExecutor.javaCaches results per tool name + canonical (key-sorted) JSON input in a ConcurrentHashMap; one instance per flag; exceptions not cachedSonnet reuses Haiku's reads; Arena is queried once per flag.02
tools/ArenaReader.javaRead-only Arena port: player, recentMatches, moderationHistory (empty = 404)Implemented by the service's resilient client and by the evaluation's plain one.17
tools/ArenaToolExecutor.javaTurns Arena responses into compact tool results: profile (status, country, account age in days, matches, rating), stats and percentiles (or a note when under 5 matches), matches (headshot rate and accuracy to 3 decimals), history (reason cut to 200 characters); reuses the pre-check's player responseFewer tokens, no irrelevant fields, one Arena read. Lives in core (a documented decision) so collect records with the same code.11
tools/ArenaUnavailableException.javaArena unreachable after its retries, or its breaker is openAborts the investigation so the flag is retried, never judged on missing data.12

5. judge-core: routing, policy, cost and the domain model

FileWhat it isWhy it is like thisLearn
routing/ModelRouter.javaHaiku first; escalate per the rule or on INVALID / TURN_LIMIT; Sonnet in a fresh conversation, final; records every attempt with cost and latency; modes via triageEnabled (baseline) and escalationEnabled (haiku-only)Cost-aware routing with independent second opinions.11
routing/EscalationRule.javaFirst match wins: LOW_CONFIDENCE (certainty max(c, 1−c) < 0.85), HIGH_TIER_BAN, THIN_EVIDENCE (< 2 items)The symmetric reading is a documented, still-open decision.16
routing/EscalationReason.javaINVALID, TURN_LIMIT, LOW_CONFIDENCE, HIGH_TIER_BAN, THIN_EVIDENCEThe judge.escalations metric tag.–
policy/DecisionPolicy.javaThe 7-row table from verdict + status to exactly one ArenaAction, then state adjustments (FLAG only from ACTIVE or REINSTATED, never BAN a banned player); reasons prefixed [Judge v1, model] and cut to 500 charactersA pure function: the model never chooses an HTTP call.11
cost/CostCalculator.java(input·p_in + cacheWrite·p_cw + cacheRead·p_cr + output·p_out) / 106, BigDecimal, 6 dp HALF_UPMeasured cost per verdict, exact money arithmetic.02
cost/PriceTable.javaPer-model prices from settings; an unknown model throwsSettings validation already ensures both models are priced.–
model/SuspiciousPlayer.javaThe Judge's copy of Telemetry's flag (ignores unknown fields); playerUuid() is a method, not a JSON fieldThe contract is the JSON, not a shared jar.01
model/Evidence.java, Rule.java, SkillTier.javaCopies of Telemetry's types; SkillTier.isTopTier() is PLATINUM or PROEach service owns its model of the contract.–
model/Verdict.javadecision, cheatType, confidence, evidence, explanation, needsHumanReview (null-checked)Plain data; validity comes from the validator.–
model/Decision.java, CheatType.javaBAN/WATCH/CLEAR and AIMBOT/SPEEDHACK/NONE, each with a case-insensitive parseModels may change capitalisation even under a schema.–
model/TokenUsage.javainput, cacheWrite, cacheRead, output; fromSdk (absent cache fields count as 0); plusTotal input = input + cacheRead + cacheWrite.11
model/Attempt.java, AttemptRecord.javaOne model's try (model, outcome FINAL/ESCALATED/INVALID/TURN_LIMIT, stats, cost, latency, reason, escalation, invalid kind) and its flattened JSON formAttemptRecord is in core because the evaluation reuses it.–
model/Investigation.javaThe final verdict (or null) plus every attempt; final model, escalated, total cost and latencyEverything the models did, before any action.–
model/ArenaAction.javaThe chosen action: kind, label, reason, evidence, confidence (or null), review flagAlways sent with source JUDGE.–
model/ActionKind.javaBAN, FLAG, CLEAR, NOTE: what is sent to Arena––
model/JudgeAction.javaBAN, FLAG_FOR_REVIEW, NOTE, CLEAR, SKIPPED: what is recordedA review request sent as a NOTE (for an already-flagged player) still counts as FLAG_FOR_REVIEW.–
model/JudgeOutcome.javaInvestigation + the chosen action––
model/PlayerStatus.javaArena's five statuses; canBeFlagged() (ACTIVE, REINSTATED); isBanned() (BANNED, APPEALED)Mirrors Arena's state machine on the client side.–

6. judge-service: the running application

Paths relative to judge-service/src/main/java/dev/fairplay/judge/service/.

FileWhat it isWhy it is like thisLearn
JudgeServiceApplication.javaThe Boot entry point––
config/StartupChecks.javaRefuses to start without ANTHROPIC_API_KEY and the Arena password, naming what's missing and never printing valuesFail fast, never leak secrets.05
config/JudgeSettingsConfig.javaBinds judge.* onto the Spring-free JudgeSettings with Binder and checks the prompt files exist; enables JudgePropertiesCore stays free of Spring.03
config/JudgeProperties.javaService-only settings: Kafka topics and retry interval; the Arena URL, account, timeouts, retryValidated in constructors.–
config/AnthropicConfig.javaOne SDK client (maxRetries(0), request timeout, key and optional base URL from the Environment); the resilience stack bound to Micrometer; the gateway; the JudgeEngineReading ANTHROPIC_BASE_URL from the Environment lets tests point it at WireMock (a documented decision).12
config/ArenaConfig.javaThe UTC clock bean and the ArenaClient bean with its retry and breaker bound to MicrometerTakes StartupChecks as a parameter so the check runs first.03
config/KafkaConfig.javaDeclares only its own topics (judge-verdicts: compacted, 6 partitions; the DLT: 7 days); DefaultErrorHandler with FixedBackOff(5 s, unlimited) and non-retryable exceptions to the DLT on the same partitionNever touches Telemetry's topics; never drops a flag during an outage.09
arena/ArenaClient.javaImplements ArenaReader and sends actions; RestClient on the JDK HTTP client with 2 s connect and 5 s read timeouts; token login and one refresh on 401; retry (3 attempts from 500 ms) around a breaker (20/10/50%, open 15 s) for 5xx and I/O; 409 → ArenaConflictException; other 4xx on an action → non-retryable; persistent 401 or 403 → unavailable (held)A long-running client deserves a breaker; credentials problems are held, not dropped.12
arena/ArenaConflictException.javaArena answered 409 to an action (someone acted in between)Triggers re-read and re-decide.–
arena/PreChecks.javaFree checks from one GET /players/{id}: unknown (missing, not a PLAYER, or an unknown status) or already banned (BANNED or APPEALED) means skipNo tokens spent on pointless flags.11
arena/ArenaActionApplier.javaSends the chosen action; on 409 re-reads the status and runs the policy once more; a second 409 (or a vanished player) records a NOTE with no Arena actionHandles the one race the state machine allows, without involving the model.12
verdict/JudgeService.javaOne flag end to end: player read → pre-checks + "recently judged" → JudgeEngine.judge with an ArenaToolExecutor reusing the player → apply → build the record → metrics → one structured log linePublishing is the listener's job, after Arena succeeded.01
verdict/JudgeVerdictRecord.javaThe judge-verdicts record: ids, flag info, decision, cheat type, confidence, evidence, explanation, review flag, action, Arena action id, skip reason, attempts, total cost and latency, prompt version, decidedAtThe Judge's only state, keyed by flag id on a compacted topic.09
verdict/VerdictRecords.javaBuilds judged and skipped records (no verdict means review requested)––
verdict/SkipReason.javaALREADY_BANNED, UNKNOWN_PLAYER, RECENTLY_JUDGED––
kafka/FlagListener.java@KafkaListener(id = "judge", autoStartup = "false"): parse (bad → non-retryable, never echoing the text), skip already-judged flags, handle, publish and wait for the ack, rememberIdempotent at-least-once processing.09
kafka/VerdictPublisher.javaSends the record keyed by flag id; the future completes on the broker's ack (acks=all)A re-judged flag replaces its old record after compaction.–
kafka/ProcessedFlags.javaIn-memory: every flag id seen, and per player the latest BAN or FLAG_FOR_REVIEW verdict (newest wins regardless of replay order)Only actionable verdicts count, so a later skip can't hide an earlier review flag (a documented decision).16
kafka/VerdictSnapshotReader.javaReads the whole verdicts topic as of now (no group, read_committed, to the recorded end offsets, 30 s timeout), skipping unreadable recordsThe same technique as Telemetry's snapshot reader.–
kafka/ProcessedFlagsLoader.javaOn ApplicationReadyEvent: replay the verdicts, then start the listener containerA restart never judges a flag twice.03
metrics/JudgeMetrics.javaAll judge.* meters: verdicts by decision and action, latency timer, cost distribution (min 1e-6, max 1.0), attempts, escalations, invalid verdicts, LLM calls by stop reason, tokens by kind, tool calls, skips, duplicates; low-cardinality tags onlyThe cost histogram needed custom bounds: the default started at 1, putting every sub-cent verdict in one bucket.14
resources/application.ymlImports judge-defaults.yml; virtual threads; consumer group judge-agent, read_committed, 1 record per poll, 10-minute poll interval; idempotent producer; ack per record; concurrency 3; port 8083; topics and Arena settings–09

7. judge-eval: the evaluation harness

Paths relative to judge-eval/src/main/java/dev/fairplay/judge/eval/. Built and tested on synthetic fixtures; not yet run on real data.

FileWhat it isWhy it is like thisLearn
EvalApplication.javaA non-web Boot app that exits with the command's code––
EvalCommandLine.javaApplicationRunner + ExitCodeGenerator delegating to EvalCli––
EvalArgs.javaParses one command word plus --name=value options; unknown options are errors; dotted options (--judge.caching.enabled=false) are left to SpringA typo like --mdoe is reported, not ignored.–
EvalCli.javaThe three commands: collect, run (prompt, mode, subset, limit) and compare; secrets and network behind suppliers; exit 0 or 2compare needs no key, Arena or Kafka; missing secrets are named before any work.–
config/EvalConfig.javaWiring: the same JudgeSettings binding as the service; one SDK client with maxRetries(0); lazily memoised suppliers for the gateway and collectorNothing networked is built until a command needs it.–
config/EvalProperties.javaeval.*: dataset path, results dir, concurrency 4, collect targets (120 cheaters, 80 non-cheaters, seed 7), Arena account––
collect/TopicSource.javaAn interface for reading a whole topicA seam so collect is tested without a broker.13
collect/KafkaTopicSource.javaThe real implementation: no group, read_committed, never commitsCan't disturb Telemetry's consumers.–
collect/GroundTruth.javaTelemetry's ground-truth record, read only hereThe Judge never sees it while judging.–
collect/CaseSampler.javaOne candidate per player (a cheater's earliest flag at or after the cheat start; any flag for others); proportional split across cheat types (largest remainder); PRO before HONEST for non-cheaters; ids c-001... in shuffled order; stratified 50-case dev subsetPure and deterministic; ids don't reveal labels; dev keeps the overall ratio.11
collect/ArenaHttpReader.javaA small Arena reader for collect (login, three GETs, a few retries, one re-login)Runs once for a minute, so no breaker needed.–
collect/DatasetCollector.javaReads ground truth (latest per key, tombstones) and flags, samples, records the four tools' answers per case (lists at 20 items) with the production ArenaToolExecutor, writes cases.jsonl and a READMESnapshots make runs repeatable and cheap.11
model/EvalCase.javacaseId, subset, source run, a Label (isCheater, segment, cheatType), the flag, tool snapshotsThe label lives only in label.–
model/Subset.java, SubsetSelection.javaDEV/HOLDOUT (written lowercase) and ALL/DEV/HOLDOUT for --subset––
run/EvalMode.javaROUTED, BASELINE (Sonnet only, no caching), HAIKU_ONLY (never escalate), each deriving settings from the shared defaultsThe modes that split cost savings into routing and caching.11
run/RecordedToolExecutor.javaAnswers from snapshots, cutting lists to the requested limit; refuses other players like productionLives in eval, not core (a documented decision).–
run/EvalRunner.javaReplays cases through JudgeEngine on virtual threads (at most concurrency at once, sharing the rate limiter); an unanswerable case becomes an error result; results keep dataset orderBypasses pre-checks so every case is judged; the player is assumed ACTIVE.02
run/CaseResult.javaOne case's outcome: decision, action, attempts, cost, latency, final outputs and parse or validation failures, or an errorThe base for schema validity.–
run/DatasetFiles.javaReads and writes JSONL; SHA-256 of the datasetThe hash ties reports to the exact dataset.–
score/Scorer.javaPure scoring: banned only if the policy produced BAN; precision, recall, false-ban reduction, review rates, escalation rate, schema validity, cost and latency percentiles (nearest rank), confusion table, recall per cheat type, false bans per segment; lists of false bans and missed cheatersError cases counted but not scored.11
score/EvalMetrics.javaAll of the above as a record; undefined ratios are null ("n/a")––
report/RunInfo.javaPrompt version and hashes, models, mode, caching, subset, dataset hash, start time; modeLabel() adds "(no cache)"Every result is traceable.–
report/RunSummary.javaThe .summary.json: run info plus overall, dev and holdout metrics––
report/ReportWriter.javaWrites .jsonl, .summary.json and a Markdown report ending with every false ban and missed cheater with evidence and explanationThat list is what you read to write the next prompt version.–
report/ComparisonWriter.javaBuilds COMPARISON.md from all summaries: rows per version, mode and subset; the cost saving split into routing and caching; total schema validity––
report/Format.javaPercent, USD, number and table-cell formatting (pipes and newlines removed)Model text can't break a Markdown table.–
resources/application.ymlNon-web, no banner; imports judge-defaults.yml; eval settings with environment overrides––

8. Tests: what each class proves

judge-core: 194 tests (97.2% lines); judge-service: 77 unit + 12 integration (96.9%); judge-eval: 105 unit + 1 integration (96.2%). Counts in brackets are test methods.

judge-core tests and shared helpers (a test-jar)

ClassWhat it proves or provides
agent/ScriptedGatewayA fake MessagesGateway that plays back scripted responses and records requests
agent/TestMessagesBuilders for SDK Message objects (tool use, final text, stop reasons, usage)
agent/StubAnthropicServerA local HTTP stub of the Messages API, so the real SDK client can be tested
tools/FakeToolExecutor, TestSettings, MutableClockCanned tool answers with call recording; default settings; a controllable clock
SdkSmokeTest (1, tagged live)One real tool round trip ending in a parseable verdict; confirms the real request shape and prints cache token counts. Never run yet.
agent/AgentLoopTest (21)Tool rounds, multiple tools per turn in order, tool errors returned with isError, another player refused whatever executor is passed, backend exceptions abort, invalid stop reasons and unparseable text with costs reported, limits
agent/ResilientMessagesGatewayTest (16)Against the real SDK and a stub server: 429 (honouring retry-after), 529, 500, resets and timeouts retried; max attempts become unavailable; 400 non-retryable; 401/403 held as unavailable; the breaker opens, fails fast and closes again
agent/AnthropicResilienceTest (9)Retry-After parsing and capping; only transient failures retried; exponential jittered backoff stays below the maximum; one shared breaker named anthropic
agent/VerdictParserTest (13)Valid parse; integer confidence; enum case; whitespace; every kind of PARSE failure; an out-of-range value is a VALIDATION failure
agent/VerdictValidatorTest (8)Every boundary of confidence, evidence count and length, explanation length; BAN/CLEAR cheat-type rules
agent/VerdictSchemaTest (3)The schema matches the contract; every object forbids additional properties; the output config carries it
tools/ToolExecutorGuardTest (11)Other players never reach the backend; non-UUID ids count as other players; unknown or null tools; long names cut before echoing; unexpected properties; limit boundaries; the backend sees only canonical input
tools/MemoizingToolExecutorTest (7)Same call once; argument order doesn't matter; different limit or tool calls again; error results cached; exceptions not cached; nested keys sorted
tools/ToolCatalogTest (4)Four tools with exact names in order; all strict, requiring only player_id; only list tools take a limit; definitions stable across calls
tools/ArenaToolExecutorTest (14)Documented fields only; one Arena read for both player tools (and none when the pre-check's response is passed); compact rates; reasons cut; unknown player is an error; unreachable Arena aborts
routing/ModelRouterTest (24)When Haiku is final, every escalation reason, symmetric confidence boundaries, Sonnet's fresh conversation with the same tool cache, failures on both sides, modes
policy/DecisionPolicyTest (17)Every row (row 2 inclusive at exactly 0.90), state adjustments, and no BAN below the Arena floor is ever produced
cost/CostCalculatorTest (10)Worked prices for each kind of token, rounding, cached reads cheaper than uncached, no overflow, unpriced model rejected
prompt/PromptLibraryTest (7)v1 loads; cache control only when enabled; block order; no per-flag content in system blocks; identical text every call; missing version fails; the rulebook describes the catalog's tools
prompt/UserPromptBuilderTest (4)ISO time first; the flag round-trips; unused evidence fields omitted; no label
model/SuspiciousPlayerTest (6)Reads the contract example; unknown fields ignored; missing username and tier allowed; UUID handling; top tiers
JudgeEngineTest (7)End-to-end in core: confident ban; reason names the final model; no verdict means review; evaluation assumes ACTIVE; policy rerun after a conflict; tool cache per flag, not per engine
JudgeSettingsTest (13)Defaults valid; auto-ban below Arena's floor rejected; every range and section check; prompt version must be a folder name, not a path

judge-service tests

ClassWhat it proves or provides
support/WireMockContainerRuns wiremock/wiremock:3.13.2 and programs stubs and fault scenarios over its admin API (no WireMock Java dependency)
support/KafkaTestSupport, support/FlagsKafka container access, topic helpers, flag builders and awaitVerdict
arena/StubArenaServerA fake Arena for the client unit tests
JudgeFlowIT (5)One flag gives one verdict and exactly one JUDGE action in (WireMock) Arena; replaying the flag adds no Arena and no model call; a banned player is skipped with no model call; a malformed flag lands in the DLT; /actuator/prometheus exposes the dashboard's meters
ProcessedFlagsIT (1)Earlier verdicts are loaded first and their flags are not judged again
ResilienceIT (6)429, 529, timeout and reset are retried to a verdict; a 400 goes straight to the DLT; a 20-second outage drops no flag and opens the breaker (20 of 20 judged afterwards)
arena/ArenaClientTest (18)Bearer token and single login; 404 is empty; paging; 401 refresh once; refused credentials, failed login and closed port are "unavailable"; 5xx and dropped connections retried; the breaker opens and fails fast; action requests
arena/ArenaActionApplierTest (5)Accepted action reported; a conflict re-reads and decides once more; a second conflict or a vanished player gives a NOTE without an action
arena/PreChecksTest (5)Active judged; missing or non-player unknown; banned and appealed skipped; unknown status treated as unknown
verdict/JudgeServiceTest (10)Ban applied and recorded; one Arena read; skips (banned, unknown, recently judged; older judgements don't block); no verdict means review; an unreachable Arena stops the flag for retry; contract field names
kafka/FlagListenerTest (7)Judge, publish, remember; duplicates ignored and counted; not remembered when publishing fails; outages rethrown so the offset isn't committed; malformed flags non-retryable; the error message never contains the flag text
kafka/ProcessedFlagsTest (6)The 30-minute window per player; notes, clears and skips don't start it; a skip doesn't hide an earlier review flag; newest record wins regardless of replay order
config/KafkaConfigTest (6)Verdicts topic compacted with 6 partitions; DLT kept 7 days; no Telemetry topic declared; outages and unexpected errors retried, never recovered; malformed flags to the DLT on the same partition
config/ConfigWiringTest (9)Startup with both secrets; each missing secret named without values; SDK client from the environment with or without a base URL; engine, gateway and Arena client wired and bound to metrics; UTC clock
config/JudgeConfigBindingTest (8)Shared defaults bind; environment overrides; auto-ban below the floor stops startup; the shipped v1 prompts load; a missing prompt folder is rejected
metrics/JudgeMetricsTest (3)An escalated verdict counts every meter; no verdict counts as NONE; skips and duplicates

judge-eval tests

ClassWhat it proves or provides
FixturesSynthetic cases and flags (never real data)
EvalCliTest (14)run writes reports and honours subset and limit; the baseline mode appears in file names; a missing key or password is named; bad options reported; compare works without any key; usage on unknown commands; exit codes
collect/CaseSamplerTest (12)Early cheater flags excluded; PRO before HONEST; proportional split; warnings when short; same seed, same ids; ids from c-001; dev subset stratified
collect/DatasetCollectorTest (8)Writes the dataset and README; the label is only in the label field, never in the flag or snapshots; latest ground truth wins; bad flags skipped
collect/ArenaHttpReaderTest (9)Login once, reads, retries, refresh once, error cases
collect/KafkaTopicSourceIT (1)Reads every record of a real topic; nothing for a missing topic
config/EvalConfigTest (8)Binding; the same judging settings as the service; nothing networked built early; the gateway needs a key
run/EvalRunnerTest (8)Every case goes through the production policy; the label never reaches the model; order kept under concurrency; failures become error results; modes use the right models and caching
run/EvalModeTest (6)Each mode's settings, parsing
run/RecordedToolExecutorTest (8)Snapshots unchanged; lists cut to the limit; another player refused like production
run/DatasetFilesTest (4)Round trip with contract names; blank lines; a missing dataset says "collect first"; the hash changes with content
score/ScorerTest (14)Precision, recall, false-ban reduction, review and escalation rates, confusion table, undefined ratios, nearest-rank percentiles, schema validity, error cases not scored
report/ReportWriterTest (7)File names; the three files; header, tables and every mistake; n/a for undefined; model text can't break tables
report/ComparisonWriterTest (7)Rows per version, mode and subset; the cost split into routing and caching; no cost section without a baseline; schema validity summed

9. Build, ops, evaluation and documentation files

FileWhat it isLearn
pom.xml (parent)Three modules; pins anthropic-java 3.10.0 and Resilience4j 2.4.0 in dependency management; test.excludedGroups=live (a property, because Surefire ANDs groups with excludedGroups); Failsafe; JaCoCo 85% per module14
judge-core/pom.xmlOnly anthropic-java, Jackson, the three Resilience4j cores, Micrometer core and SLF4J; builds a test-jar of its helpers17
judge-service/pom.xmlCore (and its test-jar), webmvc, Kafka, Actuator, Prometheus, resilience4j-micrometer, Boot plugin–
judge-eval/pom.xmlCore (and its test-jar), Kafka, RestClient starter, Boot plugin–
mvnw, mvnw.cmd, .mvn/wrapper/The Maven Wrapper: a pinned Maven version, no installation needed14
DockerfileMulti-stage, ARG MODULE=judge-service, cache mount, JRE, non-root, port 8083 (the eval runs on the host)14
compose.yamlThe service on Telemetry's external network telemetry_default (Kafka at kafka:19092), Arena through host.docker.internal, secrets from .env, 1 GB limit14
.env.exampleANTHROPIC_API_KEY, optional ANTHROPIC_BASE_URL (tests only), JUDGE_SERVICE_PASSWORD, Arena URL, Kafka bootstrap, prompt version05
.dockerignore, .gitignoreKeep secrets, build output and the local notes out–
.gitattributesLine endings by file type: LF for mvnw and *.sh, CRLF for *.cmd, so the scripts run on Linux CI, in the container and on Windows whichever machine checked them out–
.github/workflows/ci.ymlsh ./mvnw -B verify with no API key (no test needs one); uploads JaCoCo reports13
judge-service/src/test/resources/wiremock/tool-use-turn.json, final-ban.json, final-clear.jsonCanned Messages API responses: a tool-use turn (with cache-read usage) and two final verdicts13
ops/README.mdHow to add the Judge's scrape job and dashboard to Telemetry's Prometheus and Grafana by hand (this project never modifies Telemetry)–
ops/prometheus-judge.ymlThe scrape job for judge-service:808314
ops/grafana/dashboards/judge.json7 panels: verdicts/min by action, average cost per verdict, p95 latency, cache-hit share of input tokens, escalation rate, human-review share, breaker state14
evals/README.mdExact steps to produce the dataset (a Telemetry run with loosened thresholds, Judge stopped) and the four measurement runs. States clearly that nothing has been produced yet11
evals/dataset/.gitkeep, evals/results/.gitkeepKeep the empty folders in git until real files exist–
README.mdOverview, architecture, quick start, how a verdict is made, evaluation, cost, resilience, the results table with labelled imaginary values (only the fault-injection row measured), design decisions, testing, stack, limitations15
LICENSEApache License 2.0–
Fairplay Judge — Implementation Specification.mdThe 1,187-line design, including the rulebook text, routing and policy tables, Kafka integration, resilience, cost, evaluation, testing and build order (local only)17
CLAUDE.md, handoff.md, todo.md, code_explanation/Local working notes (decisions, SDK findings, status) and your existing walkthrough16

10. Code-level probe questions

Why must the prompt version match [A-Za-z0-9._-]+? Deep

The version becomes part of a classpath path, prompts/<version>/system.md, and it can come from an environment variable. Without the check, a value like ../../something would make the loader read files outside the prompts folder (path traversal). JudgeSettingsTest.promptVersionIsAFolderNameNotAPath covers it.

What's the difference between ActionKind and JudgeAction? Intermediate

ActionKind is what is sent to Arena (BAN, FLAG, CLEAR, NOTE). JudgeAction is what the Judge records and counts (BAN, FLAG_FOR_REVIEW, NOTE, CLEAR, SKIPPED). They differ because Arena only allows FLAG from ACTIVE or REINSTATED: a review request for an already-flagged player is sent as a NOTE with requiresHumanReview, but still counts as FLAG_FOR_REVIEW in metrics and evaluation.

Why is the verdict parser so strict about trailing text? Intermediate

If it accepted "here's my verdict: {...}" or JSON inside Markdown fences, it would hide the model drifting from the required format, and the evaluation's schema-validity metric would be wrong. Strict parsing (FAIL_ON_TRAILING_TOKENS, exactly six fields, correct types) turns any drift into a counted PARSE failure that escalates to Sonnet.

Why did the cost histogram need minimumExpectedValue(0.000001)? Curveball

Micrometer's default histogram buckets for a distribution summary start around 1. A verdict costs a fraction of a cent, so every value fell into the first bucket and the p95 was meaningless. Setting the expected range to 1e-6..1.0 USD gives buckets where the data actually is.

How does the evaluation make sure the model can't see the label? Deep

Three layers, each tested: the label lives only in EvalCase.label, never in the flag or snapshots (DatasetCollectorTest); case ids are assigned in shuffled order so c-017 doesn't hint at anything (CaseSamplerTest); and EvalRunnerTest.theLabelNeverReachesTheModel inspects what's actually sent. Ground truth is read only by collect.

A Claude 401 isn't dead-lettered. Why? Deep

A 401 or 403 means credentials or permissions are wrong, which is an operational problem with the service, not with the flag. Dead-lettering would throw away perfectly good flags because of a bad key. So they become LlmUnavailableException: the flag is held and retried every 5 seconds until someone fixes the key, and the breaker and metrics make the problem visible. Only a 400, which is specific to the request, goes to the DLT.

Why does ArenaToolExecutor take the pre-check's player response? Intermediate

The pre-check already did GET /players/{id} to decide whether to judge. That response also contains everything get_player_profile and get_stat_percentiles need. Passing it in means Arena is read once per flag for those three uses, and with the memoising executor, Sonnet's escalation re-reads nothing. JudgeServiceTest.theToolsReuseThePreCheckResponseSoArenaIsReadOnce checks it.

What does EvalArgs do with --judge.caching.enabled=false? Curveball

It skips any option whose name contains a dot and leaves it for Spring, which binds it as a property override. That's how the "routed without caching" run is made without a special flag. Any other unknown non-dotted option is an error, so typos are reported instead of silently ignored.

What would break if FlagListener logged the flag JSON on a parse error? Intermediate

Nothing functional, but it would put player data (ids, usernames, evidence) into logs, which the project's rules forbid. Logs are widely accessible and long-lived. The listener's messages describe the problem ("flag is not valid JSON") without the content, and FlagListenerTest.theErrorMessageNeverContainsTheFlagText enforces it.

Why can't the service's ArenaClient be used by collect? Deep

It lives in judge-service, which is packaged as a Spring Boot fat jar; the eval module can't depend on another app's repackaged jar. That's also why the read port (ArenaReader) and the tool code (ArenaToolExecutor) were moved into judge-core: the eval gets its own small ArenaHttpReader adapter, but records snapshots with exactly the production tool code.

Saved in this browser.