Telemetry, file by file
Every file in Project 2: the four Maven modules (75 main files), all 52 test files, the scripts that ran the chaos test, the nine measured reports, and every build and ops file. For each one: what it is, why it is shaped that way, and where the idea is taught. The project's own beginner tour is project2explained.html; note that it was last regenerated after build step 7, so for the simulator app, the verifier and the Arena bridge this atlas is more current.
1. The shape of the repository
Project 2/ Maven reactor, 4 modules, Boot 4.1.2, Kafka 4.2.2
├── pom.xml parent: modules, shared test deps, Failsafe, JaCoCo (85% per module)
├── telemetry-common/ contracts: topics, records, JSON serde (+ a test-jar of helpers)
├── telemetry-simulator/ the load: seeded players, events, matches, manifest, Arena registration
├── telemetry-detector/ the brain: pure Kafka Streams topology + Spring app + Arena bridge
├── telemetry-verifier/ the judge of the judge: detection and integrity reports, exit codes
├── Dockerfile compose.yaml .env.example .dockerignore .gitignore LICENSE (Apache 2.0)
├── ops/prometheus.yml ops/grafana/{provisioning, dashboards}
├── scripts/chaos-kill.sh scripts/measured-run.sh
├── reports/ the 9 measured outputs (3 runs × detection, integrity, chaos)
├── .github/workflows/ci.yml README.md
└── (local only: CLAUDE.md, handoff.md, todo.md, the spec, Code Explanation/)
Dependency direction: every app depends on telemetry-common; no app depends on another. The detector's topology package has no Spring dependency at all.
2. telemetry-common: the shared contracts
Paths below are relative to telemetry-common/src/main/java/dev/fairplay/telemetry/common/.
| File | What it is | Why it is like this | Learn |
|---|---|---|---|
Topics.java | Constants for all nine topic names, the manifest's run key __run__, and PLAYER_PARTITIONS = 6 | Topic names are contracts; one constant for partitions keeps joins co-partitioned. | 09 |
TopicsConfig.java | A NewTopic bean per topic (partitions, replicas, delete with retention or compact); KafkaAdmin creates missing ones | Topics are declared in code; the broker's auto-create is off. | 09 |
TopicsProperties.java | telemetry.topics.replication-factor (default 1), validated in the constructor | No validation starter needed in a library module. | 03 |
json/JsonSerde.java | A generic Jackson 3 Serde<T> that is its own serializer and deserializer; a shared JsonMapper that ignores unknown fields; wraps failures in SerializationException | Spring's JsonSerde is Jackson 2; one mapper for the whole project. | 02 |
json/JsonSerdes.java | Factories for each contract type plus a generic of(Class) | The detector builds serdes for its own types with of(), since common can't reference them. | – |
model/MatchEvent.java | One gameplay event (ids, per-player seq, type, epoch-ms ts, and type-specific fields) with isValid(): an exhaustive switch per type, dtMs > 0, and a strict UUID round-trip check | The single definition of a valid event, shared by producer tests and the detector. | 02 |
model/EventType.java | SHOT, POSITION, KILL | – | – |
model/SuspiciousPlayer.java | The flag: flagId, playerId, username, skillTier, rule, score, window start and end, detectedAt, evidence, detectorVersion | The contract the Judge consumes; fields must never be renamed. | 01 |
model/Evidence.java | Optional fields with two factories: aimbot(shots, hits, headshotHits, rate, avgReaction) and speedhack(moving, overspeed, ratio, maxSpeed) | One record, two shapes; NON_NULL hides the other rule's fields. | – |
model/Rule.java | AIMBOT, SPEEDHACK | The cooldown key and metric tag. | – |
model/PlayerProfile.java | playerId, username, country, skillTier, registeredAt | Published to the compacted player-profiles topic and joined as a KTable. | 10 |
model/SkillTier.java | BRONZE, SILVER, GOLD, PLATINUM, PRO | The Judge escalates bans of PLATINUM and PRO players. | 11 |
model/IntegrityStats.java | Per player: count, lastSeq, duplicates, gaps, updatedAt; empty() | The exactly-once proof's data. | 10 |
model/ManifestEntry.java | Per player: runId, finalSeq, acked, failed | What the simulator says it sent. | 10 |
model/RunInfo.java | The run record under key __run__: id, start, end, players, total acked, failed, malformed | Its presence means "the run finished" for the verifier. | – |
model/GroundTruth.java | runId, playerId, segment, cheatType, cheatStartsAt | Only the simulator writes it and only the verifier reads it. | 10 |
model/Segment.java | HONEST, PRO, CHEATER | Added beyond the spec (a documented decision). | – |
model/CheatType.java | NONE, AIMBOT, AIMBOT_SUBTLE, SPEEDHACK, SPEEDHACK_SUBTLE | Subtle variants test the detector's limits. | – |
model/DeadLetter.java | sourceTopic, partition, offset, key, Base64 payload, error (capped at 500 characters in the compact constructor), receivedAt | Enough to find and replay the bad record. | 10 |
model/MatchSummary.java, ParticipantSummary.java | A finished match shaped exactly like Arena's POST /matches body (ISO-8601 time strings) | A contract with Arena: the bridge forwards it unchanged. | 04 |
3. telemetry-simulator: generating the load
Paths relative to telemetry-simulator/src/main/java/dev/fairplay/telemetry/simulator/.
| File | What it is | Why it is like this | Learn |
|---|---|---|---|
SimulatorApplication.java | Boot app with @ConfigurationPropertiesScan and @Import(TopicsConfig.class) | Common's package is outside the component scan. | 03 |
SimulatorProperties.java | Every tunable (seed, players, duration, match size and length, tick rate, shots per second, kill chance, shard threads, cheater and pro ratios, cheat start window, malformed ratio), validated in the compact constructor, including cross-field rules (players a multiple of match size × shards; 1000 % tickHz == 0; ratios sum ≤ 1; cheat start before the run ends); tickMs() | Invalid runs fail at startup with the property name. | 03 |
ClockConfig.java | The UTC Clock bean | Tests drive ticks with a mutable clock. | – |
population/BehaviorProfile.java | How a player aims (hit and headshot chance, reaction mean and spread, floor) and moves (move chance, speed range, multiplier). Honest: hits 0.20–0.45, headshots 0.25–0.55, reaction ~240 ms; pro: 0.40–0.55, 0.55–0.68, 155–185 ms; aimbot: 0.60–0.90, 0.75–0.95, 40–80 ms; subtle aimbot: 0.45–0.60, 0.60–0.70, 110–135 ms; speedhack ×1.5–2.0, subtle ×1.12–1.20. tierOf(headshotChance) | Cheaters change only their cheat's family of values, so an aimbot still moves like a human. | – |
population/CheaterAssignment.java | Shuffles player indices with new Random(seed); the first 5% become cheaters (40% aimbot, 20% subtle aimbot, 24% speedhack, 16% subtle speedhack, remainder to aimbot), the next 2% pros, the rest honest | Same seed, same assignment: 20 + 10 + 12 + 8 = 50 cheaters for 1,000 players. | – |
population/PlayerState.java | A record per player (id, username, country, segment, cheat type, tier, base and cheat profiles, cheat start, and an AtomicLong seq); profileAt(now), nextSeq(), withPlayerId() | Shallow immutability: the counter changes, so tests compare projections (module 02). | 02 |
population/PopulationFactory.java | Builds the population: ids nameUUIDFromBytes(seed + ":" + i), usernames sim_0001..., countries from a seeded Datafaker, profiles and cheat starts from SplittableRandom(seed ^ constant); cheaters keep their base tier "so the tier does not give them away" | A separate random stream from the shards; fully reproducible from the seed. | 02 |
engine/EventFactory.java | One tick for one player: exactly one POSITION (with ±0.15 measurement noise), then a Poisson number of SHOTs (Knuth's algorithm, mean 4.5/20 = 0.225 per tick), each maybe followed by a KILL; one shared timestamp; every event takes the next seq; event ids are name-based UUIDs of run, player and seq | About 20 + 4.5 + 0.3 ≈ 25 events per player per second, hence 25,000/s for 1,000 players. | 18 |
engine/MalformedInjector.java | With probability malformed-ratio (0.1%), emits one bad record, rotating truncated JSON, valid JSON missing fields, and 16 random bytes whose first byte is forced to 0xFF (never valid UTF-8 or JSON) | Proves the dead-letter path under load; bad records carry no seq so they never affect integrity. | 10 |
engine/ShardWorker.java | One thread's share: owns rooms, its own SplittableRandom(seed + k) and injector; tick() plays every player; when the round ends, publishes all summaries and reshuffles players into new rooms; run() catches Throwable | Thread confinement means no locks; the catch keeps scheduleAtFixedRate alive. All rooms of a shard end together (a documented decision). | 02 |
engine/SimulationEngine.java | The run: (optional Arena registration first) → population → publish profiles and ground truth and wait → one single-thread scheduler per shard at the tick rate → log events/s every 10 s → stop → flush → manifest + RunInfo → "SIMULATION COMPLETE"; runs on a simulation-main thread; @ConditionalOnProperty autostart | Registration happens before the run clock starts, so cheat start times aren't eaten by slow bcrypt registrations. | 02 |
match/MatchRoom.java | One match: members, a sim-<run>-<n> external id, a rotating map name, and per-player running totals in a small mutable class; randomOpponent never picks the shooter | A thread-confined accumulator updated thousands of times a second, so a mutable class rather than a record (documented). | – |
match/MatchSummaryBuilder.java | Ranks players by kills descending, deaths ascending, then id; average reaction = total reaction / shots (minimum 1); builds the Arena-shaped summary | Guarantees every Arena invariant (placements 1..n, headshots ≤ kills, reaction ≥ 1). | – |
output/EventPublisher.java | The only Kafka sender: events as bytes keyed by player with the event time as the record timestamp; acks and failures recorded in the manifest; malformed records counted separately; profiles, ground truth, summaries (keyed by external id) and manifest; flush() | Serialising to bytes first lets garbage go through the same producer; no hand retries (the producer is idempotent). | 09 |
output/ManifestTracker.java | Per-player LongAdder acked and failed counts and the highest acked seq (in a ConcurrentHashMap); entries() uses the generated last seq as finalSeq | Callbacks run on Kafka's I/O thread, hence the concurrent counters; using the generated seq means a lost send can't hide. | 02 |
arena/ArenaProperties.java | telemetry.arena.* for registration: base URL, service account, player password (8–72 characters, Arena's rule), concurrency 16, mapping file; a blank service password fails startup only when enabled | One record per app (the detector has its own); fail fast on misconfiguration. | – |
arena/ArenaClient.java | Registers a player via POST /auth/register; on 409 looks up the id with GET /players?q= using a cached service token, re-logging once on 401; timeouts 2 s and 10 s | Registration is repeatable (idempotent) across reruns. | 04 |
arena/ArenaPlayerRegistrar.java | Loads sim-players.json, registers only the missing players on virtual threads bounded by a Semaphore(16), saves the mapping even on partial failure, then fails if anything failed | Resumable, bounded and honest about failures. | 02 |
src/main/resources/application.yml | The idempotent producer (acks=all, idempotence, lz4, 64 KB batches, 10 ms linger, 5 in flight, 120 s delivery timeout), port 8081, Actuator, ECS logs, all telemetry.sim.* and telemetry.arena.* defaults | – | 09 |
4. telemetry-detector: the Kafka Streams application
Paths relative to telemetry-detector/src/main/java/dev/fairplay/telemetry/detector/.
The pure topology (topology/, no Spring)
| File | What it is | Why it is like this | Learn |
|---|---|---|---|
topology/TelemetryTopology.java | Builds the whole graph on a StreamsBuilder: the profiles KTable, two stores, parse and DLT split, integrity, two key-preserving window branches, merge, the named flags repartition, left join, cooldown, sinks | Testable with TopologyTestDriver; Spring only wires it. | 10 |
topology/EventParser.java | The single definition of a good record: non-null, parseable, isValid(), and record key = player id; returns an Outcome(event, problem) | Used by both the DLT path and the window path, so they can never disagree. | 10 |
topology/ParseProcessor.java | Processor API: forwards Parsed or Failed(DeadLetter) with topic, partition and offset from recordMetadata() and a Base64 payload | Bad data becomes data, not a crashed thread. | 10 |
topology/ParseResult.java | Sealed interface with records Parsed and Failed | An algebraic data type for the branch. | 02 |
topology/IntegrityProcessor.java | Per-player seq accounting in the seq-tracker store (in order, duplicate, gap) and a wall-clock punctuator re-emitting every player every 5 s | The exactly-once proof; re-emitting everything keeps consumers and restored tasks stateless. | 10 |
topology/AimStats.java | Immutable window aggregate: shots, hits, headshot hits, total reaction; derived average reaction and headshot rate (@JsonIgnore) | Immutable, so add() returns a new value, as aggregators expect. | 02 |
topology/SpeedStats.java | Moving samples, overspeed samples, max speed; speed = hypot(dx, dy) / (dtMs/1000); idle samples count toward nothing | Idle samples would otherwise dilute the overspeed ratio. | – |
topology/DetectionRules.java | The thresholds (inclusive) and the score formulas (0.5 at the line, 1.0 far over it), with evidence rounded HALF_UP | Pure maths, exhaustively unit-tested at the boundaries. | 10 |
topology/FlagCandidate.java | A window that crossed a threshold; factories for both rules; withProfile() as the join's value joiner | Travels through the repartition topic, so it's a serialisable record. | 10 |
topology/CooldownProcessor.java | Drops repeats per playerId|rule for 10 minutes using the injected clock; builds the final SuspiciousPlayer with a random flag id | Windows re-fire on every update; an aimbot flag never hides a speedhack flag. | 10 |
topology/TopologyMetrics.java | An interface (parsed, dead-lettered, flagged per rule) with a NOOP implementation | Keeps Micrometer out of the topology. Its comment warns the counts can double after a retried transaction, so they're for dashboards only. | 16 |
The Spring application around it
| File | What it is | Why it is like this | Learn |
|---|---|---|---|
DetectorApplication.java | @SpringBootApplication, @ConfigurationPropertiesScan, @EnableKafkaStreams, @Import(TopicsConfig.class) | @EnableKafkaStreams makes Spring Kafka create and manage the KafkaStreams client. | 03 |
DetectionProperties.java | Detection tunables with compact-constructor validation, including "advance must not exceed window size" (Kafka Streams would reject it) | Fail at startup, not at topology build. | 03 |
ClockConfig.java | The UTC clock bean | – | – |
StreamsWiring.java | The only Spring–topology touch point: receives the auto-configured StreamsBuilder and calls addTo | The topology stays framework-free. | 10 |
StreamsRuntimeConfig.java | A StreamsBuilderFactoryBeanConfigurer: uncaught exceptions replace the thread (REPLACE_THREAD); a state listener logs every transition | Those log lines are what chaos-kill.sh greps to time the failover. | 12 |
MicrometerTopologyMetrics.java | Micrometer counters telemetry.detect.events.parsed, .dead_lettered and telemetry.detect.flags tagged by rule | Dashboard-only (see above). | 14 |
KafkaStreamsHealthIndicator.java | UP when RUNNING, OUT_OF_SERVICE while CREATED or REBALANCING (or not yet created), DOWN otherwise | Readiness that understands rebalances. | 03 |
src/main/resources/application.yml | Streams settings (exactly_once_v2 from PROCESSING_GUARANTEE, 2 threads, 1 standby, 500 ms commits, 64 MB cache, session 6 s, heartbeat 2 s, state dir), the bridge's consumer (read_committed) and producer, port 8082, detection and bridge defaults | – | 10 |
The Arena bridge (bridge/, off by default)
| File | What it is | Why it is like this | Learn |
|---|---|---|---|
bridge/ArenaBridgeListener.java | @KafkaListener(id = "arena-bridge", idIsGroup = false, autoStartup = "${telemetry.arena.enabled:false}") on match-summaries; parses the String value (bad JSON becomes non-retryable) and calls the client | Reading a String lets one template dead-letter both bad JSON and Arena rejections. | 09 |
bridge/ArenaMatchClient.java | Posts to /api/v1/matches with a lazily cached token; 201 counts "created", 200 "replayed"; 401 re-login once; 429/5xx/unreachable → retryable; other 4xx → non-retryable carrying Arena's problem code and detail; timeouts 2 s and 5 s | Failure classification drives retry vs DLT. | 12 |
bridge/BridgeErrorHandlingConfig.java | DefaultErrorHandler with ExponentialBackOffWithMaxRetries(3) from 1 s ×2, non-retryable exceptions skip straight to a counting DeadLetterPublishingRecoverer (same partition of match-summaries.DLT) | Temporary failures retried, permanent ones parked. | 09 |
bridge/ArenaProperties.java | The detector's telemetry.arena.*: base URL, service account, retries, backoff; a blank password fails startup when enabled | A separate record from the simulator's (different keys). | – |
bridge/RetryableArenaException.java | A temporary failure (429, 5xx, no connection, timeout) | The type is the retry decision. | 12 |
bridge/NonRetryableArenaException.java | A permanent failure with Arena's code (for example PLAYER_NOT_ELIGIBLE) or ours (MALFORMED_SUMMARY) | The code ends up in the DLT headers for diagnosis. | 12 |
5. telemetry-verifier: scoring the run
Paths relative to telemetry-verifier/src/main/java/dev/fairplay/telemetry/verifier/.
| File | What it is | Why it is like this | Learn |
|---|---|---|---|
VerifierApplication.java | A non-web Boot app whose main calls System.exit(SpringApplication.exit(context)) | The exit code comes from the ExitCodeGenerator bean. | 03 |
VerifierProperties.java | reports dir, read timeout (30 s), settle poll interval (5 s, a documented addition so tests can poll fast) | – | – |
VerifierRunner.java | CommandLineRunner + ExitCodeGenerator: --mode=detection (optional --min-recall, --max-fpr) or --mode=integrity (optional --settle: re-read until all players pass or time runs out). Exit codes 0 ok, 1 integrity failed, 2 threshold missed, 3 run not finished, 4 bad arguments. Ignores stale manifest entries from older runs | Scripts and CI can act on the code; code 4 was added so bad arguments never look like a result. The two-constructor bug lived here. | 03 |
TopicSnapshotReader.java | Reads a whole topic "as it stands now": no group, read_committed, assign all partitions, seek to the beginning, poll until the end offsets recorded at the start; readLatest applies tombstones | An exact snapshot without committing offsets or joining a group. | 09 |
DetectionEvaluator.java | Pure maths: ignores flags before the run or for unknown players; a cheater counts as detected by the first flag at or after cheat start; latency per type; correct-rule check; early flags; false positives by segment; nearest-rank percentiles | No Kafka, no clock, so it's unit-tested to the decimal. | 11 |
DetectionReport.java | Counts, recall, FPR (overall, honest, pro), precision, flags by rule, latency rows, missed cheaters, false positives with evidence; every undefined ratio is 0.0, never NaN | A documented decision so reports never contain NaN. | – |
IntegrityChecker.java | A player passes only if stats exist, no failed sends, 0 duplicates, 0 gaps, and both lastSeq and count equal finalSeq; lists at most 20 failures but counts all | Strict per-player equality is the proof. | 10 |
IntegrityReport.java | Totals plus failures side by side; passed() requires at least one player | An empty manifest is not a pass. | – |
ReportWriter.java | Renders Markdown reports and writes reports/<mode>-<runId>.md | Human-readable evidence you can commit or quote. | – |
src/main/resources/application.yml | web-application-type: none, a read_committed consumer, report settings | – | – |
6. Tests: what each class proves
269 unit tests and 17 integration tests, all passing, with line coverage of 100% (common), 99.2% (simulator), 100% (detector) and 100% (verifier). The number after each class is how many test methods it has.
telemetry-common tests and the shared test-jar
| Class | What it proves or provides |
|---|---|
TopicsConfigTest (4) | All nine topics with their partitions and cleanup policies; replication factor from properties, default 1, non-positive rejected |
json/JsonSerdeTest (9) | Round trips for every event type and contract record; null fields omitted and contract names kept; isValid not serialised; nulls both ways; garbage throws SerializationException; unknown fields ignored |
model/MatchEventTest (7) | Validity per type, positive dt, seq ≥ 1, ts > 0, strict UUIDs, required ids |
model/ModelHelpersTest (4) | DeadLetter error capping, IntegrityStats zeros, Evidence factories fill only their own fields |
support/MutableClock | A settable, advanceable Clock |
support/TestEvents | Builders for valid events with deterministic ids |
support/KafkaTestContainer | One apache/kafka-native broker per JVM |
support/KafkaTestContainerIT (1) | Smoke test: Failsafe runs, Docker is reachable, the pinned image starts |
support/StubArenaServer | A fake Arena on the JDK HttpServer: scripted responses per route, every request recorded |
telemetry-simulator tests
| Class | What it proves |
|---|---|
SimulatorPropertiesTest (8) | YAML defaults bind; environment overrides; startup fails on a bad multiple, bad counts, ratios, cheat window, match or tick settings; boundary values accepted |
TestSimProps | Default simulator settings for tests |
population/BehaviorProfileTest (9) | Every draw stays inside its table; cheat profiles keep the other family's values; tier boundaries |
population/CheaterAssignmentTest (6) | Expected counts for 1,000 players; same seed, same result; different seed, different; the remainder goes to AIMBOT |
population/PlayerStateTest (1) | withPlayerId keeps everything but the id and starts a new sequence |
population/PopulationFactoryTest (9) | Naming formulas; same seed, same population (compared through a projection); cheaters' start inside the window; tiers; profile switch exactly at cheat start; gapless sequences |
engine/EventFactoryTest (12) | Every event valid with seq +1; one POSITION per tick first; dt 50 ms; headshot implies hit; shot rate matches; profile switches at cheat start; honest players don't exceed sprint speed by much; same seed and clock reproduce the same events; Poisson mean |
engine/MalformedInjectorTest (5) | Rotation of three kinds; no payload ever parses as a valid event; ratio respected; returned buffers can't corrupt the templates |
engine/ShardWorkerTest (6) | Round-robin room partition; gapless seqs; finished matches published and rooms reshuffled; bad records follow real ones; a failing tick is logged and the schedule survives |
engine/SimulationEngineTest (4) | A failed setup send fails the run instead of hanging; stopping interrupts a run; with Arena enabled everything uses Arena's ids; stop before start is harmless |
match/MatchSummaryBuilderTest (11) | Placements 1..n; ranking and tie-breaks; reaction average rules; every Arena invariant holds; ISO-8601 times survive JSON; map rotation; opponent selection |
output/EventPublisherTest (10) | Keyed by player with event time as record timestamp; acks and failures (async and thrown) counted correctly; malformed never in the manifest; keys per topic |
output/ManifestTrackerTest (4) | Per-player counts; highest acked seq ignores out-of-order acks; finalSeq is the generated seq |
arena/ArenaClientTest (6) | Registration returns Arena's id; an existing username is looked up; the token is cached; one re-login on 401; error cases |
arena/ArenaPlayerRegistrarTest (5) | Registers all and saves the mapping; later runs skip known players; partial failure keeps what worked; unreadable file is an error |
arena/ArenaPropertiesTest (3) | A blank password is fine when disabled and fails startup when enabled |
SimulatorIT (2) | Against a real broker: every player's seqs arrive in order with no gaps and match the manifest; profiles, ground truth and finished matches are published |
telemetry-detector tests
| Class | What it proves |
|---|---|
topology/TopologyFixture | Wraps TopologyTestDriver with a MutableClock, input and output topics, and test metrics |
TestDetection | The default detection settings for tests |
topology/AimbotTopologyTest (10) | One flag with exact evidence (12/12/10, 0.8333, score 0.843); honest stream never flags; cooldown per player; slow reactions, few headshots or few hits don't flag; shots spread over more than a window don't accumulate; invalid or mis-keyed events never count; POSITION and KILL never feed the aimbot |
topology/SpeedhackTopologyTest (7) | 60 positions at speed 11 give one flag; speeds 6.9–7.3 never flag; 39 moving samples aren't enough; idle samples don't dilute; cooldown; an aimbot flag doesn't hide a speedhack flag |
topology/DeadLetterTopologyTest (7) | Garbage, invalid events, wrong keys, null and JSON-null payloads all dead-lettered with reason, offset and Base64; valid events after bad ones still processed |
topology/IntegrityTopologyTest (12) | Counts, duplicates, gaps (including a first event that isn't 1); nothing emitted before the interval; every player re-emitted every interval; per-player independence; all event types count; bad records never reach the tracker |
topology/ProfileJoinTopologyTest (5) | Known player gets username and tier; unknown still emitted with nulls; latest profile wins; no cross-player leakage |
topology/DetectionRulesTest (15) | Every boundary is inclusive (12 hits, 140 ms, 0.62, 40 samples, 0.15); scores 0.5 at the line and 1.0 far over; formulas; HALF_UP rounding; clamping |
topology/StatsTest (8) | AimStats and SpeedStats accumulation, immutability, zero denominators, speed maths, JSON round trip through the window-store serde |
topology/TopologyShapeTest (5) | Sources are exactly events, profiles and the flags repartition; no window repartition topics; sinks; the ground-truth topic is never mentioned |
topology/TopologyMetricsTest (1) | The NOOP implementation accepts every call |
DetectionPropertiesTest (7) | YAML defaults and overrides bind; each broken rule fails startup; boundary values accepted |
WiringUnitTest (7) | Metrics per rule; the health indicator's mapping for every state; a dead stream thread is replaced; the clock is UTC system time. Spring classes built directly, because a @SpringBootTest would need a broker |
bridge/ArenaBridgeUnitTest (4) | The listener parses and hands over; garbage and null values are non-retryable and never reach Arena; broken properties rejected |
bridge/ArenaMatchClientTest (12) | 201 created, 200 replay; token cached; 401 re-login once, a second 401 non-retryable; 429/5xx/refused connection retryable; 4xx non-retryable with Arena's code; long bodies cut; login failure cases |
KafkaIt | Plain-client helpers for ITs: produce events, read whole topics with read_committed |
DetectorEndToEndIT (2) | On a real broker: the cheater flagged once (with username and tier), the honest player never, garbage dead-lettered; internal topics have the expected shape (no window repartition, flags repartition has 6 partitions) |
ExactlyOnceIT (1) | 20 players × 500 events each counted exactly; a re-sent seq is detected as one duplicate (the negative control) |
ArenaBridgeIT (6) | Against StubArenaServer: 201 = one call, no DLT; two temporary failures then success = three calls; endless temporary failure = three calls then DLT; permanent rejection = one call then DLT; garbage never reaches Arena; an expired token is refreshed |
telemetry-verifier tests
| Class | What it proves |
|---|---|
DetectionEvaluatorTest (13) | Segment counts; recall, precision and FPR; ignoring pre-run and unknown flags; correct rule from the earliest qualifying flag; early flags don't count as detection; latency rows; missed and false-positive lists; other runs ignored; no NaN; nearest-rank percentiles |
IntegrityCheckerTest (10) | Pass on exact match; each failure reason fails a player (duplicates, gaps, wrong lastSeq or count, failed sends, missing stats); at most 20 listed; empty manifest isn't a pass |
ReportWriterTest (7) | Report headers, tables, lists; file written under a created directory; an unwritable directory is an error |
VerifierRunnerTest (12) | Every exit code path (0, 1, 2, 3, 4), the settle loop, stale entries ignored, interrupted settle keeps the interrupt flag |
VerifierPropertiesTest (3) | Validation of directory and durations |
VerifierContextTest (1) | The whole Spring context starts and the runner receives the arguments (the guard against the two-constructor bug) |
TopicSnapshotReaderIT (5) | Latest value per key with tombstones; typed reads skip nulls; empty and missing topics; never shows records of an aborted transaction |
7. Build, delivery, evidence and documentation files
| File | What it is | Learn |
|---|---|---|
pom.xml (parent) | packaging=pom, the four modules, shared test dependencies (Boot test, Kafka test, Testcontainers Kafka, Awaitility), Failsafe for *IT, JaCoCo with jacoco.skip=true at the parent and an 85% per-module minimum | 14 |
telemetry-common/pom.xml | Kafka starter, kafka-streams, Jackson; builds a test-jar of the support classes | 14 |
telemetry-simulator/pom.xml | Common, webmvc (for Actuator over HTTP), Kafka, Actuator, validation, Prometheus registry, Datafaker (countries), common's test-jar, Boot plugin | – |
telemetry-detector/pom.xml | Common, webmvc, Kafka, kafka-streams, Actuator, validation, Prometheus, kafka-streams-test-utils (TopologyTestDriver), common's test-jar | – |
telemetry-verifier/pom.xml | Common, Kafka, validation (no web), common's test-jar | – |
mvnw, mvnw.cmd, .mvn/wrapper/ | The Maven Wrapper: a pinned Maven version, no installation needed | 14 |
Dockerfile | One file for all apps via ARG MODULE; BuildKit cache mount for ~/.m2; JRE runtime; non-root; directories for state, reports and data | 14 |
compose.yaml | KRaft broker with INTERNAL/EXTERNAL/CONTROLLER listeners and auto-create off; Kafka UI on 8090; 3 detector replicas (1 GB each); simulator (sim profile); verifier (tools profile); Prometheus; Grafana; an x-app anchor | 14 |
.env.example | Kafka version 4.2.2, bootstrap, RF 1, seed 42, 1,000 players, 5 minutes, processing guarantee, state dir, port 8082, Arena switch and URL, the telemetry-sim password, the simulated players' password | – |
.dockerignore, .gitignore | Keep secrets, build output, reports and data out of images and the repository | – |
.gitattributes | Line endings by file type: LF for mvnw and *.sh, CRLF for *.cmd, so the scripts run on Linux CI, in the containers and on Windows whichever machine checked them out | – |
.github/workflows/ci.yml | ./mvnw -B verify on Ubuntu with Docker available for Testcontainers; uploads every module's JaCoCo report | 13 |
ops/prometheus.yml | Scrapes the simulator statically and the 3 detector replicas through DNS service discovery | 14 |
ops/grafana/provisioning/datasources/prometheus.yml | Provisions Prometheus as the default datasource | – |
ops/grafana/provisioning/dashboards/dashboards.yml | Tells Grafana to load dashboards from a mounted folder | – |
ops/grafana/dashboards/telemetry.json | The 8 panels: events produced/sec, processed/sec, process rate by instance, max consumer lag, flags/min by rule, dead letters/min, cheaters active, bridge outcomes | 14 |
scripts/chaos-kill.sh | Kills one detector, waits until both survivors log "REBALANCING to RUNNING", prints the failover time, then describes the consumer group to show all partitions owned | 12 |
scripts/measured-run.sh | One complete measured run: fresh stack, wait for 3 detectors, start the simulation, chaos kill at about 90 s, wait for completion, wait for lag 0, run both verifier modes, record exit codes | 14 |
reports/detection-r-49fe8f.md, integrity-r-49fe8f.md, chaos-exactly_once_v2.txt | The exactly-once run: recall 100%, FPR 0.11%, precision 98.04%, 7,466,329 = 7,466,329, failover 14,407 ms. The chaos file also shows the Git Bash path bug that broke the group description in this first run (see module 16) | 16 |
reports/detection-r-df576c.md, integrity-r-df576c.md, chaos-at_least_once.txt | The at-least-once run: 0 duplicates, 0 gaps, failover 16,668 ms, with the full consumer-group table showing per-partition lag of up to about 72,000 right after failover | 10 |
reports/detection-r-8bfb5e.md, integrity-r-8bfb5e.md, chaos-at_least_once_nocache.txt | At-least-once with the state-store cache off: still 0 duplicates; detection latency rose sharply | 10 |
README.md | What it does, architecture, topics, rules, quick start, design decisions, all measured results including the misses, tech stack, modules, testing, next steps | 15 |
LICENSE | Apache License 2.0 | – |
Fairplay Telemetry — Implementation Specification.md | The 1,309-line design: overview, rules, stack and pitfalls, structure, configuration, topics and contracts, simulator, topology, bridge, verifier, observability, testing, Docker and chaos, how each number is measured, build order (local only) | 17 |
CLAUDE.md, handoff.md, todo.md, Code Explanation/ | Local working notes and your existing walkthrough (git-ignored) | 16 |
8. Code-level probe questions
Where does 25,000 events per second come from? Intermediate
From EventFactory and the settings: 20 ticks per second, each with exactly one POSITION (20/s), a Poisson number of SHOTs with mean 4.5/s, and a KILL after some hits (about 0.3/s). That's roughly 25 events per player per second, times 1,000 players. The measured rate was about 24,900/s.
How can the simulator be "seeded" if match boundaries use the wall clock? Deep
Everything about who and how is seeded: player ids are name-based UUIDs of the seed and index, roles come from a seeded shuffle, profiles and cheat start offsets from a seeded SplittableRandom, each shard's behaviour from SplittableRandom(seed + k), and event ids are name-based UUIDs. But ticks and match ends follow real time, so the exact interleaving differs between runs. The population and behaviour are reproducible; the whole run isn't bit-for-bit.
Why does MalformedInjector set the first byte to 0xFF? Curveball
Random bytes could, by bad luck, form valid JSON (like {} or a number), which would make a test flaky. 0xFF can never appear in valid UTF-8, so a payload starting with it can never parse. MalformedInjectorTest.noPayloadParsesIntoAValidEvent checks it.
Why is finalSeq the generated sequence and not the highest acknowledged one? Deep
If it were the highest acked seq, a send that failed at the end of the run would silently lower the expectation, and the integrity check would still pass. Using the last generated seq means a failed send shows up as finalSeq > acked, and the verifier fails that player (it also checks failed == 0). It's a documented decision in ManifestTracker.
Why do the counters in MicrometerTopologyMetrics say "dashboard only"? Deep
Exactly-once covers Kafka writes and offsets, not side effects in your process. If a transaction is aborted and the records are reprocessed after a crash, the processor code runs twice and increments the counters twice, even though the output topics contain each result once. So exact numbers come from the verifier reading committed topics, and the counters are only for trends.
How does the verifier read a topic without a consumer group? Intermediate
TopicSnapshotReader creates a consumer with no group id, assigns all partitions manually, seeks to the beginning, records the end offsets once, and polls until each partition's position reaches its recorded end (or a timeout). It uses read_committed and no auto-commit. It takes a snapshot as of now and leaves no group state behind.
What does StreamsRuntimeConfig change, and how did it help measurement? Intermediate
It sets an uncaught-exception handler that replaces a dead stream thread instead of shutting down the instance, and a state listener that logs every transition. chaos-kill.sh greps those "REBALANCING to RUNNING" log lines on both survivors to compute the failover time, so the logs double as the measurement instrument.
Why does the bridge read the value as a String instead of a MatchSummary? Intermediate
If a deserializer turned bytes into MatchSummary, bad JSON would fail inside Kafka's deserialization, before the listener, and need a separate handling path. Reading a String and parsing in the listener turns bad JSON into a NonRetryableArenaException, so one DefaultErrorHandler and one KafkaTemplate<String, String> dead-letter both garbage and Arena rejections the same way.
Which test proves the verifier can't be fooled by aborted transactions? Deep
TopicSnapshotReaderIT.neverShowsRecordsOfAnAbortedTransaction: it writes records in a Kafka transaction, aborts it, and checks the reader returns none of them. That's what makes read_committed a tested guarantee rather than a configuration line.
Why does PopulationFactory give cheaters their base profile's skill tier? Curveball
The tier is computed from the base profile's headshot chance (base.skillTier()), never from the cheat profile. An aimbot's cheat profile (headshots 0.75–0.95) would always come out as PRO, so tiers would correlate with cheating and leak the answer into the flags (and into the Judge, which treats top-tier bans specially). Keeping the base tier means the profile reflects the player as the platform knew them before the cheat, so "the tier does not give them away" (the code's own comment).