Kafka Streams
Telemetry's detector is a Kafka Streams application: it parses 25,000 events per second, aggregates them in hopping windows, joins flags with player profiles, de-duplicates with a state store, and does all of it exactly once across three instances, one of which you killed mid-run. This is the most advanced technology in your portfolio. Learn it well and it becomes your strongest interview topic.
By the end you can answer
- "What is Kafka Streams, and how is it different from Flink or Spark?"
- "KStream vs KTable? Stateless vs stateful operations?"
- "How are state stores made fault-tolerant? What are standby replicas?"
- "Explain tumbling, hopping, sliding and session windows. What is grace?"
- "What is repartitioning, and what was your 'repartition trap'?"
- "How does
exactly_once_v2work, and how did you prove it?" - "Why did failover take 14 seconds instead of 6?"
- "How do you test a topology?"
1. What Kafka Streams is
Kafka Streams is a Java library, not a cluster. You add it to an ordinary application; it reads from Kafka topics, processes records, keeps state, and writes results back to Kafka. To scale, you start more copies of the application with the same application.id. They form a consumer group and split the partitions between them automatically.
| Kafka Streams | Apache Flink / Spark Streaming | |
|---|---|---|
| Deployment | A library inside your app: a normal jar, a normal container | A separate processing cluster (job managers, workers) |
| Sources and sinks | Kafka in, Kafka out | Many (Kafka, files, databases...) |
| State backup | Changelog topics in Kafka | Checkpoints to distributed storage |
| Fits when | Kafka-centric microservices, moderate complexity | Very large state, complex event-time pipelines, many sources |
The detector is three replicas (Compose deploy.replicas: 3), each with num.stream.threads: 2, so six processing threads for six input partitions.
2. Topologies: the DSL and the Processor API
A topology is a directed graph of processing steps: source nodes read topics, processor nodes transform records, sink nodes write topics. You build it with the high-level DSL (filter, mapValues, groupByKey, windowedBy, join...) and drop to the low-level Processor API when you need custom logic with direct state-store access.
match-events (raw bytes)
├─► process(ParseProcessor) ─► split ─┬─ "failed" ─► mapValues(deadLetter) ─► match-events.DLT
│ └─ "ok" ─► process(IntegrityProcessor, store seq-tracker)
│ └─(punctuator every 5 s)─► integrity-stats
└─► mapValues(EventParser) ─► filter(valid) ← key-preserving re-parse
├─► filter(SHOT) ─► groupByKey ─► windowedBy(5 s hop 1 s, grace 2 s) ─► aggregate(AimStats)
│ ─► toStream ─► filter(aimbotTriggered) ─► map(to plain key + FlagCandidate) ─┐
└─► filter(POSITION) ─► ... same shape ... SpeedStats ─► speedhackTriggered ─────────┤
merge
─► repartition("flags") ─► leftJoin(KTable player-profiles) ─► process(CooldownProcessor)
─► suspicious-players
TelemetryTopology is plain Java: its constructor takes DetectionProperties, a Clock and a TopologyMetrics, and addTo(StreamsBuilder) adds every node. There is no Spring in it, so tests build it directly. The only Spring glue is StreamsWiring, a component whose constructor receives Spring Kafka's auto-configured StreamsBuilder and calls new TelemetryTopology(...).addTo(builder). Spring Kafka then starts the KafkaStreams client.
3. KStream, KTable and GlobalKTable
| Abstraction | Each record means | Your use |
|---|---|---|
| KStream | An independent event ("player X fired a shot at t"). Two records with the same key are two facts. | match-events, the flag streams |
| KTable | An update to the latest value for its key (an upsert). Two records with the same key: the second replaces the first. Backed by a local state store; partitioned like its topic. | builder.table("player-profiles", ..., Materialized.as("player-profiles-store")) |
| GlobalKTable | Like a KTable, but every instance holds the entire table | Not used. It would avoid co-partitioning at the cost of replicating all data to every instance |
This "stream-table duality" is central: a table is the latest-value-per-key view of a stream, and a stream is the changelog of a table. A compacted topic is exactly that changelog.
4. Stateless operations, and why map vs mapValues matters
filter: drop records.split().branch(...): route to named sub-streams (the parse step splits intoparse-failedandparse-ok).merge: combine streams (aimbot and speedhack flags).mapValues: change the value, keep the key. Streams knows partitioning is unchanged.map/selectKey: may change the key. Streams marks the stream "key-changing": if a stateful operation or a join comes next, it must repartition (section 7).process(...)(Processor API): Streams can't see what your code does to the key, so it conservatively marks the output as possibly key-changing too. That fact caused the repartition trap in section 7.
5. Stateful operations and state stores
Aggregations need memory: "how many hits has this player had in this window?". Kafka Streams keeps that in state stores, local key-value (or window) stores, by default backed by RocksDB, an embedded on-disk key-value database under state.dir, so state can exceed RAM.
| Store | Type | Holds |
|---|---|---|
aimbot-windows, speedhack-windows | Window stores (from aggregate) | AimStats / SpeedStats per player per window |
seq-tracker | Key-value (addStateStore) | IntegrityStats per player: count, last seq, duplicates, gaps |
flag-cooldown | Key-value | Last flag time per playerId|rule |
player-profiles-store | Key-value (the KTable) | Latest profile per player |
Fault tolerance: changelog topics
A local disk isn't safe: if the instance dies, its state is gone. So every write to a store is also sent to an internal, compacted changelog topic in Kafka (named <application.id>-<store>-changelog). When a task moves to another instance, that instance restores the store by replaying the changelog, then continues. Kafka is both the input and the backup of the state.
Record caches
With statestore.cache.max.bytes: 67108864 (64 MB), updates to the same key are coalesced in memory and flushed downstream (and to the changelog) at commit time or on eviction. That cuts traffic dramatically, but changes when results appear. Telemetry's notes warn not to assert exact evidence numbers in integration tests, because with caching the first emitted flag can come from a different window than in a TopologyTestDriver test. The README records the effect of turning the cache off: every update went to the changelog, the detector fell behind, and median detection latency rose from about 3 s to about 30 s.
6. Time and windows
Which "time"?
- Event time: when the event happened (the record timestamp, set from
MatchEvent.ts). Windows use this. - Processing (wall-clock) time: when your code runs. Punctuators can use it; the cooldown uses the injected
Clock. - Stream time: per task, the largest event timestamp seen so far. It only advances when new records arrive. It decides when windows close.
Window types
| Type | Shape | Use |
|---|---|---|
| Tumbling | Fixed size, no overlap: [0,5) [5,10) ... | "Per 5-second bucket" reports |
| Hopping | Fixed size, start every advance; windows overlap | Moving statistics with fixed boundaries: the detector |
| Sliding | Windows defined by the time difference between records | "Any two events within 5 s" |
| Session | Grow with activity; close after an inactivity gap | User sessions, bursts of play |
Hopping windows: size 5 s, advance 1 s (each event lands in 5 overlapping windows)
time (s) 0 1 2 3 4 5 6 7 8
[=========================) window [0,5)
[=========================) window [1,6)
[=========================) window [2,7)
[=========================) window [3,8)
▲
a cheat crossing the threshold here is seen
by the next window to close over it, within about 1 s
windowEvents
.filter((key, event) -> event.type() == EventType.SHOT)
.groupByKey(Grouped.with(Serdes.String(), JsonSerdes.matchEvent()))
.windowedBy(TimeWindows.ofSizeAndGrace(aim.windowSize(), aim.grace()) // 5 s, grace 2 s
.advanceBy(aim.advance())) // 1 s
.aggregate(AimStats::empty, (key, event, stats) -> stats.add(event),
Materialized.<String, AimStats, WindowStore<Bytes, byte[]>>as(AIMBOT_WINDOWS) ...)
.toStream()
.filter((windowedKey, stats) -> stats != null && rules.aimbotTriggered(stats))
.map((windowedKey, stats) -> KeyValue.pair(windowedKey.key(),
FlagCandidate.aimbot(windowedKey.key(), windowedKey.window(), stats, rules)));
- Why hopping, not tumbling? With tumbling 5 s windows, a burst that straddles a boundary (half in [0,5), half in [5,10)) might satisfy neither window. Hopping windows overlap, so some window always covers any 5-second stretch (to 1 s resolution), and a cheater is flagged about a second after crossing the line. Wider windows would be slower; narrower ones noisier.
- Grace (2 s): events can arrive out of order. A window keeps accepting late events for 2 s of stream time after its end; after that it is closed and later events are dropped (and counted in a metric).
- Emission: by default a windowed aggregation emits an updated result whenever the aggregate changes (subject to caching). The detector wants that: it flags as soon as a threshold is crossed rather than waiting for the window to close (which
suppress(untilWindowCloses)would do). The price is repeated matches from the same and overlapping windows, which the cooldown handles. - Inclusive boundaries in the rules (
>=,<=), and scores from 0.5 (just over) to 1.0 (far over), in the pureDetectionRulesclass.
AimbotTopologyTest sends 15 shots, one every 200 ms from T0, all hits, the last 13 headshots, at 60 ms reaction time. It asserts exactly one flag whose window is [T0 − 2 s, T0 + 3 s). The 12th shot (at T0 + 2.2 s) is the first moment any window holds 12 hits, and the earliest window containing it starts at T0 − 2 s. That window holds shots 1 to 12, 10 of them headshots, giving evidence (12, 12, 10, 0.8333, 60.0) and score 0.843. Being able to derive that number by hand in an interview is very convincing.
7. Repartitioning, co-partitioning and the "repartition trap"
A stateful operation (aggregate, join) must see all records for a key in the same task, so the data must be partitioned by that key. If the key may have changed upstream, Kafka Streams inserts an internal repartition topic: it writes every record back to Kafka with the new key and reads it again. That's correct, but it costs a full extra write and read through the broker for every record.
The specification's snippet grouped the output of ParseProcessor (a process() call) for the windows. Because KStream#process marks its output as possibly key-changing, Kafka Streams would add aimbot-windows-repartition and speedhack-windows-repartition topics and push every one of about 24,000 events per second through the broker again, twice.
The fix: the window branches don't consume the parse processor's output. They re-parse the raw stream with mapValues (key-preserving) using the same pure EventParser, so both paths accept exactly the same records and no repartition topic appears. Parsing JSON twice is far cheaper than an extra broker round trip per event. TopologyShapeTest asserts the topology description contains no window repartition topics, and DetectorEndToEndIT checks against a real broker, so nobody "fixes" it back.
The one repartition you want: flags
After windowing, the key is a Windowed<String>; mapping it back to the plain player id is a key change, and the join needs partitioning by player id. Here a repartition is unavoidable, but flags are rare, so it's cheap. The code makes it explicit and named:
aimFlags.merge(speedFlags)
.repartition(Repartitioned.with(Serdes.String(), JsonSerdes.of(FlagCandidate.class))
.withName(FLAGS_REPARTITION)) // internal topic "...-flags-repartition"
.leftJoin(profiles, FlagCandidate::withProfile, Joined.with(...))
Naming it keeps the internal topic name stable across code changes. Auto-generated names contain node numbers (KSTREAM-...-0000000017) that shift when you add a step, which would orphan the old topic and lose its data on redeploy.
Co-partitioning for joins
A KStream-KTable join happens locally in each task: flag partition 2 is joined against profile partition 2. That only works if both topics are co-partitioned: same number of partitions (6 and 6), same key (player id) and same partitioner (default). That's why every player-keyed topic has exactly 6 partitions. A GlobalKTable would remove the requirement by copying the whole table to every instance.
The join itself
A left join of a stream with a table looks up the table's current value for the key when each flag arrives. "Left" means a flag with no profile is still emitted, with username and skillTier as null, rather than dropped. Dropping it would hide cheaters whose profile hadn't arrived yet. The ValueJoiner is the record method FlagCandidate::withProfile.
8. The Processor API: custom logic with stores and punctuators
public final class CooldownProcessor implements Processor<String, FlagCandidate, String, SuspiciousPlayer> {
public void init(ProcessorContext<String, SuspiciousPlayer> context) {
this.context = context;
this.store = context.getStateStore(TelemetryTopology.COOLDOWN_STORE);
}
public void process(Record<String, FlagCandidate> record) {
FlagCandidate c = record.value();
String storeKey = c.playerId() + "|" + c.rule(); // an aimbot flag never hides a speedhack flag
long now = clock.millis();
Long last = store.get(storeKey);
if (last != null && now - last < cooldown.toMillis()) return; // drop repeats for 10 min
store.put(storeKey, now);
context.forward(record.withValue(new SuspiciousPlayer(UUID.randomUUID().toString(), ...)));
}
}
- Implement
Processor<KIn, VIn, KOut, VOut>fromorg.apache.kafka.streams.processor.api(the newer, typed API). Attach it withstream.process(() -> new CooldownProcessor(...), COOLDOWN_STORE): a supplier, because each task needs its own instance, and the store names it may access. context.forward(record)emits downstream; emitting nothing filters the record out.context.recordMetadata()gives the source topic, partition and offset.ParseProcessorputs them in each dead letter, along with the Base64 payload and the reason, so a bad record can be found and replayed.- Telemetry's rules forbid the older
transform/transformValues(deprecated in favour ofprocess) and alsoprocessValues, the key-preserving variant. That restriction is why the windows usemapValuesplusEventParser.
Punctuators: doing work on a schedule
public void init(ProcessorContext<String, IntegrityStats> context) {
this.store = context.getStateStore(TelemetryTopology.SEQ_STORE);
context.schedule(emitInterval, PunctuationType.WALL_CLOCK_TIME, timestamp -> emitAll()); // every 5 s
}
public void process(Record<String, MatchEvent> record) {
store.put(event.playerId(), apply(current, event.seq())); // nothing forwarded here
}
static IntegrityStats apply(IntegrityStats s, long seq) {
if (seq == s.lastSeq() + 1) return /* count+1, lastSeq=seq */; // next in order
if (seq <= s.lastSeq()) return /* duplicates+1 */; // seen before
return /* count+1, gaps += seq - lastSeq - 1, lastSeq=seq */; // something missing
}
A punctuator runs a callback periodically, by WALL_CLOCK_TIME (real time, even if no records arrive) or STREAM_TIME (advances only with incoming event timestamps). The integrity processor re-emits the stats of every player every 5 seconds. A consumer then only needs the latest value per key, and a task restored on another instance after failover needs no memory of "what changed since the last emit".
9. Exactly-once: how it works and how you proved it
The mechanism
With processing.guarantee: exactly_once_v2, each stream thread wraps its work in Kafka transactions (module 09). Within one commit interval (commit.interval.ms: 500) it:
- reads input records,
- writes output records (flags, integrity stats, dead letters) and state-store changelog records,
- adds the input offsets it consumed to the same transaction,
- commits: all of it becomes visible atomically, or on failure none of it does.
If an instance dies mid-interval, its open transaction is aborted. The task's new owner restores the store from the changelog (which only contains committed updates) and resumes from the last committed input offset, so every record affects state and output exactly once. Downstream consumers must read with read_committed to ignore the aborted attempt. "v2" uses one transactional producer per thread rather than per task (v1), which scales much better with many partitions.
Why "at-least-once" can double count
Without transactions, a store update can reach the changelog and the output before the input offset is committed. Kill the instance in that gap and the new owner re-reads those events from the old offset and applies them again on top of the restored state: duplicates in the counts.
The proof: per-player sequence numbers
- The simulator gives every player's events a gapless
seq1, 2, 3, ... and records each player's finalseqin thesimulation-manifesttopic. IntegrityProcessorcounts what arrives per player: in-order, duplicates and gaps.- The verifier compares the two for every player: count must equal
finalSeq, with zero duplicates and zero gaps.
This is stronger than comparing totals: one lost event plus one duplicated event would leave the total unchanged, while the per-player check catches both. A test that can't fail proves nothing, so ExactlyOnceIT includes a negative control: it re-sends one event on purpose and asserts that exactly one duplicate is detected.
| Measured (1,000 players, 5 min, 3 replicas, one killed ~90 s in) | exactly_once_v2 |
|---|---|
Events expected (sum of finalSeq) | 7,466,329 |
| Events counted | 7,466,329 |
| Duplicates / gaps | 0 / 0 |
| Players passing | 1,000 of 1,000 |
You also ran the same scenario with at_least_once, expecting duplicates. You got none, twice: once with defaults and once with the state-store cache off, the condition under which the window should be widest. So the runs prove the exactly-once run was clean and that the checker can detect duplicates (the negative control), but they do not show exactly-once beating at-least-once. The likely reason, unverified, is that a duplicate needs the kill to land in the short gap between a changelog write and the offset commit, and one docker kill per run didn't hit it.
How to say it: "My comparison didn't show the difference I expected, and I reported that instead of tuning until it did. To demonstrate it, I'd run many kills per run, or inject a fault exactly between the store write and the commit."
10. Tasks, scaling and failover
- Kafka Streams splits the topology into sub-topologies (separated by repartition topics) and creates one task per input partition of each sub-topology. A task owns its partitions and its slice of every state store. A task is processed by one thread at a time, so no locks are needed inside processors.
- Tasks are spread over all threads of all instances in the group: 3 instances × 2 threads.
- Standby replicas (
num.standby.replicas: 1): each store is also continuously copied, by reading its changelog, into a standby task on another instance. On failover that instance takes over with a warm store instead of restoring from scratch. - Detection speed:
session.timeout.ms: 6000(the minimum) withheartbeat.interval.ms: 2000lets the group notice a killed instance in about 6 s instead of the default 45 s.
The failover you measured: 14.4 s against a target of under 10 s (missed)
The survivors' logs gave a clear timeline:
- About 5 s until the group noticed the dead member (the session timeout).
- Three rebalances back to back, about 8–9 s together. The README's own explanation: task migration happens in steps, and each step triggers a follow-up rebalance. One known reason for steps is the cooperative rebalancing protocol: a partition that moves between two running instances is first revoked from its old owner in one round and only assigned to its new owner in a follow-up round. A crashed instance's partitions have no running owner to give them up, so they can be reassigned in the very first round. The exact cause of the two follow-up rounds wasn't pinned down, so say "in steps" rather than guessing.
Meanwhile two instances had to absorb the full 25,000 events/s with only about 5,000/s of headroom, so lag peaked near 380,000 events (per-partition maximum) and took about 2 minutes to drain. Afterwards all 6 partitions were owned by the two survivors.
Tune the warm-up and probing settings (acceptable.recovery.lag, probing.rebalance.interval.ms, max.warmup.replicas); try the new Kafka Streams rebalance protocol; and, for planned restarts, use static membership (group.instance.id) so a quick restart doesn't trigger a rebalance at all. And add capacity: with 3 instances running near their limit, losing one leaves too little headroom to keep lag flat.
11. Bad data and errors
| Option | What it can do |
|---|---|
Deserialization exception handler (LogAndContinue / LogAndFail) | Only skip or crash; the record is lost or the thread stops |
| Parse inside the topology (Telemetry's choice) | Read raw byte[]; a processor turns bad bytes, invalid events or key mismatches into a DeadLetter (topic, partition, offset, key, Base64 payload, reason, time) routed to match-events.DLT; the stream carries on |
| Uncaught exception handler | Decide whether to replace the thread or shut down the client after an unexpected error |
| Native DLQ support in newer Kafka Streams | Listed in the README as a next step, replacing hand-routing |
The simulator deliberately injects malformed records (0.1%, 7,503 in the measured run) to prove this path works under load.
12. Testing a topology
TopologyTestDriverruns a topology in memory, synchronously: no broker, no threads, no Spring. You pipe records in with exact timestamps and read output topics.TopologyFixturewraps it with aMutableClockand helper methods;driver.advanceWallClockTime(...)fires wall-clock punctuators on demand. Window boundaries, grace, joins and cooldowns can be tested to the millisecond.- Shape tests on
topology.describe():TopologyShapeTestasserts the only sources arematch-events,player-profilesand theflagsrepartition topic; that no window repartition topics exist; and thatcheater-ground-truthnever appears anywhere, so the detector can't "cheat" by reading the answers. - Integration tests (
*IT) run the real Spring app againstapache/kafka-nativein Testcontainers, with a randomapplication.idandstate.dirper test class and a short commit interval, so parallel apps don't share a group.
13. Interview questions
What is Kafka Streams and how does it scale? Basic
A Java library for stream processing on Kafka, running inside a normal application. Instances sharing an application.id form a consumer group; the topology is split into tasks, one per input partition, which are spread over all instances' threads. To scale, run more instances, up to the partition count. My detector runs 3 replicas with 2 threads each for 6 partitions.
KStream vs KTable? Basic
A KStream is a sequence of independent events: two shots by the same player are two facts. A KTable is a changelog where each record updates the latest value for its key. I read match-events as a stream and player-profiles (a compacted topic) as a table, then left-join flags against the table to add username and skill tier.
How is state fault-tolerant? Intermediate
State lives in local RocksDB stores, and every write also goes to a compacted changelog topic. If an instance dies, the task moves and its new owner restores the store by replaying the changelog. Standby replicas keep a continuously updated copy on another instance so that restore is near-instant. Under exactly-once, the changelog contains only committed updates, so the restored state matches the committed offsets.
Explain hopping windows and grace, and why you chose 5 s / 1 s / 2 s. Intermediate
A hopping window has a fixed size and starts every advance, so windows overlap; with 5 s and 1 s each event is in 5 windows. Overlap means any 5-second burst is fully inside some window, so detection happens about 1 s after a threshold is crossed, which tumbling windows can miss at boundaries. 5 s is long enough to hold 12+ shots and 40+ movement samples. Grace of 2 s accepts events arriving up to 2 s late in stream time before a window closes. Measured median detection latency was 3.9 s for obvious aimbots and 0.9 s for speedhacks.
What triggers a repartition, and what was your repartition trap? Deep
Any stateful operation or join after a possibly key-changing step (map, selectKey, groupBy, or process(), which Streams must treat as key-changing) makes Streams insert an internal repartition topic, rewriting every record through the broker. The spec grouped the parse processor's output for the windows, which would have added two repartition topics and sent all 24,000 events/s through Kafka twice more. I re-parse the raw stream with key-preserving mapValues and the same parser instead, and guard it with a topology-shape test and an end-to-end test. The one repartition I keep is the explicit, named flags step before the join: the windowed key must become a plain key, and flags are rare.
What is co-partitioning and why do all your player-keyed topics have 6 partitions? Intermediate
A KStream-KTable join runs locally per task, so partition n of the stream must contain the same keys as partition n of the table. That requires the same key, the same partitioner and the same number of partitions. Flags (after the flags repartition) and player-profiles both have 6 partitions keyed by player id with the default partitioner. A GlobalKTable would avoid the requirement by copying the whole table to every instance.
How does exactly_once_v2 work? Deep
Each stream thread uses a transactional producer. Per commit interval it writes outputs and state-store changelog records, adds the consumed input offsets to the same transaction, and commits atomically. If the instance dies, the transaction aborts; the new owner restores state from committed changelog data and resumes from committed offsets, so every input affects state and output exactly once. Consumers read with read_committed to skip aborted output. "v2" shares one producer per thread instead of one per task, which reduces producers and transactional overhead.
How did you prove no event was lost or duplicated? Deep
Every event carries a per-player gapless sequence number, and the simulator records each player's final value in a manifest topic. A processor in the detector counts in-order events, duplicates and gaps per player in a state store, and a punctuator publishes all players' stats every 5 seconds. The verifier then checks count = finalSeq, duplicates = 0 and gaps = 0 for all 1,000 players. With one of three instances killed mid-run, all 7,466,329 events were counted exactly once. It's stronger than comparing totals, because a loss and a duplicate could cancel out. And a negative-control test injects one duplicate to prove the checker can fail.
Why did failover take 14.4 s when the session timeout is 6 s? Deep
Detection took about 5 s, and then Kafka Streams ran three rebalances back to back. Task migration happens in steps, and each step triggers a follow-up rebalance; with cooperative rebalancing, work that moves between two running instances is revoked in one round and reassigned in the next, while the dead instance's work can be reassigned straight away. Together that added 8–9 s. I report it as a missed target. Next I'd tune warm-up and probing settings, try the newer rebalance protocol, use static membership for planned restarts, and add headroom: two survivors could only process about 5,000 events/s above the input rate, so lag took 2 minutes to drain.
Your at-least-once run showed no duplicates either. So is exactly-once pointless? Curveball
No. Absence of evidence in two runs isn't evidence of absence. At-least-once only duplicates if the instance dies in the narrow window after state or output is written but before offsets commit; a single docker kill per run probably didn't land there. The exactly-once run was verified clean, and the checker is proven to detect duplicates by a negative control, but my comparison didn't demonstrate the difference. I reported that honestly. To demonstrate it, I'd automate many kills per run or inject a fault precisely between the changelog write and the commit.
Why parse inside the topology instead of using a deserialization exception handler? Intermediate
A deserialization handler can only log and skip, or fail the thread. Reading raw bytes and parsing in a processor turns every failure (unparseable JSON, an invalid event, a key that doesn't match the player id) into a dead-letter record with the source topic, partition, offset, Base64 payload and reason, routed to a DLT by ordinary topology code, while the stream continues. The simulator injects 0.1% malformed records to exercise this path.
How do you test windowing logic deterministically? Intermediate
TopologyTestDriver: the topology runs synchronously in memory with record timestamps I control and wall-clock time I advance by hand, with no broker and no Spring. My fixture wraps it with a mutable clock. That lets me assert, for example, that 15 headshots at 200 ms spacing produce exactly one flag from window [T0−2 s, T0+3 s) with score 0.843. I also assert the topology's shape from describe(), and run a few integration tests on a real broker, where I don't assert exact evidence numbers because caching changes which window emits first.
Why name the repartition topic? Curveball
Unnamed internal topics get generated names with processor numbers. Adding a step anywhere upstream shifts the numbers, so a redeployed app would create a new internal topic and abandon in-flight data in the old one, and state store names can shift the same way. Naming the repartition (flags) and the stores keeps them stable across code changes, which is important for upgrades of a stateful app.
How does the detector make sure it never peeks at who really cheats? Curveball
Ground truth is written by the simulator to its own topic, cheater-ground-truth, that only the verifier reads. A topology-shape test fails if that topic appears anywhere in the detector's topology description, as a source or a sink. It's an evaluation-hygiene guard, like keeping a test set away from training.
14. Quick quiz
1. With 5 s windows advancing by 1 s, each event belongs to how many windows?
Size / advance = 5 overlapping windows.
2. Which operation keeps the key, so no repartition is needed before a stateful step?
process is treated as possibly key-changing; that was the trap.
3. A local state store survives the loss of its instance because:
The new owner restores from the changelog (or a warm standby).
4. A KStream-KTable join needs both topics to be:
Unless the table is a GlobalKTable.
5. The integrity check compares per player:
Per-player sequence accounting catches losses and duplicates even when they would cancel out in a total.
6. Most of the 14.4 s failover came from:
Task migration happens in steps, each triggering a follow-up rebalance. With cooperative rebalancing, work moving between two running instances needs a revoke round and then an assign round; a crashed instance's work can be reassigned in the first round.
7. A WALL_CLOCK_TIME punctuator fires:
STREAM_TIME would only advance with incoming timestamps.
15. Cheat sheet
- Kafka Streams = a library; instances with the same
application.id= a group; one task per input partition; 3 replicas × 2 threads for 6 partitions. - Topology: parse (DLT) → integrity (store + 5 s punctuator) → re-parse with mapValues → hopping windows (aggregate) → rules → merge → named
flagsrepartition → leftJoin profiles KTable → cooldown (store) →suspicious-players. - KStream = events; KTable = latest per key; GlobalKTable = a full copy everywhere.
- State: RocksDB stores + compacted changelogs; restore on failover; standby replicas = warm copies; a 64 MB cache coalesces (and shifts emission timing).
- Time: event, stream and wall-clock. Windows: tumbling, hopping (5 s / 1 s, grace 2 s), sliding, session. Emits on update; the cooldown de-duplicates.
- Repartition after key change + stateful op; trap:
process()is key-changing → re-parse withmapValues; guarded byTopologyShapeTest. Name internal topics. - Co-partitioning: same key, partitioner and count (6) for joins.
- Processor API:
process(supplier, stores),init/process,forward,getStateStore,recordMetadata, punctuators (WALL_CLOCK vs STREAM_TIME). - EOS v2: outputs + changelog + offsets in one transaction per commit interval; restore from committed;
read_committeddownstream. - Proof: per-player
seqvs manifestfinalSeq; 7,466,329 = 7,466,329, 0 dup, 0 gaps; a negative control. ALOS also showed 0 (be honest). - Failover: 14.4 s (target < 10 s missed) = ~5 s detection + 3 rebalances; lag peak ~380k, ~2 min to drain.
- Tests: TopologyTestDriver + MutableClock; shape tests; ITs on kafka-native.