Fairplay Interview Prep
Module 09

Apache Kafka and Spring Kafka

Kafka is the spine between Telemetry and the Judge: about 25,000 events per second flow in, flags flow out, and the Judge even stores its only state in a compacted topic. This module builds Kafka up from "an append-only log" to producers, consumer groups, offsets, replication, compaction, transactions and error handling with dead-letter topics, all tied to your configuration files.

P2P3· about 2.5 hours

By the end you can answer

  • "What is Kafka? How is it different from a traditional message queue?"
  • "What are partitions for? How is ordering guaranteed?"
  • "How do consumer groups and rebalancing work? What is consumer lag?"
  • "Explain acks, idempotent producers and transactions. What does read_committed do?"
  • "At-most-once, at-least-once, exactly-once: which did you use where?"
  • "Retention vs compaction? How can a topic be a database?"
  • "How did you handle poison messages and outages in consumers?"

1. The big idea: a distributed, append-only log

A log here means an ordered, append-only sequence of records. New records are added at the end and never changed. Each record gets a number, its offset. Readers don't remove anything; each reader just remembers how far it has read.

 topic "match-events", partition 3:
 offset:   0      1      2      3      4      5      6  ...  (append →)
         [shot] [pos]  [shot] [kill] [shot] [pos]  [shot]
                         ▲                    ▲
          detector group's committed offset   verifier is here (reads independently)
Analogy: a newspaper archive

A traditional queue is a to-do pile: someone takes a task and it's gone. Kafka is a newspaper archive: every issue is kept (for a configured time), and every reader has a bookmark. The detector, the verifier and a brand-new analytics job can all read the same issues at their own pace, and anyone can re-read from an older bookmark ("replay").

Kafka (log)Classic queue (RabbitMQ, SQS)
After readingThe record stays; the reader advances its offsetThe message is acknowledged and deleted
Many independent readersNatural (each group has its own offsets)Needs fan-out exchanges or one queue per reader
Replay historyYes (reset offsets)No
OrderingPer partitionUsually per queue, weakened by multiple consumers or redelivery
ThroughputVery high (sequential disk I/O, batching, zero-copy)High, with more per-message work
Per-message routing, priorities, delaysLimitedRich

2. Core vocabulary

3. Partitions, keys and ordering

Partitions exist for two reasons: parallelism (different partitions can be written and read by different machines at once) and ordering (order is guaranteed within a partition, and only there).

With a key, the default partitioner computes murmur2(keyBytes) mod numPartitions, so the same key always lands in the same partition. Every event of player X goes to one partition, in the order it was produced, and one consumer sees all of them in order. That's exactly what per-player windows and per-player sequence checks need.

TopicPartitionsCleanupWhy that shape
match-events6delete, 24 hHigh volume; 6 partitions allow up to 6 parallel detector tasks
player-profiles6compactA table (latest profile per player); co-partitioned with flags for the join
suspicious-players6delete, 7 dOutput flags, keyed by player
integrity-stats6compactLatest count, duplicates and gaps per player
match-events.DLT6delete, 7 dBad events, same keying
match-summaries / .DLT3delete, 7 dLow volume, one per finished match
cheater-ground-truth, simulation-manifest1compactSmall reference data read whole by the verifier
judge-verdicts (Judge)6compactThe Judge's only state: latest verdict per flag
Partition gotchas interviewers love
  • Max consumer parallelism = number of partitions. A 7th consumer in a group for a 6-partition topic sits idle.
  • Adding partitions later remaps keys (hash mod 6 ≠ hash mod 8), breaking per-key ordering across the change and breaking co-partitioning. Choose counts up front.
  • Hot partitions: if one key dominates, its partition is overloaded no matter how many partitions exist.
  • Telemetry's rules say: use the default partitioner and never write a custom one, so producers in any language agree on placement.

4. Producers: batching, acks and idempotence

telemetry-simulator/src/main/resources/application.yml
spring.kafka.producer:
  key-serializer:   org.apache.kafka.common.serialization.StringSerializer
  value-serializer: org.apache.kafka.common.serialization.ByteArraySerializer   # bytes, so garbage can be injected on purpose
  acks: all
  batch-size: 65536                 # up to 64 KB per partition batch
  compression-type: lz4
  properties:
    enable.idempotence: true
    linger.ms: 10                   # wait up to 10 ms to fill a batch
    max.in.flight.requests.per.connection: 5
    delivery.timeout.ms: 120000     # give up after 2 minutes of retries

Throughput: batching and compression

send() doesn't hit the network immediately. Records are appended to an in-memory batch per partition, which is sent when it's full (batch.size) or after linger.ms. Bigger batches mean fewer requests and better compression (lz4 is fast). 25,000 small JSON events per second is easy for one producer this way. The trade-off is a few milliseconds of added latency.

Durability: acks

acksThe producer's send succeeds whenRisk
0It's on the wireSilent loss
1The partition leader wrote itLoss if the leader dies before followers copy it
allAll in-sync replicas wrote itSlowest; the safest (with min.insync.replicas ≥ 2)

No duplicates from retries: the idempotent producer

A producer retries when it doesn't get an acknowledgement, but the first attempt may actually have succeeded, so a naive retry writes a duplicate. With enable.idempotence=true (the default since Kafka 3.0) the producer gets a producer id and numbers each batch per partition. The broker remembers the last sequence per producer and partition, and discards duplicates and rejects gaps. Retries become safe and order is preserved, even with up to 5 requests in flight.

That's why EventPublisher's comment says sends are "never retried by hand: the idempotent producer already retries safely". A send that still fails after delivery.timeout.ms is counted in the manifest (failed), so the integrity check can account for it. The measured run had 0 failed sends.

Idempotence is per producer session

It deduplicates retries within one producer's lifetime. If the application itself crashes and re-sends after restart, that's a new producer id and a new record. End-to-end duplicates need transactions (section 8) or idempotent consumers.

5. Brokers, replication and KRaft

Be upfront: your setup is single-broker

Telemetry runs one broker with replication factor 1 (TOPIC_RF=1), fine on a laptop but with no tolerance for a broker failure. The failure you tested was a consumer instance being killed, not a broker. Telemetry's README lists "a second broker with replication factor 3" as next work. Say this plainly if asked about durability.

KRaft: no more ZooKeeper

Older Kafka kept cluster metadata (topics, partitions, leaders) in a separate ZooKeeper cluster. KRaft moves that into Kafka itself: a quorum of controller nodes agrees on metadata using the Raft consensus protocol. Kafka 4.0 removed ZooKeeper support entirely. Your Compose broker runs in combined mode, KAFKA_PROCESS_ROLES: broker,controller, with KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093. Telemetry's notes forbid any ZooKeeper setting in source.

Listeners: the classic Docker confusion

compose.yaml (Telemetry)
KAFKA_LISTENERS:            INTERNAL://:19092,EXTERNAL://:9092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:19092,EXTERNAL://localhost:9092

A client first connects to a bootstrap address, and the broker then tells it which address to use (the advertised listener). Containers on the Compose network must be told kafka:19092; your laptop's IntelliJ must be told localhost:9092. One listener per audience. That's why the apps use KAFKA_BOOTSTRAP=kafka:19092 in Compose and localhost:9092 by default outside it. Getting this wrong ("connects, then hangs") is the most common Kafka-in-Docker bug.

6. Consumers, groups, offsets and rebalancing

Consumers pull records in a loop (poll()). Consumers with the same group.id form a consumer group: Kafka assigns each partition to exactly one member of the group, so the group divides the work. A different group gets its own full copy of the stream.

 suspicious-players (6 partitions)          group "judge-agent" (one Judge app, concurrency 3)
   P0 P1 ─────────────────────────────────► consumer thread 1
   P2 P3 ─────────────────────────────────► consumer thread 2
   P4 P5 ─────────────────────────────────► consumer thread 3
 the verifier reads the same topic in its own group: a full independent copy

Offsets and commits

A group stores its progress, the next offset to read for each partition, in the internal topic __consumer_offsets. On restart or reassignment, consumption resumes from the committed offset. auto.offset.reset decides where a group with no committed offset starts: earliest (from the beginning, which both your consumers use, so nothing published before startup is missed) or latest.

Delivery semantics come from when you commit

SemanticsOrder of workOn a crash
At-most-onceCommit, then processThe record is skipped (lost)
At-least-onceProcess, then commitThe record is processed again (duplicate), so make processing idempotent
Exactly-onceProcess and commit atomically in one Kafka transactionOnly for Kafka-to-Kafka pipelines (module 10)

The Judge and the Arena bridge are at-least-once with idempotent effects. The bridge relies on Arena's externalId (a replay gets 200 and changes nothing). The Judge checks processed.hasFlag(flagId) against its replayed verdicts, re-reads player status in pre-checks, and handles Arena's 409.

Liveness: heartbeats, sessions and the poll interval

Rebalancing

When members join, leave or die, the group rebalances: partitions are redistributed. Classic "eager" rebalancing stops everyone's consumption during the shuffle; cooperative incremental rebalancing moves only the partitions that need to move. Kafka 4 also brings a new consumer group protocol in which the broker coordinates assignments. Rebalances are a big part of failover time, which module 10 measures (three back-to-back rebalances, 14.4 s).

Consumer lag

Lag = the partition's latest offset minus the group's committed offset: how far behind the consumer is. It's the key health metric of any streaming system. During Telemetry's failover, the per-partition maximum lag peaked near 380,000 events and took about 2 minutes to drain. Client-side lag metrics (records-lag-max) are windowed maxima and lag behind reality; the authoritative view is kafka-consumer-groups.sh --describe, or Kafka UI on port 8090.

7. Retention vs compaction: logs and tables

cleanup.policyKeepsThink of it asYour topics
deleteEverything newer than retention.ms (or under a size limit)An event historymatch-events (24 h), flags, DLTs (7 d)
compactAt least the latest record per keyA table / changelog / snapshotplayer-profiles, integrity-stats, cheater-ground-truth, simulation-manifest, judge-verdicts
In your projects: a topic as the Judge's database

The Judge has no database. Its state is the compacted judge-verdicts topic (latest verdict per flag id). At startup, ProcessedFlagsLoader reads the whole topic into memory before starting the flag listener (autoStartup = "false", then registry.getListenerContainer("judge").start()). The listener can then skip flags it already judged, and apply the "judged in the last 30 minutes" pre-check. The topic is both the output and the memory. This is a light form of event sourcing: state rebuilt by replaying a log.

8. Transactions and read_committed

A transactional producer (configured with a transactional.id) can write to several partitions and commit consumer offsets as one atomic unit:

producer.beginTransaction();
producer.send(record to suspicious-players P2);
producer.send(record to a state-store changelog P2);
producer.sendOffsetsToTransaction(offsets of match-events P2, groupMetadata);
producer.commitTransaction();          // all visible, or (abort) none

This read-process-write cycle is how Kafka Streams achieves exactly-once (module 10). Under the hood, the broker writes special commit or abort markers into each partition.

But aborted records are still physically in the log. A consumer with the default isolation.level=read_uncommitted reads them anyway. With read_committed, the consumer only reads up to the last stable offset and filters out records from aborted transactions. That's why every consumer of detector output (the verifier and the Judge, plus Telemetry's own bridge) sets isolation-level: read_committed. Without it, a detector killed mid-transaction could leak flags that were then rolled back and re-emitted, which would look like duplicates.

Exactly-once stops at Kafka's edge

Kafka transactions cover Kafka writes and Kafka offsets only. The Judge's HTTP call to Arena can't take part. That's why the Judge is at-least-once with idempotent, state-checked effects, and why the bridge leans on externalId. Saying "exactly-once inside Kafka; at-least-once plus idempotency at the boundaries" shows you really understand it.

9. Creating topics deliberately

telemetry-common/.../TopicsConfig.java
@Bean
NewTopic matchEventsTopic() {
    return TopicBuilder.name(Topics.MATCH_EVENTS)
            .partitions(6)
            .replicas(replicationFactor)
            .config(TopicConfig.CLEANUP_POLICY_CONFIG, TopicConfig.CLEANUP_POLICY_DELETE)
            .config(TopicConfig.RETENTION_MS_CONFIG, "86400000")      // 24 h
            .build();
}

Spring's KafkaAdmin creates any missing NewTopic beans at startup. The broker has KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false", so a typo in a topic name fails loudly instead of silently creating a new topic with default settings. The Judge declares NewTopics only for the topics it owns (judge-verdicts and its DLT) and never creates or changes Telemetry's: ownership again.

10. Message formats and schema evolution

All values are JSON produced by your own Jackson 3 serde. JSON is readable and flexible, but nothing enforces the shape: a producer can rename a field and every consumer breaks, or silently loses data (the skillTier curveball in module 01).

11. Spring Kafka

Producing with KafkaTemplate

template.send(topic, key, value) returns a CompletableFuture<SendResult> that completes when the broker acknowledges. The simulator collects thousands of these and waits with allOf(...).join(). The Judge's listener calls publisher.publish(verdict).join() so it doesn't return (and let the offset be committed) until the verdict is safely stored.

Consuming with @KafkaListener

judge-service/.../kafka/FlagListener.java
@KafkaListener(id = LISTENER_ID, topics = "${judge.kafka.flags-topic}", autoStartup = "false")
void onFlag(ConsumerRecord<String, String> record) {
    SuspiciousPlayer flag = parseFlag(record.value());     // bad JSON → NonRetryableJudgeException
    if (processed.hasFlag(flag.flagId())) {
        metrics.duplicateFlag();
        return;                                            // already judged: idempotent
    }
    JudgeVerdictRecord verdict = judgeService.handle(flag);   // pre-checks, models, Arena action
    publisher.publish(verdict).join();                     // wait for the broker's ack
    processed.remember(verdict);
}

Error handling: retries, back-off and dead-letter topics

When the listener throws, Spring's DefaultErrorHandler decides what to do. It doesn't commit the failed record; it seeks back so the record is delivered again after a back-off. After the retries are exhausted, a recoverer runs, typically the DeadLetterPublishingRecoverer, which publishes the record (with exception details in headers) to a dead-letter topic (DLT) and lets consumption move on.

Telemetry bridge: BridgeErrorHandlingConfig
ExponentialBackOffWithMaxRetries backOff =
    new ExponentialBackOffWithMaxRetries(3);
backOff.setInitialInterval(1000);   // 1 s
backOff.setMultiplier(2.0);         // 2 s, 4 s
DefaultErrorHandler handler =
    new DefaultErrorHandler(countingDltRecoverer, backOff);
handler.addNotRetryableExceptions(
    NonRetryableArenaException.class);

Temporary Arena failures (429, 5xx, timeouts): 4 attempts, then match-summaries.DLT. Permanent ones (400/404/409/422): straight to the DLT.

Judge: KafkaConfig
DefaultErrorHandler handler = new DefaultErrorHandler(
    recoverer,
    new FixedBackOff(5000,                    // every 5 s
        FixedBackOff.UNLIMITED_ATTEMPTS));    // forever
handler.addNotRetryableExceptions(
    NonRetryableJudgeException.class);

Outages (Claude or Arena down, circuit open): hold the flag and retry forever, never dropping it. Malformed flags and Claude 400s: straight to suspicious-players.judge.DLT.

The design principle behind both

Classify failures. A transient failure (the other side is down or busy) deserves retries, because waiting can fix it. A permanent failure (the message itself is bad: a poison pill) will fail forever, so retrying just blocks the partition. Send it to a DLT for a human to inspect. The two projects choose different retry budgets for good reasons: a match summary is low-value and the bridge gives up after about 7 seconds; a flag is a possible cheater, so the Judge never gives up during an outage. ResilienceIT proves it: during a 20-second injected outage no flag was dead-lettered, and all 20 were judged afterwards.

Trade-off: blocking retries cause head-of-line blocking

While a record is being retried, the rest of its partition waits behind it. That's correct when order matters and the cause is a shared outage (everything would fail anyway). When failures are per-record, Spring Kafka's non-blocking retries (@RetryableTopic, with retry topics and delays) let the partition keep flowing, at the cost of ordering. Another subtle point: the DLT recoverer writes to the same partition number as the source record, so the DLT needs at least as many partitions, which is why they match.

12. Kafka vs alternatives

NeedGood fit
High-throughput event streams, replay, many independent consumers, stream processingKafka
Task queues, per-message acknowledgement, routing, priorities, delayed messagesRabbitMQ, SQS
Small scale streams alongside an existing RedisRedis Streams
Managed, serverless pub/subGoogle Pub/Sub, AWS Kinesis, SNS

13. Interview questions

What is a partition and why does Kafka have them? Basic

A partition is an ordered, append-only log; a topic is a set of them. Partitions give parallelism (different brokers store them, different consumers read them) and are the unit of ordering: order is guaranteed only within a partition. I key every player event by player id, so one player's events land in one partition, in order, which per-player windows and sequence checks require.

How do consumer groups work? Basic

Consumers sharing a group.id split the partitions: each partition goes to exactly one member, so the group processes the topic once, in parallel. Another group gets its own complete copy with its own offsets. Parallelism is capped by the partition count. When members join or leave, the group rebalances.

What does acks=all plus idempotence give you? Intermediate

acks=all means the leader acknowledges only once all in-sync replicas have the record, so an acknowledged write survives a leader failure (given min.insync.replicas ≥ 2 and RF ≥ 3). Idempotence gives each producer an id and a sequence number per partition so the broker discards duplicate retries and keeps order. Together: no loss of acknowledged data and no duplicates from producer retries. In my setup RF is 1 on a single broker, so I'm honest that broker loss isn't covered.

At-least-once vs exactly-once: which did you use where? Intermediate

Inside Telemetry's detector, Kafka Streams with exactly_once_v2: reads, state updates, outputs and offset commits are one Kafka transaction, which I verified per player with sequence numbers. At the edges, the bridge to Arena and the Judge, it's at-least-once with idempotent effects, because HTTP calls can't join a Kafka transaction: Arena dedups matches by externalId, and the Judge skips flags already in its verdict log and re-checks player status.

What does isolation.level=read_committed do and why do your consumers need it? Deep

Transactional producers write records before committing; on abort the records stay in the log with an abort marker. A read_uncommitted consumer would read them. read_committed reads only up to the last stable offset and skips aborted records. The detector writes inside transactions, and if an instance is killed mid-transaction, its partial output is aborted and the new owner re-produces it. Without read_committed, the Judge and the verifier would see those aborted flags plus the re-emitted ones, which look like duplicates.

Retention vs compaction? Intermediate

Delete retention keeps records for a time or size and then drops whole old segments: a history. Compaction keeps at least the latest value per key and removes older ones in the background: a table. I use compaction for profiles, integrity stats, the manifest, ground truth and the Judge's verdicts, which it replays at startup as its only state. Tombstones (null values) delete keys, and because compaction lags, a replay must keep the last value per key.

How do you handle a poison message? Intermediate

Classify the exception. A poison message fails deterministically, so retrying blocks its partition forever. Spring Kafka's DefaultErrorHandler with addNotRetryableExceptions sends those straight to a dead-letter topic via DeadLetterPublishingRecoverer, with exception headers for diagnosis, and consumption continues. Transient failures get back-off retries. In Telemetry's detector I go further: raw bytes are parsed inside the topology, so a corrupt event becomes a dead-letter record with its offset and Base64 payload instead of a deserialization exception.

Why does the Judge retry forever while the bridge gives up after 3 retries? Deep

Different values and different failure causes. A flag is a possible cheater: dropping it is costly, and the failures that block it (Claude or Arena unavailable, circuit open) are outages that end, so the Judge holds the flag and retries every 5 seconds without committing the offset. Only a malformed flag or a Claude 400 goes to the DLT. A match summary that keeps failing after about 7 seconds is either a real outage or a permanent rejection; parking it in the DLT keeps the partition moving and it can be replayed. Same mechanism, different budgets.

Why set max.poll.interval.ms to 10 minutes in the Judge? Deep

If the app doesn't call poll() within that interval, Kafka assumes it's stuck, removes it from the group and rebalances, and the uncommitted record goes to another member, which may also time out. Judging one flag can involve two model investigations with tool calls, plus resilience retries, so the default 5 minutes could be exceeded. I take one record per poll and allow 10 minutes. The alternative is to pause partitions and process asynchronously.

What is consumer lag and how did you watch it? Intermediate

Lag is the log-end offset minus the group's committed offset: how many records are waiting. I graphed it in Grafana and checked it with kafka-consumer-groups.sh --describe, which is authoritative. Client metrics like records-lag-max are windowed maxima and trail reality. During the chaos test the per-partition max peaked around 380,000 and took about 2 minutes to drain, because two survivors had only about 5,000 events/s of headroom over the 25,000/s input.

Your app connects to Kafka in Docker and then hangs. What do you check? Curveball

Advertised listeners. The bootstrap connection works, but the broker then hands back its advertised address, for example kafka:19092, which the host can't resolve, or localhost:9092, which inside a container points to the container itself. Configure one listener per audience: INTERNAL://kafka:19092 for containers, EXTERNAL://localhost:9092 for the host, as Telemetry's Compose file does.

Can you just add partitions to match-events later to scale? Curveball

You can, but it remaps keys: hash mod 6 differs from hash mod 12, so a player's new events land in a different partition from their old ones. Per-key ordering breaks around the change, and stateful processing (windows keyed by player) and co-partitioned joins break, because player-profiles would still have 6. You'd plan capacity up front, or create a new topic with more partitions and migrate.

What did KRaft change? Basic

Kafka no longer needs ZooKeeper. A quorum of controller nodes stores cluster metadata in an internal log and agrees using Raft. It's one system to run, with faster controller failover and better metadata scalability. Kafka 4 is KRaft-only; my broker runs as a combined broker and controller.

14. Quick quiz

1. Kafka guarantees order:

Key by entity to keep each entity's events in one partition.

2. A group has 8 consumers for a 6-partition topic. How many are idle?

One partition per consumer at most.

3. Processing first and committing the offset afterwards gives:

A crash in between means reprocessing, so make effects idempotent.

4. A compacted topic keeps:

It behaves like a table, and is great for state.

5. Without read_committed, a consumer of transactional output may see:

Aborted records remain in the log; read_committed filters them.

6. Which failure should go straight to a DLT without retries?

Deterministic failures are poison pills.

7. The idempotent producer prevents:

Per-session sequence numbers let the broker drop retried batches.

15. Cheat sheet

  • Kafka = a distributed, partitioned, replicated append-only log; readers keep offsets; replay is possible.
  • Partitions: parallelism + per-partition order; key → murmur2 mod n; max consumers = partitions; adding partitions remaps keys.
  • Producer: batching (batch.size, linger.ms), lz4, acks=all, idempotence (producer id + sequence: no retry duplicates).
  • Replication: leader + followers, ISR, min.insync.replicas. Yours: 1 broker, RF 1 (say it).
  • KRaft: Raft controller quorum, no ZooKeeper (Kafka 4). Advertised listeners: INTERNAL kafka:19092, EXTERNAL localhost:9092.
  • Groups: a partition per member; offsets in __consumer_offsets; earliest; heartbeats + session.timeout.ms (6 s in the detectors); max.poll.interval.ms (10 min in the Judge); rebalances; lag.
  • Semantics: commit-then-process = at-most-once; process-then-commit = at-least-once (+ idempotency); a transaction = exactly-once (Kafka only).
  • Retention (history) vs compaction (latest per key; tombstones; keep last on replay). judge-verdicts = the Judge's state, replayed before the listener starts.
  • Transactions: atomic multi-partition writes + offsets; consumers need read_committed.
  • Spring Kafka: KafkaTemplate (future), @KafkaListener (concurrency, idIsGroup=false, autoStartup), DefaultErrorHandler + BackOff + not-retryable + DeadLetterPublishingRecoverer (same partition).
  • Retry budgets: bridge 1/2/4 s then DLT; Judge every 5 s forever on outages, DLT only for poison.
  • Next: Avro + Schema Registry; RF 3 cluster.
Saved in this browser.