CCDAK Cheat Sheet: Kafka Developer Decisions
Review Kafka ordering, offsets, transactions, schema evolution, Streams state and Connect delivery boundaries before your next practice session.
Use this reference to recall a distinction after studying it. Confirm version-sensitive settings in the documentation for the client and runtime you use.
Records, keys and retention
| Decision | Boundary to remember |
|---|---|
| Preserve entity order | Kafka orders records within a partition. Stable keys and routing help; concurrent side effects can still finish out of order. |
| Increase partitions | Existing records stay in their partitions. A partition-count-dependent key mapping can change for later records. |
| Reconstruct current state | Compaction retains useful latest keyed state; it is not a complete audit history. |
| Preserve every business change | Retain an event history long enough for the required replay window. Time and size limits both matter. |
| Recover after missing tombstones | A stale local view may retain deleted entities. Rebuild from an appropriate complete retained source rather than assuming offset validity proves state completeness. |
Producer outcomes
send() returns a future for an asynchronous result. Successful preparation or buffer acceptance does not establish a broker acknowledgment. Inspect the completed result and effective acknowledgment settings.
- Idempotence: protects against supported producer-retry duplicates; it does not deduplicate a new application send by business identity.
- Transactions: can combine Kafka output and consumer-offset progress. HTTP requests and database effects need their own recovery contract.
- Transactional identity: use a distinct stable identity for each logical producer when restart fencing is required.
- Timeouts: for Kafka 4.0, effective
delivery.timeout.msmust be at leastrequest.timeout.ms + linger.ms. Explicit examples should account for all three values.
Consumer positions and effects
A committed offset identifies the next position to read. After completing offset 41, commit 42 when that progress is safe.
| Order of operations | Crash risk |
|---|---|
| Commit, then apply an external effect | The record can be skipped if the process fails between those operations. |
| Apply the effect, then commit | The effect can repeat if the process fails before the commit. |
| Apply with a stable external idempotency key, then commit | Replays can be safe when the destination preserves that key and its documented semantics. |
seek() changes a consumer’s current position; it does not itself commit group progress. A valid existing commit normally takes precedence over auto.offset.reset. Bound a replay by explicit starts and exclusive ends. Timestamp lookups require attention to out-of-order timestamps and retained history.
Schemas and runtime decoding
- A serializer encodes values; a deserializer decodes them. Kafka brokers store bytes.
- Backward compatibility: a new reader can read older data under the specified schema policy.
- Forward compatibility: an older reader can read newer data.
- Transitive checks: include more schema history than a check against only the latest registered version.
- Registration compatibility does not prove unchanged business meaning, automatic runtime defaults, or correct event routing.
- Confluent Avro framing carries a schema ID. Use the writer schema for that record rather than assuming the latest schema describes every stored payload.
Streams and Connect
| Mechanism | Developer implication |
|---|---|
mapValues | Changes the value while preserving the key; configure an appropriate output Serde when the value type changes. |
| KStream–KTable join | A stream event consults table state; later table changes do not inherently re-emit all previous stream events. |
| Logged state store | A replacement task can restore state from its changelog when local storage is lost. Preserve application identity and required Kafka history. |
| Window and grace | Use event timestamps and observed stream time. Wall-clock waiting alone does not establish that a stream-time window has closed. |
| Connect converters | Changing a converter does not rewrite historical bytes. A homogeneous migration needs a defined mapping, backfill and resume boundary. |
tasks.max | A ceiling, not a promise that every connector creates that many tasks or can use that much parallelism. |
| Single-record transforms | Reshape individual records; use an appropriate stateful processing stage for cross-record joins. |
Test the actual guarantee
Use unit tests for local decisions, topology tests for deterministic transformations and state, and broker-backed integration tests for membership, transaction visibility and recovery. TopologyTestDriver does not reproduce production cache coalescing or rebalance timing.
Zero lag demonstrates caught-up consumer progress at a measured boundary. To prove complete external effects, reconcile source identities against durable destination records and identify missing or repeated effects.