Kafka + Delivery Semantics

Exactly-once vs at-least-once delivery in Kafka

At-least-once means messages are not lost but may be processed more than once. Exactly-once means the final processing effect happens once, which usually requires transactions, idempotency, or careful system design.

KafkaDistributed SystemsReliabilityIdempotency

The Short Answer

At-least-once means the message should not be lost, but it may be processed more than once.

Exactly-once means the final processing effect should happen once, even if retries, crashes, or rebalances happen.

In interviews, the key phrase is: exactly-once is usually about the processing result, not magic duplicate-free delivery across every external system.

The Real Problem

Imagine a consumer reads a Kafka message, charges a customer, and then crashes before committing the offset.

When the consumer restarts, Kafka may give it the same message again because the offset was not committed. That protects you from losing the message, but now your application may repeat the side effect.

At-least-once

Read message
↓
Process message
↓
Crash before offset commit
↓
Message is read again

Good: message is not lost. Risk: duplicate processing.

Exactly-once goal

Read message
↓
Process message
↓
Commit output + offset atomically
↓
Final effect happens once

The goal is not just delivery. The goal is one correct final state.

Why At-least-once Is Common

At-least-once is popular because it is practical and resilient. If something fails, the system can retry.

But retrying means your application must tolerate duplicates. That is why idempotency is so important in distributed systems.

java
// Example idea: ignore duplicate event IDs
if (processedEventIds.contains(event.id())) {
    return;
}

process(event);
processedEventIds.add(event.id());
💡 At-least-once plus idempotent processing is often the most practical design in real systems.

Where Duplicates Come From

  • Producer sends a record, times out, and retries.
  • Consumer processes a record but crashes before committing.
  • A rebalance causes another consumer to resume from the last committed offset.
  • An external database write succeeds, but Kafka offset commit fails.

The Critical Interview Insight

Kafka can help with exactly-once processing when the pipeline stays inside Kafka: consume from one topic, transform, produce to another topic, and commit offsets transactionally.

But if your consumer writes to an external database, sends an email, charges a credit card, or calls another service, Kafka cannot automatically make that external side effect exactly-once.

Once external side effects are involved, you usually need idempotency keys, unique constraints, deduplication tables, or transactional outbox patterns.

Mental Model: Offset Commit vs Processing

Commit after processing

Read offset 42
↓
Process event
↓
Commit offset 43

Safer against loss, but duplicates can happen if processing succeeds and the commit fails.

Commit before processing

Read offset 42
↓
Commit offset 43
↓
Process event

Avoids duplicate processing, but risks message loss if the app crashes after commit and before processing.

How to Explain This in an Interview

At-least-once means I may process a message more than once, so my consumer should be idempotent. Exactly-once means the final effect should happen once. In Kafka, exactly-once is strongest for Kafka-to-Kafka transactional pipelines. For external systems, I need extra design patterns like idempotency keys, unique constraints, or the transactional outbox pattern.

Common Follow-ups

Is exactly-once always better?

No. It adds complexity and may add overhead. Many systems use at-least-once with idempotent consumers because it is simpler and robust.

Can Kafka prevent all duplicate effects?

Not by itself when external systems are involved. Kafka can coordinate Kafka records and offsets, but your database, payment system, or email provider needs its own idempotency strategy.

What is the safest default answer?

Assume duplicates can happen. Design consumers to be idempotent unless you have a very specific exactly-once transactional design.

Final Takeaway

At-least-once protects against data loss but allows duplicates. Exactly-once is about making the final processing result happen once. In real systems, idempotency is the concept that makes this practical.