| Kafka | Interview | Staff Engineer Interview

A true story about how superficial knowledge can be expensive

I was confident. Five years working with Kafka, dozens of producers and consumers implemented, data pipelines running in production. When I received the invitation for a Staff Engineer interview at one of the country's largest fintechs, I thought: "Kafka? That's my territory."

How wrong I was.

The Interview That Changed Everything

The interviewer, a senior architect with over 15 years of experience, started with basic questions. We talked about partitions, offsets, consumer groups. Everything flowing perfectly until he asked the question:

Interviewer: "Tell me something: does Kafka guarantee message delivery?"

Me: "Yes, of course. Kafka guarantees that messages are delivered."

Interviewer: "Interesting. And how exactly does it do that?"

Silence. An awkward silence that seemed to last forever.

Me: "Well… it uses replication, right? Messages are replicated between brokers…"

Interviewer: "Right, but explain to me: if I configure my producer with acks=0, does Kafka still guarantee delivery?"

Me: "Uh… yes?"

Interviewer: "And what if all brokers fail before replication is completed?"

More silence. My confidence crumbling with each second.

Interviewer: "Let me rephrase: can you explain the different delivery semantics in Kafka and in which scenarios each one applies?"

Me: "Look, I know there are some different configurations, but I couldn't explain them in detail…"

The interview continued, but I knew I had lost. The feedback was direct: "Good operational knowledge, but lacks conceptual depth for a staff position."

It hurt. A lot.

Unveiling Kafka's Delivery Semantics

After that humiliating interview, I dove deep into studying delivery semantics. And I discovered that the answer to "Does Kafka guarantee message delivery?" is more complex than a simple "yes" or "no."

At Most Once

This is Kafka's most "relaxed" semantic:

• How it works: The producer sends the message and doesn't wait for broker confirmation

• Configuration: acks=0

• Guarantee: Messages can be lost, but never duplicated

• Ideal scenario: Non-critical logs, real-time metrics where occasional loss is acceptable

# Producer configured for "fire-and-forget"
acks=0
retries=0

At Least Once

The intermediate, more conservative semantic:

• How it works: Producer waits for confirmation and retries in case of failure

• Configuration: acks=1 or acks=all with retries enabled

• Guarantee: Messages are never lost, but can be duplicated

• Ideal scenario: Systems where duplication is tolerable, but loss is not

# Producer configured to guarantee delivery
acks=all
retries=Integer.MAX_VALUE
retry.backoff.ms=100

Exactly Once

The "Holy Grail" of messaging:

• How it works: Combines idempotent producers with transactions

• Configuration: enable.idempotence=true + acks=all + transactional control

• Guarantee: Each message is delivered exactly once

• Ideal scenario: Financial systems, critical processing where duplication is unacceptable

# Producer configured for exactly-once
acks=all
enable.idempotence=true
transactional.id=unique-transaction-id
retries=Integer.MAX_VALUE
max.in.flight.requests.per.connection=5

The Questions Every Kafka "Expert" Should Know How to Answer

Now, let me properly answer the questions that brought me down:

Does Kafka guarantee message delivery?

Answer: It depends on the configuration. Kafka offers different levels of guarantee:

• With acks=0: Does NOT guarantee delivery (fire-and-forget)

• With acks=1: Guarantees that the leader received the message

• With acks=all + min.insync.replicas > 1: Guarantees durability with high availability

The real guarantee depends on the combination of producer, broker configurations and replication strategy.

Does Kafka guarantee message ordering?

Answer: Yes, per partition:

• Within a partition: Order is strictly maintained (FIFO)

• Between partitions: No global ordering guarantee

• Important condition: For strict ordering, use max.in.flight.requests.per.connection=1

• Strategy: Use consistent partitioning keys for related data

Can Kafka lose messages?

Answer: Yes, in specific scenarios:

• Producer with acks=0 has no delivery confirmation

• Failure of all brokers before complete replication

• Consumer that commits offset before processing the message

• Messages expired by retention policy

• Prevention: Configure acks=all, adequate replication and proper offset management

Can Kafka perform retries?

Answer: Yes, at multiple levels:

In Producer:

retries=Integer.MAX_VALUE
retry.backoff.ms=100
delivery.timeout.ms=120000

In Consumer: No built-in automatic retry, requires manual implementation with patterns like Dead Letter Topics.

In Kafka Streams/Connect: Configurable retry policies with exponential backoff.

The Knowledge That Separates "Experts" from True Specialists

The difference between someone who uses Kafka and someone who truly understands Kafka lies exactly here: in the deep understanding of delivery guarantees and their implications.

Can I configure a producer? Yes.

Can I consume messages? Yes.

But did I understand the nuances of different delivery semantics? Clearly, no.

This experience taught me that operational knowledge without conceptual foundation is a time bomb in one's career. In senior and staff positions, you need to understand not only how to do something, but why to do it in a certain way and what are the implications of each choice.

Don't Make the Same Mistake

If you've made it this far, congratulations. You already know more about Kafka than I did in that fateful interview.

My advice? Don't just be a technology user. Be a specialist. Understand the fundamentals, the nuances, the implications of each configuration.

The next time someone asks "Does Kafka guarantee message delivery?", you'll know that the answer isn't a simple "yes" or "no" – it's a conversation about trade-offs, configurations, and specific guarantees.

And who knows, you won't lose an incredible opportunity like I did.