logs are the source of truth

how append-only history became the foundation of modern systems

I started writing this piece as an article on Kafka, not Franz Kafka, but Apache Kafka.

But the more I tried to think about Kafka, the more I kept running into the same idea, over and over again: logs.

Not as debug output or something you grep when things break.
But as the thing everything else quietly depends on.

That realization made me pause the Kafka article entirely. What started as an attempt to understand a system slowly turned into an attempt to explain a pattern.

So instead of writing about Kafka, I’m writing about logs.

How we “normally” treat logs

“make sure you add informative logs…”

That’s what we are used to listening to, at least I listened initially in my career. After all, logs are the things we look up to at the first sign of trouble.

And that’s what they have always been treated as, a thing to look at only when things break.

They are rotated/deleted periodically, to not consume memory, as they are treated as “expendables”.

A log

logs as the foundation of persistence

long before Kafka, or distributed systems existed, databases already used logs as one of the key components to maintain the persistence of the data.

write-ahead log (database as a log)

Every single write/update operation in a database is first written to a separate WAL file as a log. That file is the source of truth for the entire state of the database.

Everything, from tables, indexes, constraints, and your actual data, is written to a file as a log line. And the tables/indexes are just a materialized view of the same logs.

Different database states at different points in time

Logs are what protect the data from getting wiped out in cases of database crashes. These are replayed to have the database maintain its last stable state, ensuring the data is what we stored.

And replication is just shipping the same log across different machines, because logs are the state.

The beautiful nature of logs is that they are append-only; we get a point-in-time snapshot of the database at a single point in time, across histories. They exist from the start of the system, they are the history of the system, and when we look at these logs, we do a time travel across the states of the system.

cdc-events (database operations as a log)

Just as logs are used to create tables, indexes, and the actual data inside, the inverse is also true. Any update/delete can be turned into a log, and then an event can be transmitted according to the type of event. This is what we call change data capture (CDC).

consider the example -

We want to perform an action that whenever a new user entry is inserted in our users table, we want to send it a confirmation email and other offers.

Even if you’ve never worked with event-driven systems or Kafka before, the most logical way we can think of is to maybe get this insertion data from the database and act accordingly. And one of the ways to get that is to trigger the insertion log whenever a new write happens.

CDC event triggered on insertion

So, logs can not only be the source of truths, but they can also be used to convey the change in the source of truth elsewhere efficiently.

source control (git as a log)

If we look deeper into what git actually does, we will realize that this “logs as state” is deeper than we realize from above.

git stores “changes”, and we can go to any particular revision of the file, even the entire repo, and take ONLY the exact pieces required by us.

I’ve written a more technical explanation to understand how git does it, but the underlying mechanism remains the same.

Inside Git’s Brain: How Commits Are Stored and Linked

The whole source-control system is based on logs.

You changed a file at line 308, that’s a log written by git.

git translates those change logs into a structure that makes us mutate states according to our needs. If we think about it, git is like any other database, just for storing the files, states, and changes. and like any other database, it relies on logs for persistence and state tracking.

distributed consensus (agreement as a log)

When we think about consensus protocols, we often think it’s about asking “what is the current value?”, which is often misleading.

When multiple machines are involved, it becomes more of “what history do they agree on?” and the current value becomes something that is derived by replaying this history.

This sequence of history, well… you guessed it, is the log.

Consensus protocols don’t replicate states directly -

1. the leader appends an entry to its log

2. followers replicate that log entry

3. once a quorum agrees, the entry is committed

4. each node applies the log entry to its local state machine

The state is rebuilt by replaying the log.

If a node crashes, it asks for the missing log entries till that point, and the state can be derived from it.

leader election

Once we agree that logs become the truth of the system, the next question is “who should be allowed to write the next entry?”

Leader election is about answering that very question, that only one should have the authority to do so.

why do we need a leader?

A distributed log can’t tolerate ambiguity, since it’s the very mechanism of being true to the system.

If we allow multiple nodes to append to the same log independently, the histories would diverge, and we would lose the deterministic nature of the system.

at any moment, only one node may decide what the next log entry is.

This node is called the leader.

Who’s the leader?

Based on the above distributed system and its logs, who do you think could be the leader?

voting for the leader

When I first read about it, I thought leader elections are about picking a healthy node - one that’s fast, responsive, and alive.

But in log-based consensus systems, nodes don’t vote for nodes; they vote for histories.

Whenever a candidate competes for being the leader, it sends its history as a basis on which others decide.

If the history contains everything that a node agrees upon (and has the same or higher term than itself), then the candidate gets the vote.

Let’s pick the above system, and try to see who gets whose votes.

candidate  |  votes_from
_______________________________________________________________________________________
node 1     |  node 2 (seq 27 >= seq 26)
node 2     |  NONE (it's seq 26 is the lowest among everyone else's)
node 3     |  node 1 (28 >= 27), node 2 (28 >= 26), node 4 (28 >= 28)
node 4     |  node 1 (28 >= 27), node 2 (28 >= 26), node 3 (28 >= 28) 

Now, both Node3 and Node4 get the majority (if their terms are the same), and we can’t have multiple leaders. RAFT conducts the election again, with a new term.

Voting for the wrong leader would be catastrophic for the system.

If a node that’s missing committed log entries were elected leader, it could:

• overwrite acknowledged history

• erase events that other nodes already accepted

• violate the system’s guarantees silently

Now, if we look vaguely at leader election, we might think of it as a mere coordination detail among distributed systems.

But now, knowing how logs play a larger role, there’s a deeper insight: the leader election is about protecting the log, because that is the source of truth that keeps the entire system running.

apache kafka (everything is a log)

Now that we have established that logs are more than we expect them to be. We are ready for a system like Kafka, which makes logs everything…

In the previous article, I explored the intricacies of a message queue.

Designing Message Queues from First Principles

This won’t be a detailed write-up about Kafka and its internals; I want to highlight the core idea of why logs become so powerful in Kafka.

I highly encourage anyone unfamiliar with Kafka and who wants to understand the basics, read - How Kafka Works, and also Why was Apache Kafka Created?

They are an excellent starting point for anyone who wants to understand Kafka.

to it’s bare bone, kafka is nothing but a database that stores data as a log - that’s it.

kafka and the multiple consumer problem

Storing logs is the easy part. The harder question is: who gets to read them?

If logs are the source of truth, then consumers aren’t just “workers”. They become the readers of history.

And the moment we have more than one consumer, an uncomfortable problem appears.

If multiple consumers read from the same log, we get a lot of questions -

• who reads which part?

• what happens when one consumer is slow, or if it dies?

• how do we make sure history isn’t processed twice or skipped?

Traditionally, message queues answer this by moving messages. Once a message is consumed, it disappears - the work is done.

Kafka refuses to do that. It keeps history as it is and instead tracks where each consumer is in that history.

offsets

Kafka doesn’t depend on consumers to tell if they have consumed the message.

Instead, it asks, “Where are you in the log?”

Consumers don’t delete data. They commit offsets, a kind of marker that implies “it has read till that mark”.

This main idea makes the consumption replayable and independent of time.

If you want, you can just start from the beginning of the messages.

Consumers can crash, disappear for hours, come back, and continue exactly where they left off.

Consumer Offset - Credits

the partition constraint

Kafka strictly refuses to violate ordering. because a log only makes sense if its order is preserved - only when the logs are sequential.

Kafka solves this by splitting a topic into partitions - independent logs with their own ordering.

and here’s the rule Kafka enforces strictly -

Within a consumer group, a partition can belong to only one consumer at a time.

Because splitting an ordered log between consumers would be inconsistent.

How would you ensure that histories are preserved across multiple consumers reading different partitions?

This is the reason why if you have more consumers than the number of partitions listening to a topic, then the extra consumers remain idle.

Kafka would rather leave consumers idle than violate ordering.

Multiple Consumers - Credits

even the progress itself is a log

The beautiful thing is that Kafka stores consumer offsets itself as logs.

Kafka uses a log to track how you're reading a log.

There’s an internal topic __consumer_offsets where every offset commit is written as an event.

Kafka tracks progress as history. even the coordination among brokers is append-only. And if any of the broker crashes, since the states are conserved by the log, they can replay them on restarts.

Kafka isn’t just a queue. Kafka isn’t just a message bus. Kafka isn’t just trying to “deliver messages”.

Kafka is maintaining an ordered history, and everything else, the consumption, parallelism, and recovery, is layered on top of that log.

One key thing to note here is the technical nature of append-only (sequential) writes.

Disk seeks are expensive; sequential writes are cheap. By treating everything as a log, Kafka turns a slow mechanical or SSD-based storage layer into something that rivals RAM speeds because it avoids the "random access" tax.

logs as the boundary between truth and state

By now, this pattern should be hard to ignore.

Databases, Kafka, Git, consensus protocols, very different systems, they all converge on the same idea: truth lives in a log.

State can be partially written, corrupted, or lost.
Logs are append-only. They preserve order, time, and causality.

Kafka didn’t invent this idea. Neither did Databases. Consensus protocols didn’t either.

They all arrived at it independently.

Because when systems are forced to survive crashes, replays, and disagreements, logs are the only structure that stays honest.

a message

I’ve been thinking a lot about how writers monetize their content in different ways, including paywalls. That’s not what I want for this space.

Everything I write here will stay free; that’s important to me. At the same time, writing takes time, solitude, and energy.

If you have ever felt that my work helped you in some way, and you want to support it, you can do that here —

posts worth reading

• This article by Jay Kreps, back in 2013, might be one of the most amazing articles I’ve read in a while. It scratches the surface and goes down into the rabbit-hole of logs as a fundamental aspect of real-time data - The Log: What Every Software Engineer Should Know About Real-time Data’s Unifying Abstraction

• An excellent piece for anyone starting on Kafka - How Kafka Works

• The “why” behind Kafka - Why was Kafka Created