- Not every event-driven system needs Kafka. Redis Streams gives you consumer groups, at-least-once delivery, and replay — at a fraction of the operational weight.
- The decoupling that matters most is between producers and consumers, and you can get that without a heavy broker.
- The honest tradeoff is durability: Redis AOF gives a sub-second-to-~1s loss window. Know whether your domain can tolerate that.
- Design so that swapping in Kafka later is a contained change, not a rewrite.
When a system needs to react to a stream of events, the reflex in a lot of shops is to reach straight for Apache Kafka. Kafka is excellent, and for high-volume, durability-critical pipelines it's the right answer. But it's also a real operational commitment — brokers to run, partitions to balance, a whole subsystem to keep healthy. For a meaningful class of services, that's more machine than the job needs.
I built an event-driven market-data scanner that makes the opposite choice on purpose: Redis Streams as the broker, not Kafka. It's worth walking through why, because the decision generalizes — and because "use the lighter tool" is only good advice when you can name exactly what you're giving up.
The system, briefly
The service tracks a watchlist of stocks across several independent signals — live price quotes, analyst recommendations, and company news — each refreshed on its own schedule. A set of scheduled producers pull from a market-data API and publish events. A consumer, subscribed via a consumer group, processes each event: it persists history to PostgreSQL and caches the latest value per symbol in Redis. A React dashboard reads it back.
The defining requirement isn't volume — it's decoupling. Each signal type has a totally different refresh rate and lifecycle. Price quotes update every few minutes; recommendations change twice a day; news trickles in hourly. Adding a new signal should never mean touching the existing ones.
What Redis Streams actually gives you
People underestimate Redis Streams because they think of Redis as "just a cache." But Streams is a proper append-only log with the primitives that matter for event processing:
- Consumer groups. Multiple consumers can share a stream, each event handled by exactly one member of the group — the same model that makes Kafka horizontally scalable.
- At-least-once delivery. An event stays pending until a consumer explicitly acknowledges it. A processing crash doesn't lose the event; it stays in the pending list to be inspected or replayed.
- Replay and inspection. You can see exactly which events are unacknowledged and reprocess them, which is most of what people actually want from a durable log day to day.
Because producers and the consumer are decoupled by the stream, adding a new signal is just a new producer, a new event type, and a new handler — with zero change to anything already running. That's the whole reason for the message-driven design, and it's independent of which broker sits in the middle.
The tradeoff you must say out loud: durability
This is where intellectual honesty matters. Kafka's replicated, disk-backed log is more durable than Redis. With Redis you lean on AOF (append-only file) persistence, which gives roughly a sub-second to one-second window where, in a hard crash, the most recent events could be lost.
For this system that's an acceptable trade: a missed price tick is corrected by the next scan minutes later, and news handling is idempotent (deduplicated by article id), so a replay can't create duplicates. The data self-heals. The right question is never "which broker is better" — it's "can my domain tolerate this specific loss window?" For market data refreshed on a schedule, yes. For financial transactions, no — and there you pay for Kafka.
Persistence across restarts
With AOF enabled, the streams, consumer-group offsets, pending entries, and the latest-value cache all survive a restart. In-flight events aren't lost when the broker bounces. That covers the common failure — a container restart — cleanly, and it's worth configuring deliberately rather than assuming the defaults protect you.
Design so Kafka is a swap, not a rewrite
The best way to make a lighter-tool decision safely is to make it reversible. Keep the publish/subscribe boundary thin: producers publish events through a small publisher abstraction, the consumer subscribes through a listener, and the business logic (persist, cache, dedupe) knows nothing about the broker underneath.
Get that boundary right and the day you genuinely outgrow Redis — higher volume, stricter durability, multi-region — swapping in Kafka or another replicated broker is a contained change at the edges, not a teardown. You earn the simplicity now without mortgaging the future.
"Decoupled and event-driven" does not have to mean "Kafka." Pick the broker your durability requirements demand, keep the boundary swappable, and don't run more infrastructure than the problem earns.
The interesting engineering decisions are rarely about which tool is most powerful. They're about matching the tool to the actual constraints — and being able to defend the choice when someone asks why you didn't just use Kafka.
Have a system like this to build?
This is the kind of work I take on. Tell me what you're building or rescuing.