Two different models
diagram
| Queue | Stream | |
|---|---|---|
| Model | Work distribution, competing consumers | Append-only log, many independent readers |
| After consumption | Message removed | Retained (days, or forever with compaction) |
| Ordering | Best effort (FIFO queues exist) | Guaranteed per partition |
| Replay | No | Yes, reset the offset |
| Typical use | Background jobs, emails, bulk tasks | Event backbone, CDC, analytics, event sourcing |
Semantics to know
- At-most-once / at-least-once / exactly-once. In practice, design for at-least-once + idempotent consumers.
- Kafka ordering: only within a partition, so choose the partition key (e.g.
entryId) to order what matters. - Backpressure and DLQs: poison messages go to a dead-letter queue after N attempts.
- Visibility timeout (SQS) / ack deadline: set it longer than the worst-case processing time, or you'll get duplicates.
Architect's tip
Rule of thumb: "do this job once" → queue. "This happened, and several systems care now or later" → stream.
Cheatsheet
The whole topic on one page. Click to open full screen.
Sources & further learning
Videos, courses, docs and books I recommend for this topic.
Apache Kafka — IntroductionDocs · Apache Kafka · kafka.apache.orgRabbitMQ TutorialsDocs · RabbitMQ · rabbitmq.comThe Log: What every software engineer should know about real-time dataArticle · Jay Kreps (LinkedIn Engineering) · engineering.linkedin.comKafka 101 (free course)Course · Confluent Developer · developer.confluent.ioKafka vs RabbitMQ — video searchVideo · YouTube · youtube.com
Related topics
Architecture PatternsApplied in production
Event-Driven Architecture
Services announce facts ("EntryPublished") and others react asynchronously — decoupling producers from consumers in time, space and knowledge.
Intermediate5 sources
Architecture PatternsUnderstood
Transactional Outbox
Write the business change and the outgoing event in the same local transaction, then relay the event to the broker — no more "saved to DB but message lost".
Advanced4 sources
System DesignUnderstood
Design: Notification System
Send email, SMS, push and webhooks at scale — fan-out, user preferences, retries, dedupe and provider failover.
Intermediate3 sources