Message Queues#

Design reliable message-driven systems.

Queue Types#

RabbitMQ Patterns#

Work Queues (Competing Consumers)#

Producer → Queue → Consumer 1
                 → Consumer 2
                 → Consumer 3

• Messages distributed round-robin
• Ack on success, nack on failure (requeue or DLQ)
• Prefetch count controls concurrency

Pub/Sub (Exchange → Binding → Queue)#

• Fanout: broadcast to all queues
• Direct: route by routing key
• Topic: route by pattern (user.*, user.created)
• Headers: route by header values

Kafka Patterns#

Topics & Partitions#

• Messages within a partition are ordered
• Partitions enable parallelism
• Consumer group = one instance per partition

Producer#

await producer.send({
  topic: 'order-events',
  messages: [{ key: orderId, value: JSON.stringify(order) }],
});

Consumer#

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    await processOrder(message.value);
  },
});

Dead Letter Queues#

• Messages that can't be processed go to DLQ
• Analyze DLQ periodically for systemic issues
• DLQ messages can be replayed after fix
• Set max retry count before DLQ

Best Practices#

• Idempotent consumers (same message processed twice = safe)
• Monitor queue depth, consumer lag, error rate
• Set message TTL to prevent infinite backlog
• Use structured message schemas (Avro, Protobuf)
• Test with network failures and consumer crashes