What queue service do you use? Any suggestions based on my requirements?

[u/Jordz2203]

Im looking for a new queue for a Node/TS app running on Kubernetes. My main requirements:

• Strict FIFO per key, with only one job in flight per key. Different keys run in parallel (SQS has Message groups I think)
• Cloud-agnostic and self-hostable, so no SQS
• Handles large bursts (obviously, since its a queue)

I've used SQS FIFO message groups before and they did exactly this. Right now I'm leaning towards Kafka with a key-ordered consumer, but it seems like quite a lot of complexity when in reality I need a simple queue but for scale. I'd love to hear what others use. Pulsar? RabbitMQ? Something else?

I've looked into the popular ones like RabbitMQ, but it seemed to not have exactly what Im looking for.

[u/itgforlife]

I use BullMQ with Redis on NodeJs.

[u/SolidLunch4702 → u/itgforlife]

BullMQ can do it if you set concurrency to 1 per queue and group by key, but you're managing that grouping yourself through queue naming or job IDs. Kafka's exactly what you want for large bursts though, the partition-per-key model gives you strict ordering and the throughput's absurd. I'd say bite the bullet on the complexity, you'll outgrow Redis-based queues fast if scale's a real concern.

[u/present_absence → u/SolidLunch4702]

This is correct. I use bullmq/redis for one of my low traffic nodejs projects and managing it works perfectly fine, but we use kafka at work for high burst/high volume

[u/285418]

A queue doesn't absorb bursts by itself, it just holds the backlog while your consumers fall behind. And 'strict FIFO per key, one job in flight per key' is a hard throughput ceiling: with N active keys and jobs taking T seconds you get N/T jobs per second, and no broker choice changes that. So what are your numbers, how many distinct keys at peak, what's the burst size? Because Kafka vs Pulsar vs whatever is a rounding error next to that, and one job in flight per key probably rules out whatever you'd pick anyway if the key cardinality is low.

Also curious what RabbitMQ was actually missing for you. Streams with a single active consumer per partition covers the one-in-flight bit, so saying it 'seemed to not have exactly what I'm looking for' without naming the gap makes it hard to trust the rest of the list.

[u/Maxion → u/285418]

Also FIFO + one job in flight per key WILL cause you to have stuck jobs and now an ever growing queue.

[u/PLBjt]

The thing I'd nail down first is whether you need ordering per key or only one job in flight per key, because those lead to different designs. With Kafka the partition is the ordering unit, so one hot key stalls everything behind it on that partition, and a consumer group can't go wider than the partition count. With BullMQ you get per-group concurrency, but you're the one writing the lock logic and it gets fiddly when a worker dies mid-job. A quick check is to replay a burst where one key carries 40 percent of the traffic and watch p99 latency on the other keys. The tradeoff is Kafka's ops cost versus a Redis footprint that grows with payload size.

[u/InevitableFun1272]

One thing that fixes the exact OOM problem you described: stop putting payloads in the queue. Store the payload in Postgres or S3 and pass only an ID through the job. Redis then holds tiny messages, and a 2.5M-item spike doesn't move the memory needle at all — the queue becomes a notification channel, not storage. You lose some convenience versus passing the payload straight through, but for that burst pattern it's the difference between a stable queue and one that eats your events.

[u/Mean-Lingonberry4874]

Kafka will do the per-key ordering, but if the ops cost is the real objection I'd try Postgres before standing up a cluster. Jobs table with a group_key, workers claim rows with SELECT ... FOR UPDATE SKIP LOCKED, and each worker takes a transaction-scoped advisory lock on hashtext(group_key) for the duration of the job. Same key can't be in flight twice, other keys run in parallel, nothing cloud-specific. It won't eat a Kafka-sized burst, but for "one in flight per key" on k8s it's usually enough, and you can peel the hot keys off later if the table starts lagging.

[u/Noch_ein_Kamel]

Just out of curiosity b/c I have no idea; how large are large bursts?

[u/Jordz2203 → u/Noch_ein_Kamel]

On average maybe 20,000-30,000 items added to the queue an hour, then at certain points of the month we might have 2,500,000 added in an hour.

[u/AdorableZeppelin → u/Jordz2203]

Is the burst predictable?

[u/Jordz2203 → u/AdorableZeppelin]

Not really, we can usually guess the times, but days are inconsistent because it depends when our customers do their operational processes. For example one company is a credit provider, and their batch uploads can sometimes be delayed due to weekends, public/bank holidays, or just any other reason.

[u/Maxion → u/Jordz2203]

Curious, why can you only have one job in process at a time per key? IMO that's going to lead to a world of hurt.

[u/Jordz2203 → u/Maxion]

This particular queue is for asynchronous updates to insurance policies. So we cant allow two to be run at the same time, because then one of them will be executing with outdated information on that policy.

[u/ArielCoding]

Before choosing a broker, work out how many different keys are active at the same time during a burst, and how long each job takes, since only one job per key can run at once your limit is (numbers of keys / seconds per job), so if you only have a few keys, you’ll hit a wall no matter which tool you pick.

[u/_CuriousCoder_]

I've used BullMQ with Redis on NodeJs, it handles bursts well. For strict FIFO per key, you'd need to implement lock logic. Consider storing payloads outside the queue, like in Postgres, to reduce memory usage.

[u/Used-Alternative-01]

if you're already on postgres i'd look at that before kafka. a jobs table with a key column and SELECT ... FOR UPDATE SKIP LOCKED, where you only pick the oldest job for keys that don't already have one running, gets you strict per key FIFO with parallelism across keys. kafka works but the one in flight per key part ends up being your consumer code anyway, plus you're running a cluster on k8s. NATS JetStream is also worth a look if you want something lighter that you can self host.

[u/TekiDeki]

Two things worth flagging, plus two options nobody mentioned yet.

JetStream is not a drop-in for SQS message groups. Ordering there is per subject, consumers filter by subject, so per-key serialization means subject-per-key plus a consumer per key, or hashing keys into a fixed number of buckets and consuming from those. With dynamic keys you end up owning the routing layer yourself, and unique subjects are not free memory-wise. Unless something landed very recently, there is no server-side "assign this key to this consumer" feature in JetStream. The disk-persistence part above is accurate though, and that is the real difference from Redis.

If the only objection to Kafka is operational weight: Redpanda is Kafka API-compatible, single binary, no JVM and no ZooKeeper, and it runs fine on k8s. Cheaper way to find out whether partition-per-key actually fits your traffic than standing up a real Kafka cluster.

If you are open to something that is not a broker at all: Temporal gives you this requirement as a primitive. The server enforces at most one workflow execution per Workflow ID at a time, so key = Workflow ID, work arrives as signals, and a running execution processes them serially while other keys run in parallel. There is a TS SDK, it is self-hostable and cloud-agnostic, and the backlog behind a busy key is handled for you. Honest caveat: self-hosting it is its own ops commitment, so it is the boring-correctness option, not the lightweight one.

Whichever you pick, design the lease. One in flight per key without an expiring lock means a single evicted pod, and node drains and preemption on k8s are routine, leaves that key blocked while its backlog grows. You want lease renewal with a heartbeat, redelivery on expiry, and idempotent consumers, because at-least-once is what you actually get from any of these.

[u/farzad_meow]

smoothmq or bullmq. as long as you do one runner, every queue will act as a fifo. why not use a db table to do it( very similar to bullmq)

[u/wasay-inc]

Look into BullMQ running on Redis on Kubernetes since it is native to Node and TypeScript You can use job concurrency settings and key locks to guarantee single job in flight processing per key It handles large bursts effortlessly and avoids the massive operational overhead of deploying Kafka on K8s

[u/Jordz2203 → u/wasay-inc]

The company I was at previously seemed to have an infinite amount of headaches related to their usage of BullMQ. Largely to do with costs

[u/wasay-inc → u/Jordz2203]

Redis memory costs scale quickly when holding large job payloads in BullMQ. If low infrastructure cost is key look at NATS JetStream on Kubernetes because it handles key based routing with minimal memory overhead

[u/Jordz2203 → u/wasay-inc]

I will take a look at NATS, thank you! Just checked with one of my old colleagues, the issue with Bull was we had to:
A) pay for paid bull to get message groups
B) OOM bull means we lose our queue events, this was because 90% of the time our queue sat at 2%, but then something big happens triggering a ton of events and we'd spike to many millions of queue items.

[u/wasay-inc → u/Jordz2203]

That burst pattern makes complete sense. NATS JetStream solves both issues because streams persist messages directly to disk instead of forcing everything into RAM like Redis. This prevents OOM data loss during massive event spikes while keeping idle running costs near zero.

[u/okawei]

Redis