A real concurrency cap: a worker pool and a circuit breaker

I wanted to process a list of items “a few at a time”. I reached for the tool everyone reaches for, mapped every item to a promise and handed the lot to Promise.all, and told myself I had a limit.

I did not have a limit. I had the opposite of a limit.

Promise.all is not a cap#

Here is the misunderstanding, plainly. When you map an array to promises, the work starts the moment each promise is created. Promise.all does not throttle anything, it just waits for all of them. So mapping a thousand items launches a thousand operations at once, and then you wait.

If those operations are heavy, you have just fired every one of them simultaneously at a machine that can handle a handful. That is not concurrency control, that is a stampede with a nice-looking await in front of it. On a real server that stampede is the difference between chewing through a queue steadily and toppling over under load you brought on yourself, taking every other user’s request down with it.

For a hard ceiling you need a fixed pool. You spin up min(maxParallel, N) workers, and each worker pulls the next item from a shared index cursor, does the work, writes the result into a pre-sized array at that item’s slot, and pulls again. When the cursor runs out, the workers stop.

async function runPool(items, maxParallel, task) {
  const results = new Array(items.length);
  let cursor = 0;
  async function worker() {
    while (cursor < items.length) {
      const i = cursor++;
      results[i] = await task(items[i], i);
    }
  }
  const size = Math.min(maxParallel, items.length);
  await Promise.all(Array.from({ length: size }, worker));
  return results;
}

There are never more than maxParallel operations in flight, results land in order because each slot is fixed, and the ceiling is a real ceiling.

A scheduler is a circuit breaker underneath#

The second lesson came from a background scheduler that polled a lot of external sources, each with its own rate limit. My first version just looped and synced everything on a timer. It hammered slow sources and kept calling ones that were already broken.

What I actually needed was three separate mechanisms bundled together under one word, “scheduler”.

A concurrency semaphore, so only so many syncs run at once, the pool idea again. Per source exponential backoff, so a source that just failed waits longer before its next attempt instead of being retried immediately. And a failure counter, so after N consecutive failures I flip that source into a persisted error state, with the reason recorded, and stop calling it entirely until something changes.

That last part is the circuit breaker. A connection that has failed ten times in a row is not going to succeed on the eleventh, and every call you make to it is wasted budget against a rate limit you need for the sources that actually work.

A retry loop with no memory is just a way to fail faster and more expensively.

Where the simple choice runs out#

One more thing I want to be straight about. My scheduler runs in memory, in a single process. State lives in RAM, and if the process dies, the backoff counters reset.

That is a deliberate early-stage choice, not an oversight. For the current scale it is simple and it works. But I know exactly where it breaks: the day I need more than one instance, or need the state to survive a restart, the in-memory scheduler moves to a durable queue. I have written that migration point down so future me does not discover it by surprise.

The takeaway across both lessons is the same. Promise.all and a naive loop feel like control and are not. A fixed pool gives you a real cap. A semaphore plus backoff plus a persisted error state gives you a scheduler that stops digging when it is in a hole. And knowing where your simple choice runs out is part of making the simple choice honestly.