Job queue · replay

A timeout costs one job 2 s, not the whole queue

A job that times out waits a short backoff and rejoins the back of the queue. The other jobs keep running.

Example for the replay recipe in references/motion.md · the job service is made up

fail after backoff pass queue full 3rd fail factSubmit5 jobs at once decisionAdmitqueue ≤ 8 admitted 5 stateQueue 0 / 85 / 83 / 82 / 83 / 80 / 8 J1 J3 J4 J2 J4 J5 J3 J5 J2 J4 J5 effectRuntimeout 30 s fetch run upload stateRefused429 · try later stateRetrybackoff 2 s, then 4 s J2 · try 2 of 3empty decisionResultpass or fail pass 1pass 1 · fail 1pass 5 · fail 1 stateDead letterafter 3 tries stateDone 0 jobs1 job5 jobs WORKER POOL 0 / 2 busy2 / 2 busy1 / 2 busy2 / 2 busy1 / 2 busy0 / 2 busy W1 · idle W2 · idle W1 · J1 W1 · J3 W2 · J2 1 2 3 4 5 6 7 8 9
  1. A burst of five jobs arrives at once.
  2. Admit lets all five in, because the queue holds eight (admit.ts:14).
  3. Two jobs start and three wait, because the pool has two workers (pool.ts:8).
  4. J1 uploads and passes, so it is done (worker.ts:52).
  5. The free worker takes J3 from the front, because a slot opened (pool.ts:21).
  6. J2 runs past 30 s, so it fails, waits 2 s, and rejoins the back of the queue (retry.ts:22).
  7. The rest run in order. J2 passes on its second try.
  8. All five are done. Each dot is a job; about one in ten takes the retry loop.
Fig. 1One run of five jobs, one event per step. A timeout sends J2 to the back of the queue; it never blocks J3, J4, or J5.

Sources

Illustrative: src/jobs/admit.ts:14 · src/jobs/pool.ts:8, 21 · src/jobs/worker.ts:52 · src/jobs/retry.ts:22. The service is an example, not a real codebase.