Module: Wurk::Fetcher::ManyQueues

Included in:

Reliable

Defined in:

lib/wurk/fetcher/many_queues.rb

Overview

The rest of Reliable#walk's non-blocking pass once the first queue came back empty: every remaining queue in one round trip (lib/wurk/lua/fetch_first.lua) instead of one LMOVE per queue. A sparse 100-queue capsule used to pay 100 round trips per pass to reach a job on its last queue; it now pays two. The script walks the order it is given, so strict order, the weighted shuffle and the paused filter all stay Reliable#queues_cmd's.

Why not the first queue too, for one round trip: a script pays Redis per key it is handed, before it moves anything — measured on Redis 7.0 at ~13us server time for 10 queues and ~83us for 100, against ~0.2us for a bare LMOVE. A capsule whose first queue is busy is the common shape, and folding that queue into the script made its every fetch 1.4x slower end to end on 10 queues. Leading with the LMOVE keeps that path exactly where it was and spends the script only on a pass that already missed.

Constant Summary

NO_ARGV =

[].freeze

REST_MISS_BACKOFF =

How long a script pass that found every queue empty answers for the whole fetcher. The script sends two keys per queue, per thread, per poll: 120 KB a pass at 1000 queues, which at concurrency 500 cost a process 70% of a core and Redis 26% of one while idle. A thread skipping it parks in BLMOVE on the first queue for the whole poll, unaware of anything that lands on a later queue meanwhile, so a job can wait out a poll interval (about 2s) with a thread idle. That is the same bound an empty script pass gives it. Not a config knob.

0.25