Class: Wurk::Swarm

Inherits:
Object
  • Object
show all
Includes:
Component
Defined in:
lib/wurk/swarm.rb,
lib/wurk/swarm/backoff.rb,
lib/wurk/swarm/prefork.rb,
lib/wurk/swarm/restart.rb,
lib/wurk/swarm/liveness.rb,
lib/wurk/swarm/child_boot.rb,
lib/wurk/swarm/crash_watch.rb,
lib/wurk/swarm/memory_watch.rb,
lib/wurk/swarm/orphan_guard.rb,
lib/wurk/swarm/test_mode_guard.rb,
lib/wurk/swarm/restart_counters.rb

Overview

Parent supervisor. Forks N children per the worker topology, monitors PIDs, relays signals, respawns crashed children with per-slot exponential backoff, handles rolling restart on SIGUSR1, recycles RSS-bloated children, replaces children whose heartbeat has stopped moving (Swarm::Liveness), and gives up, loudly, on a fleet that cannot stay up (Swarm::CrashWatch).

The supervise loop never sleeps on behalf of a respawn or a restart: crash backoff is tracked as per-slot due-times (Swarm::Backoff) and rolling restart / recycle run as a non-blocking state machine (Swarm::Restart) advanced one phase per tick. TERM/INT is therefore honored within a tick regardless of restart or backoff state.

Boot ordering (must be exact — see docs/idea/03-process-model.md):

1. Host app boots fully; eager loads done.
2. Railtie `after_initialize` fires.
3. `boot` closes parent-side connections (Redis, ActiveRecord).
4. `boot` forks N children.
5. Each child reconnects DB + opens a fresh Redis pool, then
 installs its own signal handlers and starts the Launcher.
6. Parent calls `supervise` to enter the wait/relay loop.

Signals (see docs/idea/04-signals.md):

TERM/INT  → `shutdown`           (graceful drain; aborts any restart)
TSTP      → relay TSTP           (quiet — stop fetching; one-way, no resume)
USR1      → `rolling_restart`    (zero-downtime cycle)

Defined Under Namespace

Modules: Prefork, TestModeGuard Classes: Backoff, ChildBoot, CrashLoop, CrashWatch, Liveness, MemoryWatch, OrphanGuard, Restart, RestartCounters

Constant Summary collapse

SUPERVISE_TICK = 
0.2
RESPAWN_BACKOFF = 
1.0
HEARTBEAT_WAIT = 
30
DEFAULT_SHUTDOWN_TIMEOUT = 
25
SUPERVISOR_POOL_SIZE =

The supervisor's own Redis pool: one connection, disjoint from every capsule pool. Post-boot the parent's only Redis use is the heartbeat probe (heartbeat_seen?) — one SISMEMBER per restart tick — and routing it through the capsule main pool reopened, at capsule size (>= 10), the very sockets step 3 had just closed; the next respawn/restart fork then inherited them. Dedicated and torn down before every fork, so no child ever sees a parent socket.

1
SUPERVISOR_POOL_NAME = 
'swarm-supervisor'
HARD_KILL_REAP_ATTEMPTS =

Poll budget for the post-SIGKILL reap sweep (250ms). Deliberately far shorter than any drain deadline — it runs after the fleet is already dead and races a kernel teardown measured in microseconds.

25
HARD_KILL_REAP_INTERVAL = 
0.01
SHUTDOWN_GRACE =

Children each hard_shutdown after their own drain deadline (bulk_requeue

  • a 3s ensure window + heartbeat cleanup); the parent must not SIGKILL them mid-tail, so its own wait always extends past theirs by this much.
5
RETERM_INTERVAL =

How often the drain re-relays TERM to whatever is still in the child table. A child forked in the instant before the drain can miss the first one outright: from Process.fork returning in the child until ChildBoot resets the inherited traps, the PARENT's handler is what a TERM finds there, and it writes the signal into the parent's self-pipe (or, once fork_child has dropped that pipe, into nothing) instead of stopping the child. The child then boots clean, drains nothing, and wait_for_children burns its whole budget before hard_kill_stragglers SIGKILLs it — taking its in-flight jobs with it. The window is the _fork hook chain (ActiveSupport's ForkTracker and friends all run child-side before Process.fork returns), so it widens with load: measured 10.5s drains against a 2.6s baseline once that chain is slowed. No child-side fix can close it — the window precedes any code the child controls — so the supervisor repeats itself instead. Re-sending is safe: by the second pass the child owns its handler, TERM to a child already draining is a no-op, and the pids come from the child table, where an exited child is a zombie until reaped and so can never be a recycled pid.

1.0
REQUIET_INTERVAL =

Quiet has the same blind spot: a TSTP relayed into that window is written to the inherited (now inert) parent trap and dropped, and the child boots fetching mid-maintenance. Once quieted, the supervisor keeps re-relaying TSTP on this cadence for the rest of its life — quiet is one-way and idempotent in the child, so repeats cost one syscall each.

RETERM_INTERVAL
SWARM_SIGNALS =

USR2 is relayed (log reopen) — without a trap, a logrotate config that signals the master pid would hit USR2's default disposition and kill the whole swarm. TTIN likewise: its default disposition stops the supervisor, so a thread-dump request is relayed to the children instead.

{ 'TERM' => :term, 'INT' => :term, 'TSTP' => :tstp, 'USR1' => :usr1, 'USR2' => :usr2,
'TTIN' => :ttin }.freeze
REDIS_UNAVAILABLE_EXIT_STATUS =

What ChildBoot exits with when Redis itself was unreachable. The parent reads it to tell an outage (keep waiting) from a broken release (give up). 75 is EX_TEMPFAIL.

75
CRASH_STREAK_ERROR_AT =

A crash streak this long logs at ERROR instead of WARN.

3

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(topology:, config: Wurk.configuration, memory_limit: config.memory_limit_kb, shutdown_timeout: DEFAULT_SHUTDOWN_TIMEOUT, heartbeat_timeout: config.swarm_heartbeat_timeout, crash_exit_after: config.swarm_crash_exit_after, exit_on_crash_loop: false) ⇒ Swarm

heartbeat_timeout: — seconds a child's heartbeat may stand still before it is replaced (Swarm::Liveness). Defaults to config.swarm_heartbeat_timeout, then 60; false or 0 turns supervision off.

exit_on_crash_loop: — the standalone runner passes true so a fleet that cannot stay up drains and lets crash_looped? decide the exit status. A swarm co-hosted in a web process only reports: an orchestrator watches the web server, and a broken worker must not take HTTP down with it.



129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
 
# File 'lib/wurk/swarm.rb', line 129

def initialize(topology:, config: Wurk.configuration, memory_limit: config.memory_limit_kb, # rubocop:disable Metrics/AbcSize
               shutdown_timeout: DEFAULT_SHUTDOWN_TIMEOUT, heartbeat_timeout: config.swarm_heartbeat_timeout,
               crash_exit_after: config.swarm_crash_exit_after, exit_on_crash_loop: false)
  @topology = topology
  @config = config
  @shutdown_timeout = shutdown_timeout
  # One reentrant lock covers the child table AND the restart machine: they
  # are mutually recursive (Restart#advance calls back into `describe` and
  # `spawn`, which read and write `@children`), so separate locks would be
  # taken in opposite orders — enqueue holds children→restart, advance holds
  # restart→children — and deadlock. Monitor's reentrancy is what lets those
  # callbacks re-enter. Never held across a sleep: `wait_for_children` and
  # the supervise tick sleep outside it.
  @lock = ::Monitor.new
  @children = {}
  @assignments = []
  @owner_pid = nil
  @stopping = false
  @shutdown_requested = false
  @quieted = false
  @requiet_at = 0
  @respawn_backoff = Backoff.new(base: RESPAWN_BACKOFF)
  @crash_watch = CrashWatch.new(exit_after: crash_exit_after, young_within: Backoff::RESET_AFTER)
  @exit_on_crash_loop = exit_on_crash_loop
  @crash_looped = false
  @restart_counters = nil
  @title_program = nil
  @restart = build_restart
  @liveness = build_liveness(heartbeat_timeout)
  @memory_watch = build_memory_watch(memory_limit)
  @orphan_reaper = Fetcher::Reaper.new(config)
  init_fork_unsafe_handles
end

Instance Attribute Details

#config ⇒ Object (readonly) Originally defined in module Component

Returns the value of attribute config.

#topology ⇒ Object (readonly)

Returns the value of attribute topology.



119
120
121
 
# File 'lib/wurk/swarm.rb', line 119

def topology
  @topology
end

Instance Method Details

#boot(install_signals: true) ⇒ Object

install_signals: is false in tests so the integration suite can drive shutdown / rolling_restart directly without poisoning the test process's signal handlers.

Traps go in BEFORE fork_children: a TERM landing in the (previously post-fork) window between fork and trap installation left the parent on its default disposition — it died instantly and orphaned live, fetching children. Installed first, the trap queues the TERM and the supervise loop drains it (relaying to children) even if it arrives mid-boot.

prepare_for_fork! stays after close_parent_sockets: it materializes the slot-independent config so every child inherits it copy-on-write, and none of what it touches opens a socket — anything that did would land back on the wrong side of step 3. settle_memory_before_fork comes last, after every parent allocation, so the compaction covers them; it opens no socket either.

Nothing that talks to Redis belongs between steps 3 and 4. #101 boot-audit: a pre-fork SCRIPT LOAD was tried here (the cache is server-global, so one upload could serve the whole fleet) and MEASURED SLOWER — bench:swarm_boot 152 -> 95 i/s, a 36.7% regression. Step 3 has just closed every parent socket, so the upload has to open its own connection, and it lands serially ahead of the first fork: ~2ms of a ~10ms boot, on every child's path. Re-measured in 2026-10 (a parent PING before step 3, warming the client code the children inherit): still slower. The children reconnect in parallel and check the cache in their own first round trip (ChildBoot#validate_redis!).

Raises:

  • (ArgumentError) 


190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
 
# File 'lib/wurk/swarm.rb', line 190

def boot(install_signals: true)
  raise 'Wurk::Swarm already booted' unless @assignments.empty?
  raise ArgumentError, 'Topology has no slots' if @topology.empty?

  @assignments = @topology.assignments.freeze
  @owner_pid = ::Process.pid
  @title_program = ::File.basename($PROGRAM_NAME) if install_signals
  install_signal_handlers if install_signals
  close_parent_sockets
  @config.prepare_for_fork!
  settle_memory_before_fork
  fork_children
  update_proctitle
  child_pids
end

#children ⇒ Object

A snapshot, not the live table. Callers read this off the supervise thread, which inserts (respawn, restart) and deletes (reap) on every tick; handing out the live Hash lets them iterate it mid-mutation.



209
210
211
 
# File 'lib/wurk/swarm.rb', line 209

def children
  @lock.synchronize { @children.dup }
end

#crash_looped? ⇒ Boolean

True once this swarm gave up on a crash loop; the caller owns the exit.

Returns:

  • (Boolean) 


257
258
259
 
# File 'lib/wurk/swarm.rb', line 257

def crash_looped?
  @crash_looped
end

#default_tag(dir = Dir.pwd) ⇒ Object Originally defined in module Component

#fire_event(event, oneshot: true, reverse: false, reraise: false) ⇒ Object Originally defined in module Component

Invokes lifecycle hooks for event. Hooks run in registration order (or LIFO when reverse: true, used for teardown). A raise in one hook is reported via handle_exception and does NOT stop the next hook unless reraise: true (used in tests / fail-fast boot). oneshot: true clears the bucket after dispatch so the event can't fire twice.

#handle_exception(ex, ctx = {}) ⇒ Object Originally defined in module Component

#hostname ⇒ Object Originally defined in module Component

#identity ⇒ Object Originally defined in module Component

#leader? ⇒ Boolean Originally defined in module Component

True iff this process currently holds the cluster dear-leader lock. Cached per Component instance for LEADER_CACHE_TTL_MS (~5s): cron and the metrics rollups call this every tick, and an uncached GET would double their Redis traffic at short intervals for no benefit — the lock's own renewal cadence (60s+, spec §6.1) easily tolerates a few-second-stale read. Returns false unconditionally when WURK_LEADER=false (or SIDEKIQ_LEADER=false) is set on the process (opt-out hot-standby). Any Redis error is swallowed → false, so a transient partition can't propagate as an exception into user code.

Spec: docs/target/sidekiq-ent.md §6.1.

Returns:

  • (Boolean)

#logger ⇒ Object Originally defined in module Component

--- delegated to config -------------------------------------------

#mono_ms ⇒ Object Originally defined in module Component

#owner? ⇒ Boolean

Only the process that forked the children may supervise or signal them. A forked child inherits this object along with the host's at_exit hooks — and rails_boot registers one that drains the swarm — so ChildBoot's exit reaches a full drain inside the child. There @children holds the child's SIBLINGS, not its own children: unguarded, that TERMs them, stalls the whole shutdown timeout waiting on PIDs it can never reap, then SIGKILLs whatever survived.

Returns:

  • (Boolean) 


252
253
254
 
# File 'lib/wurk/swarm.rb', line 252

def owner?
  ::Process.pid == @owner_pid
end

#process_nonce ⇒ Object Originally defined in module Component

#quiet_swarm ⇒ Object

TSTP quiet is one-way and GLOBAL (spec §21.3): it must survive respawns and memory recycles, or a quieted-but-crashed child's replacement would resume fetching mid-maintenance. The flag makes every future fork boot already-quieted (see ChildBoot start_quiet).



265
266
267
268
269
 
# File 'lib/wurk/swarm.rb', line 265

def quiet_swarm
  @quieted = true
  @requiet_at = monotonic + REQUIET_INTERVAL
  relay_signal('TSTP')
end

#real_ms ⇒ Object Originally defined in module Component

--- clocks ---------------------------------------------------------

#redis(idempotent: false) ⇒ Object Originally defined in module Component

#request_shutdown ⇒ Object

Cross-thread drain request: raise a flag the supervise loop observes on its next tick instead of draining on the caller's thread. shutdown walks and clears the child table that the supervise thread is concurrently mutating (reap, respawn, restart), so two threads inside it race. Every caller that isn't the supervise thread — rails_boot's at_exit, which fires on the host's main thread — comes through here.



241
242
243
 
# File 'lib/wurk/swarm.rb', line 241

def request_shutdown
  @shutdown_requested = true
end

#rolling_restart ⇒ Object

SIGUSR1: queue every live child for the rolling-restart state machine, which replaces one slot at a time (spawn replacement → await its heartbeat → TERM the old child → await its drain) without blocking the supervise thread, so TERM stays responsive throughout the cycle.



275
276
277
 
# File 'lib/wurk/swarm.rb', line 275

def rolling_restart
  @lock.synchronize { @restart.enqueue(@children.keys, :rolling) }
end

#safe_thread(name, priority: nil, &block) ⇒ Object Originally defined in module Component

Spawns a named thread that runs block under watchdog(name). The parent must retain the returned Thread; otherwise GC may not, but report_on_exception is disabled so we don't double-log on death.

Priority resolution matches Sidekiq (component.rb:44-48): explicit argument, then config.thread_priority, then -1. Ruby's default of 0 buys a 100ms timeslice; each negative step halves it, so -1 keeps a CPU-heavy capsule from starving its siblings for a whole tick.

#shutdown(timeout: @shutdown_timeout) ⇒ Object 



222
223
224
225
226
227
228
229
230
231
232
233
 
# File 'lib/wurk/swarm.rb', line 222

def shutdown(timeout: @shutdown_timeout)
  return unless owner?

  @stopping = true
  @lock.synchronize { @restart.abort }
  relay_signal('TERM')
  wait_for_children(timeout + SHUTDOWN_GRACE)
  hard_kill_stragglers
  close_supervisor_pool
  close_signal_pipe
  @restart_counters&.close
end

#supervise ⇒ Object 



213
214
215
216
217
218
219
220
 
# File 'lib/wurk/swarm.rb', line 213

def supervise
  return unless owner?

  until done?
    supervise_tick
    sleep SUPERVISE_TICK
  end
end

#tid ⇒ Object Originally defined in module Component

#watchdog(last_words) ⇒ Object Originally defined in module Component

Wraps a block at a thread boundary: any unhandled exception is reported via handle_exception (so it lands in error_handlers / the log) and then re-raised. last_words is the component label included in the context.