Class: Wurk::Swarm::Restart

Inherits:
Object
  • Object
show all
Defined in:
lib/wurk/swarm/restart.rb

Overview

Non-blocking rolling-restart / recycle state machine. One slot is in flight at a time; advance moves it a single phase per supervise tick so the supervisor never blocks on a restart and keeps honoring TERM:

spawn_replacement → await_heartbeat(heartbeat_wait) → term_old
→ await_exit(drain_timeout) → done

The reaper reports child exits via claim_exit, so a replacement that dies before it heartbeats is seen as dead (not "slow"): the old child is kept, a per-slot backoff applied, and the slot retried. If the old child is gone too, nothing is left holding the slot: it is handed to the swarm's crash-respawn backoff (respawn) instead of being requeued behind a dead pid, which next_ready_pid would drop. abort drops everything — the swarm's TERM handler then drains the in-flight replacement + old as ordinary children.

Collaborators are injected (Config) so the machine is decoupled from the swarm's fork/kill/Redis internals and unit-testable against fakes.

Defined Under Namespace

Classes: Config

Instance Method Summary collapse

Constructor Details

#initialize(config) ⇒ Restart

rubocop:disable Metrics/AbcSize



27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
 
# File 'lib/wurk/swarm/restart.rb', line 27

def initialize(config) # rubocop:disable Metrics/AbcSize
  @spawn = config.spawn            # ->(slot, idx) => replacement pid (registered by the swarm)
  @kill = config.kill              # ->(pid, sig)
  @heartbeat = config.heartbeat    # ->(pid) => truthy once the child has beaten
  @describe = config.describe      # ->(pid) => { slot:, index: } | nil
  @respawn = config.respawn        # ->(idx) => delay; arms the swarm's crash respawn for the slot
  @now = config.now                # -> monotonic seconds
  @logger = config.logger
  @heartbeat_wait = config.heartbeat_wait
  @drain_timeout = config.drain_timeout
  @backoff = config.backoff
  @restarted = config.restarted    # ->(reason) once per replacement forked; optional
  @queue = []
  @reasons = {}
  @current = nil
end

Instance Method Details

#abort ⇒ Object

TERM/INT: forget queued + in-flight work. The replacement and old child are ordinary children the swarm's shutdown TERMs and drains.



90
91
92
93
94
 
# File 'lib/wurk/swarm/restart.rb', line 90

def abort
  @queue.clear
  @reasons.clear
  @current = nil
end

#advance ⇒ Object 



78
79
80
81
82
83
84
85
86
 
# File 'lib/wurk/swarm/restart.rb', line 78

def advance
  start_next if @current.nil?
  return if @current.nil?

  case @current[:phase]
  when :await_heartbeat then advance_await_heartbeat
  when :await_exit then advance_await_exit
  end
end

#claim_exit(pid) ⇒ Object

Reaper hook. Returns true when pid belonged to the in-flight restart so the swarm skips crash-respawn for it. A replacement is only claimed while awaiting its heartbeat — once it takes over the slot (await_exit), its death is an ordinary crash the swarm respawns.



64
65
66
67
68
69
70
71
72
73
74
75
76
 
# File 'lib/wurk/swarm/restart.rb', line 64

def claim_exit(pid) # rubocop:disable Naming/PredicateMethod
  return false unless @current

  if pid == @current[:old_pid]
    @current[:old_exited] = true
    true
  elsif pid == @current[:replacement] && @current[:phase] == :await_heartbeat
    @current[:replacement_dead] = true
    true
  else
    false
  end
end

#enqueue(pids, reason = :rolling) ⇒ Object

Queue live child PIDs for restart, skipping any already queued or in flight so recycle + rolling restart can't double up on one slot. reason (:rolling, :recycle) is what restarted reports for the slot.



51
52
53
54
55
56
57
58
 
# File 'lib/wurk/swarm/restart.rb', line 51

def enqueue(pids, reason = :rolling)
  pids.each do |pid|
    next if in_flight?(pid) || @queue.include?(pid)

    @queue << pid
    @reasons[pid] = reason
  end
end

#idle? ⇒ Boolean

Returns:

  • (Boolean) 


44
45
46
 
# File 'lib/wurk/swarm/restart.rb', line 44

def idle?
  @current.nil? && @queue.empty?
end