Class: Wurk::Swarm::Liveness

Inherits:
Object
  • Object
show all
Defined in:
lib/wurk/swarm/liveness.rb

Overview

Parent-side child liveness. A child can stay alive as a pid while doing nothing — a deadlocked VM, a C extension holding the GVL, a heartbeat thread that died — and the supervisor only ever reaped exits, so such a child held its slot (and its private-list jobs) until someone noticed.

Every check_interval the swarm hands over its live pids; one pipelined HGET of each child's beat field answers whether that child's heartbeat has MOVED since the last look. A child whose beat has not moved for timeout seconds gets TERM (an ordinary drain), and SIGKILL kill_after seconds later if it is still there. Its exit then takes the swarm's normal crash-respawn path. Each step is logged.

Progress is tracked on the parent's monotonic clock — "the beat value changed" — never by comparing the child's wall-clock beat to ours, so an NTP step cannot kill (or spare) a whole fleet.

Not judged:

* a child that has never beaten gets `boot_grace` on top of `timeout`
from the first look (fork → `:startup` hooks → first beat);
* anything while `paused:` (quiet, rolling restart / recycle in
flight) — the swarm is deliberately churning, and progress is
forgotten so every child gets a fresh window afterwards;
* anything after a probe failed — children cannot beat into a Redis the
parent cannot read either, so the next successful probe starts every
child over instead of killing the fleet the moment Redis comes back.

Collaborators are injected (Config) so this is unit-testable with fakes, like Swarm::Restart.

Defined Under Namespace

Classes: Config

Constant Summary collapse

DEFAULT_TIMEOUT = 
60
CHECK_INTERVAL = 
5
MIN_TIMEOUT =

Below two beat intervals a single delayed beat would read as death.

2 * Heartbeat::BEAT_PAUSE

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(config) ⇒ Liveness

Returns a new instance of Liveness.



55
56
57
58
59
60
61
62
63
64
65
66
67
68
 
# File 'lib/wurk/swarm/liveness.rb', line 55

def initialize(config)
  @beats = config.beats            # ->(pids) => { pid => beat String | nil }; raises if Redis is unreadable
  @kill = config.kill              # ->(pid, sig)
  @now = config.now                # -> monotonic seconds
  @logger = config.logger
  @timeout = config.timeout
  @kill_after = config.kill_after
  @boot_grace = config.boot_grace
  @check_interval = config.check_interval || CHECK_INTERVAL
  @progress = {}                   # pid => [last beat value, monotonic time it was last seen moving]
  @terminating = {}                # pid => { kill_at:, killed: }
  @next_check = 0
  @blind = false
end

Class Method Details

.timeout_from(value) ⇒ Object

nil/false/0 disables supervision; anything else must clear MIN_TIMEOUT.



43
44
45
46
47
48
49
50
51
52
53
 
# File 'lib/wurk/swarm/liveness.rb', line 43

def self.timeout_from(value)
  return DEFAULT_TIMEOUT if value.nil?
  return nil if value == false || value.to_s == '0'

  seconds = Float(value)
  if seconds < MIN_TIMEOUT
    raise ArgumentError, "swarm_heartbeat_timeout must be >= #{MIN_TIMEOUT}s (got #{value})"
  end

  seconds
end

Instance Method Details

#forget(pid) ⇒ Object

Reaper hook: the child is gone, whatever killed it.



84
85
86
87
88
89
 
# File 'lib/wurk/swarm/liveness.rb', line 84

def forget(pid)
  @progress.delete(pid)
  return unless @terminating.delete(pid)

  @logger.warn { "swarm: unresponsive child #{pid} exited; slot goes through normal respawn" }
end

#terminating?(pid) ⇒ Boolean

Returns:

  • (Boolean) 


91
92
93
 
# File 'lib/wurk/swarm/liveness.rb', line 91

def terminating?(pid)
  @terminating.key?(pid)
end

#tick(pids, paused:) ⇒ Object 



70
71
72
73
74
75
76
77
78
79
80
81
 
# File 'lib/wurk/swarm/liveness.rb', line 70

def tick(pids, paused:)
  now = @now.call
  escalate(now)
  if paused
    @progress.clear
    return
  end
  return if now < @next_check

  @next_check = now + @check_interval
  check(pids - @terminating.keys, now)
end