Class: Wurk::Swarm::CrashWatch

Inherits:
Object
  • Object
show all
Defined in:
lib/wurk/swarm/crash_watch.rb

Overview

Decides when a fleet has stopped being worth respawning. Pure bookkeeping, clock passed in, like Backoff. Backoff alone retries forever, and a parent that respawns forever stays Running while processing nothing: orchestrators and process-grep probes see a healthy pod.

A slot's streak starts at its first young crash and ends when a child outlives young_within. The exit is due once EVERY slot has been in an unbroken streak for exit_after: one bad slot beside healthy ones is degraded, not dead, and stays the respawn loop's problem.

A crash excused: by the caller (Redis unreachable) ends the streak too, because the release is not shown to be broken and the swarm's job during an outage is to be there when Redis returns.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(exit_after:, young_within:) ⇒ CrashWatch

exit_after nil disables the exit; streak counting still feeds the logs.



22
23
24
25
26
27
 
# File 'lib/wurk/swarm/crash_watch.rb', line 22

def initialize(exit_after:, young_within:)
  @exit_after = exit_after
  @young_within = young_within
  @since = {}
  @streak = Hash.new(0)
end

Instance Attribute Details

#exit_after ⇒ Object (readonly)

Returns the value of attribute exit_after.



19
20
21
 
# File 'lib/wurk/swarm/crash_watch.rb', line 19

def exit_after
  @exit_after
end

Instance Method Details

#due?(slots, now) ⇒ Boolean

Returns:

  • (Boolean) 


49
50
51
52
53
54
 
# File 'lib/wurk/swarm/crash_watch.rb', line 49

def due?(slots, now)
  return false unless @exit_after
  return false if slots.empty?

  slots.all? { |slot| (at = @since[slot]) && now - at >= @exit_after }
end

#max_streak ⇒ Object

Longest unbroken streak across slots; 0 when none is crashing.



41
42
43
 
# File 'lib/wurk/swarm/crash_watch.rb', line 41

def max_streak
  @streak.values.max || 0
end

#rearm(now) ⇒ Object

Restart the clock, so a co-hosted swarm that keeps running reports once per window rather than once per tick.



58
59
60
 
# File 'lib/wurk/swarm/crash_watch.rb', line 58

def rearm(now)
  @since.transform_values! { now }
end

#record(slot, lifetime:, now:, excused: false) ⇒ Object

Returns the slot's streak length after this crash.



30
31
32
33
34
35
36
37
38
 
# File 'lib/wurk/swarm/crash_watch.rb', line 30

def record(slot, lifetime:, now:, excused: false)
  if excused || lifetime >= @young_within
    clear(slot)
    return excused ? 0 : 1
  end

  @since[slot] ||= now
  @streak[slot] += 1
end

#settled(slot, lifetime) ⇒ Object 



45
46
47
 
# File 'lib/wurk/swarm/crash_watch.rb', line 45

def settled(slot, lifetime)
  clear(slot) if lifetime >= @young_within
end