Class: Wurk::HealthProbe

Inherits:
Object
  • Object
show all
Defined in:
lib/wurk/health_probe.rb

Overview

wurk health: an exec-style liveness probe, for a pod with no HTTP port. It boots no application, so it is fast and only needs to reach Redis (REDIS_URL or --redis-url). Passes when enough processes on this host have a recent heartbeat.

A Redis it cannot reach is not this pod's fault, and restarting workers does not bring Redis back, so that passes too unless --strict.

max_job_age: (--max-job-age N, opt-in) also fails when a job on this host has been running for N seconds or more: a wedged processor keeps the heartbeat thread beating, so the beat alone cannot catch it. Set N above your longest legitimate job.

Constant Summary collapse

HEALTHY = 
0
UNHEALTHY = 
1
DEFAULT_WINDOW = 
60
CLIENT_LIMITS =

At most connect + two round trips: about 2s in all against a hung server.

{ connect_timeout: 0.5, read_timeout: 0.6, write_timeout: 0.6, reconnect_attempts: 0 }.freeze
RUN_AT_RE =

The last field of a heartbeat work entry; see Metrics::Prometheus.

/"run_at":(\d+(?:\.\d+)?)\}\z/

Instance Method Summary collapse

Constructor Details

#initialize(config, window: DEFAULT_WINDOW, min: 1, strict: false, max_job_age: nil, hostname: Component.hostname, out: $stdout) ⇒ HealthProbe

Returns a new instance of HealthProbe.



30
31
32
33
34
35
36
37
38
39
 
# File 'lib/wurk/health_probe.rb', line 30

def initialize(config, window: DEFAULT_WINDOW, min: 1, strict: false, max_job_age: nil,
               hostname: Component.hostname, out: $stdout)
  @config = config
  @window = window
  @max_job_age = max_job_age
  @min = min
  @strict = strict
  @prefix = "#{hostname}:"
  @out = out
end

Instance Method Details

#call ⇒ Object 



41
42
43
44
45
46
47
48
49
50
51
 
# File 'lib/wurk/health_probe.rb', line 41

def call
  beats, run_ats = read_local
  fresh = fresh_count(beats)
  @out.puts "wurk health: #{fresh} process(es) on this host beat within #{@window}s (need #{@min})"
  return UNHEALTHY if fresh < @min

  jobs_within_age?(run_ats) ? HEALTHY : UNHEALTHY
rescue RedisClient::Error, ConnectionPool::TimeoutError => e
  @out.puts "wurk health: cannot read Redis (#{e.class}: #{e.message})"
  @strict ? UNHEALTHY : HEALTHY
end