Class: Wurk::Lua::Loader

Inherits:
Object
  • Object
show all
Defined in:
lib/wurk/lua/loader.rb

Overview

EVALSHA wrapper with NOSCRIPT recovery. The SHA1 of each script source is precomputed in Wurk::Lua::SHAS, so the first call to eval_cached after a fork has a fast path: a single EVALSHA, no per-pool bookkeeping. If the script cache was flushed (manual SCRIPT FLUSH, replica failover, OOM eviction), EVALSHA returns NOSCRIPT — we then SCRIPT LOAD once and retry exactly once.

Spec: docs/target/sidekiq-free.md §20 (Lua script caching).

Constant Summary collapse

NOSCRIPT_PREFIX = 
'NOSCRIPT'
SHA_LIST = 
SHAS.values.freeze
SOURCE_LIST = 
SCRIPTS.values.freeze

Class Method Summary collapse

Class Method Details

.eval_cached(redis, name, keys:, argv:) ⇒ Object

Returns Lua script return value.

Parameters:

  • redis (RedisClient) —

    a single connection (not a pool)

  • name (Symbol) —

    key into Wurk::Lua::SCRIPTS

  • keys (Array<String>) —

    EVALSHA KEYS

  • argv (Array) —

    EVALSHA ARGV (coerced to strings by Redis)

Returns:

  • Lua script return value



52
53
54
55
56
57
58
59
60
61
 
# File 'lib/wurk/lua/loader.rb', line 52

def eval_cached(redis, name, keys:, argv:)
  src = SCRIPTS.fetch(name) { raise ArgumentError, "unknown Lua script: #{name.inspect}" }
  sha = SHAS.fetch(name)
  evalsha(redis, sha, keys, argv)
rescue RedisClient::CommandError => e
  raise unless noscript?(e)

  redis.call('SCRIPT', 'LOAD', src)
  evalsha(redis, sha, keys, argv)
end

.eval_once(redis, name, keys:, argv:) ⇒ Object

Run a pipeline that contains EVALSHAs, recovering from a flushed script cache the way a pipeline forces us to.

#eval_cached for a script that a re-send would apply twice: redis-client replays call on a dropped socket, never call_once. NOSCRIPT proves nothing ran, so its recovery may send again.



69
70
71
72
73
74
75
76
 
# File 'lib/wurk/lua/loader.rb', line 69

def eval_once(redis, name, keys:, argv:)
  redis.call_once('EVALSHA', SHAS.fetch(name), keys.size, *keys, *argv)
rescue RedisClient::CommandError => e
  raise unless noscript?(e)

  redis.call('SCRIPT', 'LOAD', SCRIPTS.fetch(name))
  redis.call_once('EVALSHA', SHAS.fetch(name), keys.size, *keys, *argv)
end

.eval_with_source(redis, name, keys:, argv:) ⇒ Object

Source-embedded EVAL — the slow but cache-independent counterpart to eval_cached. Used on retry from a pipelined NOSCRIPT recovery where EVALSHA can still race a freshly-loaded script under heavy CI load (cf. WorkerTest NOSCRIPT flake on test (3.4, 7.2)). EVAL ships the full source every call, so it never raises NOSCRIPT.



106
107
108
109
 
# File 'lib/wurk/lua/loader.rb', line 106

def eval_with_source(redis, name, keys:, argv:)
  src = SCRIPTS.fetch(name) { raise ArgumentError, "unknown Lua script: #{name.inspect}" }
  redis.call('EVAL', src, keys.size, *keys, *argv)
end

.load_missing(redis) ⇒ Object

Upload only the scripts the server's cache lacks, so the first real EVALSHA hits a warm cache instead of paying a NOSCRIPT reload. The cache is server-global: after the first boot against a server every SHA is already there, and this is one SCRIPT EXISTS carrying the SHAs instead of ~70KB of source — per swarm child, on the boot-critical path. A cold cache (new or flushed server) costs one more round trip for the loads. Returns how many were uploaded. An ACL may allow SCRIPT LOAD yet deny SCRIPT EXISTS; such a user gets the full upload it had before rather than a child that cannot boot.



37
38
39
40
41
42
43
44
45
 
# File 'lib/wurk/lua/loader.rb', line 37

def load_missing(redis)
  present = redis.call('SCRIPT', 'EXISTS', *SHA_LIST)
  missing = SOURCE_LIST.reject.with_index { |_, i| present[i] == 1 }
  redis.pipelined { |pipe| missing.each { |src| pipe.call('SCRIPT', 'LOAD', src) } } unless missing.empty?
  missing.size
rescue RedisClient::PermissionError
  script_load_all(redis)
  SOURCE_LIST.size
end

.pipelined_eval(redis) ⇒ Object

A pipelined EVALSHA surfaces NOSCRIPT only when the pipeline finalizes — never to #eval_cached's inline rescue, which has already returned by then. A pipeline is not a transaction: by the time that error is raised, every other command in it has already been applied; only the EVALSHAs failed. Recovery is one SCRIPT LOAD and a replay of the whole block through source-embedded EVAL, which cannot raise NOSCRIPT at all — so the plain commands run a second time.

That is why the block must be replay-safe: it is handed the pipeline and the eval method to route through, and may run twice. Both of Fetcher::Reliable's callers pipeline commands that are (a replayed LREM removes nothing, a replayed reliable_requeue is LPOS-guarded, a replayed fetch_slot claims a different job into the same private list), and so is Heartbeat's beat (every write converges).



92
93
94
95
96
97
98
99
 
# File 'lib/wurk/lua/loader.rb', line 92

def pipelined_eval(redis)
  redis.pipelined { |pipe| yield pipe, :eval_cached }
rescue RedisClient::CommandError => e
  raise unless noscript?(e)

  script_load_all(redis)
  redis.pipelined { |pipe| yield pipe, :eval_with_source }
end

.script_load_all(redis) ⇒ Object

Upload every registered script in a single pipelined round-trip (all SCRIPT LOADs, one RTT — not one per script). Idempotent on the Redis side: SCRIPT LOAD of the same source returns the same SHA no matter how often it runs. Transient connection errors are the pool wrapper's job (Wurk::RedisPool#with); this only ships the loads.



24
25
26
 
# File 'lib/wurk/lua/loader.rb', line 24

def script_load_all(redis)
  redis.pipelined { |pipe| SOURCE_LIST.each { |src| pipe.call('SCRIPT', 'LOAD', src) } }
end