One worker in an image-reading pipeline depends on two separate reading engines, and overnight both of them happened to be off. Three read requests queued up anyway, each one sitting at the gateway for ten full minutes before timing out. Three timeouts in a row is the rule that trips a worker into an outright error state. One of the two engines came back about a minute after the last timeout fired. Nothing downstream noticed. The worker stayed marked as broken for almost ten more hours, until a person noticed and cleared it by hand.
The reason it stayed stuck turned out to be simple once traced: the error flag was only ever cleared by a successful read. But a worker already marked as errored is exactly the one the dispatcher stops sending work to. No work means no chance to succeed, and no success means the flag never clears — a closed loop with no way out from the inside. Toggling the on/off switch for that worker seven times that same morning did nothing, because the switch and the flag were never connected to begin with. More telling was the explanation text sitting next to the error: it stated that the system had closed the worker's valve for safety. The valve had never been touched. Nobody had checked before writing that sentence; it was generated to sound plausible, and it looked exactly as authoritative as a true one would have.
The fix wasn't to widen the conditions that clear the flag. It was to let the system cool down on its own terms and then test itself. If a fixed interval passes — ten minutes by default — with no new failure after the last one, the error count drops back one notch and the worker is handed exactly one image, nothing more. If that single read succeeds, the flag clears completely. If it fails, the worker goes straight back to error with no partial credit and no extra chances. The system gets to retry its own recovery instead of waiting for someone to notice a sentence that wasn't true.
This is an excerpt. The full write-up was first published on Haenginro Lab.