The workflow that failed silently for six weeks

The workflow itself was simple: a new order in the store triggered an update to the inventory system, which triggered a low-stock alert to the ops team once a product dropped under its reorder threshold. It had worked for months. Then a field name changed upstream during an unrelated platform update, the integration's mapping broke, and the workflow stopped running — with no error visible anywhere a human was looking.

How it went six weeks unnoticed

Nobody was checking whether the workflow ran, because it had never needed checking before. There was no error email, because the failure was not the kind of failure that raises an exception — the trigger simply stopped matching, so the workflow stopped firing, silently, which to every system involved looked identical to "there is nothing to do right now." The first sign of a problem was a stockout on a product that should have triggered a reorder alert weeks earlier. By the time anyone noticed, the cost was not the broken integration — it was six weeks of inventory decisions made on data that had quietly stopped updating.

The actual failure was not technical

The integration breaking was a normal, unavoidable event — upstream systems change fields, that is not negligence, that is software. The real failure was architectural: a workflow that could fail completely and produce exactly the same visible output as a workflow with nothing to report. Nothing in the system distinguished "quiet because everything is fine" from "quiet because it is broken."

The alerting pattern we use now

Every workflow we build since has a heartbeat, separate from its actual function: a scheduled check that confirms the workflow ran successfully within its expected window, regardless of whether it had anything to act on. No heartbeat within the window raises an alert on its own — not "no low-stock items found," but "the check for low-stock items did not run," which are very different facts and need to stay distinguishable. It costs almost nothing to build in from the start and it is the single change that turns a silent six-week failure into a five-minute one.

What we tell clients now

The lesson was not "add more monitoring" in the abstract — it was specific: any automation that runs unattended needs a way to prove it is still running, not just a way to report what it found. If a workflow can fail into silence, it eventually will, and the six weeks between the failure and someone noticing is where the real cost lives.