Designing for failover that nobody notices

When a node goes quiet, the worst thing your infrastructure can do is tell the user about it. A dropped call, a spinning loader, a transaction that never confirms — these are the moments that turn a good product into one people stop trusting. Failover, done right, is invisible.

Most teams discover this the hard way. You point your app at an RPC endpoint, it works in development, and then one Saturday a provider has a bad hour and your users are the monitoring system. The fix is rarely glamorous: it is retries, health checks, request hedging, and a router that knows which way to send traffic before a human wakes up.

The shape of a quiet failure

A node rarely dies cleanly. More often it gets slow, returns stale data, or starts rejecting a fraction of requests. Treating "down" as binary misses most of the real-world cases, so Hexiora scores every upstream continuously on latency, error rate, and block lag — and routes against that score rather than a simple up/down flag.

"The goal isn't zero failures. It's zero failures the user can feel."

Hedging without doubling cost

For idempotent reads, Hexiora can issue a second request to a different upstream if the first is taking too long, and return whichever answers first. The trick is the threshold: hedge too eagerly and you double your traffic; too late and you've already lost the latency budget. We tie it to a rolling p95 per method, so the system hedges exactly when it starts to matter.

// pseudocode: latency-aware hedge
const deadline = p95(method) * 1.4;
const first = call(primary, req);
const backup = sleep(deadline).then(() => call(secondary, req));
return Promise.race([first, backup]);

What this buys you

The result is an endpoint that behaves like infrastructure: predictable under load, honest when something is wrong, and quiet when it routes around a problem. You see it in the trace afterward — a retry, a switched upstream — but your users never do. That is the entire point.

• Continuous scoring beats binary health checks.

• Hedge against p95, not a fixed timeout.

• Make the recovery visible to you, invisible to users.

None of this is exotic. It is the boring, expensive work that turns "it usually works" into "it works," and it is exactly the layer we think you should never have to build yourself.