Health Checks Aren't a Feature, They're a Requirement (And You're Probably Doing Them Wrong)

A service told the load balancer it was healthy. The load balancer believed it. For 14 minutes, every single request that arrived was routed to a service that could not process anything.

The service was running. The process was alive. Memory was available. CPU was fine. The health check endpoint returned 200 OK.

The database connection pool was exhausted. Every request was timing out. Every response was an error. The service was broken.

But the health check did not know that. So the load balancer did not know that. So the service stayed in the active pool, receiving traffic, returning errors, getting slower, until cascade failures brought down three other services.

Health checks are not a feature. They are a contract between your service and the infrastructure running it. And when that contract breaks, everything breaks.

What a Health Check Actually Is

Most developers think a health check is: return 200 OK if the service started.

This is the wrong mental model. A health check is: return 200 OK if the service can handle requests correctly right now.

These are not the same thing.

A service can be running, process alive, listening on a port, and still be unable to handle requests. The database might be unreachable. The cache might be down. A required third-party service might be failing. The service is up but it is dead.

A real health check answers the question: if a request arrived right now, would this service be able to handle it?

If the answer is no, the health check should return non-200. If the answer is yes, it returns 200.

The load balancer uses this information to make routing decisions. If a service is unhealthy, take it out of the rotation. Do not send traffic to it. Let it recover. Route new requests to healthy instances.

This is critical infrastructure logic. When it works, users do not notice. When it breaks, the entire system breaks.

Why Health Checks Matter

Imagine a distributed system with ten services. Each service has three instances for redundancy. A load balancer sits in front, routing requests.

Now one instance of service A has a problem. Maybe the database connection pool is exhausted. It cannot process requests anymore.

Without health checks, the load balancer does not know this. It keeps sending traffic to the broken instance. The instance returns errors. Those errors propagate to service B. Service B starts getting overwhelmed trying to handle errors. Service B starts failing. Errors propagate to service C.

Within minutes, half the system is broken because one instance was broken and nobody noticed until it was too late.

With working health checks, the broken instance reports itself as unhealthy. The load balancer removes it from the pool. Traffic routes to the two healthy instances. They handle the load fine. The broken instance recovers in the background. Once it is healthy again, the load balancer adds it back.

One instance breaks. The system stays up. The difference between these two scenarios is a health check that tells the truth.

But here is the problem. Most health checks lie. They report 200 OK when the service is actually broken. And that lie is worse than no health check at all.

How Most Teams Get It Wrong

I have seen hundreds of health checks that look like this:

app.get('/health', (req, res) => {
  res.status(200).json({ status: 'ok' });
});

This checks nothing. It says: if the HTTP server is running, we are healthy. The server is running. Everything else could be on fire. The health check does not care.

A slightly better version:

app.get('/health', (req, res) => {
  try {
    const dbConnection = db.getConnection();
    res.status(200).json({ status: 'ok' });
  } catch (e) {
    res.status(503).json({ status: 'error', error: e.message });
  }
});

This checks the database. But it does it synchronously, on the health check request, every time. On a system with 100 health check requests per second (load balancer + monitoring + sidecars + other systems), you are running 100 database connections per second just to verify health.

That is expensive. That is wasteful. And it is not even right, because it only checks the database. What about the cache? The message queue? The rate limiting service?

The wrong approach is checking too little. But the other wrong approach is checking too much.

I have also seen health checks that look like this:

app.get('/health', async (req, res) => {
  const checks = await Promise.all([
    checkDatabase(),
    checkCache(),
    checkMessageQueue(),
    checkExternalAPI(),
    checkDiskSpace(),
    checkMemory(),
    checkCPU(),
    checkFileSystemPermissions()
  ]);
  
  if (checks.every(c => c.ok)) {
    res.status(200).json({ status: 'ok' });
  } else {
    res.status(503).json({ status: 'error', checks });
  }
});

This checks everything. It is comprehensive. It is also often wrong, because checking everything on every request is expensive, and expensive health checks cause their own problems.

If the health check takes 500ms to run and it runs every 10 seconds, that is fine. But if the load balancer health check timeout is 2 seconds and the service is under load, the health check might timeout. The load balancer sees the timeout as a failure. It removes the service from the pool. The service is now isolated and cannot recover. A health check meant to detect problems just created one.

The Configuration That Actually Matters

A correct health check is a balance between three things: accuracy, cost, and safety.

Accuracy: The health check should report the actual health status of the service. If the service is broken, it should return non-200. If the service is fine, it should return 200.

Cost: The health check should be cheap to run. Ideally sub-millisecond. It should not create new dependencies that can fail. It should not require expensive operations.

Safety: The health check should not make things worse. It should not trigger expensive operations that slow down the service. It should not timeout and cause cascade failures.

Here is what a reasonable health check looks like:

let lastHealthStatus = { healthy: true, lastChecked: Date.now() };

// Run detailed checks in the background, every 10 seconds
setInterval(async () => {
  try {
    await checkDatabase();
    await checkCache();
    await checkMessageQueue();
    lastHealthStatus = { healthy: true, lastChecked: Date.now() };
  } catch (e) {
    lastHealthStatus = { healthy: false, error: e.message, lastChecked: Date.now() };
  }
}, 10000);

// Health check endpoint returns cached result — very fast
app.get('/health', (req, res) => {
  // Check if the status is stale (older than 30 seconds)
  if (Date.now() - lastHealthStatus.lastChecked > 30000) {
    return res.status(503).json({ status: 'degraded', message: 'health check stale' });
  }
  
  if (lastHealthStatus.healthy) {
    res.status(200).json({ status: 'ok' });
  } else {
    res.status(503).json({ status: 'error', error: lastHealthStatus.error });
  }
});

This approach:

• Checks the important dependencies (database, cache, message queue) regularly in the background
• Returns the cached result instantly when asked (sub-millisecond)
• Fails fast if the health check itself becomes stale (more than 30 seconds old)
• Does not create new connections or expensive operations on the health check path

It is not perfect. There is a small window where the cached status is wrong. But that window is 30 seconds at most, and the trade-off is that the service is not bogged down by health checks.

What Actually Breaks Because of Bad Health Checks

A health check that returns 200 OK when the service is broken is worse than no health check.

Why? Because the load balancer trusts it. The load balancer routes traffic based on that trust. If the trust is misplaced, the traffic goes to a broken service. The broken service fails. The errors cascade.

A service returns 200 OK but its database connection pool is exhausted. Every request fails. The load balancer does not know this, so it keeps sending traffic. The service is now a slow-failure machine. Every request takes time, fails, and wastes resources. Downstream services get errors and start failing.

A service returns 200 OK but a critical third-party API is timing out. The service tries to call the API, times out, retries, times out again. The service is using all its threads on failed API calls. New requests queue up. The queue fills. Requests timeout. The service looks broken from the outside.

A health check itself is expensive and runs on every request. The health check path becomes a bottleneck. Requests queue up waiting for health check results. The service is slow not because the main logic is slow, but because the health check is slow.

A health check checks a dependency that is not actually required for the service to function. The dependency goes down. The service reports unhealthy. The load balancer removes the service. But the service could have served requests without that dependency. The service is needlessly removed from the pool.

Each of these is a way that misconfigured health checks cause cascade failures.

When to Use Health Checks

Not every service needs a health check. Health checks add complexity. They add moving parts that can fail.

But if your service is part of a load-balanced pool, a health check is mandatory. You have no choice. The load balancer needs to know which instances are healthy.

If your service is a microservice that other services depend on, a health check is mandatory. You are part of a distributed system. Failures need to propagate cleanly. A health check makes that possible.

If your service is a singleton (only one instance), a health check is still useful but less critical. You cannot take it out of the pool if it is the only instance. But a health check is useful for visibility: you can monitor whether your service is actually healthy.

If your service is a background job, a cron task, a batch process, a health check does not make sense. There is no load balancer. There is no pool. A health check endpoint is useless.

The question to ask is: will the infrastructure running this service need to make routing decisions based on whether the service is healthy?

If yes, you need a health check. If no, you do not.

The Uncomfortable Truth

Most teams do not think about health checks until production breaks. They add a basic endpoint that returns 200 OK. It works for a while. Then it does not work.

A service fails in a way that the health check does not catch. The load balancer trusts the health check. Traffic keeps flowing to the broken service. The system breaks.

Then the team scrambles. They add more checks to the health check. The health check gets slower. The health check becomes a new bottleneck. They optimize. They add caching. It gets better.

But the fundamental issue remains: they are checking the wrong things, in the wrong way, for the wrong reasons.

A health check is not a status report. It is a contract. It is a promise to the infrastructure: if you route a request to me, I will be able to handle it.

If you cannot keep that promise, do not return 200 OK. Lie, and the infrastructure routes traffic to a broken service. Fail, and infrastructure removes you from the pool so you can recover.

The choice is clear. But it requires understanding that a health check is not a convenience feature. It is the most critical contract in your system.

---

Your health check is a lie unless it checks the right things, costs nothing to run, and actually predicts whether your service can handle requests. Most health checks fail on all three counts. Fix yours before production teaches you what failure looks like.

---