Adding Observability to a Small SaaS Without Overbuilding It

The first useful canary failure did not crash a pod. A tenant page loaded its shell, but the menu that a guest needed did not arrive. A normal health check could honestly report that the process was alive while a person at a table had very little use for it.

At first I treated monitoring as a question of covering the stack: dashboards, errors, alerts and enough green indicators to make a release feel supervised. The awkward detail was that these are different questions. A completed scheduled job says the runner ran. It does not say that a pub page rendered, that the tenant was identified correctly, or that a guest could put something in a basket.

The canary therefore starts in a browser and checks the public shells, first-party APIs, tenant identity and a real guest menu route. It discovers tenants from an explicitly configured URL, an allowlist, or controlled server-side sources; it does not ask the browser to carry an administrative database key. On a failed run it saves a screenshot and trace, which is much more useful at nine in the morning than a bare 502.

That investigation also forced a distinction I had previously blurred. A runner failure and a business failure are not the same incident. The Kubernetes job exits successfully when it has published a reliable report, even if that report says a tenant check failed. Prometheus records the business state separately; the runner only fails for its own configuration, browser or reporting problems.

What the checks mean

Signal	Question it answers
Liveness/readiness probe	Can this process accept work?
Browser canary	Can a guest reach a usable page?
Tenant checks	Is the venue identity, menu and expected state coherent?
Error tracking	What failed inside the application?
Heartbeat	Did the observer complete a run?

The canary also needs to be fair to the product. A closed venue passes when ordering is correctly blocked. A venue without a live payment connection produces a warning rather than a fictitious payment outage. Optional fonts, analytics and telemetry do not turn a working ordering surface into a critical alert. The important failures are missing required documents, first-party API errors, blank pages and fatal application text.

This is deliberately a small system: browser checks, structured JSON reports, Prometheus metrics, error tracking and an alert route with an owner. The cost is that it cannot prove every private workflow or execute a real payment. It should not: the production checker is explicitly non-mutating and uses a dedicated low-privilege account for the limited admin path it tests.

I now write the question beside every alert before I add the alert. “Is the pod alive?” and “can a customer order?” both deserve answers, but combining them is how a tidy dashboard becomes a misleading one.