Observability, Capacity, And Operational Readiness

Purpose

Observability is the ability to understand what a system is doing from the outside. Capacity planning is the practice of knowing what load the system can handle and what it will need next. Operational readiness is the discipline of launching systems that teams can actually run.

These are the practical glue between architecture diagrams and production reality.

Observability Signals

Good observability answers:

• Is the user experience healthy?
• What changed?
• Where is the bottleneck?
• Which dependency is failing?
• How many users are affected?
• Is the system recovering?

Golden Signals

For many services, start with:

• Latency.
• Traffic.
• Errors.
• Saturation.

For data systems, also consider:

• Durability.
• Replication lag.
• Queue depth.
• Processing delay.
• Data freshness.
• Correctness checks.

Alerting Principles

Good alerts are:

• Actionable.
• User-impact focused.
• Owned by a team.
• Routed correctly.
• Clear about severity.
• Linked to a runbook.

Bad alerts are:

• Noisy.
• Infrastructure-only without user impact.
• Unowned.
• Repeatedly ignored.
• Triggered by normal behavior.

If an alert wakes someone and they do nothing, the alert is training them to ignore the system.

Capacity Planning

Capacity planning asks:

• What is current load?
• What is expected growth?
• What is peak load?
• What is the bottleneck?
• What is the scaling mechanism?
• What is the cost of scaling?
• What are limits and quotas?
• What is the lead time to add capacity?

Capacity is not only servers. It includes database connections, queue consumers, storage, API limits, rate limits, licenses, team support capacity, and vendor quotas.

Operational Readiness Review

Before launch, ask:

• Who owns the service?
• What does healthy look like?
• What dashboards exist?
• What alerts exist?
• What runbooks exist?
• How do we deploy and roll back?
• How do we restore data?
• What dependencies can fail?
• What is the expected load?
• What is the cost expectation?
• What security controls are required?
• What customer/support message is needed during disruption?

Runbooks

Runbooks should include:

• Symptoms.
• Dashboard links.
• First checks.
• Common causes.
• Safe actions.
• Escalation path.
• Rollback/restart steps.
• Customer communication notes if relevant.
• Post-incident follow-up checklist.

Runbooks should be short enough to use during stress. A 40-page runbook during an incident is literature, not operations.

Dashboards

Useful dashboards:

• Service overview: traffic, latency, errors, saturation.
• Dependency dashboard.
• Business journey dashboard.
• Deployment/release dashboard.
• Cost dashboard.
• Security/reliability signals.
• Queue/job dashboard.

Dashboards should be designed for decisions:

• Is it healthy?
• What changed?
• Where should we look next?
• Should we roll back?
• Should we scale?

Team Reference Guide

Guidelines For Teams

• Build observability into services from the start.
• Alert on symptoms before causes.
• Create runbooks for common failures.
• Review capacity before major launches.
• Treat operational readiness as part of done.

Reflection Questions

• What would we look at first during an incident?
• Which important user journey lacks visibility?
• Which alert is noisy enough to remove or redesign?
• What capacity limit would surprise us under load?

Further Study

• Google SRE Monitoring Distributed Systems: https://sre.google/sre-book/monitoring-distributed-systems/
• Google SRE Practical Alerting: https://sre.google/sre-book/practical-alerting/
• OpenTelemetry documentation: https://opentelemetry.io/docs/
• Prometheus documentation: https://prometheus.io/docs/introduction/overview/
• Grafana documentation: https://grafana.com/docs/
• Azure operational excellence: https://learn.microsoft.com/en-us/azure/well-architected/operational-excellence/
• AWS operational excellence pillar: https://docs.aws.amazon.com/wellarchitected/latest/operational-excellence-pillar/welcome.html