Observability

Observability is the ability to understand what the system is doing from its external outputs — without needing to ask the system directly. Metrics tell you what happened quantitatively; logs tell you what happened in detail; alerts tell you when something needs your attention. Together, they make the platform operable.

Observability is layer 6 — it monitors everything below it. Unlike other layers, which provide services to layers above them, observability is consumed primarily by humans (operators, on-call engineers). Metrics dashboards, log queries, and alert routing are the interface between the platform and the people responsible for it.

Good observability is designed deliberately. Dashboards that show every available metric are hard to use. Alerts that fire constantly are ignored. Logs that contain everything make finding anything hard. Nyxever's observability principle: instrument intentionally, alert on user-visible symptoms, and assume that the person reading the alert has never seen it before.

The tooling for this layer is under evaluation. What is decided is the approach: build on open standards — OpenTelemetry and its protocol (OTLP), the OpenMetrics exposition format, syslog (RFC 5424) and structured logs — so that collection, storage and visualization components stay documented, explained and replaceable.

Monitoring and observability Monitoring checks whether services are up or down. Observability provides the data needed to understand why they behave as they do — before and after they fail.

Metrics

Overview

Metrics are numerical measurements taken at points in time — CPU utilization, request rate, error count, queue depth, memory usage. Time-series metrics are stored as a sequence of (timestamp, value) pairs indexed by a set of labels. Labels identify the source (host, service, datacenter) and the dimension being measured.

The right metric resolution and retention depends on how the data is used. High resolution (e.g., 15-second collection intervals) is necessary to detect short-lived spikes; lower resolution (5-minute rollups) is sufficient for capacity planning. Keeping all data at the highest resolution indefinitely is expensive. A common pattern is full resolution for recent data (e.g., 15 days), downsampled for historical analysis.

⚠  Metric naming conventions, label cardinality, and recording rules documentation in progress.

Architecture

Metrics reach a time-series store in one of two ways. In a pull model, the collector fetches metrics from HTTP endpoints exposed by each target (for example in the OpenMetrics text format) at a fixed interval: the collector controls the rate, and an unreachable target shows up immediately as a failed collection. In a push model, targets send metrics to a receiver (for example over OTLP), which suits short-lived jobs and sources behind network boundaries.

Beyond a single cluster, the requirements are long-term retention with downsampling, a global query view across sites, and storage that can itself be backed by the platform's object or block storage.

⚠  Metrics architecture (collection model, retention, multi-site aggregation) documentation in progress.

Options & Standards

Selection criteria

The metrics backend is under evaluation. Candidates are assessed on support for OpenMetrics and OTLP ingestion, a capable query language (rates, aggregations, cross-metric operations), behaviour under high label cardinality, retention and downsampling options, operability, licensing and cost. Proxmox VE and Ceph both provide built-in metric export; collecting it without custom glue is a requirement.

Dashboards & visualization

Visualization is kept separate from storage: the dashboard layer queries the metrics and log stores and renders panels, but holds no data itself, so either side can be replaced or upgraded independently. Dashboards must be definable as code and kept in version control.

⚠  Metrics and dashboard tooling evaluation, and the Nyxever dashboard library, in progress.

Logging

Overview

Logs are timestamped records of discrete events — an HTTP request completed, a certificate was issued, a VM was started, a query failed. Unlike metrics, which are aggregated numerical summaries, logs retain the raw detail of every event. This detail is valuable during incident investigation and useless for capacity planning.

Structured logging — emitting JSON or key-value pairs rather than free-form text — makes logs machine-parseable. A structured log line can be indexed by its fields and queried precisely: all errors from a specific service, all authentication failures from a specific IP, all requests that took over 500ms. Free-form text logs require regex parsing that is fragile and slow.

Two standards matter for transport: syslog (RFC 5424), which most infrastructure components — including Proxmox VE hosts and Ceph daemons — already speak, and the OpenTelemetry log data model carried over OTLP, which links log records to traces and metrics through shared resource attributes.

⚠  Log level standards, structured logging conventions, and log retention policy documentation in progress.

Options & Standards

Indexing trade-off

Log stores fall into two broad designs. Label-indexed stores index only a small set of stream labels (host, service, severity) and keep log content compressed, often in object storage: storage is cheap and ingestion fast, but searching inside log content is a slower scan. Full-text-indexed stores index the content itself: arbitrary searches are fast, at a higher cost in storage, memory and operational effort.

The log backend is under evaluation against that trade-off, plus syslog and OTLP ingestion, the ability to correlate a log query with the metrics on the same dashboard, retention controls, operability and cost.

⚠  Log pipeline evaluation (collection agents, storage backend, query language) in progress.

Alerting

Overview

An alert should answer one question: does this situation require human action right now? If the answer is no, the alert should not exist. Alert fatigue — the result of too many low-signal alerts — is one of the most dangerous operational conditions. Engineers stop reading alerts that usually resolve themselves, which means they also stop reading the ones that don't.

Every alert in Nyxever should have a runbook — a document describing what the alert means, what caused it historically, and what the responder should do. An alert without a runbook is a request for the on-call engineer to figure out from scratch what to do, at 3 AM, under pressure. Runbooks encode institutional knowledge so it doesn't have to live in people's heads.

Alert on symptoms. "Disk will fill in 4 hours" is a symptom-based alert that gives the responder time to act. "Disk write rate above 100 MB/s" is a cause-based alert that may or may not require action depending on context. Symptom-based alerts have lower false-positive rates and clearer remediation paths. Where a service has a service-level objective (SLO), alerting on the rate at which its error budget is being consumed is the most direct symptom of all.

⚠  Alert design guide, runbook template, and severity classification documentation in progress.

Options & Standards

Requirements

The alerting pipeline is under evaluation. Whatever is chosen must provide: grouping and deduplication (ten identical alerts from ten nodes become one notification); label-based routing (by service, severity, site or team, to receivers such as a paging service, chat, email or a generic webhook); inhibition (suppress lower-severity alerts while a related higher-severity alert is firing); silences (mute specific alerts for a defined window during maintenance); and a link from every alert to its runbook.

Alert rules, routing and runbooks are kept as code in version control, so every change to what pages a human is reviewed like any other change.

⚠  Alerting tooling evaluation, routing design and notification integration in progress.