Observability
Observability is the ability to understand what the system is doing from its external outputs — without needing to ask the system directly. Metrics tell you what happened quantitatively; logs tell you what happened in detail; alerts tell you when something needs your attention. Together, they make the platform operable.
Observability is layer 6 — it monitors everything below it. Unlike other layers, which provide services to layers above them, observability is consumed primarily by humans (operators, on-call engineers). Metrics dashboards, log queries, and alert routing are the interface between the platform and the people responsible for it.
Good observability is designed deliberately. Dashboards that show every available metric are hard to use. Alerts that fire constantly are ignored. Logs that contain everything make finding anything hard. Nyxever's observability principle: instrument intentionally, alert on user-visible symptoms, and assume that the person reading the alert has never seen it before.
The tooling for this layer is under evaluation. What is decided is the approach: build on open standards — OpenTelemetry and its protocol (OTLP), the OpenMetrics exposition format, syslog (RFC 5424) and structured logs — so that collection, storage and visualization components stay documented, explained and replaceable.
Metrics
Overview
Metrics are numerical measurements taken at points in time — CPU utilization, request rate, error count, queue depth, memory usage. Time-series metrics are stored as a sequence of (timestamp, value) pairs indexed by a set of labels. Labels identify the source (host, service, datacenter) and the dimension being measured.
The right metric resolution and retention depends on how the data is used. High resolution (e.g., 15-second collection intervals) is necessary to detect short-lived spikes; lower resolution (5-minute rollups) is sufficient for capacity planning. Keeping all data at the highest resolution indefinitely is expensive. A common pattern is full resolution for recent data (e.g., 15 days), downsampled for historical analysis.
Architecture
Metrics reach a time-series store in one of two ways. In a pull model, the collector fetches metrics from HTTP endpoints exposed by each target (for example in the OpenMetrics text format) at a fixed interval: the collector controls the rate, and an unreachable target shows up immediately as a failed collection. In a push model, targets send metrics to a receiver (for example over OTLP), which suits short-lived jobs and sources behind network boundaries.
Beyond a single cluster, the requirements are long-term retention with downsampling, a global query view across sites, and storage that can itself be backed by the platform's object or block storage.
Options & Standards
Selection criteria
The metrics backend is under evaluation. Candidates are assessed on support for OpenMetrics and OTLP ingestion, a capable query language (rates, aggregations, cross-metric operations), behaviour under high label cardinality, retention and downsampling options, operability, licensing and cost. Proxmox VE and Ceph both provide built-in metric export; collecting it without custom glue is a requirement.
Dashboards & visualization
Visualization is kept separate from storage: the dashboard layer queries the metrics and log stores and renders panels, but holds no data itself, so either side can be replaced or upgraded independently. Dashboards must be definable as code and kept in version control.
Logging
Overview
Logs are timestamped records of discrete events — an HTTP request completed, a certificate was issued, a VM was started, a query failed. Unlike metrics, which are aggregated numerical summaries, logs retain the raw detail of every event. This detail is valuable during incident investigation and useless for capacity planning.
Structured logging — emitting JSON or key-value pairs rather than free-form text — makes logs machine-parseable. A structured log line can be indexed by its fields and queried precisely: all errors from a specific service, all authentication failures from a specific IP, all requests that took over 500ms. Free-form text logs require regex parsing that is fragile and slow.
Two standards matter for transport: syslog (RFC 5424), which most infrastructure components — including Proxmox VE hosts and Ceph daemons — already speak, and the OpenTelemetry log data model carried over OTLP, which links log records to traces and metrics through shared resource attributes.
Options & Standards
Indexing trade-off
Log stores fall into two broad designs. Label-indexed stores index only a small set of stream labels (host, service, severity) and keep log content compressed, often in object storage: storage is cheap and ingestion fast, but searching inside log content is a slower scan. Full-text-indexed stores index the content itself: arbitrary searches are fast, at a higher cost in storage, memory and operational effort.
The log backend is under evaluation against that trade-off, plus syslog and OTLP ingestion, the ability to correlate a log query with the metrics on the same dashboard, retention controls, operability and cost.
Alerting
Overview
An alert should answer one question: does this situation require human action right now? If the answer is no, the alert should not exist. Alert fatigue — the result of too many low-signal alerts — is one of the most dangerous operational conditions. Engineers stop reading alerts that usually resolve themselves, which means they also stop reading the ones that don't.
Every alert in Nyxever should have a runbook — a document describing what the alert means, what caused it historically, and what the responder should do. An alert without a runbook is a request for the on-call engineer to figure out from scratch what to do, at 3 AM, under pressure. Runbooks encode institutional knowledge so it doesn't have to live in people's heads.
Alert on symptoms. "Disk will fill in 4 hours" is a symptom-based alert that gives the responder time to act. "Disk write rate above 100 MB/s" is a cause-based alert that may or may not require action depending on context. Symptom-based alerts have lower false-positive rates and clearer remediation paths. Where a service has a service-level objective (SLO), alerting on the rate at which its error budget is being consumed is the most direct symptom of all.
Options & Standards
Requirements
The alerting pipeline is under evaluation. Whatever is chosen must provide: grouping and deduplication (ten identical alerts from ten nodes become one notification); label-based routing (by service, severity, site or team, to receivers such as a paging service, chat, email or a generic webhook); inhibition (suppress lower-severity alerts while a related higher-severity alert is firing); silences (mute specific alerts for a defined window during maintenance); and a link from every alert to its runbook.
Alert rules, routing and runbooks are kept as code in version control, so every change to what pages a human is reviewed like any other change.