Storage

The storage layer provides durable data persistence across four access patterns: block (raw device), file (POSIX filesystem), object (HTTP API), and local (on-node). Choosing the right storage type for each workload affects performance, availability, and operational complexity.

Each storage type fits different workloads. Block storage has the lowest overhead and widest compatibility; distributed file storage enables shared access across nodes; object storage scales out behind a simple HTTP API; local storage is fastest but tied to one node. Most infrastructure environments use all four.

Decided: Ceph is the platform's distributed storage system and ZFS (OpenZFS) is the local storage choice, both integrated with Proxmox VE and consumed by Kubernetes through CSI. Other storage and backup tooling is still under evaluation; each choice will be documented with its reasoning once made.

Distributed storage is a prerequisite for compute HA VMs backed by local storage cannot be live-migrated or automatically restarted on another node after a failure. Distributed block or file storage is required for the compute layer to provide high availability.

Block Storage

Overview

Block storage presents raw storage to clients as a block device — the same abstraction as a physical hard drive or SSD. Clients can format it with any filesystem, use it as a raw device, or hand it to a database that manages its own storage layout. Block storage has no awareness of files or directories; it deals only in fixed-size blocks of data.

Block storage is the right choice for virtual machine disks (which need raw device semantics), databases with high IOPS requirements, and anything that needs a filesystem with specific characteristics that the storage layer shouldn't dictate. For data that several clients must read and write at the same time, use file or object storage.

⚠  Block storage use-case selection guide in progress.

Architecture

Distributed block storage replicates data across multiple storage nodes (OSDs — Object Storage Daemons in Ceph terminology) so that the failure of any single node does not result in data loss. The replication strategy is either full replication (N copies of each data chunk, stored on N different nodes) or erasure coding (M data chunks + K parity chunks, any M chunks sufficient to reconstruct the data, requiring fewer raw bytes than full replication at the cost of more CPU).

Ceph places data with CRUSH, a deterministic placement algorithm that lets any client compute which OSDs hold an object from the cluster map, without a central lookup table. Adding or removing OSDs triggers a rebalance, and no central metadata server sits in the data path.

⚠  Replication vs erasure coding tradeoffs, CRUSH map configuration, and OSD sizing documentation in progress.

Technologies

Local block volumes (LVM)

Logical Volume Manager (LVM) is the standard Linux kernel abstraction for local block storage. Physical volumes (disks or partitions) are pooled into volume groups and carved into logical volumes that appear as block devices, with support for copy-on-write snapshots and thin provisioning.

LVM manages storage on a single node, so it suits OS volumes, swap and local scratch space. VM disks that must migrate between nodes or survive a node failure belong on distributed storage (Ceph RBD). Whether LVM or ZFS volumes are used for local block devices is under evaluation.

⚠  Local block volume selection (LVM vs ZFS volumes) and snapshot management documentation in progress.

Ceph RBD

Ceph RADOS Block Device (RBD) provides distributed block storage built on the Ceph RADOS object store. An RBD image is stored as a series of fixed-size objects distributed across the cluster. Clients access RBD images via the librbd library or the kernel RBD driver, presenting them as block devices that can be used as VM disks, container volumes, or database storage.

RBD supports copy-on-write snapshots and clones, enabling fast VM provisioning from base images. Snapshots are lightweight — they record only what has changed since the snapshot was taken. Clones of a snapshot are instant; data is shared with the parent until a clone diverges.

RBD is the platform's distributed block storage: Proxmox VE uses it for VM disks, and Kubernetes consumes it through the Ceph CSI driver for persistent volumes.

⚠  Ceph RBD configuration, Proxmox VE integration, snapshot management, and performance tuning documentation in progress.

File Storage

Overview

File storage presents a POSIX filesystem interface — directories, files, permissions, and standard syscalls (open, read, write, stat). Shared file storage allows multiple clients to access the same filesystem simultaneously, making it appropriate for workloads that need to share data or configuration: backup targets, media repositories, configuration stores, shared home directories.

The tradeoffs versus block storage: file storage has higher latency (the storage layer parses filesystem operations), fewer IOPS for databases, and the access control model is the filesystem's (POSIX permissions or ACLs) rather than the application's. For workloads that are genuinely sharing data across nodes, file storage is the correct choice; for single-VM workloads, block storage is usually better.

⚠  File storage use-case and filesystem selection guide in progress.

Technologies

NFS

Network File System (NFS) is an open protocol for exporting a filesystem from a server and mounting it on clients. NFSv4 adds stateful sessions, optional Kerberos authentication and better performance than v3. Proxmox VE can use an NFS share as storage for ISOs, templates and backups.

A single NFS server is a single point of failure; high availability needs a clustered or failover server design. NFS is relevant to Nyxever mainly as a protocol for interoperating with existing systems and as a possible backup target. Where NFS fits in the platform is under evaluation.

⚠  NFS use cases, Proxmox VE integration, and availability considerations documentation in progress.

CephFS

CephFS is a POSIX filesystem built on Ceph's RADOS object store. File metadata (directory entries, file attributes, inodes) is managed by one or more Metadata Servers (MDS). File data is stored as RADOS objects, distributed across OSDs like all other Ceph data. Multiple MDS instances can share load; one active MDS with standbys is the most common configuration.

CephFS inherits Ceph's replication model, so file data survives the loss of a node, and with standby MDS daemons the metadata service fails over automatically. Multiple clients can mount the same CephFS volume simultaneously, and Kubernetes can consume it through the Ceph CSI driver for ReadWriteMany volumes. Which shared-file workloads use CephFS is being worked out.

⚠  CephFS deployment, MDS configuration, and mount options documentation in progress.

Object Storage

Overview

Object storage stores data as objects — named blobs with associated metadata — accessible via an HTTP API. There is no directory hierarchy in the classical sense; objects are stored in flat buckets, though key naming conventions can simulate a directory structure. The de facto standard interface is the S3 API, which began as a hyperscaler's object storage service and is now implemented by many systems.

Object storage scales out by adding capacity, suits HTTP clients, and fits backup targets, build artifacts, media libraries and anything accessed through an API rather than a filesystem. Databases, VM disks and other workloads that need low-latency random access belong on block storage.

⚠  Object storage design patterns and S3 API compatibility guide in progress.

Technologies

S3 / Ceph RGW

Ceph Object Gateway (RGW, also called RADOS Gateway) provides an S3-compatible HTTP API on top of RADOS. Applications that use the S3 API can target Ceph RGW without modification. RGW handles authentication (S3 access keys or STS-style federation), multipart uploads, bucket policies, and server-side encryption.

Since RGW stores data in RADOS, it inherits Ceph's replication properties. Running RBD, CephFS and RGW on one Ceph cluster gives block, file and object access from a single storage system: one set of OSDs to operate and monitor. Using RGW as the platform's object store is the working assumption; off-site object storage for backups is a separate decision (see Backups).

⚠  RGW deployment, S3 client configuration, and bucket lifecycle policy documentation in progress.

Local Storage

Overview

Local storage — storage physically attached to a single node — is the fastest option (no network latency, no serialization overhead) and the least available (node failure means the storage is unreachable until the node recovers). It is appropriate for ephemeral data, high-performance scratch space, and workloads where the application handles its own replication.

In a Nyxever cluster, local storage is used for: the hypervisor OS, swap, local scratch space for VMs with high I/O requirements, and Ceph OSD journals/WALs (where fast local SSDs accelerate the write path for Ceph). Data that needs to survive node failures should not be stored solely on local storage.

⚠  Local storage configuration and drive layout recommendations in progress.

Technologies

ZFS

ZFS (OpenZFS) is a combined filesystem and volume manager with end-to-end checksums, copy-on-write semantics and built-in RAID (mirrors and RAIDZ). Every block is checksummed when written and verified when read or scrubbed, so silent corruption is detected and, where redundancy exists, repaired automatically. ZFS is the platform's choice for local storage on Proxmox VE nodes.

ZFS features that matter for infrastructure use: the ARC (Adaptive Replacement Cache) provides aggressive in-memory caching; snapshots are instant and space-efficient (copy-on-write, no data copied until diverged); send/receive enables efficient incremental backups at the dataset level. RAIDZ1/2/3 provide parity-based redundancy within a single node; data that must stay available when the whole node fails belongs on distributed storage.

⚠  ZFS pool configuration, RAIDZ levels, ARC tuning, and send/receive backup guide in progress.

Backups

Overview

A backup is a separate, independent copy of data that can be restored even if the primary system is completely destroyed. Replication, RAID and on-node snapshots protect against hardware failure; backups also protect against deletion, corruption that replicates, ransomware and site loss.

The 3-2-1 rule: three copies of data, on two different media, with one copy off-site. Applied to Nyxever: primary data on Ceph (replicated across nodes), a backup copy on a separate storage system, and a third copy off-site (a remote site or a hyperscaler's object storage). RPO (Recovery Point Objective — how much data you can afford to lose) and RTO (Recovery Time Objective — how long recovery may take) set backup frequency and off-site replication targets.

⚠  Backup strategy guide, RPO/RTO targets, and off-site replication documentation in progress.

Requirements & candidates

The backup tooling is not yet decided. Candidates will be assessed against these requirements:

  • Covers Proxmox VE guests (VMs and LXC containers), Kubernetes workloads and their persistent volumes, and host-level ZFS datasets.
  • Incremental, deduplicated backups with client-side encryption at rest.
  • Writes to an S3-compatible object store or another off-site target, with immutability or retention locking to resist ransomware.
  • Built-in integrity checks, and restores that can be tested on a schedule.
  • Uses mechanisms the platform already has where they fit: ZFS send/receive, Ceph RBD snapshots and export, and CSI volume snapshots.
  • Operable by a small team, with the cost and recovery path documented.
⚠  Backup tooling evaluation against these requirements in progress.