Human + AI
Nyxever treats AI as a capability within the infrastructure, designed and operated like any other service. Where inference runs — on local hardware or through a hosted model API — is a per-workload decision, made on data sensitivity, cost, capability and risk, and documented with its reasoning. Models are selected deliberately, and humans remain accountable for what the system does.
The "Human +" reflects a design commitment: AI augments human decision-making. Workflows that invoke AI capabilities have explicit human checkpoints. Automation that acts on model output without human review requires active justification and is documented as such.
AI is layer 5 in the stack — it depends on compute (especially GPU compute), networking, storage (for model weights and inference state), and identity (for API authentication and audit logging). This layer is optional in the base platform; most infrastructure does not require AI inference to function.
AI Architecture
Local AI inference is architecturally similar to other internal services: a server process listens for requests, loads a model into GPU memory, and returns completions. The main differences from traditional services are the hardware requirements (GPU memory, fast storage for model weights) and the statefulness of the model server (a loaded model occupies GPU memory and takes time to evict).
Nyxever's AI architecture distinguishes two serving profiles with different requirements: development inference (single node, single GPU, few users, ease of experimentation matters most) and production inference (multiple GPUs, concurrent users, token-generation throughput and latency as first-class metrics). The serving software for each profile is under evaluation against the criteria below.
Model selection is a deliberate choice with its own decision record. The largest model that fits in available VRAM is not necessarily the best choice. A smaller, faster model with lower latency may serve a use case better than a larger, slower one. Every model added to the platform has a documented purpose and is evaluated against alternatives.
Local Inference
Overview
Local inference runs a language model on hardware you control. The model is loaded from local storage into GPU VRAM; inference requests arrive via HTTP API and return completions without any external network calls. Most inference servers expose a de facto standard chat-completions HTTP API, so existing clients and tools can be pointed at a local endpoint with little or no change.
The practical constraints: GPU VRAM is the binding resource. A 7B parameter model in FP16 precision requires approximately 14 GB of VRAM. Quantization (reducing precision to INT8, INT4) reduces VRAM usage at some quality cost — a 7B model at INT4 fits in roughly 4 GB. Quantization tradeoffs depend on the use case; for many infrastructure tasks, INT8 or even INT4 quality is sufficient.
Architecture
An inference server receives a prompt, tokenizes it, runs forward passes through the model to generate output tokens, and streams them back to the client. The KV cache (key-value cache for attention mechanism intermediate values) is the memory-intensive data structure during generation — it grows with the context length. Managing KV cache efficiently is the primary engineering challenge in high-throughput inference.
For production workloads with concurrent users, continuous batching is essential: rather than processing one request at a time and waiting for completion, the inference server batches tokens across multiple in-flight requests at each forward pass. This dramatically improves GPU utilization and throughput.
Options & Standards
Selection criteria
The inference-serving software is not yet decided. Candidates are assessed per serving profile against the same criteria: throughput and latency under concurrent load (continuous batching, efficient KV-cache management, multi-GPU tensor parallelism for large models); ease of setup and model management for development use; supported model formats and quantization schemes; a standard HTTP inference API, so clients stay portable between servers; and hardware support, licensing, operability, and the metrics and logs it exports.
Standards under evaluation
Model formats: GGUF (common for quantized models on a single node), safetensors (common for full-precision weights) and ONNX (a portable graph format). Models: open-weight models are preferred where they meet the quality bar, because they can run locally and be pinned to a known version. Tool access: the Model Context Protocol (MCP) as a standard way to give models scoped, auditable access to tools and data.
Decided: GPU-backed inference workloads run in virtual machines on Proxmox VE using PCIe passthrough through KVM/QEMU, or as Kubernetes workloads on GPU-equipped nodes.
GPU Scheduling
Overview
GPUs are expensive, scarce hardware resources. Unlike CPU cores, which can be oversubscribed (a VM with 4 vCPUs can run on a host with 4 cores shared with other VMs), GPU VRAM is a hard limit — a model loaded into VRAM occupies it exclusively. Scheduling GPU workloads means treating GPUs as first-class schedulable resources with explicit allocation.
There are three broad ways to share GPUs: whole-device assignment (one workload per GPU, simplest isolation), hardware partitioning (some data-centre GPUs can be split into isolated instances with dedicated memory and compute), and time-slicing (workloads take turns, with no memory isolation). Each trades utilization against isolation and predictability.
Options & Standards
Under evaluation: whole-device PCIe passthrough to VMs on Proxmox VE; the Kubernetes device plugin framework and Dynamic Resource Allocation for exposing GPUs to pods; and whether hardware partitioning or time-slicing is worth its operational cost for our workloads. Criteria: isolation, utilization, hardware support, operability and cost.
Human in the Loop
Overview
Human-in-the-loop design means that consequential actions — changes to infrastructure configuration, deployments, data modifications — require a human to review and confirm model output before execution. The model proposes; the human decides. This is a deliberate accountability structure, independent of how capable the model is.
The practical implementation depends on the action's risk profile. Generating a summary of logs can be fully automated. Generating a configuration change that will be applied to production requires human review. Generating a command that will delete data requires explicit confirmation with the human seeing exactly what will be deleted before they confirm. The risk profile sets the checkpoint depth for each action.
Automation that skips human checkpoints is opt-in, documented, and time-limited. Model access to tools (for example through MCP) is scoped to the minimum needed, and every model-initiated action is logged with the identity that approved it. Unattended automation acting on model output is treated as a failure mode, even when the model is usually right.