See the homepage for the quick start installation. This section covers building from source and service deployment options.

Requires the Rust toolchain, version 1.94 or newer. rustup installs are always current; package-manager installs such as Homebrew can lag, so upgrade or switch to rustup if the build reports an unsupported rustc.

git clone https://github.com/giovannifil-64/oxydllm.git
cd oxydllm

For using the inference engine, you must build with either the metal or cuda feature depending on your target.

If you are on a Apple Silicon Mac, run:

cargo build --release --features metal

On the other hand, for NVIDIA CUDA you need also to set the compute capability:

CUDA_COMPUTE_CAP=<value> cargo build --release --features cuda

Replace <value> with the compute capability of your GPU.

Here is a list of the supported compute capabilities and their corresponding GPUs

After installation via the quick start script, the server runs automatically as a system service (launchd on macOS, systemd on Linux). To manually restart or stop:

Verified on macOS on Apple Silicon (M-series, Metal): every checkpoint in section 4 was run through the regression sweep on an M5 with 24 GB. CUDA compiles in CI and is unverified at runtime; the CPU path serves the unit tests and small models and is not a performance target.

The OpenAI-compatible request and response shapes, the model registry (~/.oxydllm/models/.oxydllm_registry.json) and the configuration files stay backward-compatible across 0.x releases on a best-effort basis: a newer binary reads what an older one wrote, reconciles the registry with what is on disk and prunes stale entries, and any change that breaks a client or a config gets a BREAKING section in the changelog. Those formats and a written deprecation policy freeze at 1.0.

Download a model from HuggingFace. For GGUF repos, an interactive variant selection prompt appears unless --variant is specified. Variants already on disk are listed with a check mark and excluded from choices, re-downloading requires --force. A download that stops, whether the machine slept, the network dropped or the process was killed, resumes from the bytes already on disk: a stalled connection is reopened on its own, and re-running the same pull after a shutdown picks up where it stopped.

oxydllm pull <user/model> [options]

For a GGUF variant the header is read with a range request before anything is downloaded, and the file is refused with the reason when it holds a quantization the engine cannot decode, an architecture this build does not serve, or a mixture of experts, which the GGUF runtime does not serve. A precision a publisher appends after the quantization is part of the variant label (Q4_K_M-FP16), so two such files of one repository stay two variants.

Examples:

# GGUF model, interactive variant selection
oxydllm pull Qwen/Qwen3-4B-GGUF

# skip the prompt, specify variant directly
oxydllm pull Qwen/Qwen3-4B-GGUF --variant Q4_K_M

# full-precision safetensors
oxydllm pull meta-llama/Llama-3.2-1B-Instruct

# gated model with explicit token
oxydllm pull meta-llama/Llama-3.1-8B-Instruct --token hf_xxxxxxxxxxxx

Print all locally available models with their architecture and disk size. Each model is identified by its HuggingFace user/model ID, which is the same string you pass to run, estimate, and rm, and the same one the API expects in the model field. Models pulled with a custom --name appear under that name instead.

oxydllm list

Example output:

  NAME                                   ARCHITECTURE                          SIZE
  ─────────────────────────────────────────────────────────────────────────────────
  allenai/OLMoE-1B-7B-0924-Instruct      OlmoeForCausalLM                  12.89 GB
  google/gemma-3-1b-it                   Gemma3ForCausalLM                  1.86 GB
  meta-llama/Llama-3.2-1B-Instruct       LlamaForCausalLM                   2.30 GB
  microsoft/Phi-3.5-mini-instruct        Phi3ForCausalLM                    7.12 GB
  mistralai/Ministral-3-3B-Instruct-2512 Mistral3ForConditionalGeneration   4.35 GB
  openai/gpt-oss-20b                     GptOssForCausalLM                 12.82 GB
  Qwen/Qwen3-4B-AWQ                      Qwen3ForCausalLM                   2.48 GB
  Qwen/Qwen3-4B-Q4_K_M                   qwen3 (GGUF)                       2.33 GB
  Qwen/Qwen3.5-4B                        Qwen3_5ForConditionalGeneration    8.68 GB

  9 model(s) in ~/.oxydllm/models

Multiple GGUF quantizations stored in the same folder each appear as a separate entry. Like all other subcommands, list accepts --models-dir <DIR> to read from a custom directory instead of the default (~/.oxydllm/models/).

Start an interactive chat session in the terminal. Loads the model directly without starting an HTTP server. Accepts partial model names such as oxydllm run Qwen3-4B, which resolves to the first matching local model.

oxydllm run <model-name> [options]

Examples:

# basic interactive chat
oxydllm run Qwen/Qwen3-4B-Q4_K_M

# longer context with balanced KV quantization
oxydllm run Qwen/Qwen3-4B-Q4_K_M --max-context-len 8192 --kv-quant balanced

# lower temperature for more deterministic output
oxydllm run Qwen/Qwen3-4B-Q4_K_M --temperature 0.2 --top-p 0.9

Print a memory estimate for a model before downloading or running it. Accepts local model names (including partial names with fuzzy matching like run) and HuggingFace repo IDs. For FP8 safetensors models on Apple Silicon, the reported weight size accounts for load-time dequantization to BF16.

oxydllm estimate <model> [options]

Examples:

# estimate for an already-downloaded model
oxydllm estimate Qwen/Qwen3-4B-Q4_K_M

# estimate before downloading, with a larger context window
oxydllm estimate Qwen/Qwen3-4B-GGUF --context-len 8192 --num-sequences 4

Remove a model from disk and deregister from the local registry. Alias: oxydllm remove.

oxydllm rm <model-name> [options]

Examples:

oxydllm rm Qwen/Qwen3-4B-Q4_K_M
oxydllm rm Qwen/Qwen3-4B-Q4_K_M --force

Start the HTTP inference server. When installed via the quick start script, this runs automatically via the OS service manager (launchd on macOS, systemd on Linux), so you do not need to invoke it directly. Use it when building from source or to manually restart the server.

Models load on demand when the first request arrives. No model is specified at startup. Multiple models can reside in memory simultaneously; when the total exceeds --memory-budget, the least-recently-used model is evicted. Models also evict automatically after --keep-alive seconds of inactivity.

The inference engine uses a batched scheduler, where all sequences that are currently active share the same GPU forward pass. This means throughput improves as more requests arrive in parallel rather than degrading to serial execution. When a model loads, the scheduler automatically determines how many sequences can run concurrently based on the available KV cache memory, and logs the result at startup. You can override this limit with --max-num-seqs.

Batching does not change an answer. A request gets the same logits, bit for bit, whether its prompt is prefilled alone, beside other prompts or behind a cached prefix, whether its tokens decode alone or in a batch of up to eight sequences (sixteen on AWQ, thirty-two on GPTQ): each forward is either a prefill or a decode step, never both, and the kernels follow that phase rather than the row count. Past those batch sizes, batched decode can still differ from single-stream decode in the last bits.

Incoming requests are placed in a bounded queue. If the queue is full when a new request arrives, the server responds immediately with HTTP 429 so the client can retry rather than waiting indefinitely. The queue capacity is set by --max-queued-requests and defaults to 200.

Every option can be set as a CLI flag or an environment variable, useful when the server runs as a background service and you want to change settings without editing the unit file directly. CLI flags take priority when both are set.

oxydllm start [options]

Examples:

# default start, scheduler limit computed automatically
oxydllm start

# custom port, longer context
oxydllm start --port 8080 --max-context-len 8192

# memory-capped multi-model server with shorter keep-alive
oxydllm start --memory-budget 24576 --keep-alive 300

# explicit concurrency limit and larger request queue
oxydllm start --max-num-seqs 16 --max-queued-requests 500

# configure via environment variables (same effect as the flags above)
OXYDLLM_MAX_NUM_SEQS=16 OXYDLLM_MAX_CONTEXT_LEN=8192 oxydllm start

# structured JSON logs (also works in service files and docker compose)
LOG_FORMAT=json oxydllm start

# multi-GPU (CUDA)
oxydllm start --devices 0,1

# server with balanced KV quantization
oxydllm start --kv-quant balanced

Update oxydllm to a newer release. Without any flags the command queries the GitHub releases API for the latest stable non-pre-release build and compares its version tag against the version compiled into the installed binary. If the remote tag differs, the update proceeds by downloading and executing the official install.sh script with the appropriate channel variable set; install.sh handles stopping the running service, replacing the binary, and restarting the service automatically.

Passing --pre targets the most recent pre-release build (alpha, beta, or release candidate) instead of the latest stable. The command queries the full releases list, excludes the nightly tag, and picks the first result. Passing --nightly fetches the nightly release and compares its published_at timestamp against OXYDLLM_BUILD_TS, a Unix timestamp baked into the binary at compile time; the update runs only when the remote build is strictly newer than the installed one. When no stable release exists yet the command prints an informational message and exits without error. When the nightly release is temporarily unavailable during a build, the command likewise exits cleanly and suggests retrying in a few minutes. This command is only available in binaries installed via install.sh; source builds print an error and exit without making any changes.

oxydllm update [--pre | --nightly]

Examples:

# update to the latest stable release
oxydllm update

# update to the latest pre-release (alpha/beta/rc)
oxydllm update --pre

# update to the latest nightly build
oxydllm update --nightly

Remove oxydllm from the system. The command stops and removes the OS service before deleting the binary via self-removal: on macOS the launchd agent at ~/Library/LaunchAgents/com.oxydllm.oxydllmd.plist is unloaded with launchctl bootout before the plist is deleted; on Linux the systemd unit is stopped, disabled, the unit file at /etc/systemd/system/oxydllm.service is removed, and systemctl daemon-reload is called. Files that require elevated permissions are removed via sudo automatically. After service cleanup the binary is deleted and the process exits cleanly. A confirmation prompt is always shown before any changes are made.

Passing --purge also removes ~/.oxydllm/, including all downloaded models and configuration data. This operation cannot be undone and is not included in the default removal unless the flag is explicitly set. This command is only available in binaries installed via install.sh; source builds print an error and exit without making any changes.

oxydllm uninstall [--purge]

Examples:

# remove service and binary, keep downloaded models
oxydllm uninstall

# remove everything including all downloaded models and data
oxydllm uninstall --purge

oxydLLM exposes an OpenAI-compatible HTTP API on http://localhost:11313 by default. Because the wire format matches the OpenAI spec, any client library or tool that already speaks to OpenAI (OpenAI's Python SDK, LangChain, LiteLLM, curl) works without code changes. All request and response bodies are JSON.

Main inference endpoint, handles streaming and non-streaming requests, function calling, structured output, and thinking mode. The model loads on first request if not already in memory. See Request fields for the full field reference.

curl http://localhost:11313/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Q4_K_M",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Response:

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1714000000,
  "model": "Qwen/Qwen3-4B-Q4_K_M",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "Hello! How can I help you?"
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 9,
    "total_tokens": 19
  }
}

OpenAI-compatible text embeddings for encoder-only models (BERT/RoBERTa family, e.g. ibm-granite/granite-embedding-125m-english). input accepts a string or an array of strings; vectors are pooled per the model's sentence-transformers config (CLS or mean) and L2-normalised. Encoders load on first use and stay resident; embedding runs interleave safely with chat traffic under the device GPU lock.

curl http://localhost:11313/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ibm-granite/granite-embedding-125m-english",
    "input": ["first text", "second text"]
  }'

Lists all models found in the models directory, whether or not they are currently loaded. This is the endpoint most client libraries call at startup to discover available models.

curl http://localhost:11313/v1/models

Returns only the models that are currently loaded in memory. Useful for monitoring how many models are active and how much of the memory budget is occupied.

curl http://localhost:11313/v1/models/running

Returns metadata for a single model by its ID.

curl 'http://localhost:11313/v1/models/Qwen/Qwen3-4B-Q4_K_M'

Returns 200 OK when the server is up. Use this for readiness probes or to check whether the process started successfully.

curl http://localhost:11313/health

Exposes all runtime metrics in Prometheus text format. Scrape this endpoint with Prometheus, Vector, or any compatible agent. Metrics are always present in the output even before the first request arrives. See Observability for the full metric reference.

curl http://localhost:11313/metrics

Example output:

# HELP oxydllm_ttft_milliseconds Time-to-first-token in milliseconds.
# TYPE oxydllm_ttft_milliseconds histogram
oxydllm_ttft_milliseconds_bucket{model="Qwen/Qwen3-4B-Q4_K_M",le="100"} 3
...
# HELP oxydllm_requests_total Total completed chat completion requests.
# TYPE oxydllm_requests_total counter
oxydllm_requests_total{model="Qwen/Qwen3-4B-Q4_K_M",status="ok"} 5
# HELP oxydllm_queue_depth Current number of sequences in the inference engine.
# TYPE oxydllm_queue_depth gauge
oxydllm_queue_depth 0
# HELP oxydllm_vram_used_bytes Total inference memory in bytes.
# TYPE oxydllm_vram_used_bytes gauge
oxydllm_vram_used_bytes 3.14159e+09

Support is per architecture (the architectures field in a checkpoint's config.json): every checkpoint of a supported architecture loads through the same code path. The architecture table lives on the homepage; the table below lists every checkpoint verified end to end on Apple Silicon, with the weight formats exercised for each. Other sizes and finetunes of a supported architecture are expected to work but are not regularly tested.

On a 24 GB Apple Silicon machine, checkpoints up to about 13 GB of weights serve 4096-token requests with room for several concurrent sequences; the loader refuses what cannot hold a 256-block KV pool beside its weights, such as Qwen3.8-27B Q4_K_M or Mistral 7B in BF16, rather than fail on the first long prompt. GGUF files of a mixture-of-experts architecture are refused by pull from their header: the GGUF runtime serves dense models only.

Multimodal input (vision first, text-only output) and cross-encoder reranking are planned and not yet supported.

The end-to-end sweep labels every failure as runtime, model or budget, and a release ships with runtime defects at zero. The behaviours below belong to the checkpoints: they reproduce at temperature 0 on a server that passes its determinism, batching, streaming and stop checks on the same checkpoint, and where noted llama.cpp gives the same answer to the same prompt and tool definitions. The server does not paper over them.

• Tool and schema adherence is model-dependent. The server implements the protocol, but a model can still ignore tool instructions or emit invalid JSON or tool arguments.
• Llama 3.x (1B, 3B and 8B) write a tool call as JSON in the message text, with finish_reason stop and no tool_calls entry; llama.cpp returns the same text.
• Granite 3.3 (2B and 8B) and Qwen2-1.5B call an offered tool whether or not the question needs it: "What is 2+2?" beside a weather tool returns a get_weather call at a probability near one, and llama.cpp emits the same call. tool_choice: "none", or an allowed_tools subset, is the remedy when a turn must not call.
• Checkpoints under a billion parameters and 2-bit quantizations (Qwen3-0.6B at every quantization, Qwen2.5-1.5B Q2_K) lose a needle after about 1.5k tokens of repeated text, drop simple arithmetic across turns, and answer a one-word question with markdown: on Qwen3-0.6B Q8_0, "Color of a banana? One word." gets One word: **orange**. from this server and from llama.cpp alike, token for token on five such prompts. On Qwen3-1.7B the two engines agree on four prompts of five and split on the multi-turn sum, each giving an answer the other gives in another quantization: a greedy near-tie of the model, not a defect of either. Larger or less quantized checkpoints of the same family pass.
• Reasoning models (Qwen3, Qwen3.5, the DeepSeek-R1 distills, gpt-oss, Gemma 4) spend the completion budget thinking: with a small max_tokens the reply can come back empty with finish_reason length while reasoning_content holds the thought. Raise max_tokens or disable thinking for the turn.

Complete field reference for POST /v1/chat/completions. Standard OpenAI fields behave identically to the OpenAI API.

Essential fields:

Advanced fields:

Function calling lets you describe tools to the model and have it decide when and how to use them. Pass tool definitions in the tools array. When the model decides a tool call is appropriate, it returns a tool_calls array in place of regular text content. Your application executes the function, then sends the result back as a tool role message to continue the conversation. A model whose chat template renders tools itself (Qwen, Llama 3, Mistral, Granite, gemma-4, gpt-oss) sees them in that format and nothing else; a model whose template ignores them is given the definitions and a JSON calling convention in the system prompt instead.

Request with a tool definition:

curl http://localhost:11313/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Q4_K_M",
    "messages": [{"role": "user", "content": "What is the weather in Milan?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {
            "city": {"type": "string", "description": "City name"}
          },
          "required": ["city"]
        }
      }
    }],
    "tool_choice": "auto"
  }'

Response when the model calls the tool (content is null, finish_reason is "tool_calls"):

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_abc123",
        "type": "function",
        "function": {
          "name": "get_weather",
          "arguments": "{\"city\": \"Milan\"}"
        }
      }]
    },
    "finish_reason": "tool_calls"
  }]
}

Send the tool result back to continue:

"messages": [
  {"role": "user",      "content": "What is the weather in Milan?"},
  {"role": "assistant", "content": null, "tool_calls": [...]},
  {
    "role": "tool",
    "tool_call_id": "call_abc123",
    "content": "{\"temp\": 22, \"condition\": \"sunny\"}"
  }
]

tool_choice accepts the following values:

Use response_format to ask the model to respond with valid JSON. The server injects a system instruction telling the model what format is expected. With json_schema, the schema itself is included so the model knows exactly which fields to emit.

Return any valid JSON object:

"response_format": {"type": "json_object"}

Return JSON matching a schema:

"response_format": {
  "type": "json_schema",
  "json_schema": {
    "name": "person",
    "strict": true,
    "schema": {
      "type": "object",
      "properties": {
        "name":  {"type": "string"},
        "age":   {"type": "integer"},
        "email": {"type": "string"}
      },
      "required": ["name", "age"],
      "additionalProperties": false
    }
  }
}

Full example:

curl http://localhost:11313/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Q4_K_M",
    "messages": [{"role": "user", "content": "Extract: John Doe, 34 years old"}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "person",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "age":  {"type": "integer"}
          },
          "required": ["name", "age"],
          "additionalProperties": false
        }
      }
    }
  }'

Some models (such as Qwen3) support extended reasoning, emitting an internal chain-of-thought before producing their final answer. Setting "enable_thinking": true activates this mode. The reasoning trace is returned in a separate reasoning_content field, and the polished final answer appears in content as usual. In streaming mode, reasoning_content tokens arrive in dedicated chunks before content chunks.

curl http://localhost:11313/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Q4_K_M",
    "messages": [{"role": "user", "content": "Explain quantum entanglement"}],
    "enable_thinking": true
  }'

Response:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "reasoning_content": "The user wants a clear explanation of quantum entanglement...",
      "content": "Quantum entanglement is a phenomenon where two particles become correlated..."
    },
    "finish_reason": "stop"
  }]
}

Harmony models (gpt-oss). GPT-OSS models always reason: the architecture has no off switch. Instead of enable_thinking, the reasoning_effort field scales the reasoning depth: low, medium (default), or high. The harmony channel stream is parsed server-side: the analysis channel is returned in reasoning_content, the final channel in content, both in non-streaming and streaming responses.

curl http://localhost:11313/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-oss-20b",
    "messages": [{"role": "user", "content": "Explain quantum entanglement"}],
    "reasoning_effort": "low"
  }'

Set "stream": true to receive the response token-by-token as Server-Sent Events. Each event carries a JSON delta object with the incremental content. The stream ends with data: [DONE].

curl http://localhost:11313/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Q4_K_M",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }'

Stream format:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"!"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Include token usage in the stream:

"stream": true,
"stream_options": {"include_usage": true}

The HTTP API has no authentication by default. Without --api-key (or OXYDLLM_API_KEY) set, any client that can reach the listen port can invoke loaded models, enumerate weights, and read Prometheus metrics. For any deployment that is not a single-user local machine:

1. Configure an API key. Pass --api-key <KEY> or set the environment variable:

OXYDLLM_API_KEY=$(openssl rand -hex 32) oxydllm start

Once set, every request to /v1/* and /metrics must include the key:

curl http://localhost:11313/v1/models \
  -H "Authorization: Bearer $OXYDLLM_API_KEY"

# X-API-Key is also accepted, for clients that cannot set Authorization:
curl http://localhost:11313/v1/models -H "X-API-Key: $OXYDLLM_API_KEY"

Missing or wrong keys return 401 with error.type = "invalid_api_key". /health is intentionally exempt so container and systemd liveness probes keep working without credentials.

2. Restrict network exposure. The default bind is 0.0.0.0:11313, which exposes the server on every interface. For private deployments use a reverse proxy (nginx, Caddy, Traefik) or bind the listener to 127.0.0.1 and tunnel via SSH.

Request-side hardening already enforced by the server, no configuration needed:

• Per-request wall-clock timeout (--request-timeout, default 300s) bounds the time a single chat-completion request can hold a slot. On expiry the engine sequence is aborted; non-streaming responses return 408 Request Timeout, streaming responses emit a final request_timeout error chunk followed by [DONE]. Set to 0 to disable.
• Sampling parameter ranges are validated up front: temperature ∈ [0, 2], top_p ∈ [0, 1], min_p ∈ [0, 1], frequency_penalty/presence_penalty ∈ [-2, 2], top_logprobs ∈ [0, 20], repetition_penalty > 0, n ∈ [1, 128], max_tokens ≥ 1. Out-of-range values return 400 invalid_request_error rather than silently degrading the sampler.
• Context length is checked before the request is queued: a prompt that leaves no room for a completion, against the model's context or against what its KV cache can hold, returns 400 invalid_request_error with code context_length_exceeded. A max_tokens larger than the room left is clamped to it.

The KV cache consists of per-token key and value tensors stored during inference. It grows linearly with context length and can become the dominant memory consumer at long contexts. oxydLLM supports KV cache quantization via TurboQuant, compressing those tensors during the decode phase without touching model weights. The flag is available on both start and run.

Stack Stage-2 QJL key residual quantization on top of any mode for additional compression:

oxydllm start --kv-quant balanced --qjl-quantization

GGUF checkpoints route their matmuls through candle's quantized kernels, which cover every ggml type it can decode. oxydLLM carried its own bf16 Metal kernels for ten of those types until they were measured against candle's on prefill shapes: 0.4 TFLOPS against 3.1, and behind at every batch size down to a single token. They were written when the comparison went the other way and had not been rechecked since. Removing them made prompt processing 3.5 to 4 times faster and decode about 1.2 times faster on every GGUF checkpoint tested, and took 1064 lines with it.

A regression test holds candle's quantized matmul to a floor of 750 GFLOPS at a prefill shape, a quarter of what it measured when the kernels were removed, so if the comparison ever reverses again it fails loudly instead of quietly costing every user four times the prompt latency.

MoE GGUF checkpoints are not yet supported because the per-expert quantized layout differs. A GGUF whose tensors use quantizations candle cannot decode, which is how mixed-precision builds marked UD, AD or IQ are usually put together, is refused by reading its header before the download rather than after fifteen gigabytes of it.

GGUF loading itself uses a zero-copy mmap path: the file is mapped with memmap2 and tensor slices reference the mapped pages directly; the only copy is the single memcpy into a Metal buffer per tensor, parallelised with rayon. End-to-end cold load + first inference on Qwen3-4B-Q4_K_M drops from ~9.4 s to ~2.7 s.

oxydLLM auto-detects AWQ-quantized safetensors checkpoints (autoawq GEMM layout, group_size typically 128, 4-bit and 8-bit) and routes them through a fused W4A16 / W8A16 path on Metal. No CLI flag is required: drop the checkpoint into the models directory and run it like any other.

oxydllm pull Qwen/Qwen3-4B-AWQ
oxydllm run Qwen/Qwen3-4B-AWQ

At load time you will see log lines confirming the path:

INFO AWQ checkpoint detected (W4A16 fused matmul on Metal) quant=awq version=gemm bits=4 group_size=128
INFO Packed-quant attention loader engaged group_size=128 qkv_fused=true bias_present=false quant=awq
INFO Packed-quant FFN loader engaged (separate gate/up tensors) intermediate_size=9728 gate_up_fused=true quant=awq

AWQ is selected per-projection on tensor presence rather than a global flag. A linear layer takes the AWQ path when all three of {prefix}.qweight, {prefix}.qzeros, and {prefix}.scales are present. The quantization_config block in config.json is validated up-front: quant_method = "awq" with bits in {4, 8} and version = "gemm" is accepted; other variants (AWQ GEMV, AWQ Marlin) fail before any tensor I/O with an actionable error.

On Metal + BF16 activations, packed weights stay resident: the M=1 decode forward calls a fused split-K w{4,8}a16_gemv_* kernel that reads packed nibbles directly from HBM and accumulates into an F32 buffer with atomic adds; M>1 prefill calls dequantize_w{4,8}_* to produce a transient bf16 weight that then feeds candle's tuned matmul. The candle QMatMul fallback is dropped, so Qwen3-4B-AWQ's resident weight memory drops from ~7.5 GB (fp16 dequant-at-load) to ~2.5 GB. CPU and non-bf16 paths still dequantize at load and execute as a standard Linear.

Inside a single transformer block all projections must share the same format. If q_proj ships as AWQ but k_proj does not (or analogous mismatch in FFN), the loader fails with a message naming the missing tensor. This catches partially-quantized checkpoints early instead of producing silently wrong outputs at runtime.

GPTQ checkpoints (auto-gptq layout, 4-bit and 8-bit) with desc_act = false are supported on Metal via a dedicated resident kernel family. The runtime selects GPTQ vs AWQ from quantization_config.quant_method; g_idx is loaded but ignored on the supported path (only sequential routing is wired). Asymmetric (sym = false) and symmetric (sym = true) checkpoints both work.

oxydllm pull Qwen/Qwen3-0.6B-GPTQ-Int8
oxydllm run Qwen/Qwen3-0.6B-GPTQ-Int8

The kernel layout differs from AWQ in two ways: qweight is packed along in_features instead of out_features, and the zero-point convention is val = scale × (q - (zero + 1)). The Metal kernels reflect this with a gptq_gemv_impl<T, BITS> template that dispatches one thread per output column (versus AWQ's one thread per packed-out word). Decode throughput on Qwen3-0.6B-GPTQ-Int8 is ~89 tok/s on the M5 reference machine; the same checkpoint runs at ~50 tok/s when forced through the dequant-at-load fallback (CPU device or F32 dtype).

desc_act = true (act-order) is rejected at load with an actionable error. Mixtral and DeepSeek-V2/V3 GPTQ layouts are not supported (different tensor naming or MLA attention).

FP8 (E4M3) checkpoints with per-tensor or block-wise inverse scales (weight_block_size, DeepSeek- / Qwen3-FP8 style) are supported on every backend. On Metal, where the GPU has no FP8 compute kernels, weights are dequantized to BF16 at load time; the weight × scale_inv multiply is performed in F32 (not BF16) to avoid mantissa loss across deep block-wise rescaling. On CUDA / CPU a Level-2 resident path keeps FP8 weights packed and dequantizes on the fly in Fp8Linear::forward.

GPT-OSS checkpoints ship their MoE expert weights in MXFP4 (OCP microscaling FP4): blocks of 32 E2M1 values sharing one E8M0 exponent byte. Expert weights stay packed on Metal (dequantizing gpt-oss-20b's ~19 B expert parameters to BF16 would need ~38 GB) and run through fused dequant kernels: a batched GEMV for decode and a tiled GEMM for prefill. Attention, router, embeddings and the LM head remain BF16 as shipped. openai/gpt-oss-20b loads in ~13 GB resident on a 24 GB Apple Silicon machine. gpt-oss-120b shares the same architecture and should load on machines with enough memory, but is untested.

oxydllm pull openai/gpt-oss-20b
oxydllm run openai/gpt-oss-20b

oxydLLM supports sparsely-gated MoE checkpoints with the standard top-k softmax routing used by Qwen3MoeForCausalLM, OlmoeForCausalLM, Qwen3_5MoeForConditionalGeneration (Qwen3.6: adds a Qwen2-MoE-style shared expert, always active and sigmoid-gated, auto-detected from tensor presence), and GptOssForCausalLM (gpt-oss adds MXFP4 expert weights, interleaved gate/up projections with a clamped swiglu activation, and a router bias; see MXFP4 weights). Each transformer block holds a MoeFeedForward with a linear router (mlp.gate.weight) and N expert FFNs (mlp.experts.{e}.{gate,up,down}_proj.weight). The router scores are softmaxed in F32 for numerical stability across many experts, then a top-k selection is built with arg_sort_last_dim + gather; norm_topk_prob from config.json controls whether the top-k weights are renormalised to sum to 1 per token (true on Qwen3-MoE / Mixtral, false on OLMoE-1B-7B).

oxydllm pull allenai/OLMoE-1B-7B-0924-Instruct
oxydllm run allenai/OLMoE-1B-7B-0924-Instruct

Dispatch strategy. The forward pass picks one of two paths per call based on n_tokens vs top_k:

QK-norm layout. OLMoE applies q_norm / k_norm to the flat [B, T, hidden] tensor before reshape into heads (weight shape [hidden]), whereas Qwen3 / Gemma3 apply them per-head on [B, H, T, head_dim] (weight shape [head_dim]). The two layouts compute variance over different axes and are not interchangeable. Attention::load inspects the q_norm weight tensor's elem_count and selects the correct path automatically; there is no config flag.

Not yet supported. Mixtral (block_sparse_moe.experts.{e}.{w1,w2,w3} tensor naming), DeepSeek-V2/V3 (MoE + MLA attention), and GGUF-quantized MoE checkpoints all fall outside the current loader.

MoE checkpoints larger than memory can run by keeping only the router-selected experts resident. At load time oxydLLM sums the checkpoint's expert and non-expert tensor sizes (scaled to the runtime dtype, so an FP8 file counts double once dequantized to BF16) and compares them against the memory envelope: the operator's --memory-budget when set, otherwise the kernel's own availability metric (kern.memorystatus_level on macOS, which also counts purgeable and compressor-reclaimable pages). If everything fits, the model loads fully resident as before, which is always fastest. If it does not, expert streaming engages automatically and the decision is logged.

When streaming, attention, norms, routers, embeddings, and shared experts load resident; the expert weights stay on disk. The safetensors headers are parsed once, and misses are fetched with parallel uncached positioned reads (one task per tensor, F_NOCACHE on macOS so the page cache never holds a second copy of the expert pool) and built in parallel in disk order. Hits come from an LRU cache whose byte budget is whatever memory remains after the resident weights and a fixed headroom (override with --expert-cache-mb); evicted experts are released in batches behind a single synchronization every 512 MB. A model that cannot fit even its non-expert weights plus a minimum cache is rejected with a clear error instead of thrashing. On unload the server logs the cache hit rate.

Block-scaled FP8 experts (Qwen3.6 family) cache in their file encoding: raw F8 bytes plus the scale grid, half the bytes of the dequantized form, so double the experts per cache budget. Decode-sized batches run native F8 GEMV kernels that read the bytes and block scales directly with a fixed-order deterministic reduction; larger batches dequantize on the device through a kernel proven bitwise identical to the resident loader's cast chain. Measured on Qwen3.6-35B-A3B-FP8 with a pinned 3.4 GB cache: 4.0 tok/s decode on a 24 GB machine.

The Qwen3.5 family is a hybrid model: most layers replace softmax attention with a Gated DeltaNet linear-attention mixer that carries a fixed-size recurrent state instead of a growing KV cache, while the remaining layers use gated full attention. Both mixer kinds live in the same transformer block and share the paged cache infrastructure. The family is text-only.

The DeltaNet path runs a chunked scan during prefill and a recurrent state update during decode (stored in the paged cache in place of K/V for those layers), with a short causal depthwise convolution on the q/k/v projections, Gemma-style gated RMSNorm, and partial RoPE applied only on the attention layers. Per-layer mixer kinds and the shared DeltaNet geometry come from the model config; no flag is required.

Supported formats: BF16 safetensors, compressed-tensors INT4 (fully quantized, and mixed BF16 DeltaNet + INT4 attention/MLP), and GGUF (qwen35 architecture). Correctness is covered by GatedDeltaNet golden fixtures checked against the reference implementation, plus an adversarial end-to-end test battery.

oxydllm pull Qwen/Qwen3.5-4B
oxydllm run Qwen/Qwen3.5-4B

The fused Metal Scaled Dot-Product Attention kernel accelerates the hot attention path on Apple Silicon. It currently supports a fixed set of head dimensions: 32, 64, 72, 80, 96, 128, 256. Models with other head dimensions remain functionally correct but fall back to the non-fused attention path with a measurable throughput cost.

Prometheus metrics are exposed at GET /metrics. The table below describes every metric.

Example Prometheus queries:

# 95th-percentile TTFT over the last 5 minutes
histogram_quantile(0.95, rate(oxydllm_ttft_milliseconds_bucket[5m]))

# Prefix cache hit ratio
rate(oxydllm_prefix_cache_requests_total{result="hit"}[5m])
  / rate(oxydllm_prefix_cache_requests_total[5m])

# Request throughput by status
rate(oxydllm_requests_total[1m])

Structured logs and request tracing. Every request is assigned a request_id (UUID v4) at the HTTP handler entry point. This ID appears in all log events for that request, from template rendering to the final generated token, making it possible to trace a single request end-to-end even under concurrent load:

grep 'request_id=abc-123' app.log

By default logs use a compact human-readable format. Setting LOG_FORMAT=json switches to one JSON object per line, compatible with Loki, Datadog, AWS CloudWatch, and jq. The variable is read at startup and applies to all commands, not just start:

LOG_FORMAT=json oxydllm start
LOG_FORMAT=json oxydllm run Qwen/Qwen3-4B-Q4_K_M

Each log line becomes a self-contained JSON object:

{"timestamp":"2024-01-01T12:00:00.123Z","level":"INFO","fields":{"request_id":"abc-123","ttft_ms":123.4,"model_id":"Qwen/Qwen3-4B-Q4_K_M"},"message":"first token emitted"}

Query a single request's lifecycle in Loki:

{app="oxydllm"} | json | request_id="abc-123"

Or filter locally with jq:

oxydllm start 2>&1 | jq 'select(.fields.request_id=="abc-123")'

Distributed tracing (OpenTelemetry). For per-request visibility that aggregate metrics cannot give (the queue/prefill vs decode breakdown of a single request, the slowest individual requests, correlation by request_id), oxydLLM can export OpenTelemetry traces over OTLP/HTTP. This is additive: the Prometheus endpoint above is unchanged, and tracing stays off unless an endpoint is configured. Point it at any OTLP/HTTP collector (Grafana Alloy, the OpenTelemetry Collector, Jaeger, Grafana Tempo):

oxydllm start --otel-endpoint http://localhost:4318
# or, equivalently, via environment:
OXYDLLM_OTEL_ENDPOINT=http://localhost:4318 oxydllm start

The value is treated as the OTLP base, so traces are sent to <endpoint>/v1/traces (matching the OTEL_EXPORTER_OTLP_ENDPOINT convention, which is also honored). Spans are batched in the background and flushed on shutdown.

Each request produces a trace tree http.request → inference.request → decode: the gap before decode starts is the time-to-first-token. Span attributes include request_id, model_id, prompt_tokens, max_tokens, ttft_ms, completion_tokens, tokens_per_second, finish_reason, and outcome (ok / error / unloaded); the first token emitted and request completed events are attached to the span. A W3C traceparent header on the incoming request is honored, so the spans join the caller's distributed trace (e.g. through an upstream gateway); without it, a fresh trace is started. Loading a model emits a separate model.load span (weight/KV bytes, layer count, warm-up time). In Grafana Tempo, search service.name = oxydllm and filter by any of those attributes.

Configure oxydLLM without passing CLI flags. Useful for containers, shell profiles, and CI environments. Every start flag has a matching OXYDLLM_* variable (see the start table); the variables below are the ones without a CLI flag, plus the most commonly set ones.

Off unless set, and none of them change what the server computes.