See the homepage for the quick start installation. This section covers building from source and service deployment options.

Requires the Rust toolchain.

git clone https://github.com/giovannifil-64/oxydllm.git
cd oxydllm

For using the inference engine, you must build with either the metal or cuda feature depending on your target.

If you are on a Apple Silicon Mac, run:

cargo build --release --features metal

On the other hand, for NVIDIA CUDA you need also to set the compute capability:

CUDA_COMPUTE_CAP=<value> cargo build --release --features cuda

Replace <value> with the compute capability of your GPU.

Here is a list of the supported compute capabilities and their corresponding GPUs

After installation via the quick start script, the server runs automatically as a system service (launchd on macOS, systemd on Linux). To manually restart or stop:

Download a model from HuggingFace. For GGUF repos, an interactive variant selection prompt appears unless --variant is specified. Variants already on disk are listed with a check mark and excluded from choices, re-downloading requires --force.

oxydllm pull <user/model> [options]

Examples:

# GGUF model, interactive variant selection
oxydllm pull Qwen/Qwen3-4B-GGUF

# skip the prompt, specify variant directly
oxydllm pull Qwen/Qwen3-4B-GGUF --variant Q4_K_M

# full-precision safetensors
oxydllm pull meta-llama/Llama-3.2-1B-Instruct

# gated model with explicit token
oxydllm pull meta-llama/Llama-3.1-8B-Instruct --token hf_xxxxxxxxxxxx

Print all locally available models with their architecture and disk size. Each model is identified by its HuggingFace user/model ID, which is the same string you pass to run, estimate, and rm, and the same one the API expects in the model field. Models pulled with a custom --name appear under that name instead.

oxydllm list

Example output:

  NAME                                   ARCHITECTURE                          SIZE
  ─────────────────────────────────────────────────────────────────────────────────
  allenai/OLMoE-1B-7B-0924-Instruct      OlmoeForCausalLM                  12.89 GB
  google/gemma-3-1b-it                   Gemma3ForCausalLM                  1.86 GB
  meta-llama/Llama-3.2-1B-Instruct       LlamaForCausalLM                   2.30 GB
  microsoft/Phi-3.5-mini-instruct        Phi3ForCausalLM                    7.12 GB
  mistralai/Ministral-3-3B-Instruct-2512 Mistral3ForConditionalGeneration   4.35 GB
  openai/gpt-oss-20b                     GptOssForCausalLM                 12.82 GB
  Qwen/Qwen3-4B-AWQ                      Qwen3ForCausalLM                   2.48 GB
  Qwen/Qwen3-4B-Q4_K_M                   qwen3 (GGUF)                       2.33 GB
  Qwen/Qwen3.5-4B                        Qwen3_5ForConditionalGeneration    8.68 GB

  9 model(s) in ~/.oxydllm/models

Multiple GGUF quantizations stored in the same folder each appear as a separate entry. Like all other subcommands, list accepts --models-dir <DIR> to read from a custom directory instead of the default (~/.oxydllm/models/).

Start an interactive chat session in the terminal. Loads the model directly without starting an HTTP server. Accepts partial model names such as oxydllm run Qwen3-4B, which resolves to the first matching local model.

oxydllm run <model-name> [options]

Examples:

# basic interactive chat
oxydllm run Qwen/Qwen3-4B-Q4_K_M

# longer context with balanced KV quantization
oxydllm run Qwen/Qwen3-4B-Q4_K_M --max-context-len 8192 --kv-quant balanced

# lower temperature for more deterministic output
oxydllm run Qwen/Qwen3-4B-Q4_K_M --temperature 0.2 --top-p 0.9

Print a memory estimate for a model before downloading or running it. Accepts local model names (including partial names with fuzzy matching like run) and HuggingFace repo IDs. For FP8 safetensors models on Apple Silicon, the reported weight size accounts for load-time dequantization to BF16.

oxydllm estimate <model> [options]

Examples:

# estimate for an already-downloaded model
oxydllm estimate Qwen/Qwen3-4B-Q4_K_M

# estimate before downloading, with a larger context window
oxydllm estimate Qwen/Qwen3-4B-GGUF --context-len 8192 --num-sequences 4

Remove a model from disk and deregister from the local registry. Alias: oxydllm remove.

oxydllm rm <model-name> [options]

Examples:

oxydllm rm Qwen/Qwen3-4B-Q4_K_M
oxydllm rm Qwen/Qwen3-4B-Q4_K_M --force

Start the HTTP inference server. When installed via the quick start script, this runs automatically via the OS service manager (launchd on macOS, systemd on Linux), so you do not need to invoke it directly. Use it when building from source or to manually restart the server.

Models load on demand when the first request arrives. No model is specified at startup. Multiple models can reside in memory simultaneously; when the total exceeds --memory-budget, the least-recently-used model is evicted. Models also evict automatically after --keep-alive seconds of inactivity.

The inference engine uses a batched scheduler, where all sequences that are currently active share the same GPU forward pass. This means throughput improves as more requests arrive in parallel rather than degrading to serial execution. When a model loads, the scheduler automatically determines how many sequences can run concurrently based on the available KV cache memory, and logs the result at startup. You can override this limit with --max-num-seqs.

Incoming requests are placed in a bounded queue. If the queue is full when a new request arrives, the server responds immediately with HTTP 429 so the client can retry rather than waiting indefinitely. The queue capacity is set by --max-queued-requests and defaults to 200.

Every option can be set as a CLI flag or an environment variable, useful when the server runs as a background service and you want to change settings without editing the unit file directly. CLI flags take priority when both are set.

oxydllm start [options]

Examples:

# default start, scheduler limit computed automatically
oxydllm start

# custom port, longer context
oxydllm start --port 8080 --max-context-len 8192

# memory-capped multi-model server with shorter keep-alive
oxydllm start --memory-budget 24576 --keep-alive 300

# explicit concurrency limit and larger request queue
oxydllm start --max-num-seqs 16 --max-queued-requests 500

# configure via environment variables (same effect as the flags above)
OXYDLLM_MAX_NUM_SEQS=16 OXYDLLM_MAX_CONTEXT_LEN=8192 oxydllm start

# structured JSON logs (also works in service files and docker compose)
LOG_FORMAT=json oxydllm start

# multi-GPU (CUDA)
oxydllm start --devices 0,1

# server with balanced KV quantization
oxydllm start --kv-quant balanced

Update oxydllm to a newer release. Without any flags the command queries the GitHub releases API for the latest stable non-pre-release build and compares its version tag against the version compiled into the installed binary. If the remote tag differs, the update proceeds by downloading and executing the official install.sh script with the appropriate channel variable set; install.sh handles stopping the running service, replacing the binary, and restarting the service automatically.

Passing --pre targets the most recent pre-release build (alpha, beta, or release candidate) instead of the latest stable. The command queries the full releases list, excludes the nightly tag, and picks the first result. Passing --nightly fetches the nightly release and compares its published_at timestamp against OXYDLLM_BUILD_TS, a Unix timestamp baked into the binary at compile time; the update runs only when the remote build is strictly newer than the installed one. When no stable release exists yet the command prints an informational message and exits without error. When the nightly release is temporarily unavailable during a build, the command likewise exits cleanly and suggests retrying in a few minutes. This command is only available in binaries installed via install.sh; source builds print an error and exit without making any changes.

oxydllm update [--pre | --nightly]

Examples:

# update to the latest stable release
oxydllm update

# update to the latest pre-release (alpha/beta/rc)
oxydllm update --pre

# update to the latest nightly build
oxydllm update --nightly

Remove oxydllm from the system. The command stops and removes the OS service before deleting the binary via self-removal: on macOS the launchd agent at ~/Library/LaunchAgents/com.oxydllm.oxydllmd.plist is unloaded with launchctl bootout before the plist is deleted; on Linux the systemd unit is stopped, disabled, the unit file at /etc/systemd/system/oxydllm.service is removed, and systemctl daemon-reload is called. Files that require elevated permissions are removed via sudo automatically. After service cleanup the binary is deleted and the process exits cleanly. A confirmation prompt is always shown before any changes are made.

Passing --purge also removes ~/.oxydllm/, including all downloaded models and configuration data. This operation cannot be undone and is not included in the default removal unless the flag is explicitly set. This command is only available in binaries installed via install.sh; source builds print an error and exit without making any changes.

oxydllm uninstall [--purge]

Examples:

# remove service and binary, keep downloaded models
oxydllm uninstall

# remove everything including all downloaded models and data
oxydllm uninstall --purge

oxydLLM exposes an OpenAI-compatible HTTP API on http://localhost:11313 by default. Because the wire format matches the OpenAI spec, any client library or tool that already speaks to OpenAI (OpenAI's Python SDK, LangChain, LiteLLM, curl) works without code changes. All request and response bodies are JSON.

Main inference endpoint, handles streaming and non-streaming requests, function calling, structured output, and thinking mode. The model loads on first request if not already in memory. See Request fields for the full field reference.

curl http://localhost:11313/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Q4_K_M",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Response:

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1714000000,
  "model": "Qwen/Qwen3-4B-Q4_K_M",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "Hello! How can I help you?"
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 9,
    "total_tokens": 19
  }
}

Lists all models found in the models directory, whether or not they are currently loaded. This is the endpoint most client libraries call at startup to discover available models.

curl http://localhost:11313/v1/models

Returns only the models that are currently loaded in memory. Useful for monitoring how many models are active and how much of the memory budget is occupied.

curl http://localhost:11313/v1/models/running

Returns metadata for a single model by its ID.

curl 'http://localhost:11313/v1/models/Qwen/Qwen3-4B-Q4_K_M'

Returns 200 OK when the server is up. Use this for readiness probes or to check whether the process started successfully.

curl http://localhost:11313/health

Exposes all runtime metrics in Prometheus text format. Scrape this endpoint with Prometheus, Vector, or any compatible agent. Metrics are always present in the output even before the first request arrives. See Observability for the full metric reference.

curl http://localhost:11313/metrics

Example output:

# HELP oxydllm_ttft_milliseconds Time-to-first-token in milliseconds.
# TYPE oxydllm_ttft_milliseconds histogram
oxydllm_ttft_milliseconds_bucket{model="Qwen/Qwen3-4B-Q4_K_M",le="100"} 3
...
# HELP oxydllm_requests_total Total completed chat completion requests.
# TYPE oxydllm_requests_total counter
oxydllm_requests_total{model="Qwen/Qwen3-4B-Q4_K_M",status="ok"} 5
# HELP oxydllm_queue_depth Current number of sequences in the inference engine.
# TYPE oxydllm_queue_depth gauge
oxydllm_queue_depth 0
# HELP oxydllm_vram_used_bytes Total inference memory in bytes.
# TYPE oxydllm_vram_used_bytes gauge
oxydllm_vram_used_bytes 3.14159e+09

Complete field reference for POST /v1/chat/completions. Standard OpenAI fields behave identically to the OpenAI API.

Essential fields:

Advanced fields:

Function calling lets you describe tools to the model and have it decide when and how to use them. Pass tool definitions in the tools array. When the model decides a tool call is appropriate, it returns a tool_calls array in place of regular text content. Your application executes the function, then sends the result back as a tool role message to continue the conversation.

Request with a tool definition:

curl http://localhost:11313/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Q4_K_M",
    "messages": [{"role": "user", "content": "What is the weather in Milan?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {
            "city": {"type": "string", "description": "City name"}
          },
          "required": ["city"]
        }
      }
    }],
    "tool_choice": "auto"
  }'

Response when the model calls the tool (content is null, finish_reason is "tool_calls"):

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_abc123",
        "type": "function",
        "function": {
          "name": "get_weather",
          "arguments": "{\"city\": \"Milan\"}"
        }
      }]
    },
    "finish_reason": "tool_calls"
  }]
}

Send the tool result back to continue:

"messages": [
  {"role": "user",      "content": "What is the weather in Milan?"},
  {"role": "assistant", "content": null, "tool_calls": [...]},
  {
    "role": "tool",
    "tool_call_id": "call_abc123",
    "content": "{\"temp\": 22, \"condition\": \"sunny\"}"
  }
]

tool_choice accepts the following values:

Use response_format to ask the model to respond with valid JSON. The server injects a system instruction telling the model what format is expected. With json_schema, the schema itself is included so the model knows exactly which fields to emit.

Return any valid JSON object:

"response_format": {"type": "json_object"}

Return JSON matching a schema:

"response_format": {
  "type": "json_schema",
  "json_schema": {
    "name": "person",
    "strict": true,
    "schema": {
      "type": "object",
      "properties": {
        "name":  {"type": "string"},
        "age":   {"type": "integer"},
        "email": {"type": "string"}
      },
      "required": ["name", "age"],
      "additionalProperties": false
    }
  }
}

Full example:

curl http://localhost:11313/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Q4_K_M",
    "messages": [{"role": "user", "content": "Extract: John Doe, 34 years old"}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "person",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "age":  {"type": "integer"}
          },
          "required": ["name", "age"],
          "additionalProperties": false
        }
      }
    }
  }'

Some models (such as Qwen3) support extended reasoning, emitting an internal chain-of-thought before producing their final answer. Setting "enable_thinking": true activates this mode. The reasoning trace is returned in a separate reasoning_content field, and the polished final answer appears in content as usual. In streaming mode, reasoning_content tokens arrive in dedicated chunks before content chunks.

curl http://localhost:11313/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Q4_K_M",
    "messages": [{"role": "user", "content": "Explain quantum entanglement"}],
    "enable_thinking": true
  }'

Response:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "reasoning_content": "The user wants a clear explanation of quantum entanglement...",
      "content": "Quantum entanglement is a phenomenon where two particles become correlated..."
    },
    "finish_reason": "stop"
  }]
}

Harmony models (gpt-oss). GPT-OSS models always reason: the architecture has no off switch. Instead of enable_thinking, the reasoning_effort field scales the reasoning depth: low, medium (default), or high. The harmony channel stream is parsed server-side: the analysis channel is returned in reasoning_content, the final channel in content, both in non-streaming and streaming responses.

curl http://localhost:11313/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-oss-20b",
    "messages": [{"role": "user", "content": "Explain quantum entanglement"}],
    "reasoning_effort": "low"
  }'

Set "stream": true to receive the response token-by-token as Server-Sent Events. Each event carries a JSON delta object with the incremental content. The stream ends with data: [DONE].

curl http://localhost:11313/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Q4_K_M",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }'

Stream format:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"!"}}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Include token usage in the stream:

"stream": true,
"stream_options": {"include_usage": true}

The HTTP API has no authentication by default. Without --api-key (or OXYDLLM_API_KEY) set, any client that can reach the listen port can invoke loaded models, enumerate weights, and read Prometheus metrics. For any deployment that is not a single-user local machine:

1. Configure an API key. Pass --api-key <KEY> or set the environment variable:

OXYDLLM_API_KEY=$(openssl rand -hex 32) oxydllm start

Once set, every request to /v1/* and /metrics must include the key:

curl http://localhost:11313/v1/models \
  -H "Authorization: Bearer $OXYDLLM_API_KEY"

# X-API-Key is also accepted, for clients that cannot set Authorization:
curl http://localhost:11313/v1/models -H "X-API-Key: $OXYDLLM_API_KEY"

Missing or wrong keys return 401 with error.type = "invalid_api_key". /health is intentionally exempt so container and systemd liveness probes keep working without credentials.

2. Restrict network exposure. The default bind is 0.0.0.0:11313, which exposes the server on every interface. For private deployments use a reverse proxy (nginx, Caddy, Traefik) or bind the listener to 127.0.0.1 and tunnel via SSH.

Request-side hardening already enforced by the server, no configuration needed:

• Per-request wall-clock timeout (--request-timeout, default 300s) bounds the time a single chat-completion request can hold a slot. On expiry the engine sequence is aborted; non-streaming responses return 408 Request Timeout, streaming responses emit a final request_timeout error chunk followed by [DONE]. Set to 0 to disable.
• Sampling parameter ranges are validated up front: temperature ∈ [0, 2], top_p ∈ [0, 1], min_p ∈ [0, 1], frequency_penalty/presence_penalty ∈ [-2, 2], top_logprobs ∈ [0, 20], repetition_penalty > 0, n ∈ [1, 128], max_tokens ≥ 1. Out-of-range values return 400 invalid_request_error rather than silently degrading the sampler.

The KV cache consists of per-token key and value tensors stored during inference. It grows linearly with context length and can become the dominant memory consumer at long contexts. oxydLLM supports KV cache quantization via TurboQuant, compressing those tensors during the decode phase without touching model weights. The flag is available on both start and run.

Stack Stage-2 QJL key residual quantization on top of any mode for additional compression:

oxydllm start --kv-quant balanced --qjl-quantization

GGUF checkpoints loaded on Apple Silicon route through a bf16-aware Metal fast path that covers ten quant types: Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, about 98% of mainstream GGUF checkpoints. Each quant has a dedicated gguf_*_gemv_bf16 kernel for M=1 decode and a fused gguf_*_mul_mm_bf16 kernel for M>1 prefill that dequantizes inline (no transient bf16 weight tensor is materialised). The fast path activates automatically when the activation dtype is bf16 on a Metal device; the candle QMatMul fallback is dropped at the same time, so weights are never held twice in memory.

i-quants (IQ*) and ternary (TQ*) types are not in the fast path; they fall back to candle's reference F32 GEMV. MoE GGUF checkpoints are not yet supported because the per-expert quantized layout differs.

GGUF loading itself uses a zero-copy mmap path: the file is mapped with memmap2 and tensor slices reference the mapped pages directly; the only copy is the single memcpy into a Metal buffer per tensor, parallelised with rayon. End-to-end cold load + first inference on Qwen3-4B-Q4_K_M drops from ~9.4 s to ~2.7 s.

oxydLLM auto-detects AWQ-quantized safetensors checkpoints (autoawq GEMM layout, group_size typically 128, 4-bit and 8-bit) and routes them through a fused W4A16 / W8A16 path on Metal. No CLI flag is required: drop the checkpoint into the models directory and run it like any other.

oxydllm pull Qwen/Qwen3-4B-AWQ
oxydllm run Qwen/Qwen3-4B-AWQ

At load time you will see log lines confirming the path:

INFO AWQ checkpoint detected (W4A16 fused matmul on Metal) quant=awq version=gemm bits=4 group_size=128
INFO Packed-quant attention loader engaged group_size=128 qkv_fused=true bias_present=false quant=awq
INFO Packed-quant FFN loader engaged (separate gate/up tensors) intermediate_size=9728 gate_up_fused=true quant=awq

AWQ is selected per-projection on tensor presence rather than a global flag. A linear layer takes the AWQ path when all three of {prefix}.qweight, {prefix}.qzeros, and {prefix}.scales are present. The quantization_config block in config.json is validated up-front: quant_method = "awq" with bits in {4, 8} and version = "gemm" is accepted; other variants (AWQ GEMV, AWQ Marlin) fail before any tensor I/O with an actionable error.

On Metal + BF16 activations, packed weights stay resident: the M=1 decode forward calls a fused split-K w{4,8}a16_gemv_* kernel that reads packed nibbles directly from HBM and accumulates into an F32 buffer with atomic adds; M>1 prefill calls dequantize_w{4,8}_* to produce a transient bf16 weight that then feeds candle's tuned matmul. The candle QMatMul fallback is dropped, so Qwen3-4B-AWQ's resident weight memory drops from ~7.5 GB (fp16 dequant-at-load) to ~2.5 GB. CPU and non-bf16 paths still dequantize at load and execute as a standard Linear.

Inside a single transformer block all projections must share the same format. If q_proj ships as AWQ but k_proj does not (or analogous mismatch in FFN), the loader fails with a message naming the missing tensor. This catches partially-quantized checkpoints early instead of producing silently wrong outputs at runtime.

GPTQ checkpoints (auto-gptq layout, 4-bit and 8-bit) with desc_act = false are supported on Metal via a dedicated resident kernel family. The runtime selects GPTQ vs AWQ from quantization_config.quant_method; g_idx is loaded but ignored on the supported path (only sequential routing is wired). Asymmetric (sym = false) and symmetric (sym = true) checkpoints both work.

oxydllm pull Qwen/Qwen3-0.6B-GPTQ-Int8
oxydllm run Qwen/Qwen3-0.6B-GPTQ-Int8

The kernel layout differs from AWQ in two ways: qweight is packed along in_features instead of out_features, and the zero-point convention is val = scale × (q - (zero + 1)). The Metal kernels reflect this with a gptq_gemv_impl<T, BITS> template that dispatches one thread per output column (versus AWQ's one thread per packed-out word). Decode throughput on Qwen3-0.6B-GPTQ-Int8 is ~89 tok/s on the M5 reference machine; the same checkpoint runs at ~50 tok/s when forced through the dequant-at-load fallback (CPU device or F32 dtype).

desc_act = true (act-order) is rejected at load with an actionable error. Mixtral and DeepSeek-V2/V3 GPTQ layouts are not supported (different tensor naming or MLA attention).

FP8 (E4M3) checkpoints with per-tensor or block-wise inverse scales (weight_block_size, DeepSeek- / Qwen3-FP8 style) are supported on every backend. On Metal, where the GPU has no FP8 compute kernels, weights are dequantized to BF16 at load time; the weight × scale_inv multiply is performed in F32 (not BF16) to avoid mantissa loss across deep block-wise rescaling. On CUDA / CPU a Level-2 resident path keeps FP8 weights packed and dequantizes on the fly in Fp8Linear::forward.

GPT-OSS checkpoints ship their MoE expert weights in MXFP4 (OCP microscaling FP4): blocks of 32 E2M1 values sharing one E8M0 exponent byte. Expert weights stay packed on Metal (dequantizing gpt-oss-20b's ~19 B expert parameters to BF16 would need ~38 GB) and run through fused dequant kernels: a batched GEMV for decode and a tiled GEMM for prefill. Attention, router, embeddings and the LM head remain BF16 as shipped. openai/gpt-oss-20b loads in ~13 GB resident on a 24 GB Apple Silicon machine. gpt-oss-120b shares the same architecture and should load on machines with enough memory, but is untested.

oxydllm pull openai/gpt-oss-20b
oxydllm run openai/gpt-oss-20b

oxydLLM supports sparsely-gated MoE checkpoints with the standard top-k softmax routing used by Qwen3MoeForCausalLM, OlmoeForCausalLM, and GptOssForCausalLM (gpt-oss adds MXFP4 expert weights, interleaved gate/up projections with a clamped swiglu activation, and a router bias; see MXFP4 weights). Each transformer block holds a MoeFeedForward with a linear router (mlp.gate.weight) and N expert FFNs (mlp.experts.{e}.{gate,up,down}_proj.weight). The router scores are softmaxed in F32 for numerical stability across many experts, then a top-k selection is built with arg_sort_last_dim + gather; norm_topk_prob from config.json controls whether the top-k weights are renormalised to sum to 1 per token (true on Qwen3-MoE / Mixtral, false on OLMoE-1B-7B).

oxydllm pull allenai/OLMoE-1B-7B-0924-Instruct
oxydllm run allenai/OLMoE-1B-7B-0924-Instruct

Dispatch strategy. The forward pass picks one of two paths per call based on n_tokens vs top_k:

QK-norm layout. OLMoE applies q_norm / k_norm to the flat [B, T, hidden] tensor before reshape into heads (weight shape [hidden]), whereas Qwen3 / Gemma3 apply them per-head on [B, H, T, head_dim] (weight shape [head_dim]). The two layouts compute variance over different axes and are not interchangeable. Attention::load inspects the q_norm weight tensor's elem_count and selects the correct path automatically; there is no config flag.

Not yet supported. Mixtral (block_sparse_moe.experts.{e}.{w1,w2,w3} tensor naming), DeepSeek-V2/V3 (MoE + MLA attention), and GGUF-quantized MoE checkpoints all fall outside the current loader.

The Qwen3.5 family is a hybrid model: most layers replace softmax attention with a Gated DeltaNet linear-attention mixer that carries a fixed-size recurrent state instead of a growing KV cache, while the remaining layers use gated full attention. Both mixer kinds live in the same transformer block and share the paged cache infrastructure. The family is text-only.

The DeltaNet path runs a chunked scan during prefill and a recurrent state update during decode (stored in the paged cache in place of K/V for those layers), with a short causal depthwise convolution on the q/k/v projections, Gemma-style gated RMSNorm, and partial RoPE applied only on the attention layers. Per-layer mixer kinds and the shared DeltaNet geometry come from the model config; no flag is required.

Supported formats: BF16 safetensors, compressed-tensors INT4 (fully quantized, and mixed BF16 DeltaNet + INT4 attention/MLP), and GGUF (qwen35 architecture). Correctness is covered by GatedDeltaNet golden fixtures checked against the reference implementation, plus an adversarial end-to-end test battery.

oxydllm pull Qwen/Qwen3.5-4B
oxydllm run Qwen/Qwen3.5-4B

The fused Metal Scaled Dot-Product Attention kernel accelerates the hot attention path on Apple Silicon. It currently supports a fixed set of head dimensions: 32, 64, 72, 80, 96, 128, 256. Models with other head dimensions remain functionally correct but fall back to the non-fused attention path with a measurable throughput cost.

Prometheus metrics are exposed at GET /metrics. The table below describes every metric.

Example Prometheus queries:

# 95th-percentile TTFT over the last 5 minutes
histogram_quantile(0.95, rate(oxydllm_ttft_milliseconds_bucket[5m]))

# Prefix cache hit ratio
rate(oxydllm_prefix_cache_requests_total{result="hit"}[5m])
  / rate(oxydllm_prefix_cache_requests_total[5m])

# Request throughput by status
rate(oxydllm_requests_total[1m])

Structured logs and request tracing. Every request is assigned a request_id (UUID v4) at the HTTP handler entry point. This ID appears in all log events for that request, from template rendering to the final generated token, making it possible to trace a single request end-to-end even under concurrent load:

grep 'request_id=abc-123' app.log

By default logs use a compact human-readable format. Setting LOG_FORMAT=json switches to one JSON object per line, compatible with Loki, Datadog, AWS CloudWatch, and jq. The variable is read at startup and applies to all commands, not just start:

LOG_FORMAT=json oxydllm start
LOG_FORMAT=json oxydllm run Qwen/Qwen3-4B-Q4_K_M

Each log line becomes a self-contained JSON object:

{"timestamp":"2024-01-01T12:00:00.123Z","level":"INFO","fields":{"request_id":"abc-123","ttft_ms":123.4,"model_id":"Qwen/Qwen3-4B-Q4_K_M"},"message":"first token emitted"}

Query a single request's lifecycle in Loki:

{app="oxydllm"} | json | request_id="abc-123"

Or filter locally with jq:

oxydllm start 2>&1 | jq 'select(.fields.request_id=="abc-123")'

Distributed tracing (OpenTelemetry). For per-request visibility that aggregate metrics cannot give (the queue/prefill vs decode breakdown of a single request, the slowest individual requests, correlation by request_id), oxydLLM can export OpenTelemetry traces over OTLP/HTTP. This is additive: the Prometheus endpoint above is unchanged, and tracing stays off unless an endpoint is configured. Point it at any OTLP/HTTP collector (Grafana Alloy, the OpenTelemetry Collector, Jaeger, Grafana Tempo):

oxydllm start --otel-endpoint http://localhost:4318
# or, equivalently, via environment:
OXYDLLM_OTEL_ENDPOINT=http://localhost:4318 oxydllm start

The value is treated as the OTLP base, so traces are sent to <endpoint>/v1/traces (matching the OTEL_EXPORTER_OTLP_ENDPOINT convention, which is also honored). Spans are batched in the background and flushed on shutdown.

Each request produces a trace tree http.request → inference.request → decode: the gap before decode starts is the time-to-first-token. Span attributes include request_id, model_id, prompt_tokens, max_tokens, ttft_ms, completion_tokens, tokens_per_second, finish_reason, and outcome (ok / error / unloaded); the first token emitted and request completed events are attached to the span. A W3C traceparent header on the incoming request is honored, so the spans join the caller's distributed trace (e.g. through an upstream gateway); without it, a fresh trace is started. Loading a model emits a separate model.load span (weight/KV bytes, layer count, warm-up time). In Grafana Tempo, search service.name = oxydllm and filter by any of those attributes.

Configure oxydLLM without passing CLI flags. Useful for containers, shell profiles, and CI environments. Every start flag has a matching OXYDLLM_* variable (see the start table); the variables below are the ones without a CLI flag, plus the most commonly set ones.