oxydLLM is a Rust-based LLM inference engine, targeting Apple Silicon and Linux (with NVIDIA GPUs). It exposes an OpenAI-compatible HTTP API, so any tool or library that already talks to ChatGPT, or any other OpenAI-compatible client, can use it as a drop-in local backend.
curl -fsSL https://github.com/giovannifil-64/oxydllm/raw/main/install.sh | sh
cargo build --release --features {metal or cuda}.
When building from source, start the server manually with
cargo run --release -- start
oxydllm pull Qwen/Qwen3-4B-GGUF
oxydllm estimate Qwen/Qwen3-4B-GGUF --context-len 8192
curl http://localhost:11313/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-4B-Q4_K_M",
"messages": [{"role": "user", "content": "Hello"}]
}'
oxydllm run Qwen/Qwen3-4B-Q4_K_M
lossless, balanced, aggressive)
enable_thinking, gpt-oss reasoning_effort)tool_choice: auto, required, forced, noneSupport is per architecture (the architectures field in a
checkpoint's config.json): every checkpoint of a supported architecture loads through the
same code path. Checkpoints in the table below are verified end to end on Apple Silicon; other sizes
and finetunes of the same architecture are expected to work but are not regularly tested.
| ARCHITECTURE | EXAMPLE MODELS | VERIFIED CHECKPOINT |
|---|---|---|
| LlamaForCausalLM | Llama 3.x, TinyLlama | meta-llama/Llama-3.2-1B-Instruct |
| MistralForCausalLM | Mistral 7B v0.x | same code path as Ministral, unverified |
| Mistral3ForConditionalGeneration | Ministral 3 | mistralai/Ministral-3-3B-Instruct-2512 |
| Qwen2ForCausalLM | Qwen2, Qwen2.5 | Qwen/Qwen2.5-1.5B / 3B-Instruct |
| Qwen3ForCausalLM | Qwen3 dense | Qwen/Qwen3-0.6B / 1.7B / 4B |
| Qwen3MoeForCausalLM | Qwen3-30B-A3B | same MoE runtime as OLMoE/Qwen3.6, unverified |
| Qwen3_5ForConditionalGeneration | Qwen3.5 (hybrid, text-only) | Qwen/Qwen3.5-4B |
| Qwen3_5MoeForConditionalGeneration | Qwen3.6 MoE hybrid | Qwen/Qwen3.6-35B-A3B-FP8 |
| Gemma / Gemma2 / Gemma3 ForCausalLM, Gemma4ForConditionalGeneration | Gemma 1-4 | google/gemma-2b-it, gemma-2-2b-it, gemma-3-1b-it, gemma-4-E2B-it |
| Phi3ForCausalLM | Phi-3, Phi-3.5 | microsoft/Phi-3.5-mini-instruct |
| GraniteForCausalLM | Granite 3.x dense | ibm-granite/granite-3.3-2b-instruct |
| OlmoeForCausalLM | OLMoE | allenai/OLMoE-1B-7B-0924-Instruct |
| GptOssForCausalLM | gpt-oss | openai/gpt-oss-20b |
| ARCHITECTURE | EXAMPLE MODELS | VERIFIED CHECKPOINT |
|---|---|---|
| RobertaModel / BertModel | granite-embedding r1, BERT family | ibm-granite/granite-embedding-125m-english |
| qwen3 causal embedder | Qwen3-Embedding | Qwen/Qwen3-Embedding-0.6B |
Multimodal input (vision first, text-only output) and cross-encoder reranking are on the roadmap and not yet supported.
The full list of verified checkpoints with per-format coverage is in the documentation.
| OS | DEVICE | BACKEND | NOTES |
|---|---|---|---|
| macOS 14 / 15 / 26 | Apple Silicon | Metal | primary platform |
| Linux (x86_64) | NVIDIA Ada Lovelace / Hopper / Blackwell / Blackwell Ultra / Blackwell consumer (RTX 50xx) | CUDA | functional, not perf-tuned |
| Linux (arm64) | NVIDIA GH200 / DGX Spark / GB300 (Blackwell Ultra) / Jetson Thor | CUDA | functional, not perf-tuned |
oxydLLM is built on top of the Candle tensor library. The model layer implements a single unified transformer that covers all supported architectures (LLaMA, Qwen, Gemma, Mistral, Phi) with minimal per-family branching, so adding a new model family rarely requires touching the core inference path. The Qwen3.5 family additionally runs a hybrid token mixer, interleaving Gated DeltaNet (linear attention) layers with gated full-attention layers on the same block and cache infrastructure.
Memory is managed through a paged KV allocator: instead of pre-allocating a fixed buffer per sequence, the cache is divided into fixed-size blocks assigned on demand from a shared pool. A rolling hash over each block enables a prefix cache, so when two requests share a common prompt prefix, the KV blocks for those tokens are reused directly without running a new forward pass.
The scheduler runs prefill and decode concurrently across multiple active sequences, dispatching to either the Metal backend (Apple Silicon) or the CUDA backend (NVIDIA Ada Lovelace, Hopper, Blackwell and newer). Metal has additional fused kernels for attention, RMSNorm, RoPE, and softmax; the CUDA path relies on Candle's generic kernels.