oxydLLM is a Rust-based LLM inference engine, targeting Apple Silicon and Linux (with NVIDIA GPUs). It exposes an OpenAI-compatible HTTP API, so any tool or library that already talks to ChatGPT, or any other OpenAI-compatible client, can use it as a drop-in local backend.
curl -fsSL https://github.com/giovannifil-64/oxydllm/raw/main/install.sh | sh
cargo build --release --features {metal or cuda}.
When building from source, start the server manually with
cargo run --release -- start
oxydllm pull Qwen/Qwen3-4B-GGUF
oxydllm estimate Qwen/Qwen3-4B-GGUF --context-len 8192
curl http://localhost:11313/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-4B-Q4_K_M",
"messages": [{"role": "user", "content": "Hello"}]
}'
oxydllm run Qwen/Qwen3-4B-Q4_K_M
lossless, balanced, aggressive)
enable_thinking, gpt-oss reasoning_effort)tool_choice: auto, required, forced, none| MODEL | ARCHITECTURE | WEIGHTS |
|---|---|---|
| meta-llama/Llama-3.2-1B-Instruct | LlamaForCausalLM | safetensors BF16 |
| Qwen/Qwen2.5-1.5B-Instruct | Qwen2ForCausalLM | safetensors BF16, GGUF Q2_K, Q3_K_M, Q4_0, Q4_K_M (incl. bartowski Q4_0 mirror) |
| Qwen/qwen2-1_5b-instruct-q4_0 | Qwen2 (GGUF) | GGUF Q4_0 |
| Qwen/Qwen2.5-3B-Instruct | Qwen2ForCausalLM | safetensors BF16 |
| Qwen/Qwen3-0.6B | Qwen3ForCausalLM | safetensors BF16, GPTQ Int8 (W8A16 resident) |
| Qwen/Qwen3-1.7B | Qwen3ForCausalLM | GGUF Q8_0, GPTQ Int8 (W8A16 resident) |
| Qwen/Qwen3-4B | Qwen3ForCausalLM | GGUF Q4_K_M, Q5_0, Q5_K_M, Q6_K, AWQ 4-bit (W4A16 resident), FP8 (E4M3 block-wise) |
| Qwen/Qwen3.5-4B | Qwen3_5ForConditionalGeneration | safetensors BF16; compressed-tensors INT4 (cyankiwi/Qwen3.5-4B-AWQ-4bit) and mixed BF16-DeltaNet + INT4 (cyankiwi/Qwen3.5-4B-AWQ-BF16-INT4); GGUF qwen35 (unsloth/Qwen3.5-4B-GGUF) |
| google/gemma-2b-it | GemmaForCausalLM | safetensors BF16 |
| google/gemma-2-2b-it | Gemma2ForCausalLM | safetensors BF16 |
| google/gemma-3-1b-it | Gemma3ForCausalLM | safetensors BF16 |
| google/gemma-4-E2B-it | Gemma4ForConditionalGeneration | safetensors BF16 |
| mistralai/Ministral-3-3B-Instruct-2512 | Mistral3ForConditionalGeneration | safetensors BF16 |
| microsoft/Phi-3.5-mini-instruct | Phi3ForCausalLM | safetensors BF16 |
| allenai/OLMoE-1B-7B-0924-Instruct | OlmoeForCausalLM (MoE) | safetensors BF16, 64 experts × top-k 8 |
| openai/gpt-oss-20b | GptOssForCausalLM (MoE) | MXFP4 experts + BF16, 32 experts × top-k 4 |
oxydllm pull <hf-repo>.| OS | DEVICE | BACKEND | NOTES |
|---|---|---|---|
| macOS 14 / 15 / 26 | Apple Silicon | Metal | primary platform |
| Linux (x86_64) | NVIDIA Ada Lovelace / Hopper / Blackwell / Blackwell Ultra / Blackwell consumer (RTX 50xx) | CUDA | functional, not perf-tuned |
| Linux (arm64) | NVIDIA GH200 / DGX Spark / GB300 (Blackwell Ultra) / Jetson Thor | CUDA | functional, not perf-tuned |
oxydLLM is built on top of the Candle tensor library. The model layer implements a single unified transformer that covers all supported architectures (LLaMA, Qwen, Gemma, Mistral, Phi) with minimal per-family branching, so adding a new model family rarely requires touching the core inference path. The Qwen3.5 family additionally runs a hybrid token mixer, interleaving Gated DeltaNet (linear attention) layers with gated full-attention layers on the same block and cache infrastructure.
Memory is managed through a paged KV allocator: instead of pre-allocating a fixed buffer per sequence, the cache is divided into fixed-size blocks assigned on demand from a shared pool. A rolling hash over each block enables a prefix cache, so when two requests share a common prompt prefix, the KV blocks for those tokens are reused directly without running a new forward pass.
The scheduler runs prefill and decode concurrently across multiple active sequences, dispatching to either the Metal backend (Apple Silicon) or the CUDA backend (NVIDIA Ada Lovelace, Hopper, Blackwell and newer). Metal has additional fused kernels for attention, RMSNorm, RoPE, and softmax; the CUDA path relies on Candle's generic kernels.