I've been working on a side project called aether , a Rust LLM inference engine that can load GGUF models and run them with WGPU GPU acceleration.
It started as a way to understand how LLMs actually work under the hood. One thing led to another, and now it has:
- Loads GGUF models (Llama/Mistral/Phi/Qwen)
- WGPU GPU backend (Metal/Vulkan/DX12)
- Custom fused WGSL compute shaders for Q8_0 and Q4_K quantized matmul (dequantize inline instead of a separate pass)
- Concurrent request pool for serving multiple users
- OpenAI-compatible API server (axum)
- Pure Rust, no Python dependencies in the hot path
The GPU path is still experimental (CPU mode is the safe default), but the dequant shaders and the fused matmul kernels were honestly the most fun part to write.
I'm not trying to compete with llama.cpp or MLX, this was primarily a learning project that grew into something actually useful. Happy to answer questions or take feedback.
Stack: Rust, WGPU, WGSL, GGUF, axum, Tokio
(Full transparency, the majority of this code and post were written with AI assistance. I drove the design decisions, architecture, and testing; AI handled a lot of the implementation. Treat it accordingly.)
