Mojo and Vulkan - Running Local AI on Any GPU

For about a decade, NVIDIA's CUDA monopoly has been the single biggest lock-in in AI. Two things happened in 2026 that are finally breaking it: a cross-vendor inference backend that runs on any GPU, and a programming language that makes CUDA optional for kernel development.

The problem with CUDA#

CUDA is proprietary to NVIDIA. AMD, Intel, Apple, and Qualcomm GPUs cannot run it natively. AMD's official ROCm stack has historically had limited GPU support, mostly professional cards with poor Radeon desktop support. It has been improving: ROCm 10 (August 2026) finally added official RDNA4 support, which covers the RX 9000 series. Each vendor still pushes its own compute stack: ROCm, oneAPI, Metal.

The result: if you want to run local AI, you buy NVIDIA. If you want to write GPU kernels, you learn CUDA. The moat is total.

Vulkan compute in llama.cpp#

llama.cpp added a Vulkan compute backend. Build with GGML_VULKAN=on and it runs LLM inference on any GPU that has a Vulkan driver.

Vulkan is a cross-platform, cross-vendor graphics and compute API backed by the Khronos Group. It works on NVIDIA, AMD, Intel, Apple (via MoltenVK translation), and Qualcomm GPUs. Models are loaded in GGUF format and run entirely locally.

Performance findings#

The picture is workload-dependent rather than a clean sweep. ROCm generally wins prompt processing, Vulkan generally wins token generation, and the gap flips between models. For chat and coding sessions, which are generation-heavy, Vulkan is usually the faster backend on RDNA3/4. That the open-source RADV driver from Mesa trades blows with (and often beats) AMD's own HIP stack for LLM inference is still an awkward result for AMD's software team and a massive win for open-source driver development.

The Vulkan backend itself keeps improving. The standalone Vulkan 0.6.4 release (August 2026) added wave32 and shared-memory optimizations that lifted prompt processing by 5 to 10% on most shapes and up to 72% on one large-prefill coding workload, with decode unchanged. Against current ROCm builds it now wins both phases on several Qwen3-Coder model shapes, including more than doubling decode speed at 32K context depth.

What this enables#

• AMD Radeon RX 7000/9000 series users get first-class AI performance without ROCm headaches
• Intel Arc users can run llama.cpp via Vulkan
• Qualcomm Adreno GPUs (phones, laptops) become viable for on-device AI
• Apple Silicon works via MoltenVK translation layer
• Older cards (even RDNA2 like the RX 6700 XT) keep working through RADV, though ROCm is often faster there on long prompts

The Mojo programming language#

Mojo is a superset of Python designed for high-performance AI compute. Created by Modular (Chris Lattner, creator of LLVM, Swift, and MLIR), it compiles via MLIR to target CPUs, GPUs (NVIDIA, AMD, Intel), and accelerators.

The claim: match or beat CUDA/CUTLASS while being as readable as Python.

Mojo 1.0 is out, and the whole language is now open source#

Three things landed in the last two months that change Mojo's status from promising experiment to real option:

1. Mojo hit 1.0 on August 11, 2026, with a source stability guarantee: code you write today will not break under you
2. One week later, at ModCon 2026, Modular open sourced the entire compiler and toolchain under Apache 2.0
3. Mojo 1.1 shipped on September 17, 2026, with faster kernels on AMD and NVIDIA GPUs, better compile times, and community contributions to the compiler now accepted

ModCon also brought the platform news. MAX (the inference framework built on Mojo) is now source-available with no device-usage restrictions, Modular Cloud went generally available, and the platform added support for AWS Trainium, Google TPUs, Qualcomm Cloud AI 100 Ultra, and Qualcomm's Dragonfly accelerators alongside NVIDIA, AMD, and Apple GPUs. Native Windows support for Mojo is coming through a collaboration with Microsoft.

Qualcomm acquires Modular#

Qualcomm announced its intent to acquire Modular in June 2026 and completed the deal on July 29, 2026, in an all-stock deal worth $3.9B. Chris Lattner now runs advanced AI software and platforms at Qualcomm. Mojo, MAX, and Modular Cloud keep their brands and their open ecosystem mission, which Qualcomm's own press release reiterates.

The concrete payoff already exists: Qualcomm's data center inference silicon (Cloud AI 100 Ultra and the Dragonfly AI 200 line) is integrated into MAX, making it the first NPU target in the stack. The same model code that runs on NVIDIA and AMD GPUs now serves on Qualcomm's accelerators, including a Gemma 4 31B deployment spread across four Cloud AI 100 chips. Mojo compiles natively to Qualcomm's Hexagon NPU backend rather than going through a translation layer.

So Mojo is not going anywhere. It is open source, stable at 1.x, and has serious corporate backing from a mobile and edge AI chip giant.

How it all fits together#

Application Layer:  Ollama / LM Studio / Text Generation WebUI
                          |
Inference Engine:         llama.cpp (GGUF models)
                          |
GPU Backend:              Vulkan Compute API
                          |
Driver Layer:             RADV (Mesa) / NVIDIA Vulkan / AMD Vulkan
                          |
Hardware:                 ANY GPU (NVIDIA, AMD, Intel, Apple, Qualcomm)

For kernel development, Mojo replaces the CUDA-dependent path:

Traditional:   Python (PyTorch) → C++/CUDA → NVIDIA GPU only
Mojo path:     Python/Mojo → Mojo compiler → Any GPU

Practical recommendations#

What it means#

For NVIDIA, the CUDA moat is weakening. Vulkan plus llama.cpp runs on AMD GPUs competitively. But NVIDIA still dominates training, since CUDA, cuDNN, and TensorRT have no real alternative. Inference is where the disruption happens.

For AMD, Vulkan outperforming ROCm is embarrassing, but users do not care which backend works. The Radeon RX 9070 XT (16GB VRAM, ~$600) becomes a compelling local AI card for the first time.

For everyone else, any GPU is now an AI GPU. The hardware you already own can run local models. GGUF + llama.cpp + Vulkan is the most portable local AI stack in existence. Mojo adds a development path that is not CUDA-dependent, and with MAX now reaching TPUs, Trainium, and Qualcomm silicon, the "write once, run on any accelerator" pitch is no longer just a pitch.