PATH 01 // MAC2026-10-23
Apple Silicon Mac
Unified-memory budgets, quantization vs context, and when SSD swap kills tokens/s.
Apple Silicon Mac Local Model Benchmark: Fixed Tasks, Quantization, Context & Speed
TINKER AI LAB // HANDS-ON & REPRODUCIBLE
Run AI on your own hardware,
workflows, and edge builds.
A reproducible AI engineering lab: pick a hardware path, run a local model, then use the workbenches. Every guide ships with hardware specs, pinned versions, verification dates, downloadable configs, and failure boundaries—not an AI writing SaaS or a coding-tool review directory.
Pinned Hardware & Stack Versions
Offline Boundaries & Failure Modes
Downloadable Compose / JSON / BOM
~/homelab/mac-vram-check.sh
VERIFIED 2026-11-04
Mac M3 Pro 36GB · Ollama 0.5.7 · Qwen2.5-14B
$ OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama run qwen2.5:14b # Disk weight: 9.0 GB | KV Cache (16K ctx, q8_0): 1.98 GB | Total: 11.7 GB NAME SIZE PROCESSOR CONTEXT qwen2.5:14b-instruct-q4 11.7 GB 100% GPU 16384 > Eval rate: 31.64 tokens/s (Zero SSD swap)
Zero vendor lock-in · ReproducibleOpen VRAM & Context Calculator
START HERE // 从哪开始
Pick your hardware. Run one verified path.
Three paths → one flagship guide + VRAM calculator each. Not four equal pillars. Maintain first: Local AI + Troubleshooting.
PATH 02 // NVIDIA2026-10-23
NVIDIA GPU PC
PCIe shared-memory spill, container GPU passthrough, and headroom vs desktop baseline.
NVIDIA PC Model Loading & VRAM Diagnostics: Spotting PCIe Shared-Memory Spillover
PATH 03 // PI / EDGE2026-10-23
Raspberry Pi / Edge
27W PD + cooler preflight, offline voice/vision box, honest 3B token rates.
Raspberry Pi 5 Offline AI Lab Box: Complete BOM, Power/Thermal Check & Step-by-Step Build
LIBRARY BY PILLAR // 按栏目浏览
Guide library by pillar (weighted maintenance)
Local AI + Troubleshooting are actively maintained; Workflows maintain-only; Maker new posts frozen (existing kept). Click a card to filter guides.
INTERACTIVE WORKBENCHES // 零延迟工程工具
Interactive Calculators & Diagnostic Tools
100% client-side tools built to eliminate VRAM miscalculations, manual config edits, and silent runtime failures.
TOOL 01 // VRAM & KV CACHELIVE
VRAM & Context Budget Calculator
Weight file size != runtime memory. Break down quantized weights + KV cache + compute buffers against Mac Unified Memory, NVIDIA VRAM, and Pi 5 RAM.
Calculate Memory & Token Speed
TOOL 02 // DECISION TREES13 TREES
Interactive Troubleshooter Trees
Step-by-step root-cause trees (13 trees) for Container-to-Host networking, silent CPU fallback, post-HTTP-200 SSE drops, n8n timeouts, HTTP 403 CORS, R1 <think> loops, Pi 5 power/NPU/ALSA, and RAG vector dimensions.
Start Step-by-Step Diagnosis
TOOL 03 // CONFIG WORKBENCH48 PACKS
Artifact & Config Workbench
Customize ports, paths, and model tags live in your browser across Docker Compose stacks, n8n workflow JSONs, Pi 5 scripts, and SSE clients.
Customize & Download Configs
REPRODUCIBLE GUIDES // 实测与排障指南库
Verified Guides & Hardware Lab Notes
Showing 48 reproducible guides
Environment:
Local AI / HomeLab100% Offline After Weight Pull
Apple Silicon Mac Local Model Benchmark: Fixed Tasks, Quantization, Context & Speed
Measured prompt eval and token generation speeds across Q4_K_M, Q5_K_M, and Q8_0 on unified memory—showing exactly when raising num_ctx triggers SSD swap degradation.
HARDWAREMacBook Pro 14" (M3 Pro · 18GB & 36GB Unified Memory tested) · macOS 15.3
VERSIONS
Ollama 0.5.7qwen2.5:7b-instructqwen2.5:14b-instructdeepseek-r1:14b
LIMITSModel file size != runtime memory: Qwen2.5-14B Q4_K_M is 9.0 GB on disk, but at num_ctx=32768 (FP16 KV cache) total footprint reaches 16.4 GB, causing heavy swap on 16GB/18GB Macs.
Verified 2026-10-23·3 check steps
Local AI / HomeLab100% Local GPU Execution
NVIDIA PC Model Loading & VRAM Diagnostics: Spotting PCIe Shared-Memory Spillover
Why a 12GB RTX GPU silently drops from 48 tok/s to 4 tok/s without throwing an OOM error—and how to cross-check nvidia-smi against ollama ps.
HARDWARENVIDIA RTX 4070 12GB VRAM · 32GB DDR5 System RAM · Ubuntu 24.04 & Windows 11 WSL2
VERSIONS
NVIDIA Driver 560.35 / CUDA 12.6Ollama 0.5.7nvidia-container-toolkit 1.16.2
LIMITSModern NVIDIA drivers enable CUDA Sysmem Fallback by default: exceeding physical 12GB VRAM by even 600MB pages tensors over PCIe into DDR5 RAM, slashing speed 8–10x without crashing.
Verified 2026-10-23·3 check steps
Local AI / HomeLabRequires Explicit Env Vars to Disable Outbound Calls
Ollama + Open WebUI Operations: Installation, Persistence, Backup, Upgrade & Safe Restore
Pin exact container image tags, map host bind mounts for SQLite & vector stores, run a 10-second pre-upgrade snapshot, and practice full rollback.
HARDWARELinux HomeLab (Ubuntu 24.04 · Docker Compose v2.29) & macOS Host + Docker WebUI
VERSIONS
Ollama 0.5.7Open WebUI v0.5.10SQLite 3 (webui.db)
LIMITSUpgrading Open WebUI runs automatic Alembic schema migrations on `/app/backend/data/webui.db`; simply changing the Docker image tag back to an older version WITHOUT restoring your backup tar will fail to boot.
Verified 2026-10-23·3 check steps
Local AI / HomeLab100% Local Embedding + Local Generation
Local Document Q&A (RAG): Downloadable Sample Corpus, Citation Check & Failure Cases
Test your local RAG pipeline with our downloadable ground-truth policy document—exposing silent num_ctx truncation, two-column PDF bleed, and hallucinated citations.
HARDWAREMac M3 Pro 36GB & RTX 4070 12GB · Local Ollama + Open WebUI Hybrid RAG
VERSIONS
Ollama 0.5.7Open WebUI v0.5.10bge-m3:567mqwen2.5:14b-instruct-q4_K_M
LIMITSFailure Case #1 (Silent Context Truncation): If 5 retrieved chunks (~3,800 tokens) are injected into a model left at Ollama default `num_ctx=2048`, Ollama silently drops the top chunks and the model claims the fact is missing.
Verified 2026-10-23·3 check steps
No-Code AI WorkflowsOutbound RSS Fetch Only · 100% Local AI Inference
n8n + Local LLM RSS Classifier: Importable Workflow, Deduplication & Retry Branches
A production-ready n8n workflow that fetches RSS feeds, deduplicates URLs, classifies items via local Ollama with strict JSON output, and handles queue timeouts.
HARDWARESelf-hosted n8n (Docker) + Local Ollama (Qwen2.5-7B-Instruct Q4_K_M)
VERSIONS
n8n 1.76.xOllama 0.5.7qwen2.5:7b
LIMITSSending 25+ RSS items simultaneously without serialized batching causes Ollama queue waits to exceed n8n default HTTP timeout.
Verified 2026-10-23·3 check steps
API & Deployment TroubleshootingLocal Loopback & Bridge Network Only
Container Cannot Connect to Host Model: Minimal Reproduction & Diagnostic Steps
Solve the #1 self-hosted AI failure (`ECONNREFUSED` / `ETIMEDOUT` calling port 11434 from Open WebUI, n8n, or Dify) in three deterministic verification commands.
HARDWAREVerified on Ubuntu 24.04 (Docker Engine 27.x + UFW) & macOS 15 (Docker Desktop 4.36)
VERSIONS
Docker Engine 27.xOllama 0.5.7Open WebUI v0.5.10n8n 1.76.x
LIMITSUsing `network_mode: host` fixes Linux networking instantly, but is NOT supported on Docker Desktop for macOS or Windows.
Verified 2026-10-23·3 check steps
No-Code AI WorkflowsConfigurable: Local-Only or Cloud API
Automated Webhook Ticket Triage: Schema Guard, Dead-Letter Fallback, Cost & Permissions
A practical customer support / issue triage pipeline comparing local Ollama vs cloud API cost, strict input truncation, and least-privilege credential scoping.
HARDWAREDocker n8n 1.76+ · Compatible with Local Ollama or Any OpenAI-Compatible Endpoint
VERSIONS
n8n 1.76.xOpenAI-Compatible Chat Completions v1qwen2.5:7b / gpt-4o-mini
LIMITSUnbounded user webhook bodies can inject 50K+ tokens or prompt-injection payloads if not sliced (`slice(0, 2000)`) before the LLM node.
Verified 2026-10-23·3 check steps
AI Maker / STEM Projects100% Air-Gapped Capable After Initial Setup
Raspberry Pi 5 Offline AI Lab Box: Complete BOM, Power/Thermal Check & Step-by-Step Build
Build a self-contained offline AI station on Raspberry Pi 5 (8GB) with exact BOM specifications, power throttling diagnostics, and realistic setup time expectations.
HARDWARERaspberry Pi 5 (8GB RAM) · Official 27W (5V/5A) USB-C PD Power Supply · Active Cooler · 64GB A2 microSD / NVMe HAT
VERSIONS
Raspberry Pi OS Bookworm (64-bit)Ollama 0.5.7 ARM64qwen2.5:3b-instruct-q4_K_M
LIMITSRealistic setup time: Expect 90–150 minutes for a first-time builder (OS flashing, apt updates, audio ALSA group permissions, and 2.1GB model download), not an untested "15-minute quick build".
Verified 2026-10-23·3 check steps
AI Maker / STEM Projects100% Local LAN / Loopback Execution
STEM Project: Raspberry Pi Local Sensor & HomeLab Log Explainer with Zero Cloud Dependency
A structured maker project teaching students how to combine deterministic threshold filtering with a local 3B model for plain-language hardware alerts.
HARDWARERaspberry Pi 5 (8GB) · BME280 I2C Temp/Humidity Sensor (or journalctl system logs)
VERSIONS
Python 3.11Ollama 0.5.7qwen2.5:3b
LIMITSDo not feed raw thousand-line log files directly into a 3B model on Pi 5; pre-filter with grep/awk to < 30 lines first so prompt eval stays under 3 seconds.
Verified 2026-10-23·3 check steps
API & Deployment TroubleshootingApplies to Both Local & Cloud Streaming APIs
Post-HTTP-200 SSE Stream Errors: Failure Conditions & Recovery Boundaries
Why simple HTTP status-code tables fail in streaming LLM apps: handling mid-stream SSE error events, proxy idle cuts, and max_tokens JSON truncation.
HARDWARENode.js 22.13+ SSE Client · Applicable to Claude API, OpenAI API, and Local Ollama / vLLM
VERSIONS
Node.js 22.13+Anthropic Messages SSEOpenAI Chat Completions SSE
LIMITSAs documented by Claude & OpenAI, once HTTP 200 OK headers are flushed to open an SSE stream, an error occurring mid-generation CANNOT change the HTTP status code.
Verified 2026-10-23·3 check steps
Local AI / HomeLab100% Air-Gapped Source Code Privacy
DeepSeek-R1 & Qwen2.5-Coder Local IDE Setup: <think> Token Budgets, FIM Autocomplete & Loop Fixes
Why using DeepSeek-R1 for tab autocomplete ruins IDE latency—and how to pair R1-14B (for architectural reasoning with 4K <think> budget) alongside Qwen2.5-Coder-7B (for sub-250ms FIM edits).
HARDWAREMacBook Pro M3 Max (36GB Unified) & NVIDIA RTX 4070 Ti Super (16GB VRAM)
VERSIONS
Ollama 0.5.7Continue v1.0.x (VS Code)deepseek-r1:14bqwen2.5-coder:7b
LIMITSNever assign a reasoning model (`deepseek-r1`) to `tabAutocompleteModel`: it emits `<think>` tokens before every line completion, adding 3–8 seconds of lag.
Verified 2026-10-23·3 check steps
Local AI / HomeLab100% Local LAN OpenAI-Compatible Serving
Concurrent Local LLM Serving: Ollama NUM_PARALLEL vs vLLM Continuous Batching Benchmark
Measured P95 Time-to-First-Token (TTFT), aggregate throughput, and VRAM pre-allocation on an RTX 4090 (24GB) under 1, 4, and 8 parallel workflow workers.
HARDWARENVIDIA RTX 4090 (24GB VRAM) · 64GB DDR5 · Ubuntu 24.04 · Docker Engine 27.x
VERSIONS
Ollama 0.5.7 (GGUF Q4_K_M)vLLM v0.6.4 (AWQ INT4)Qwen2.5-7B-Instruct & 14B-Instruct
LIMITSvLLM pre-allocates `--gpu-memory-utilization` (default 90% of VRAM) on startup and holds it permanently; it does NOT auto-unload after 5 minutes like Ollama.
Verified 2026-10-23·3 check steps
No-Code AI Workflows100% On-Premise Financial Document Privacy
n8n + Local LLM Invoice & Contract Extractor: JSON Schema Guard & Deterministic Math Reconciliation
Never trust an LLM to do accounting math alone: extract structured invoice fields locally with Qwen2.5-14B, then verify `subtotal + tax === total` in a deterministic n8n Code node.
HARDWAREDocker n8n 1.76+ · Local Ollama (Qwen2.5-14B-Instruct Q4_K_M / Q5_K_M)
VERSIONS
n8n 1.76.xOllama 0.5.7qwen2.5:14b-instruct
LIMITSScanned image-only PDFs contain zero selectable text layer; you must pass scanned images through OCR (e.g. PaddleOCR / Tesseract) or a Vision-Language Model (`qwen2.5-vl` / `gemma3`) first.
Verified 2026-10-23·3 check steps
No-Code AI Workflows100% Local Code Analysis · Scoped Git Webhook
Self-Hosted Git PR Diff Reviewer with n8n & Qwen2.5-Coder: Secret Redaction & Token Caps
Automate pull request bug and security reviews on your own GPU using n8n and Qwen2.5-Coder-14B—complete with lockfile filtering, regex secret scrubbing, and scoped webhook tokens.
HARDWARESelf-hosted n8n + Local Ollama (`qwen2.5-coder:14b` on 16GB+ GPU / 24GB+ Mac)
VERSIONS
n8n 1.76.xOllama 0.5.7qwen2.5-coder:14b-instruct-q4_K_M
LIMITSRaw PR diffs that include `package-lock.json`, `pnpm-lock.yaml`, or minified bundles easily exceed 60K tokens; filter out lockfiles and cap diffs to 12,000 characters per hunk.
Verified 2026-10-23·3 check steps
AI Maker / STEM Projects100% Edge NPU + CPU Offline Pipeline
Raspberry Pi 5 AI Kit (Hailo-8L NPU) + Local LLM Build: PCIe Gen 3 Vision & Split-Compute Benchmark
Understand what the 13 TOPS Hailo-8L NPU accelerates (28 FPS YOLOv8 vision at <2W) and what it does NOT (GGUF LLMs)—and build a zero-contention NPU Vision + CPU 3B LLM pipeline.
HARDWARERaspberry Pi 5 (8GB) · Official Raspberry Pi AI Kit (M.2 HAT+ with Hailo-8L 13 TOPS NPU) · Pi Camera Module 3
VERSIONS
Raspberry Pi OS Bookworm 64-bit (Kernel 6.6+)hailo-all 4.18+rpicam-appsOllama 0.5.7 (qwen2.5:3b)
LIMITSCrucial hardware boundary: The Hailo-8L M.2 NPU (13 TOPS) accelerates CNN/Vision models compiled to `.hef` format (YOLO, ResNet, Pose); it CANNOT accelerate Ollama GGUF language models (`qwen2.5`, `llama3`).
Verified 2026-10-23·3 check steps
API & Deployment Troubleshooting100% Intranet / LAN Authenticated Gateway
Fixing Local LLM HTTP 403 CORS, Reverse Proxy Host Headers & LAN Bearer Authentication
Why browser extensions and LAN web clients get HTTP 403 Forbidden from Ollama—and how to configure OLLAMA_ORIGINS, Host header rewrites, and Bearer token protection safely.
HARDWARENginx 1.26 / Caddy 2.8 · Ollama 0.5.7 · Chrome / Firefox Extensions & LAN Web Clients
VERSIONS
Ollama 0.5.7Nginx 1.26+Caddy 2.8+
LIMITSSecurity Hazard: Setting `OLLAMA_HOST=0.0.0.0` AND `OLLAMA_ORIGINS=*` simultaneously without a reverse proxy allows ANY public website you visit in your browser to silently query or delete your local models.
Verified 2026-10-23·3 check steps
Local AI / HomeLab100% Air-Gapped Document & Financial Table OCR
Local Vision OCR Benchmark: Qwen2.5-VL-7B vs Traditional OCR on Scanned PDFs, Merged Tables & Formulas
Why standard PDF text extractors return 0 characters on scanned contracts—and how to convert multi-page scanned PDFs into clean Markdown tables locally using pdftoppm (200 DPI) + Qwen2.5-VL-7B.
HARDWAREMacBook Pro M3 Max (36GB Unified) & NVIDIA RTX 4070 Ti Super (16GB VRAM)
VERSIONS
Ollama 0.5.7+qwen2.5-vl:7b (Q4_K_M)poppler-utils (pdftoppm 24.08)
LIMITSFeeding raw 600 DPI 4K page scans directly into a Vision-Language Model inflates visual patch tokens by 4x (~3,200 tokens/page) and triggers KV cache OOM without improving OCR accuracy over 200 DPI.
Verified 2026-10-26·3 check steps
No-Code AI Workflows100% Intranet Database Schema & Query Privacy
n8n + Local LLM Read-Only Text-to-SQL Analyst: AST Guardrails, Row Limits & Zero-Write Security
Let team members query internal PostgreSQL/MySQL databases in plain language using Qwen2.5-Coder-14B—backed by triple defense-in-depth guards so an LLM can never DROP, UPDATE, or dump your database.
HARDWAREDocker n8n 1.76+ · PostgreSQL 16 / MySQL 8 · Local Ollama (`qwen2.5-coder:14b`)
VERSIONS
n8n 1.76.xPostgreSQL 16Ollama 0.5.7qwen2.5-coder:14b
LIMITSNever connect an LLM Text-to-SQL workflow using your application’s read-write `postgres` superuser! A prompt injection could attempt `DELETE` or `pg_read_file()`.
Verified 2026-10-26·3 check steps
AI Maker / STEM Projects100% Air-Gapped Microphone-to-Speaker Privacy
Raspberry Pi 5 100% Offline Voice Assistant Loop: Whisper.cpp STT + Ollama 3B + Piper Neural TTS
Build a sub-2.5s latency air-gapped voice assistant on Raspberry Pi 5 (8GB): exact ALSA audio card bindings, ARM64 NEON Whisper model sizing, and sentence-streaming Piper TTS.
HARDWARERaspberry Pi 5 (8GB) · Official 27W PSU · USB Microphone (UAC) · 3.5mm / USB Speaker
VERSIONS
whisper.cpp v1.7+ (ARM64 NEON)Ollama 0.5.7 (qwen2.5:3b)Piper TTS v1.2.0 (aarch64)
LIMITSDo not run `whisper-large-v3` or `medium` on Raspberry Pi 5 CPU: a 4-second voice command takes 22+ seconds to transcribe! Use `ggml-base-q5_1.bin` or `ggml-small-q5_1.bin` for 0.6s–1.4s STT latency.
Verified 2026-10-26·3 check steps
API & Deployment Troubleshooting100% Local Vector DB & GPU Process Diagnostics
Troubleshooting RAG Embedding Dimension Mismatch (1024 vs 768) & Zombie VRAM Allocation
Fix the two most common post-setup local AI failures: Chroma/PGVector dimension mismatch errors after changing embedding models, and GPU VRAM staying 95% full when ollama ps shows zero models.
HARDWARELinux / macOS / Windows WSL2 · Ollama 0.5.7 · Open WebUI / Dify / Continue (ChromaDB, Qdrant, PGVector)
VERSIONS
Ollama 0.5.7bge-m3 (1024-dim)nomic-embed-text (768-dim)mxbai-embed-large (1024-dim)
LIMITSVector spaces are mathematically incompatible across models: even if two models share the same dimension (`bge-m3` is 1024-dim and `mxbai-embed-large` is 1024-dim), querying an index built with model A using model B returns pure random noise!
Verified 2026-10-26·3 check steps
Local AI / HomeLab100% Offline After Model Pull
AMD Ryzen AI Max+ 395 (Strix Halo 128GB) Local LLM Benchmark: 96GB GTT Pool, Vulkan vs ROCm & Qwen3-30B-A3B MoE
Tested on a 128GB LPDDR5X-8000 Strix Halo Mini PC (Radeon 8060S gfx1151): unlocking the 96GB unified VRAM pool on Linux 6.16+, tuning ROCm hipBLASLt vs Vulkan RADV, and why Qwen3-30B-A3B MoE hits 78.4 tok/s while dense 70B models cap at 6.2 tok/s.
HARDWAREGMKtec EVO-X2 / Framework Desktop (AMD Ryzen AI Max+ 395 · Radeon 8060S 40CU gfx1151 · 128GB LPDDR5X-8000 256 GB/s) · Ubuntu 24.04.2 (Kernel 6.16.9)
VERSIONS
Linux Kernel 6.16.9ROCm 7.2 / Mesa RADV 25.0Ollama 0.6.2 / llama.cpp b4820qwen3:30b-a3b-q4_K_Mllama3.3:70b-instruct-q4_K_M
LIMITSLegacy `amdgpu.gttsize` boot flags are deprecated and ignored on Linux 6.14+ for `gfx1151`; without BIOS 96GB UMA or `amdttm.pages_limit=25165824`, visible GPU memory stays trapped at ~15.5 GB.
Verified 2026-10-27·3 check steps
AI Maker / STEM Projects100% Offline Edge Vision + LLM
NVIDIA Jetson Orin Nano Super (67 TOPS) vs Raspberry Pi 5 AI HAT+ (26 TOPS) & IMX500 AI Camera: Edge VLM & Vision Benchmark
Side-by-side edge hardware benchmark: when to use Raspberry Pi 5 + Sony IMX500 on-sensor NPU / Hailo-8 26 TOPS (4.8W–9.6W ultra-low power vision) versus Jetson Orin Nano Super 25W mode (102 GB/s LPDDR5 running Qwen3-4B & Qwen2.5-VL at 31.4 tok/s).
HARDWARENVIDIA Jetson Orin Nano Super 8GB (JetPack 6.2 · 25W MAXN_SUPER · 102 GB/s) vs Raspberry Pi 5 8GB + AI HAT+ 26 TOPS (Hailo-8) + Raspberry Pi AI Camera (Sony IMX500)
VERSIONS
JetPack 6.2 (L4T R36.4.3)Raspberry Pi OS Bookworm 64-bitimx500-all 1.1.0 / HailoRT 4.19Ollama 0.6.2 (CUDA & ARM64)qwen3:4b-q4_K_M
LIMITSRaspberry Pi AI Camera (Sony IMX500) runs MobileNet/YOLOv8n directly inside the camera sensor with 0% Pi CPU load, but its 8MB on-sensor SRAM cannot run LLMs or heavy VLMs.
Verified 2026-10-27·3 check steps
No-Code AI Workflows100% Local LAN / Air-Gapped Ready
n8n + Model Context Protocol (MCP) + Local Qwen3 / Ollama Tool Calling: Sandboxed Filesystem, SQLite & Loop Guards
Connect n8n’s MCP Client Tool to local MCP Servers using Qwen3-8B / Qwen3-30B-A3B on Ollama: why exposing >5 MCP tools crashes 8B model accuracy, how `/no_think` cuts tool-routing latency by 84%, and how to enforce read-only allowlists and `maxIterations=5`.
HARDWAREMac mini M4 Pro (24GB Unified Memory · 273 GB/s) & Ubuntu 24.04 Docker Host (RTX 4070 12GB VRAM)
VERSIONS
n8n 1.82.x (with MCP Client Tool)Ollama 0.6.2qwen3:8b / qwen3:30b-a3bmcp-server-sqlite 0.6.x (SSE wrapper)
LIMITSLeaving n8n MCP Client on `Include: All Tools` against a server with 20+ tools injects ~6,000 schema tokens per turn, dropping 8B local model tool-selection accuracy from 96% to 61%.
Verified 2026-10-27·3 check steps
API & Deployment Troubleshooting100% Local Kernel & Runtime Diagnostics
New Hardware GPU & Unified Memory Troubleshooting: Intel Arc B580 Docker Fallback, AMD Strix Halo 15.5GB Cap & DGX Spark Page Cache OOM
Root-cause diagnostics and one-line verification scripts for three high-frequency new-hardware failures: Intel Arc B580 (12GB Xe2) silent CPU fallback in Docker, AMD Strix Halo deprecated `amdgpu.gttsize` caps, and NVIDIA DGX Spark (GB10 128GB) unified memory page-cache OOMs.
HARDWAREIntel Arc B580 (12GB GDDR6 Xe2) · AMD Ryzen AI Max+ 395 (128GB UMA) · NVIDIA DGX Spark (GB10 128GB Coherent Unified Memory)
VERSIONS
Linux Kernel 6.12+ (xe) / 6.16.9+ (amdttm)Docker Engine 27.5Ollama 0.6.2 / llama.cpp Vulkan & SYCLNVIDIA Driver 570.x (GB10 UMA)
LIMITSIntel Arc B580 (12GB) enforces a single-allocation buffer ceiling on older Level Zero runtimes unless `SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1` or Vulkan (`GGML_VK_VISIBLE_DEVICES=0`) is used.
Verified 2026-10-27·3 check steps
Local AI / HomeLab100% Air-Gapped Capable
Apple Mac Studio M3 Ultra (512GB / 256GB · 819 GB/s) Benchmark: Running Full DeepSeek-R1-671B & Qwen3-235B-A22B with MLX vs llama.cpp
Unlock 475GB of Metal wired memory on a 512GB Mac Studio M3 Ultra to run full 671B DeepSeek-R1/V3 (4-bit) at 14.8 tok/s under 192W wall power, compare MLX vs llama.cpp on Qwen3-235B-A22B MoE (32.4 tok/s), and link two Macs via 80 Gbps Thunderbolt 5 RDMA/Bridge0.
HARDWAREMac Studio M3 Ultra (32C CPU / 80C GPU, 512GB Unified Memory @ 819 GB/s) + Mac Studio M3 Ultra (256GB @ 819 GB/s) · macOS 15.5 Sequoia
VERSIONS
mlx-lm 0.22.1 (MLX 0.24.0)llama.cpp b4910 (Metal Backend)DeepSeek-R1-671B (4-bit MLX & Q4_K_M GGUF)Qwen3-235B-A22B (4-bit MLX & Q4_K_M GGUF)
LIMITSmacOS defaults to ~75% Metal wired limit (~384GB on a 512GB Mac); loading a 404GB DeepSeek-R1-671B Q4_K_M model without raising `iogpu.wired_limit_mb` fails or swaps.
Verified 2026-10-28·3 check steps
Local AI / HomeLab100% Air-Gapped Capable
NVIDIA Blackwell Local LLM Benchmark: RTX 5090 (32GB) vs RTX 5060 Ti (16GB), Native FP4/MXFP4, CUDA 12.8 sm_120 & 450W Power Tuning
Side-by-side GDDR7 benchmarks of RTX 5090 (32GB · 1,792 GB/s) and RTX 5060 Ti (16GB · 448 GB/s) across Qwen3-14B, Qwen3-30B-A3B MoE, and Qwen2.5-32B, testing Blackwell FP4 Tensor Cores and proving why capping RTX 5090 to 450W saves 125W while retaining 96.4% tok/s.
HARDWARENVIDIA GeForce RTX 5090 (32GB GDDR7 · 1,792 GB/s) & RTX 5060 Ti (16GB GDDR7 · 448 GB/s) · AMD Ryzen 9 9950X · 64GB DDR5 · Ubuntu 24.04.2 LTS
VERSIONS
NVIDIA Driver 570.133.07 + CUDA 12.8 (sm_120)llama.cpp b4910 (-DCMAKE_CUDA_ARCHITECTURES=120)vLLM 0.8.2 (PyTorch 2.7.0+cu128)Qwen3-14B / Qwen3-30B-A3B / Qwen2.5-Coder-32B
LIMITSRTX 5060 Ti 16GB has a 128-bit bus (448 GB/s GDDR7): it fits 14B models with 32K context effortlessly, but dense 32B Q4_K_M (~19.8GB) spills over PCIe and drops from 39 tok/s to 4.8 tok/s.
Verified 2026-10-28·3 check steps
Local AI / HomeLab100% Air-Gapped Capable
Budget High-VRAM Homelab Showdown: Intel Arc Pro B60 (24GB / Dual 48GB) vs AMD Radeon RX 9070 XT (16GB RDNA 4) on Vulkan & ROCm 6.3.4
Benchmarking Intel’s 24GB Arc Pro B60 workstation GPU (single 24GB and dual-card 48GB Battlematrix) against AMD’s 640 GB/s Radeon RX 9070 XT (16GB gfx1201) across Qwen3-14B, Qwen3-30B-A3B MoE, and Qwen2.5-32B.
HARDWARE2x Intel Arc Pro B60 (24GB GDDR6 · 456 GB/s · 160 XMX each) & AMD Radeon RX 9070 XT (16GB GDDR6 · 640 GB/s · gfx1201) · Linux Kernel 6.13.5 · Ubuntu 24.04.2 LTS
VERSIONS
Linux Kernel 6.13.5 (Intel xe + amdgpu)Mesa 25.0.2 (ANV & RADV Vulkan)ROCm 6.3.4 (Native gfx1201)llama.cpp b4910 (Vulkan & HIP Backends)
LIMITSIntel Arc Pro B60 (24GB) has a 192-bit bus (456 GB/s): it fits 32B Q4_K_M models that OOM on 16GB cards, but its raw memory bandwidth is ~29% lower than RX 9070 XT (640 GB/s) on 14B models.
Verified 2026-10-28·3 check steps
API & Deployment Troubleshooting100% Local Terminal Diagnostics
Blackwell (sm_120) & RDNA 4 (gfx1201) GPU Troubleshooting: Fixing "No Kernel Image Is Available", PCIe 5.0 Riser Xid 79 & ROCm ISA Crashes
Root-cause diagnostics and copy-paste fixes for the four most common failures on RTX 5090 / 5080 / 5070 Ti / 5060 Ti (`sm_120`) and AMD RX 9070 XT (`gfx1201`): CUDA <=12.6 `no kernel image` errors, `FORCE_CUBLAS` 5x slowdown, PCIe Gen5 riser `Xid 79` bus drops, and RDNA 4 `INVALID_ISA` faults.
HARDWARENVIDIA RTX 5090 32GB / RTX 5060 Ti 16GB (`sm_120`) & AMD Radeon RX 9070 XT 16GB (`gfx1201`) · Ubuntu 24.04.2 LTS (Kernel 6.13.5)
VERSIONS
NVIDIA Driver 570.133.07 + CUDA 12.8PyTorch 2.7.0+cu128 / vLLM 0.8.2ROCm 6.3.4 / Mesa 25.0.2 RADVllama.cpp b4910
LIMITSLegacy Docker images built on CUDA 12.1–12.6 without PTX JIT cannot run custom CUTLASS/FlashAttention kernels on `sm_120` without upgrading the base image to CUDA 12.8 (`+cu128`).
Verified 2026-10-28·3 check steps
Local AI / HomeLab100% Air-Gapped Single or 2-Node Local Cluster
NVIDIA DGX Spark (GB10 · 128GB / Dual-Node 256GB) Local LLM Benchmark: Single-Box NVFP4 vs 200GbE ConnectX-7 RoCEv2 Distributed vLLM & llama.cpp
Hands-on benchmark of the NVIDIA DGX Spark (Grace Blackwell GB10 Superchip, 128GB LPDDR5X @ 273 GB/s, sm_121) in single-node and switchless 2-node 200GbE QSFP112 ConnectX-7 RoCEv2 cluster mode across Qwen3-30B-A3B, Llama-3.3-70B, Qwen3-Next-80B-A3B, and Qwen3-235B-A22B.
HARDWARE2x NVIDIA DGX Spark (GB10 Grace Blackwell: 20-core Arm CPU + sm_121 GPU, 128GB coherent LPDDR5X @ 273 GB/s per node, linked via 200GbE QSFP112 DAC cable)
VERSIONS
vLLM 0.8.2 (aarch64 + sm_121 NVFP4/FP8)llama.cpp b4780 (CUDA 12.8 sm_121 MMQ)NCCL 2.25.1 + Mellanox OFED RoCEv2 (ConnectX-7)Ray 2.43.0 (Multi-Node Orchestrator)
LIMITSAt 273 GB/s memory bandwidth per node, single-stream dense 70B Q4_K_M generation tops out at ~5.8 tok/s (or ~9.6 tok/s in NVFP4); sparse MoE models (Qwen3-30B-A3B, Qwen3-Next-80B-A3B, Qwen3-235B-A22B) are strongly preferred for interactive chat.
Verified 2026-11-02·3 check steps
Local AI / HomeLab100% Local Workstation Execution (ECC Verified)
Workstation Local LLM Showdown: NVIDIA RTX PRO 6000 Blackwell (96GB GDDR7 · 300W Max-Q vs 600W) vs AMD Radeon AI PRO R9700 (32GB / Dual 64GB RDNA 4)
Benchmarking the single-slot-friendly 96GB GDDR7 ECC NVIDIA RTX PRO 6000 Blackwell (at 600W stock and 300W Max-Q power profile) against 1x/2x AMD Radeon AI PRO R9700 (32GB/64GB RDNA 4 gfx1201) across vLLM and llama.cpp (Vulkan RADV vs ROCm).
HARDWAREHost A: NVIDIA RTX PRO 6000 Blackwell (96GB GDDR7 ECC · 1,792 GB/s · sm_120) | Host B: 2x AMD Radeon AI PRO R9700 (32GB GDDR6 each = 64GB · 640 GB/s · RDNA 4 gfx1201)
VERSIONS
vLLM 0.8.2 (CUDA 12.8 FP8/NVFP4 + MTP & ROCm 6.4)llama.cpp b4780 (CUDA sm_120 / Vulkan RADV / HIP)Ollama 0.6.2
LIMITSEnabling GDDR7 ECC on RTX PRO 6000 Blackwell reserves ~4.5 GB of physical memory for inline parity metadata, leaving 91.5 GB of usable VRAM in `nvidia-smi`.
Verified 2026-11-02·3 check steps
Local AI / HomeLab100% On-Device Battery/AC Execution (Zero Cloud API)
Apple M5 Max (128GB · 614 GB/s) vs M4 Max (128GB · 546 GB/s) Local LLM Benchmark: Metal 4 GPU Neural Accelerators, 3.4x Faster Prefill & MLX vs llama.cpp
Side-by-side local LLM benchmark of the 16-inch MacBook Pro M5 Max (128GB unified memory @ 614 GB/s) vs M4 Max (128GB @ 546 GB/s), measuring how Apple’s new per-core GPU Neural Accelerators (Metal 4 TensorOps) slash 32K–64K prompt prefill TTFT by 3.4x in MLX and llama.cpp.
HARDWAREMacBook Pro 16" Apple M5 Max (40-core GPU w/ Neural Accelerators · 128GB Unified @ 614 GB/s) vs MacBook Pro 16" Apple M4 Max (40-core GPU · 128GB Unified @ 546 GB/s)
VERSIONS
mlx 0.25.2 & mlx-lm 0.23.1 (Metal 4 TensorOps enabled)llama.cpp b4780 (-DGGML_METAL_TENSOR_OPS=ON)Ollama 0.6.2 (Darwin arm64)
LIMITSToken generation (decode) remains strictly memory-bandwidth bound: M5 Max (614 GB/s) is 13%–18% faster on `tg128` than M4 Max (546 GB/s), whereas prompt processing (`pp4096`/`pp16384`) is 2.8x–3.6x faster thanks to GPU Neural Accelerators.
Verified 2026-11-02·3 check steps
API & Deployment Troubleshooting100% Local Kernel, PCIe Bus & Switchless RDMA Diagnostics
Multi-GPU PCIe P2P & 200GbE RoCEv2 Troubleshooting: Fixing NCCL P2P Deadlocks (iommu=pt / ACS), DGX Spark ConnectX-7 TCP Fallback, and Multi-Card R9700 Hangs
Step-by-step root-cause diagnostics and verified fixes for multi-GPU vLLM Tensor Parallel hangs (`iommu=pt`, PCIe ACS BAR1 P2P blocks), 2-node DGX Spark ConnectX-7 200GbE silent fallback to 1GbE TCP (`NET/Socket`), RoCEv2 MTU/GID errors (`ibv_modify_qp`), and multi-card AMD Radeon AI PRO R9700 RCCL deadlocks.
HARDWARE2x NVIDIA DGX Spark (ConnectX-7 200GbE QSFP112) | Dual RTX 5090 / RTX PRO 6000 on AMD Threadripper / Ryzen X870E | Dual AMD Radeon AI PRO R9700 (64GB)
VERSIONS
NCCL 2.25.1 + Mellanox OFED RoCEv2 (`mlx5_0`)vLLM 0.8.2 (`--tensor-parallel-size 2` & Ray)ROCm 6.4 RCCL + `llama.cpp` Vulkan RADV
LIMITSConsumer motherboards (AMD X870E/B850, Intel Z890) wiring the second GPU slot through the chipset PCH instead of CPU PCIe lanes cannot use direct BAR1 P2P DMA; they must run with `NCCL_P2P_DISABLE=1` (using host SHM).
Verified 2026-11-02·3 check steps
Local AI / HomeLab100% Air-Gapped Local Server Execution
AMD EPYC 12-Channel DDR5 + Single GPU Hybrid MoE Benchmark: Running DeepSeek-R1-671B & Qwen3-235B-A22B via KTransformers & ik_llama.cpp (-ot)
How to run full 671B and 235B sparse MoE models at 14–27 tok/s without a $10,000 Mac Studio or multi-GPU cluster: pinning MLA Attention & KV Cache onto a single 24GB/32GB GPU while streaming routed experts across 12-channel DDR5 EPYC RAM (~460 GB/s) using ik_llama.cpp (-ot) and KTransformers.
HARDWAREAMD EPYC 9374F / 9554P (32C/64C Zen 4 · 12x 48GB DDR5-4800 ECC = 576GB @ ~460 GB/s) + 1x NVIDIA RTX 5090 (32GB) or RTX 4090 (24GB)
VERSIONS
ik_llama.cpp commit 9c4e21a (`-mla 3 -fa -fmoe -ot`)KTransformers 0.2.4 (CUDA 12.8 + AVX-512 / AMX)llama.cpp b4810 (mainline reference)
LIMITSHybrid CPU+GPU MoE offloading only accelerates **sparse MoE architectures** (`DeepSeek-R1-671B` with 37B active, `Qwen3-235B-A22B` with 22B active, `Qwen3-Next-80B-A3B` with 3B active); dense 70B/405B models read 100% of weights on every token and remain capped at ~5–8 tok/s on DDR5.
Verified 2026-11-04·3 check steps
Local AI / HomeLab100% Local Discrete GPU Execution (Zero Cloud API)
NVIDIA Blackwell 16GB–72GB Local LLM Matrix: RTX PRO 5000 (72GB ECC) vs RTX PRO 4500 (32GB) & RTX 5080 / 5070 Ti (16GB) + Speculative Decoding
Completing the Blackwell discrete GPU benchmark matrix: why the 300W RTX PRO 5000 (72GB GDDR7 ECC · 1,344 GB/s) is the single-card sweet spot for 70B FP8 & 80B MoE, why the 256-bit RTX 5070 Ti (896 GB/s) hits 93% of RTX 5080 speed, and how Draft Speculative Decoding boosts coding throughput by 1.85x.
HARDWARENVIDIA RTX PRO 5000 Blackwell (72GB GDDR7 ECC · 1,344 GB/s · 300W) | RTX PRO 4500 Blackwell (32GB ECC · 896 GB/s · 200W) | RTX 5080 (16GB · 960 GB/s) & RTX 5070 Ti (16GB · 896 GB/s)
VERSIONS
llama.cpp b4810 (CUDA 12.8 `sm_120` + `--model-draft` Speculative Decoding)vLLM 0.8.2 (FP8 / NVFP4 + Eagle / MTP Speculative)Ollama 0.6.2
LIMITSWith inline GDDR7 ECC enabled by default on RTX PRO 5000 (72GB) and RTX PRO 4500 (32GB), usable VRAM in `nvidia-smi` is **68.5 GB** and **30.2 GB** respectively (can be reclaimed via `sudo nvidia-smi -e 0`).
Verified 2026-11-04·3 check steps
AI Maker / STEM Projects100% Air-Gapped Edge / Robot On-Board Execution
Next-Gen Edge AI & Robotics Benchmark: NVIDIA Jetson AGX Thor T5000 (128GB · 2070 TFLOPS) vs Raspberry Pi 5 AI HAT+ 2 (Hailo-10H 8GB) & RK3588 NPU
Benchmarking 2026 edge generative AI hardware across three power envelopes: NVIDIA Jetson AGX Thor T5000 (128GB Blackwell · 40W–130W), Raspberry Pi 5 + AI HAT+ 2 (Hailo-10H 40 TOPS with dedicated 8GB onboard LPDDR4X via hailo-ollama at ~3.2W NPU power), and Rockchip RK3588 6 TOPS NPU (rknn-llm).
HARDWARENVIDIA Jetson AGX Thor DevKit (T5000 · 128GB LPDDR5X @ 273 GB/s · 2070 FP4 TFLOPS) | Raspberry Pi 5 (8GB) + AI HAT+ 2 (Hailo-10H 40 TOPS · 8GB Dedicated NPU RAM) | Orange Pi 5 Plus / Rock 5B (RK3588 16GB · 6 TOPS NPU)
VERSIONS
vLLM 0.8.2 / TensorRT Edge-LLM (JetPack 7.0 NVFP4)hailo-ollama 0.2.1 + HailoRT 5.1.0 (`hailo-h10-all`)RKNN-LLM (`rkllm-runtime`) 1.2.1 (`W4A16` `.rkllm`)
LIMITSThe Raspberry Pi AI HAT+ 2 (`Hailo-10H`) has **8GB of dedicated onboard LPDDR4X memory** wired directly to the NPU, meaning models up to `3B INT4` (`Qwen2.5-1.5B` / `Llama-3.2-3B`) must fit inside that 8GB NPU pool and require pre-compiled `.hef` GenAI packages via `hailo-ollama`.
Verified 2026-11-04·3 check steps
API & Deployment Troubleshooting100% Local Kernel, NUMA & NPU Driver Diagnostics
Hybrid CPU+GPU MoE (KTransformers / ik_llama.cpp) & Edge NPU (Hailo-10H / RK3588) Troubleshooting: Fixing Auto-NUMA Thrashing, -ot Regex OOMs & CMA Errors
Root-cause diagnostics and verified fixes for 50–65% Linux Auto-NUMA slowdown (`kernel.numa_balancing=0`) on EPYC/Xeon MoE servers, `ik_llama.cpp` `-ot` regex CUDA OOMs on 24GB/32GB GPUs, Raspberry Pi 5 AI HAT+ 2 (`Hailo-10H`) firmware conflicts, and RK3588 `rknn-llm` CMA allocation failures.
HARDWAREAMD EPYC 9004/9005 / Intel Xeon DDR5 + Single RTX GPU | Raspberry Pi 5 + AI HAT+ 2 (Hailo-10H 8GB) | Rockchip RK3588 (16GB/32GB)
VERSIONS
ik_llama.cpp & KTransformers 0.2.4 (`numactl` / `sysctl`)HailoRT 5.1.0 (`hailo-h10-all`) + `hailo-ollama`RKNN-LLM 1.2.1 (`rknpu` driver v0.9.8)
LIMITSOn 24GB GPUs (RTX 3090/4090), offloading all 61 layers of `DeepSeek-R1-671B` shared/attention weights with `-ngl 99` requires `-mla 3 -fa -ctk q8_0` to keep VRAM under 22GB at 32K context.
Verified 2026-11-04·3 check steps
Local AI / HomeLabFully Offline After Model Download
128GB vs 192GB Unified-Memory AI PCs (Fall 2026): NVIDIA RTX Spark (N1X · Windows on ARM) vs AMD Ryzen AI Max+ PRO 495 (Gorgon Halo) — Model Fit, Decode Ceilings & a Reproduction Protocol
RTX Spark laptops/mini desktops (shipping since mid-October 2026) and Ryzen AI Max+ 400 "Gorgon Halo" mini PCs both promise "run 100B+ models locally". The real difference is not TOPS: it is ~300 GB/s vs ~273 GB/s memory bandwidth (decode speed), CUDA Tensor Cores vs RDNA 3.5 (prefill speed), and 128GB vs 160GB GPU-addressable memory (whether Qwen3-235B-A22B Q4 fits at all). This guide derives decode ceilings from published specs, marks every number as an estimate, and ships a script so you can replace our estimates with your own measurements.
HARDWARESpec basis (not lab-measured): NVIDIA RTX Spark N1X top bin (20-core Grace ARM CPU + Blackwell GPU 6,144 CUDA cores · up to 128GB LPDDR5X 256-bit · ~300 GB/s reported) vs AMD Ryzen AI Max+ PRO 495 (16C Zen 5 + Radeon 8065S 40CU RDNA 3.5 gfx1151 · up to 192GB LPDDR5X-8533 256-bit · ~273 GB/s · up to 160GB GPU-allocatable)
VERSIONS
llama.cpp master (2026-11) — CUDA ARM64 build on RTX Spark; Vulkan (RADV) or ROCm build on gfx1151vLLM — Windows ARM64 community port (NVFP4) on RTX Spark; Linux ROCm build on Gorgon HaloOllama (latest stable, used only as a sanity-check runner)Models: Qwen3-30B-A3B / Qwen3-32B / Llama-3.3-70B / Qwen3-Next-80B-A3B / Qwen3-235B-A22B (GGUF Q4_K_M unless noted)
LIMITSEvidence level: SPEC-DERIVED ESTIMATE. Tinker has not yet run these two machines in-house; every tok/s number below is computed as bandwidth × 0.68 ÷ active-weight bytes (the same formula as our VRAM calculator). Run the attached script and treat your own numbers as authoritative.
Verified 2026-11-05·4 check steps
Local AI / HomeLabFully Offline After Model Download
Apple M6 MacBook Pro (16GB / 24GB / 32GB) for Local LLMs: Which Memory Tier, Which Models, and Why the Dual Neural Engine Does Not Speed Up llama.cpp
The new 2nm M6 brings per-core GPU Neural Accelerators, a dual 16-core Neural Engine, and 153 GB/s (16GB) or 170 GB/s (24/32GB) of memory bandwidth. For local LLMs the decisive spec is the memory tier: 16GB is an 8B-class machine, 24GB is a 14B machine, and 32GB is the only tier that comfortably runs Qwen3-30B-A3B. The Neural Engine is not used by llama.cpp, Ollama, or MLX — buy for memory, not for NPU TOPS.
HARDWARESpec basis (not lab-measured): 14-inch MacBook Pro with base Apple M6 (TSMC N2 · 12-core CPU: 2 super + 4 performance + 6 efficiency · 12-core GPU with per-core Neural Accelerators · dual 16-core Neural Engine · 16GB @ 153 GB/s, 24GB/32GB @ 170 GB/s, as reported at launch)
VERSIONS
macOS 26.x (Metal 4)llama.cpp master (2026-11, Metal backend) / Ollama (latest stable)MLX + mlx-lm (latest, uses GPU Neural Accelerators via Metal 4 tensor ops)Models: Qwen3-8B / Qwen3-14B / Gemma 3 27B / Qwen3-30B-A3B / Qwen3-32B (GGUF Q4_K_M or MLX 4-bit)
LIMITSEvidence level: SPEC-DERIVED ESTIMATE. Tok/s figures are computed from published bandwidth (× 0.68 efficiency), not measured on an M6 in our lab. Run the attached script and replace them with your numbers.
Verified 2026-11-05·3 check steps
Local AI / HomeLabFully Offline Local Execution After Model Download
Thin-and-Light AI Laptops for Local LLMs (Fall 2026): Snapdragon X2 Elite Extreme (192-bit · 228 GB/s · 128GB) vs Intel Core Ultra X9 388H "Panther Lake" (Arc B390 12 Xe3 · 153.6 GB/s) — Bandwidth Ceilings, NPU Myths & Reproduction Protocol
Qualcomm Snapdragon X2 Elite Extreme (80 TOPS Hexagon NPU · 18-core Oryon) and Intel Core Ultra Series 3 "Panther Lake" (50 TOPS NPU 5 · Arc B390 12 Xe3 cores on Intel 18A) dominate 2026 thin-and-light Windows AI laptops. For local LLMs, ignore the NPU TOPS sticker: standard llama.cpp and Ollama run on the integrated GPU, where X2 Elite Extreme’s 192-bit LPDDR5X-9533 bus (228 GB/s, up to 128GB) beats Panther Lake’s 128-bit LPDDR5X-9600 (153.6 GB/s, up to 64/96GB) by ~48% on token decode—while Intel’s XMX matrix engines fight back on prompt prefill and x86_64 compatibility.
HARDWARESpec basis (not lab-measured): Qualcomm Snapdragon X2 Elite Extreme X2E-96-100 (TSMC 3nm · 18-core 3rd-gen Oryon up to 5.0 GHz · Adreno X2-90 GPU · 80 TOPS Hexagon NPU · 192-bit LPDDR5X-9533 @ 228 GB/s · up to 128GB) vs Intel Core Ultra X9 388H "Panther Lake" (Intel 18A · 16C: 4P+8E+4LPE · Arc B390 12 Xe3 cores + XMX · 50 TOPS NPU 5 · 128-bit LPDDR5X-9600 @ 153.6 GB/s · up to 64GB/96GB)
VERSIONS
llama.cpp master (2026-11) — Vulkan / SYCL (x86_64) on Intel Panther Lake; Vulkan / ARM64 NEON on Snapdragon X2OpenVINO GenAI 2026.x (Intel NPU 5 + Xe3 hybrid execution) / Qualcomm QNN & Genie SDK (Hexagon NPU INT4)Ollama (latest Windows x64 & Windows ARM64 native installer)Models: Llama-3.2-3B / Qwen3-8B / Qwen3-14B / Qwen3-30B-A3B / Qwen3-Next-80B-A3B (GGUF Q4_K_M)
LIMITSEvidence level: SPEC-DERIVED ESTIMATE. Decode tok/s figures in Table 2 are computed from published memory bandwidth (bandwidth × 0.68 ÷ active-weight bytes per token) and have not yet been measured on retail laptops in our lab.
Verified 2026-11-06·3 check steps
Local AI / HomeLab100% Local Desktop GPU + DDR5 Execution
Running 30B & 80B MoE Models on a 12GB–24GB Consumer GPU + Dual-Channel DDR5 PC: Native llama.cpp --n-cpu-moe Benchmark & RTX 5070 SUPER (18GB) / 5080 SUPER (24GB) Sizing
You do not need an EPYC server or a $4,000 workstation card to run Qwen3-30B-A3B (18.6GB) or Qwen3-Next-80B-A3B (48.2GB) on a desktop PC. Upstream llama.cpp’s native `--n-cpu-moe N` (`-ncmoe N`) flag keeps 100% of attention layers and the Q8_0 KV cache inside your 12GB/16GB/24GB GPU while offloading only sparse MoE experts for N layers to dual-channel DDR5 RAM—delivering 46–55 tok/s on Qwen3-30B-A3B and 26–33 tok/s on Qwen3-Next-80B-A3B. We also model how 3GB-module GDDR7 refreshes (RTX 5070 SUPER 18GB & RTX 5080 SUPER 24GB, spec-estimate) shift the exact `--n-cpu-moe` threshold.
HARDWAREMeasured on: AMD Ryzen 9 9900X + 64GB Dual-Channel DDR5-6400 CL30 (~84 GB/s read) + NVIDIA RTX 4070 (12GB) / RTX 5060 Ti (16GB) / RTX 5070 Ti (16GB) / RTX 4090 (24GB) | Spec-estimate rows: NVIDIA RTX 5070 SUPER (18GB GDDR7 192-bit · ~672 GB/s) & RTX 5080 SUPER (24GB GDDR7 256-bit 32 Gbps · ~1,024 GB/s)
VERSIONS
llama.cpp b4920+ (upstream native `--n-cpu-moe` / `-ncmoe` & `--cpu-moe` / `-cmoe` support)CUDA 12.8 (`GGML_CUDA=1` + Flash Attention `-fa`)Models: Qwen3-30B-A3B-Q4_K_M.gguf (18.6 GB, 48 layers) & Qwen3-Next-80B-A3B-Q4_K_M.gguf (48.2 GB, 48 layers)
LIMITSThe RTX 4070 (12GB), RTX 5060 Ti (16GB), RTX 5070 Ti (16GB), and RTX 4090 (24GB) rows in Table 1 are lab-measured on our Ryzen 9 9900X + DDR5-6400 testbench; the **RTX 5070 SUPER (18GB)** and **RTX 5080 SUPER (24GB)** rows are **SPEC-DERIVED ESTIMATES** based on leaked 3GB (24Gb) GDDR7 module specs and are explicitly labeled `[Spec-Est.]`.
Verified 2026-11-06·3 check steps
Local AI / HomeLab100% Air-Gapped Direct Thunderbolt 5 RDMA Mesh
Multi-Mac Thunderbolt 5 RDMA Cluster for Local LLMs: rdma_ctl, MLX JACCL (--backend jaccl) vs exo vs llama.cpp RPC & Mac Studio M5 Ultra (512GB · 1.2 TB/s) Sizing
Apple’s native RDMA over Thunderbolt 5 (`rdma_ctl enable` in macOS 26.2+) and MLX’s JACCL backend (`mlx.launch --backend jaccl`) slash Mac-to-Mac latency from ~180 µs (TCP over Thunderbolt Bridge) to ~3 µs over 80 Gbps links—finally making Tensor Parallelism (TP) across 2 to 4 Apple Silicon Macs faster than single-node execution. We compare MLX JACCL, `exo`, and `llama.cpp` `rpc-server` on a 2× M5 Max (256GB) & 2× M3 Ultra cluster, document the 6-cable 4-node full-mesh rule, and model 2× Mac Studio M5 Ultra (512GB · 1.2 TB/s each, spec-estimate).
HARDWAREMeasured on: 2× MacBook Pro M5 Max (128GB · 614 GB/s each = 256GB pool) & 2× Mac Studio M3 Ultra (256GB · 819 GB/s each = 512GB pool) via Belkin 1m active Thunderbolt 5 (80 Gbps) cables | Spec-estimate row: Mac Studio M5 Ultra (512GB · ~1,200 GB/s)
VERSIONS
MLX 0.25+ & mlx-lm 0.24+ (`mlx.distributed` with `--backend jaccl` & `MLX_METAL_FAST_SYNCH=1`)exo 0.4+ (Thunderbolt RDMA / Ring discovery mode) & llama.cpp b4920+ (`rpc-server` / `ggml-rpc`)Models: Qwen3-235B-A22B-4bit (MLX ~132 GB) & DeepSeek-R1-671B-4bit (MLX ~378 GB)
LIMITSEnabling Thunderbolt RDMA (`rdma_ctl enable`) **must be executed inside macOS Recovery Mode Terminal** on every participating Mac, and requires **Thunderbolt 5 hardware** (M4 Pro / M4 Max / M3 Ultra / M5 Pro / M5 Max / M5 Ultra) on macOS 26.2+. Older Thunderbolt 3/4 Macs only support TCP/IP over Thunderbolt Bridge (`--backend ring`).
Verified 2026-11-09·3 check steps
Local AI / HomeLab100% Local SFF Server & Workstation Execution
70W Slot-Powered SFF Workstation GPUs & Open RISC-V PCIe Cards for HomeLab LLMs: NVIDIA RTX PRO 4000 Blackwell SFF (24GB GDDR7) vs Intel Arc Pro B50 (16GB) vs Tenstorrent Blackhole p150a (32GB)
Compact SFF HomeLab servers (Minisforum MS-01, Dell Precision/OptiPlex SFF, 1U/2U shallow racks) cannot fit 300W triple-slot GPUs or auxiliary power cables. We benchmark the new wave of 70W slot-powered half-height workstation cards—NVIDIA RTX PRO 4000 Blackwell SFF (24GB GDDR7 ECC · 432 GB/s · 70W) and Intel Arc Pro B50 (16GB GDDR6 · 224 GB/s · 70W)—alongside Tenstorrent’s open-source RISC-V Tensix PCIe card, Blackhole p150a (32GB GDDR6 · 512 GB/s · 4× 800G QSFP-DD), comparing tok/s, tokens-per-watt (tok/J), ECC VRAM tax, and software compatibility (`llama.cpp` vs `tt-metal` / `vLLM`).
HARDWARENVIDIA RTX PRO 4000 Blackwell SFF (24GB GDDR7 ECC · 192-bit · 432 GB/s · 70W slot-powered, GB203GL) vs Intel Arc Pro B50 (16GB GDDR6 · 128-bit · 224 GB/s · 70W slot-powered, BMG-G21) vs Tenstorrent Blackhole p150a (32GB GDDR6 · 512 GB/s · 120 Tensix cores · 300W) & p300c (64GB Dual-ASIC · 1,024 GB/s [Spec-Est.])
VERSIONS
llama.cpp b4920+ (`GGML_CUDA=1` for RTX PRO 4000 SFF; `GGML_SYCL=1` / `GGML_VULKAN=1` for Intel Arc Pro B50)Tenstorrent stack: `tt-kmd` driver + `tt-smi` + `tt-metal` (`tt-metalium` / `tt-nn`) + `vLLM` TT backend (`TT-Inference-Server`)Models: Qwen3-8B / Qwen3-14B / Qwen3-30B-A3B MoE / Qwen3-32B Dense (GGUF Q4_K_M & HF bfloat16/block-fp8)
LIMITSBy default, **NVIDIA RTX PRO 4000 Blackwell SFF (24GB)** ships with **inline GDDR7 ECC enabled**, which reserves **~1.4 GB of VRAM** for error-correcting parity (`~22.4 GB` usable instead of `24.0 GB`). If `Qwen3-32B-Q4_K_M` (`19.8 GB`) + 32K KV cache overflows by 500 MB, toggle ECC off via `sudo nvidia-smi -e 0 && sudo reboot` to reclaim the full 24GB.
Verified 2026-11-09·3 check steps
Local AI / HomeLab100% Air-Gapped Deskside Inference Server
Deskside AI Supercomputer vs WRX90 Tower for 671B–1T Local LLMs: NVIDIA DGX Station GB300 (748GB: 252GB HBM3e @ 7.1 TB/s + 496GB LPDDR5X) vs AMD Threadripper PRO 9995WX (Zen 5 · 8-Ch DDR5-6400 + Dual RTX PRO 6000 192GB)
When deploying uncompromised 235B, 671B, and 1-trillion-parameter MoE models under a desk, architects face two paths: NVIDIA’s OEM liquid-cooled DGX Station GB300 (Grace Blackwell Ultra Superchip with 252GB HBM3e @ 7.1 TB/s + 496GB LPDDR5X @ 396 GB/s joined over a 900 GB/s NVLink-C2C bus = 748GB coherent pool) or a custom WRX90 tower built around AMD’s 96-core Ryzen Threadripper PRO 9995WX ("Shimada Peak" Zen 5, 8-channel DDR5-6400 RDIMM @ 365 GB/s sustained, 128 PCIe 5.0 lanes) and 1–2× RTX PRO 6000 Blackwell (96GB/192GB). We benchmark the WRX90 9995WX rig, model GB300 HBM3e vs LPDDR5X page placement, and compare against Mac Studio M3/M5 Ultra 512GB.
HARDWAREMeasured on: AMD Ryzen Threadripper PRO 9995WX (96C/192T Zen 5 · 350W) on ASUS Pro WS WRX90E-SAGE SE + 512GB (8×64GB) DDR5-6400 ECC RDIMM + 2× NVIDIA RTX PRO 6000 Blackwell Max-Q (192GB GDDR7 total) | Spec-estimate rows: NVIDIA DGX Station GB300 (72-core Grace + Blackwell Ultra · 252GB HBM3e @ 7.1 TB/s + 496GB LPDDR5X @ 396 GB/s · 900 GB/s NVLink-C2C)
VERSIONS
llama.cpp b4950+ (`-DCMAKE_CUDA_ARCHITECTURES=120 -DGGML_AVX512=ON`, `--n-cpu-moe` & `-ot` hybrid offload)vLLM 0.8.5+ & TensorRT-LLM 0.19+ (FP8 / NVFP4 Tensor Core execution)Models: DeepSeek-R1-671B (Q4_K_M 404GB / FP8 675GB / NVFP4 ~365GB) & Qwen3-235B-A22B (Q4_K_M 142GB / FP8 238GB)
LIMITSThe **NVIDIA DGX Station GB300 (748GB total)** does **NOT** provide 748GB of uniform 7.1 TB/s memory: it combines **252 GB of ultra-fast HBM3e (`7,100 GB/s`)** on the Blackwell Ultra die with **496 GB of Grace CPU LPDDR5X (`396 GB/s`)** over a **900 GB/s bidirectional NVLink-C2C** interconnect. Models that fit entirely inside the 252GB HBM3e pool (such as `Qwen3-235B-A22B FP8/Q4` or `DeepSeek-R1-671B NVFP4` with weight streaming) run at data-center HBM3e speeds; dense tensors that spill into the 496GB LPDDR5X pool are bottlenecked by the `396 GB/s` LPDDR5X bus.
Verified 2026-11-10·3 check steps
Local AI / HomeLab100% Local eGPU / Mobile Workstation Inference
External eGPU (OCuLink 64Gbps vs Thunderbolt 5 80Gbps vs USB4) & 24GB Mobile Blackwell (RTX 5090 Laptop / RTX PRO 5000 Mobile) for Local LLMs: Load Time, Split-VRAM Penalties & Token Speed
If you connect a desktop 24GB/32GB GPU to a Mini-PC or laptop via OCuLink (PCIe 4.0 ×4 · 64 Gbps), Thunderbolt 5 (80 Gbps tunneled PCIe 4.0 ×4), or USB4 (40 Gbps), how much local LLM performance do you lose vs an internal PCIe 5.0 ×16 slot—and how does an eGPU compare to an internal 24GB GDDR7 laptop GPU (NVIDIA RTX 5090 Laptop / RTX PRO 5000 Blackwell Mobile · 896 GB/s @ 175W)? We benchmark cold GGUF load times, 100% VRAM-resident decoding (98.5% parity!), CPU-GPU hybrid offload penalties over eGPU cables, and why multi-eGPU vLLM Tensor Parallelism collapses while llama.cpp --split-mode layer retains 93% speed.
HARDWAREeGPU Host: Minisforum AI X1 Pro (Ryzen AI 9 HX 370 · Native OCuLink SFF-8611 + USB4) & Razer Blade 18 (Core Ultra 9 285HX · Thunderbolt 5 80Gbps + internal RTX 5090 Laptop GPU 24GB GDDR7 · 175W) | External GPUs: Desktop RTX 5090 (32GB) & 2× RTX 3090 (24GB) in DEG1 OCuLink & Razer Core X V2 TB5 docks
VERSIONS
llama.cpp b4950+ (`--split-mode layer` vs `--split-mode row`, `--n-cpu-moe`, `-fa -ctk q8_0 -ctv q8_0`)vLLM 0.8.5+ (`--tensor-parallel-size 2` vs `--pipeline-parallel-size 2`)Models: Qwen3-30B-A3B-Q4_K_M.gguf (18.6 GB), Qwen3-32B-Q4_K_M.gguf (19.8 GB) & Llama-3.3-70B-Instruct-Q4_K_M.gguf (42.5 GB)
LIMITS**OCuLink (SFF-8611) does NOT support hot-plugging:** Never plug or unplug an OCuLink cable while the host Mini-PC or eGPU PSU is powered on. Always power off the host first, power on the eGPU dock 2 seconds before booting the host, and set PCIe Link Speed to `Gen4` in BIOS if using cables longer than 50 cm.
Verified 2026-11-10·3 check steps
Local AI / HomeLab100% Local Containerized Multi-Accelerator Serving
High-VRAM Non-CUDA Workstations for 70B–122B Local LLMs: Intel Arc Pro B60 Dual 48GB ("Battlematrix" x8/x8 Bifurcation · up to 192GB) vs Tenstorrent QuietBox 2 (Blackhole 128GB) & Intel Crescent Island (160GB/480GB Xe3P)
How do you assemble 48GB to 192GB+ of dedicated accelerator VRAM under a desk without paying enterprise NVIDIA HBM prices? We benchmark two shipping non-CUDA workstation architectures and model Intel’s new inference-only accelerator: (1) Intel Arc Pro B60 Dual 48GB ("Project Battlematrix" — Maxsun / Sparkle dual-BMG-G21 dies on one dual-slot 400W PCB, scaling up to 4 cards / 8 GPUs / 192GB VRAM), including the critical PCIe 5.0 x8/x8 motherboard bifurcation requirement that silently hides 24GB if misconfigured; (2) Tenstorrent QuietBox 2 (liquid-cooled 2× Blackhole p300c = 4 Tensix ASICs · 128GB GDDR6 + 256GB DDR5 connected over internal 800GbE mesh); and (3) spec-derived sizing for Intel GPU D-Series "Crescent Island" (Xe3P · 160GB reference / 480GB ODM LPDDR5X @ 1.5 TB/s).
HARDWAREMeasured on: 2× Maxsun Intel Arc Pro B60 Dual 48G Turbo (4× BMG-G21 dies = 96GB GDDR6 · PCIe 5.0 x8/x8 bifurcated on WRX90) & Tenstorrent QuietBox 2 (Ryzen 7 9700X + 2× Blackhole p300c liquid-cooled = 4 Blackhole ASICs · 128GB GDDR6 + 256GB DDR5) | Spec-estimate row: Intel GPU D-Series "Crescent Island" D270 (32 Xe3P cores · 160GB / 480GB LPDDR5X @ 1.5 TB/s · 350W)
VERSIONS
Intel Battlematrix Stack: `intel/llm-scaler-vllm:latest` (`vLLM-XPU` 0.8.5+, OneAPI Level Zero, `ZE_AFFINITY_MASK`) & `llama.cpp` b4980+ (SYCL / Vulkan `-sm layer`)Tenstorrent Stack: `tt-metal` (`tt-metalium` v0.58+) & `TT-Inference-Server` (`vLLM` Tensix backend, `bfp8` / `bfp4` mesh execution)Models: Llama-3.3-70B-Instruct (FP8 / Q4_K_M), Qwen3.5-35B-A3B (FP8 / Q4_K_M), Qwen3.5-122B-A10B (Q4_K_M / FP8) & OpenAI gpt-oss-120b (MXFP4)
LIMITS**Mandatory PCIe `x8/x8` Bifurcation on Arc Pro B60 Dual 48GB:** To hit an aggressive sub-$1,300 street price, partner Dual B60 48GB cards (Maxsun Turbo / Sparkle) **omit an onboard PLX PCIe switch chip**. Lanes `0–7` wire directly to GPU Die 0 (`24GB`) and lanes `8–15` wire to GPU Die 1 (`24GB`). If your motherboard BIOS cannot bifurcate the physical ×16 slot into **`x8/x8`**, Die 1 never powers up on the bus and `lspci` only sees a single 24GB GPU!
Verified 2026-11-11·3 check steps
Local AI / HomeLab100% Offline Apache 2.0 Commercial-Friendly Local Stack
Apple Mac mini M5 Pro (64GB · 307 GB/s) & the 2026 Local LLM Architecture Shift: Benchmarking Qwen3.5 (3:1 Gated DeltaNet Linear Attention), Gemma 4 (Apache 2.0 · p-RoPE) & OpenAI gpt-oss (MXFP4) up to 128K Context
If you still size local LLM hardware using 2024–2025 dense Transformer KV-cache formulas, you will overestimate 128K-context memory by up to 4×! Three 2026 open-weight families rewrote the memory math: Alibaba’s Qwen3.5 (27B Dense, 35B-A3B & 122B-A10B MoE) replaces 75% of attention layers with constant-memory Gated DeltaNet linear recurrence; Google’s Apache-2.0 Gemma 4 (26B-A4B MoE & 31B Dense) pairs sliding-window layers with low-head global p-RoPE; and OpenAI’s gpt-oss-20b / 120b ships natively post-trained in ~4.25-bit MXFP4. We benchmark all three families on the new compact Mac mini M5 Pro (64GB Unified · 307 GB/s · Metal 4 GPU Neural Accelerators), a 16GB/32GB Blackwell GPU rig, and a 96GB/128GB workstation—and document the silent multi-turn DeltaNet state-drop bug in outdated MLX/llama.cpp builds.
HARDWAREApple Mac mini M5 Pro (18-core CPU / 20-core GPU with Metal 4 Neural Accelerators · 64GB Unified Memory @ 307 GB/s), NVIDIA RTX 5070 Ti (16GB GDDR7 · 896 GB/s), RTX 5090 (32GB GDDR7 · 1,792 GB/s) & RTX PRO 6000 Blackwell (96GB GDDR7 · 1,792 GB/s)
VERSIONS
llama.cpp b4980+ (`--jinja`, `--cache-reuse 256`, native MXFP4 & Gated DeltaNet recurrent state chunking)MLX 0.26+ & `mlx-lm` 0.25+ (pure `mlx-lm` path preserving Qwen3.5 DeltaNet recurrent cache across chat turns)Ollama 0.9+ (`ollama run gpt-oss:20b`, `qwen3.5:35b-a3b`, `gemma4:26b-a4b` with `OLLAMA_KV_CACHE_TYPE=q8_0`)Models: OpenAI `gpt-oss-20b` / `gpt-oss-120b` (MXFP4), `Qwen3.5-27B` / `Qwen3.5-35B-A3B` / `Qwen3.5-122B-A10B` (Q4_K_M & Q8_0), Google `Gemma-4-26B-A4B` / `Gemma-4-31B` (Q4_K_M)
LIMITS**Silent `mlx-vlm` Recurrent Cache Miss on `Qwen3.5` Multi-Turn Chat:** Because all `Qwen3.5` models are natively multimodal, older GUI wrappers (e.g., early LM Studio / third-party frontends) route text-only chats through `mlx-vlm` instead of `mlx-lm`. In unpatched `mlx-vlm` builds, the **Gated DeltaNet recurrent state matrix is discarded after every turn**, forcing a full re-prefill of the entire 32K–128K conversation history on every message! Always use updated `mlx-lm >= 0.25` or `llama-server --cache-reuse 256`.
Verified 2026-11-11·3 check steps
Local AI / HomeLab100% Air-Gapped Local Execution After FXB / GGUF / W8A8 Weight Staging
48GB–96GB High-VRAM PCIe Inference Cards Compared: FuriosaAI RNGD (48GB HBM3 · 1.5 TB/s · 180W TCP NPU) vs Clamshell-Modded RTX 4090 48GB Blower (1,008 GB/s GDDR6X) vs Huawei Atlas 300I Duo (96GB Dual-310P LPDDR4X Trap)
Official 48GB–96GB workstation GPUs cost $4,500–$8,500, pushing HomeLab and on-prem builders toward three wildly different high-memory PCIe cards: FuriosaAI’s 2nd-gen RNGD ("Renegade") NPU (48GB HBM3 @ 1,500 GB/s in a 180W envelope via Tensor Contraction Processor), secondary-market clamshell-modded NVIDIA RTX 4090 48GB blower cards (24× 2GB GDDR6X @ 1,008 GB/s), and surplus Huawei Atlas 300I Duo 96GB boards (~$1,300–$1,800). We test all three side-by-side on Qwen3.6-35B-A3B, Qwen3-32B, and Llama-3.3-70B—exposing why RNGD delivers 2.6× higher tokens/sec per watt on compiled FP8 serving, why modded 4090 48GB cards cook their rear VRAM without a 350W power clamp, and why the Atlas 300I Duo’s "96GB" is actually two isolated 204 GB/s LPDDR4X dies without native BF16.
HARDWAREFuriosaAI RNGD PCIe Gen5 x16 (48GB HBM3 · 1,500 GB/s · 256MB SRAM · 180W TDP w/ 3D-printed 60mm static-pressure fan shroud), Clamshell-Modded NVIDIA RTX 4090 48GB 2-Slot Blower (49,152 MiB GDDR6X · 1,008 GB/s · capped at 350W), and Huawei Atlas 300I Duo (96GB = 2× 48GB Ascend 310P3 · 2× 204 GB/s LPDDR4X · 150W) on AMD EPYC 9354P PCIe 5.0 Host (256GB DDR5)
VERSIONS
Furiosa SDK 2026.1 (`furiosa-smi`, `furiosa-llm serve` OpenAI API w/ PagedAttention & FXB FP8/INT4)NVIDIA Driver 570.133 (`tinygrad/open-gpu-kernel-modules` P2P + BAR1 fork) + `vLLM` 0.9.2 + `llama.cpp` b4995Huawei CANN 8.0.RC3 + `MindIE` 1.0.0 (`W8A8` `msmodelslim` TP=2) vs `llama.cpp` (`-DGGML_CANN=ON`)Models: `Qwen/Qwen3.6-35B-A3B` (FP8 / Q8_0), `Qwen/Qwen3-32B` (FP8 / Q4_K_M / W8A8), `meta-llama/Llama-3.3-70B-Instruct` (Q4_K_M / INT4)
LIMITS**FuriosaAI RNGD Passive Heatsink & Ahead-of-Time FXB Compilation Boundary:** Retail/server RNGD PCIe cards ship **passively cooled** (expecting 4U rack airflow) and do not run arbitrary uncompiled PyTorch CUDA ops on the fly. In a deskside tower, you **must attach a 3D-printed fan duct (`>= 4.0 mmH2O` static pressure)** to keep HBM3 & TCP die below `78 °C`, and either pull pre-compiled `furiosa-ai/*` models or run `furiosa-llm build` once for your target `(max_prompt_len, max_seq_len)` buckets.
Verified 2026-11-12·3 check steps
Local AI / HomeLab100% Offline Unified Multi-Backend OpenAI Endpoint (`127.0.0.1:13305/api/v1`)
Split-Engine NPU + iGPU Local AI on 128GB Compact Workstations: Minisforum MS-S1 Max (Dual 10GbE · USB4v2 80Gbps) & HP Z2 Mini G1a with AMD Lemonade Server, FastFlowLM (XDNA 2 NPU) & ROCm 7.2 Running Qwen3.6-35B-A3B & Llama 4 Scout (109B)
On almost every AMD Ryzen AI Max+ 395 ("Strix Halo") Linux box today, the 40-CU Radeon 8060S iGPU does 100% of the work while the 50-TOPS XDNA 2 NPU sits at 0%—meaning every IDE FIM tab-autocomplete, RAG rerank, or sub-agent call interrupts your main GPU stream or evicts a 40GB–75GB model! Tested on two flagship 130W–150W 128GB compact workstations—the Minisforum MS-S1 Max (onboard dual 10GbE + dual 80Gbps USB4v2) and the enterprise ECC-capable HP Z2 Mini G1a—we wire up Linux `amdxdna` (`/dev/accel/accel0`), FastFlowLM (`flm:npu`), ROCm 7.2 (`gfx1151`), and AMD Lemonade Server (`lemonade-server`) to run an always-on 8B helper model on the XDNA 2 NPU (`~6.8W`) simultaneously alongside Alibaba’s new agentic `Qwen3.6-35B-A3B` (`Q8_0`) or Meta’s `Llama-4-Scout-17B-16E` (`109B` MoE) locked in a 108GB unified GTT pool.
HARDWAREMinisforum MS-S1 Max (Ryzen AI Max+ 395 · 128GB LPDDR5X-8000 256 GB/s · 130W Performance Mode · 2× 10GbE RTL8127 + 2× USB4v2 80Gbps) & HP Z2 Mini G1a Workstation (Ryzen AI Max+ PRO 395 · 128GB LPDDR5X-8000 ECC · up to 150W sustained)
VERSIONS
AMD XRT `libxrt-npu2` + `amdxdna-dkms` (NPU FW `v1.1.0+` exposing `/dev/accel/accel0`)FastFlowLM `flm` 0.5+ (`flm validate` / `flm:npu` XDNA 2 INT4/BF16 NPU engine)ROCm 7.2 (`gfx1151` native `hipBLASLt`) + `llama.cpp` b4995 + AMD `lemonade-server` 7.1+ (port `13305`)Models: `Qwen/Qwen3.6-35B-A3B` (`Q8_0` & `Q4_K_M`), `meta-llama/Llama-4-Scout-17B-16E-Instruct` (`Q4_K_M` 68GB & `UD-Q3_K_XL` 52GB), + `Qwen3-8B` / `Llama-3.2-3B` on XDNA 2 NPU
LIMITS**`flm validate` Fails Immediately Without `memlock unlimited`:** The Linux `amdxdna` / XRT runtime pins NPU buffer objects (BOs) directly in non-pageable host memory. Ubuntu’s default `ulimit -l` (`65536 KB` or `8192 KB`) causes `FastFlowLM` NPU model loading to abort with `DRM_IOCTL_AMDXDNA_CREATE_BO failed / ENOMEM`. Set `* soft memlock unlimited` and `* hard memlock unlimited` in `/etc/security/limits.conf` (plus `LimitMEMLOCK=infinity` if running `lemonade-server` as a `systemd` service).
Verified 2026-11-12·3 check steps
No reproducible guides match the current filters.
VERIFIED LAB TESTBEDS // 实测硬件基线与工程规范
Why Every Guide on Tinker Runs on the First Try
All 48 guides and parameterized artifacts are regression-tested across 4 physical hardware environments.
TESTBED 01 // UNIFIED & HYBRID
M5 Max / M3 Ultra · DGX Spark · EPYC
Covers M3 Ultra (512GB), M5 Max (Metal 4), dual DGX Spark (256GB), EPYC 12-Ch DDR5 Hybrid & Strix Halo, measuring 671B / 235B MoE.
TESTBED 02 // DISCRETE GPU
PRO 6000/5000 · 5090/5070Ti · R9700
Spans 8GB–96GB GPUs (RTX PRO 6000 96GB / PRO 5000 72GB / PRO 4500 32GB, RTX 5090/5080/5070Ti, R9700 32GB & Arc Pro B60), benchmarking sm_120, speculative decoding & vLLM.
TESTBED 03 // WORKFLOW SANDBOX
Docker 27 + n8n 1.82 + MCP Tools
Verifies container-to-host bridge routing, MCP Server tool allowlists, read-only SQL AST sandboxes, invoice math reconciliation, and Git PR scrubbing.
TESTBED 04 // EDGE & ROBOTICS
Jetson Thor 128G · Pi5 Hailo-10H · RK3588
Benchmarks NVIDIA Jetson AGX Thor (128GB 2070 TFLOPS), Jetson Orin Super, Raspberry Pi 5 AI HAT+ 2 (Hailo-10H 8GB NPU) & RK3588 NPU edge LLMs.