Watch The Video Or Read The Article (Or Do Both )

The Problem: Model Choice is Harder Than It Looks

When a new model releases, the community is usually flooded with benchmark numbers, vibe checks, and Twitter threads — but almost none of that information is specific to your hardware. A 70B model that runs beautifully on an A100 cluster might reduce your RTX 3070 to a space heater.

The real variables you need to reason about are:

• How much VRAM does the model need at your desired quantization?
• What quantization level gives the best quality that still fits in your memory?
• Will the model run GPU-only, CPU+GPU hybrid, or CPU-only? Each changes performance dramatically.
• For Mixture-of-Experts (MoE) models like DeepSeek or Mixtral, only a fraction of parameters are active per token — so the effective memory requirement is much lower than the parameter count implies.
• How does context window length affect memory requirements?

These are all solvable questions, but answering them manually for each model you want to try is tedious. llmfit automates all of it.

What is llmfit?

llmfit is described as: "Hundreds of models & providers. One command to find what runs on your hardware."

It's a terminal-first tool that:

• Detects your CPU, RAM, GPU model, VRAM, and acceleration backend (CUDA, Metal, ROCm, etc.)
• Maintains a database of hundreds of models from providers like Meta, Mistral, Qwen, Google, DeepSeek, Microsoft, IBM, and more
• Scores each model across four dimensions: Quality, Speed, Fit, and Context
• Dynamically selects the best quantization for each model on your hardware
• Shows you which models are Perfect, Good, Marginal, or Too Tight to run
• Integrates with Ollama, llama.cpp, and MLX so you can download models directly from the UI

It ships with an interactive TUI (the default) and a classic CLI mode for scripting and automation. The entire thing is a single binary — no Python dependencies, no Docker, no drama.

Installation

Getting started takes about thirty seconds. Pick your platform:

macOS / Linux — One-liner

Homebrew

Windows / Cross-platform via Cargo

If you don't have Rust installed yet, grab it from rustup.rs. Once llmfit is installed, just type:

And you're in the interactive TUI.

The Interactive TUI: Your Hardware-Aware Model Browser

When you launch llmfit, the TUI greets you with a system bar at the top showing your detected specs — total RAM, CPU cores, GPU name, VRAM, and the active acceleration backend. Below that is a scrollable, filterable table of every model in the database, ranked by composite score.

Each row in the table shows:

• Composite score (0–100)
• Estimated tokens per second
• Best quantization for your hardware (Q8_0, Q4_K_M, Q2_K, etc.)
• Run mode (GPU / MoE / CPU+GPU / CPU)
• Memory usage as a % of available memory
• Use-case category (Coding, Reasoning, Chat, Embedding, Multimodal, General)

Navigation is keyboard-driven and fast:

Pressing Enter on any model opens a detailed view with full quantization breakdown, context window specs, and provider availability. Pressing d opens a provider picker if multiple runtimes are installed — and kicks off the download with a live progress indicator right in the terminal.

The Scoring Engine: Four Dimensions, One Composite Score

The heart of llmfit is its multi-dimensional scoring system. Every model gets four sub-scores (0–100 each), which are combined into a weighted composite score. The weights shift depending on the use-case category.

For example, Chat workloads weight Speed at 0.35 because responsiveness matters most in interactive sessions. Reasoning workloads weight Quality at 0.55 because you'd rather wait a bit longer for a correct answer.

Models that don't fit in memory at all (Too Tight) are always ranked at the bottom, regardless of their quality score.

Fit Levels Explained

Beyond the composite score, llmfit assigns every model a Fit Level that summarises memory compatibility:

The Fit filter (press f in the TUI) lets you instantly narrow the list to only models at or above a chosen fit level. If you want guaranteed smooth GPU performance, filter to Perfect. If you're willing to push boundaries, include Good or Marginal.

Speed Estimation: The Math Behind the Numbers

The token generation speed estimates in llmfit aren't guesses — they're derived from how LLM inference actually works at the hardware level.

Token generation in an LLM is memory-bandwidth-bound: generating each token requires reading the full model weights from VRAM once. So throughput is fundamentally limited by how fast your GPU can move data from memory to compute units.

The formula llmfit uses:

The 0.55 efficiency factor accounts for kernel overhead, KV-cache reads, and memory controller effects. This approach is validated against published llama.cpp benchmarks on Apple Silicon and NVIDIA hardware.

llmfit maintains a bandwidth lookup table covering around 80 GPUs across NVIDIA (consumer and datacenter), AMD (RDNA and CDNA), and Apple Silicon families. For GPUs it doesn't recognise, it falls back to per-backend speed constants.

Plan Mode: What Hardware Do I Need?

Most hardware-matching tools ask 'what fits my current hardware?' Plan Mode flips the question: given a model and a target performance profile, what hardware would I need?

Press p on any model in the TUI and you'll see an interactive planner where you can set:

• Context length (how long your prompts and responses will be)
• Quantization level (Q8, Q4, Q2, or model-specific variants)
• Target tokens per second

Plan Mode responds with:

• Minimum and recommended VRAM / RAM / CPU cores
• Feasibility of GPU, CPU+GPU hybrid, and CPU-only run paths
• Upgrade deltas — how much more VRAM you'd need to hit a better fit tier

This is extremely useful if you're deciding between GPU upgrades. You can pick your dream model, set a performance target, and see exactly what hardware delta you're shopping for.

CLI Mode and JSON Output for Automation

The TUI is great for exploration, but llmfit also has a full CLI mode for scripting, CI pipelines, and agent integration. Add --cli or any subcommand:

The --json flag is available on all subcommands and produces stable, machine-readable output. This makes llmfit a natural fit as a hardware-aware model selector in AI agent pipelines and automation scripts.

Runtime Provider Integration

llmfit integrates with three local AI runtimes out of the box:

• Ollama — Detects installed models via the local API, shows a green ✓ next to each, and handles downloads via POST /api/pull.
• llama.cpp — Maps HuggingFace model names to known GGUF repos and downloads files directly to the local cache.
• MLX — Native support for Apple Silicon, pulling from mlx-community on Hugging Face.

When multiple providers support a model, pressing d in the TUI opens a provider picker modal so you can choose. Remote Ollama instances are supported via the OLLAMA_HOST environment variable — handy if you run a GPU server and connect from a laptop.

Who Should Use llmfit?

llmfit is genuinely useful across a range of use cases:

• Local AI beginners who are overwhelmed by model choices and just want to know what will work on their machine.
• Mid-range hardware users who want to squeeze the most out of 8–16 GB VRAM and need to be strategic about quantization.
• Researchers and power users who want to understand the quality vs. speed vs. memory tradeoff for different model families.
• Developers building AI pipelines who need programmatic, hardware-aware model selection via JSON output.

Hardware upgraders who want to know exactly what GPU they need before spending money.

Final Thoughts

The local AI space moves fast. New models drop every week, hardware varies wildly across users, and the gap between 'this model runs great' and 'this model melts my machine' can be surprisingly small.

llmfit fills a real gap by bringing hardware-awareness to model selection. It's not trying to replace Ollama or llama.cpp — it's the layer on top that tells you what to run before you run it. That's a small but important distinction.

With 8,700+ GitHub stars, active maintenance, Homebrew availability, and a genuinely thoughtful scoring engine, this tool punches well above its weight for a terminal utility. If you run local AI models on any kind of consumer hardware, llmfit deserves a permanent spot in your toolkit.

Resources

GitHub Repository: github.com/AlexsJones/llmfit

Install script: llmfit.axjns.dev

Disclaimer

This article is not endorsed by Salesforce, Google, or any other company in any way. I shared my knowledge on this topic in this blog post. Please always refer to Official Documentation for the latest information.