Sizing Your AI: How to Choose the Right Local Model for Your VRAM

Sizing Your AI: How to Choose the Right Local Model for Your VRAM

Running your own AI models locally gives you unparalleled privacy and control, shielding your sensitive business data from corporate cloud filters and 30-day logging traps. However, when you bring AI in-house, your computer's graphics memory (VRAM) becomes the ultimate bottleneck.

Choosing the right model is a delicate balancing act. Here is how to correctly size your local AI to survive the massive memory demands of agent frameworks without bringing your system to a grinding halt. 

I set out to run openclaw on 6GB of VRAM (GTX1060) and although the AI model ran very well on the GTX1060, as soon as I started to use Openclaw the massive context window the agent needs to run became an immediate problem. So in my case openclaw worked (just about) "toolless" with Qwen 3B on 6GB VRAM, but as soon as tools were used, the context window became a big problem. 

Despite being very keen on running locally I actually ended up only getting this all to work with cloud model. . .and I really did try. You need 12GB or more for local openclaw.  This is becuase heavy skills like gog need a massive context window to work well. 

The Magic of 4-Bit Quantization

You might wonder how an 8-billion parameter (8B) model can possibly fit on a standard graphics card. The secret is a compression process called 4-bit quantization.

• Software engines like Ollama automatically compress the precision of the model's weights to drastically lower the VRAM usage.

• Through 4-bit quantization, a 7B or 8B model shrinks down to roughly 4.5 GB to 5 GB of storage space.

• Massive 32B models, which normally require massive server farms, can be compressed down to roughly 20GB, fitting onto high-end consumer cards or older 32GB server cards.

The OpenClaw Context Window Tax

Fitting the compressed model into your graphics card is only half the battle. The most common mistake new builders make is underestimating the "Context Window."

OpenClaw is not a simple chatbot; it is an autonomous agent framework with serious context demands. The context window is the total memory space the AI uses to understand your current task—and OpenClaw fills it incredibly fast.

• The System Prompt: OpenClaw's baseline system prompt alone can consume 17,000 tokens.

• The Injections: This baseline context includes the agent's rules, your custom skill instructions (SKILL.md), runtime metadata, and injected workspace files.

• The Tool Burden: When you use tools (like Google Maps scraping or browser automation), the JSON tool schemas, the tool execution results, and raw scraped web text are all injected directly into the model's current window.

Because of this massive baseline overhead, it is highly recommended to use a context window of at least 64k tokens when running local models to prevent the agent from failing mid-task.

The Overspill and "Amnesia" Problem

When you force a very small model (like an 8B model) to handle OpenClaw's massive context requirements on a budget graphics card, two distinct failures occur:

1. VRAM Overspill (The Speed Killer): If the combined size of the AI model and its bloated context window exceeds your graphics card's physical VRAM limit, the system does not crash. Instead, it gracefully "spills over" the excess memory burden into your slower motherboard system RAM. When this happens, processing speed can plummet from a snappy 40 tokens per second down to an unusable 2 tokens per second. (On an old PC the system becomes unusable)

2. Context Truncation (The Amnesia Bug): Small graphics cards often truncate the context window when they run out of room, leading to severe "amnesia" during multi-step tasks.

3. Cognitive Overload: Even if an 3B model technically loads on a 16GB machine, asking it to process OpenClaw's complex system rules alongside massive tool data often overwhelms it. It will frequently hallucinate tool calls, produce malformed JSON, and loop endlessly on tasks that a larger model handles in a single pass.

Matching the Model to Your Hardware

To survive the OpenClaw context tax without overspilling, you must size your hardware and models realistically:

The 16GB RAM / 8GB VRAM Tier (The Budget Setup)

• Best Models: Qwen3:8B or Gemma 4 8B.

• The Reality: Well-suited for light tasks like simple email drafts or basic file management. However, expect occasional failures, JSON errors, and context drift on complex multi-step chains due to the framework's heavy context tax.

The 32GB RAM / 24GB+ VRAM Tier (The Production Sweet Spot)

• Best Models: Devstral-Small-2-24B or Qwen3-Coder:32B.

• The Reality: This is the community consensus for reliable production use. A 32GB VRAM setup provides ample room to house a smart 24B–32B model while leaving a massive, dedicated memory buffer (4–6GB) purely for the KV cache to handle a 65K context window.

When sizing your AI for agent frameworks, you aren't just making room for the model's brain—you must leave massive real estate open for its memory.

Lighting Assessment For Bats

Downwash Analysis

BRE 365 Soakaway