Local AI Models for Coding — Ollama, Open Source, and Privacy

Every prompt you send to Claude, GPT, or Gemini goes through the internet to a data center. For most coding tasks, that is fine. But there are real reasons to run AI models locally — on your own machine, with no data leaving your network.

This article covers when local models make sense, how to set them up with Ollama, and how to connect them to your existing tools. You will also get an honest comparison of local vs cloud models for coding tasks.

Why Run AI Models Locally?

Privacy. Your code never leaves your machine. For companies with strict data policies, government contractors, or anyone working with proprietary algorithms, this matters. No terms of service, no data processing agreements, no wondering if your code is used for training.

Cost. After the hardware investment, running local models costs only electricity. If you are running hundreds of prompts per day, the monthly savings add up. Cloud AI tools cost $20-$200 per month per developer. A one-time GPU purchase pays for itself in 6-12 months of heavy use.

Offline access. Airplanes, remote locations, unreliable internet — local models work everywhere. No latency, no rate limits, no service outages.

Enterprise restrictions. Many companies ban sending code to external AI services. Local models are often the only option for developers in those environments.

The honest trade-off: Local models are significantly less capable than cloud models for most coding tasks. You are trading quality for privacy, cost savings, and independence. That trade-off is worth it for some use cases and not others.

Hardware Requirements

Local AI models run on your GPU (preferred) or CPU. The model size determines how much memory you need.

Minimum setup (7B models):

• 16 GB RAM
• GPU with 8 GB VRAM (NVIDIA RTX 4060 or equivalent)
• 20 GB storage per model
• Speed: 40-50 tokens per second

Recommended setup (32B models):

• 32 GB RAM
• GPU with 24 GB VRAM (NVIDIA RTX 4090 or equivalent)
• 50 GB storage per model
• Speed: 20-30 tokens per second

Apple Silicon:

• M1/M2/M3 with 16 GB unified memory: runs 7B models well
• M1/M2/M3 with 32+ GB unified memory: runs 32B models
• M4 Pro/Max with 48+ GB: runs 70B models
• No driver installation needed — Metal acceleration works automatically with Ollama

CPU-only (no GPU):

• Possible but slow — expect 2-5 tokens per second for 7B models
• Only practical for small models and short prompts
• Not recommended for real coding workflows

If you are on a budget, Apple Silicon MacBooks offer the best value for local AI. The unified memory architecture means all your RAM is available to the model, unlike discrete GPUs where you are limited by VRAM.

Setting Up Ollama

Ollama is the simplest way to run local models. One command to install, one command to run a model.

Install Ollama:

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
winget install Ollama.Ollama

# Verify installation
ollama --version

Start the Ollama server:

ollama serve

The server runs on http://localhost:11434 by default. Keep this terminal open — Ollama needs the server running to respond to requests.

Pull your first model:

# Small and fast — good for testing
ollama pull qwen2.5-coder:7b

# Larger and more capable
ollama pull qwen2.5-coder:32b

# Best open-source coding model as of early 2026
ollama pull deepseek-coder-v2:16b

Test the model:

ollama run qwen2.5-coder:7b "Write a Python function
that checks if a string is a valid email address"

Code-Specific vs General Models

Not all models are equal for coding. Code-specific models are trained on large amounts of code and outperform general models for programming tasks — even when the code model is smaller.

Best local coding models in 2026:

Key insight: A 32B coding model often outperforms a 70B general model on programming tasks. DeepSeek Coder V2 16B beats Llama 3.1 70B on many coding benchmarks despite being four times smaller. Use code-specific models for coding.

Quantization matters. All sizes above assume Q4_K_M quantization — a compression technique that reduces memory by 40-50% with minimal quality loss (1-3% higher error rate). Ollama uses Q4_K_M by default. You can pull higher-quality quantizations if you have the VRAM:

# Default quantization (Q4_K_M) — balanced
ollama pull qwen2.5-coder:32b

# Higher quality (needs more VRAM)
ollama pull qwen2.5-coder:32b-instruct-q8_0

Using Local Models with Cursor

The most practical workflow for local models is connecting them to Cursor. This lets you use your familiar editor with a local model backend.

Step 1: Start Ollama and pull a model:

ollama serve
ollama pull qwen2.5-coder:32b

Step 2: Set up a tunnel (required by Cursor):

Cursor routes requests through its servers, so it needs a publicly accessible endpoint. Use ngrok to tunnel your local Ollama server:

# Install ngrok
brew install ngrok

# Create a tunnel to Ollama
ngrok http 11434 --host-header="localhost:11434"

ngrok gives you a URL like https://abc123.ngrok.io. Copy it.

Step 3: Configure Cursor:

1. Open Cursor Settings > Models
2. Click “Add Model” under OpenAI API
3. Set the base URL to https://abc123.ngrok.io/v1
4. Enter any string as the API key (Ollama does not require one, but Cursor requires the field to be non-empty)
5. Add the model name exactly as it appears in Ollama: qwen2.5-coder:32b

Step 4: Allow Ollama to accept external connections:

# Set environment variable before starting Ollama
OLLAMA_ORIGINS="*" ollama serve

Common issues:

• “Model not found” error. The model name in Cursor must exactly match the Ollama tag. If you pulled qwen2.5-coder:32b, do not type qwen-coder-32b.
• “Request failed” error. Make sure you added /v1 to the end of the ngrok URL.
• Slow responses. Local models are slower than cloud APIs. Expect 5-15 seconds for a response with a 32B model, compared to 1-3 seconds with cloud models.

Using Local Models with Continue

Continue is a VS Code extension designed for local model integration. It does not require a tunnel — it connects directly to Ollama on localhost.

Setup:

1. Install the Continue extension in VS Code
2. Open the Continue configuration file
3. Add Ollama as a provider:

{
  "models": [{
    "title": "Qwen 2.5 Coder 32B",
    "provider": "ollama",
    "model": "qwen2.5-coder:32b"
  }],
  "tabAutocompleteModel": {
    "title": "Qwen 2.5 Coder 7B",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b"
  }
}

Continue is simpler to set up than Cursor with local models. The trade-off is that Continue has fewer features than Cursor’s agent mode. Use Continue if your primary goal is local model usage. Use Cursor with the tunnel if you want Cursor’s full feature set.

Honest Performance Comparison

Let me be direct: local models are significantly worse than Claude Sonnet, Claude Opus, or GPT-4o for most coding tasks. Here is a real comparison.

Task: Implement a pagination component in React with TypeScript.

Task: Debug a race condition in async code.

Where local models are good enough:

• Code completion (fill in the next few lines)
• Simple refactoring (rename variable, extract function)
• Boilerplate generation (models, interfaces, basic CRUD)
• Explaining code (reading comprehension is strong even in smaller models)

Where local models fall short:

• Complex debugging (multi-file issues, race conditions)
• Architecture decisions (design patterns, system design)
• Security-sensitive code (authentication, encryption)
• Large context windows (local models typically support 4K-32K tokens vs 100K-200K for cloud models)

The Hybrid Approach

The most practical setup is a hybrid: use local models for routine tasks and cloud models for complex work.

Local model for:

• Autocomplete while typing
• Quick code explanations
• Simple code generation
• Anything involving proprietary or sensitive code

Cloud model for:

• Complex multi-file refactoring
• Architecture and design decisions
• Debugging tricky issues
• Tasks that need a large context window

In Cursor, you can switch between models per request. Start with your local model and switch to Claude when you need more power. This balances cost, privacy, and capability.

Cost Analysis

Cloud AI costs for an active developer:

• Claude Pro: $20/month
• Claude Max (for heavy users): $100-200/month
• Cursor Pro: $20/month
• Total: $40-220/month ($480-2,640/year)

Local AI costs (one-time):

• NVIDIA RTX 4090 (24 GB): $1,600
• Electricity: ~$5-10/month at heavy usage
• Total first year: ~$1,700
• Total subsequent years: ~$80/year

Break-even analysis:

• If you spend $100/month on cloud AI, local pays for itself in 17 months
• If you spend $200/month, it pays for itself in 9 months
• Apple Silicon users: an M3 Pro MacBook with 36 GB costs $2,500 — but you were buying a laptop anyway. The AI capability is a bonus.

Local models make financial sense for heavy users who run hundreds of prompts per day. For occasional use (under 50 prompts per day), cloud models are more cost-effective because you get much better quality per dollar.

Key Takeaways

• Local models are real and practical in 2026. Ollama makes setup trivial. Qwen 2.5 Coder 32B is genuinely useful for many coding tasks.
• They are not a replacement for cloud models. Be honest about the quality gap. Use local models where they are strong and cloud models where they are strong.
• Code-specific models outperform general models. Always pick a coding-focused model for programming tasks.
• The hybrid approach is the best strategy. Local for routine and sensitive work, cloud for complex tasks.
• Hardware matters. A GPU with 24 GB VRAM (or an Apple Silicon Mac with 32+ GB) is the sweet spot for running useful coding models.
• Privacy is the killer feature. If your code cannot leave your machine, local models are not just nice to have — they are your only option.

What’s Next?

In the next article, you will step back from tools and techniques to look at the bigger picture. AI-first development is changing how we think about building software — from the skills we need to the workflows we follow.

Part 20 of the Vibe Coding series.