My AI Runs on $0 Compute: The Local Stack That Powers Everything
Most of my AI work runs on a machine I already owned. No cloud bills, no API quotas, no waiting on rate limits. Here’s the actual stack.
The setup
I run Ollama on a refurbished workstation with a used GPU I grabbed off eBay for about $180. Total hardware cost was maybe $400 once you count the case and PSU. The model weights are free. The runtime is free. My power bill went up by around $4 a month.
That’s the whole compute budget.
Why I stopped paying for API calls
I used to push everything through OpenAI and Anthropic. Then I looked at my December statement. $312 for what amounted to code reviews, draft emails, and a handful of agent experiments. That stung.
For batch jobs, summarization, and rough drafts I don’t need GPT-4 class intelligence. I need something that handles 80% of the work at zero cost. Local inference does that fine. I keep the paid APIs in the rotation for things that genuinely need them.
What the stack looks like
Here’s what’s actually running on my desk right now:
- Ollama serving Llama 3.1 8B plus a couple of smaller specialized models
- n8n inside Docker for workflow orchestration, talking to Ollama over the HTTP node
- Make for the multi-app glue work that n8n gets clumsy with
- Python scripts for anything weird, and a lot of it is weird
- VS Code with Continue.dev pointed at the local model for inline completions
- Ollama itself, obviously
Ollama listens on 11434, n8n hits it over the LAN, my scripts call it directly. Nothing leaves the box unless I tell it to.
The honest tradeoffs
An 8B model isn’t going to write your landing page copy. It hallucinates more than I’d like on anything involving recent events or precise numbers. Long context tasks choke it. If you need real reasoning, you’re either running something bigger (which means real hardware money) or you’re calling a paid API.
I keep the cloud APIs in reserve. They’re not the default anymore.
When local doesn’t make sense
If your workload is bursty and unpredictable, local won’t help. You’ll overprovision or get crushed during spikes. Image generation eats VRAM fast and the hardware math gets ugly. And if you’re shipping a product that needs consistent quality across thousands of users, a hosted model with an SLA is probably the right call.
This isn’t a “cloud is dead” post. It’s a “stop paying rent on compute you’re barely using” post.
The hardware I used
If you’re building this from scratch, here’s the rough kit. A used RTX 3060 12GB handles 8B models without breaking a sweat and you can find them cheap. Here’s the current pricing on Amazon. Pair it with any modern multi-core CPU, 32GB of system RAM, and you’re set. I run mine in a normal mid-tower, nothing exotic.
Total damage: somewhere between $300 and $600 depending on how thrifty you get.
Getting started
Install Ollama. Pull a model. Run it. That’s the whole onboarding. The first time you watch it spit out a response without a network call is weirdly satisfying.
If you want a workflow layer, n8n has an Ollama node now, or you can use the HTTP node with maybe 30 seconds of config. Make works the same way.
The hard part isn’t the software. It’s accepting that a smaller model on your desk can do real work if you point it at the right problems.
The Cloud Tax
Cloud AI gets expensive the moment you start using it for real work. Not toy prompts — actual work that ships.
- GPT-4: $30/month for moderate usage
- Claude 3.5: $20/month
- Cloud hosting: $15-50/month
- API quotas: the thing that wakes you up at 2am
I pay $0 for compute. Here’s what runs the stack instead.
The Hardware
My desktop cost about $2,500 to build a couple years back. Here’s what’s in the box:
- AMD Ryzen 9800X3D, 16 cores, 32 threads
- Radeon 9070 XT, 32 GB VRAM
- 32 GB DDR5 RAM
- 2 TB NVMe SSD
- Windows 10. Not Linux. I like things that work.
Why AMD over NVIDIA?
People ask me this one a lot. Few reasons.
- 32 GB of VRAM for $800 versus 24 GB for $1,200 on an RTX 4090. That VRAM gap matters more than raw FLOPs for the models I actually run locally.
- DirectML support for PyTorch. It’s not as mature as CUDA, and you’ll feel that if you try anything exotic. For what I do day to day, it gets the job done.
- No vendor lock-in. I don’t trust any of these companies enough to hand them the keys to my workflow.
What Runs Locally
LLMs (Ollama):
| Model | Size | VRAM | Use Case |
|---|---|---|---|
| ——- | —— | —— | ———- |
| Kimi K2.5 | 32B | ~24 GB | Premium reasoning |
| Qwen3:14b | 14B | ~10 GB | Fast classification |
| Qwen3:30b | 30B | ~20 GB | Balanced writing |
| nomic-embed-text | 137M | ~1 GB | Embeddings |
Ollama’s my daily driver. Models pull fast and most load in under 10 seconds. First token lands in 2-5 seconds depending on size.
Image Generation (ComfyUI):
ComfyUI wasn’t part of my setup when I started this. I pulled it in and poked around to see if it fit. Here’s what I found:
- Stable Diffusion v1.5 via DirectML
- ~8 seconds for a 512×512 image
- ~25 seconds for 1024×1024
- Slower than CUDA, but no cloud bill
Vector Database (LanceDB):
I considered LanceDB for semantic search across my playbooks. It’s files on disk with a query layer, which is exactly what I wanted.
- Local storage, no server
- ~200ms queries based on community benchmarks
- 54 playbooks, 31K words, zero hosting cost
Code Execution:
- Python 3.14 (bleeding edge, because I like problems)
- Node.js for web servers
- Everything runs on the same machine
What Still Needs Cloud
I’d love to run everything off a box in my closet. It doesn’t work that way. A few things still need the cloud:
Web Search: Perplexity API ($0.002/query) – local models can’t browse the web
Solana RPC: Helius free tier – need a reliable node
Discord Bot: Runs on OpenClaw Gateway (local, but needs internet)
WordPress Hosting: $15/month for 3 sites – can’t self-host from home IP
I run three WordPress sites. I thought about self-hosting from home for about five minutes. Then I looked at my ISP upload speeds and pictured my home IP getting blacklisted. Fifteen a month is the trade I made.
Total cloud costs: ~$20/month vs. $100+ if everything was cloud-based.
The Real Savings
| Category | Cloud-First Cost | Local-First Cost | Savings |
|---|---|---|---|
| ———- | —————— | —————— | ——— |
| LLM compute | $40-60/month | $0 | 100% |
| Image generation | $20-50/month (Midjourney/DALL-E) | $0 | 100% |
| Vector DB | $15-30/month (Pinecone/Weaviate) | $0 | 100% |
| Code execution | $10-20/month (cloud runners) | $0 | 100% |
| Total | $85-160/month | $20/month | 75-90% |
Hardware runs $2,500. Skip $85-160 a month in cloud bills and it pays for itself in 15-30 months. That’s the math. But the real win isn’t the savings — it’s owning the thing. No API deprecation, no rate-limit shuffle, no vendor sunsetting the model my workflow runs on. The upfront cost stings. I’ll take it anyway.
The Tradeoffs
Slower than cloud for some tasks
Qwen3:14b is quick enough for most of what I throw at it. Kimi K2.5 on local hardware doesn’t match the cloud version for speed though. When timing matters, like trade execution, I still hit the cloud API. Everything else stays local because the wait doesn’t really bother me.
Setup complexity
Ollama makes the model side painless. DirectML on AMD is still rough around the edges. I had to patch ComfyUI manually when I first set it up. Not “sign up and get an API key” easy, but not bad if you’ve wrestled with dev tooling before.
Power consumption
The GPU pulls around 300W under load. Georgia rates sit about $0.12/kWh, so 8 hours a day lands near $10/month. Way under cloud costs, and the heat isn’t a bad thing in January.
No automatic scaling
If I need to run ten models at once, I can’t just spin up more instances. Hardware hits its ceiling and that’s that.
What’s Next
Three things on my list right now:
1. Second GPU – an RX 7900 XTX would give me another 24 GB of VRAM. That’s enough to run a coding model and a chat model side by side without one getting evicted.
2. Linux dual-boot – ROCm runs better on Linux. Annoying, but it’s true. Throughput should climb once I get it set up.
3. NAS offload – moving LanceDB and file storage over to the Synology. Frees up local disk and gives the NAS something to actually do.
The Bottom Line
Local AI isn’t for everyone. If you need 99.9% uptime, auto-scaling, or enterprise SLAs, cloud makes sense.
But for a solo operator running personal automations? Local is a no-brainer. Models are good enough. Hardware is cheap. And I never have to worry about getting rate-limited or having a provider quietly pull a model I rely on.
My monthly AI compute bill: $20. That covers web search, Solana RPC, and WordPress hosting. Everything else runs on silicon sitting in my office.
Want the setup guide? The Nova Operations blueprint has the full local stack config: Ollama model setups, DirectML notes, and hardware recommendations based on what I’ve actually run.
—
Not anti-cloud. Pro-ownership. I use cloud where it makes sense (web search, hosting). I use local where it makes sense (LLMs, images, databases). The trick is knowing which is which, and being honest about the tradeoff.