AI & ML

Local LLMs: Running AI on Your Own Hardware

No API key. No usage fees. No data leaving your machine. A complete guide to running Llama 3.1, Mistral, and Qwen with Ollama.

By Elena Rostova · 2025-05-02 · 15 min read

Local LLMs: Running AI on Your Own Hardware

Running AI models locally has shifted from a researcher's privilege to an afternoon project. This guide walks through running Llama 3.1, Mistral 7B, and Qwen on your own hardware using Ollama — no API keys, no usage fees, no data leaving your machine. We cover hardware requirements, installation, the REST API, model selection, and building a streaming Python client.

What Is Ollama?

Ollama is an open-source runtime that bundles model weights, the llama.cpp inference engine, and a local HTTP server into a single binary. It handles GGUF model downloading from Hugging Face, CPU/GPU layer allocation, and exposes both a native API and an OpenAI-compatible /v1 endpoint — meaning any application built against the OpenAI SDK works against your local model by changing exactly one line of code.

Hardware Requirements

You need RAM, not necessarily a GPU. All inference can happen on CPU — slowly, but correctly. A GPU dramatically improves token generation speed. The table below shows requirements per model size at Q4_K_M quantization, which is the recommended default.

Model Size	Min RAM	Recommended	CPU tok/s	Apple Silicon tok/s	NVIDIA GPU tok/s
3B	4 GB	8 GB	18–28	65–90	80–120 (RTX 3080)
7B–8B	8 GB	16 GB	8–14	35–55	45–75 (RTX 3080)
13B	12 GB	24 GB	4–8	20–35	25–45 (RTX 4090)
32B	24 GB	48 GB	2–4	8–16	10–20 (2× RTX 4090)
70B	48 GB	64 GB	0.5–2	3–8	Requires 80GB A100+

Installing Ollama

Installation is a single command on all platforms.

macOS

# Via Homebrew
brew install ollama

# Start the server
ollama serve

Linux

# Official install script (auto-detects NVIDIA/AMD GPUs)
curl -fsSL https://ollama.com/install.sh | sh

# Start as a systemd service
sudo systemctl enable --now ollama

Running Your First Model

shell example

# Pull and start an interactive session
ollama run llama3.1

# List downloaded models
ollama list

# Pull without running (for pre-downloading)
ollama pull mistral

# Remove a model
ollama rm llama3.1

# Show model info and parameters
ollama show llama3.1

Understanding Quantization

The Ollama REST API

Ollama exposes two API surfaces: its native /api endpoints and an OpenAI-compatible /v1 endpoint. Use the /v1 surface to drop Ollama into any existing application built on the OpenAI SDK.

Native API (streaming)

curl http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1",
    "prompt": "Explain the attention mechanism in one paragraph",
    "stream": false
  }'

OpenAI-compatible endpoint

from openai import OpenAI

# Point the SDK at your local Ollama server
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required by SDK, ignored by Ollama
)

response = client.chat.completions.create(
    model="llama3.1",
    messages=[
        {"role": "system", "content": "You are a concise technical assistant."},
        {"role": "user", "content": "What is retrieval-augmented generation?"},
    ],
    temperature=0.7,
)
print(response.choices[0].message.content)

Choosing the Right Model

Model	Size (Q4)	Strengths	Best For
Llama 3.1 8B	4.7 GB	Balanced, fast, instruction-tuned	General assistant, Q&A
Mistral 7B v0.3	4.1 GB	Strong reasoning, compact	Analysis, summarization
Qwen2.5-Coder 7B	4.7 GB	Excellent code generation	Programming assistant
DeepSeek-R1 8B	4.9 GB	Chain-of-thought reasoning	Math, logic, step-by-step
Phi-4 Mini	2.5 GB	Tiny, surprisingly capable	CPU-only, fast inference
Llama 3.1 70B	40 GB	Near-GPT-4 output quality	64GB+ RAM / workstation

Ollama — Full Setup Walkthrough on macOS & Linux

Watch Ollama — Full Setup Walkthrough on macOS & Linux on YouTube

Privacy and Security

Building a Streaming Python Client

streaming_client.py

import httpx
import json
import sys

def stream_response(prompt: str, model: str = "llama3.1") -> None:
    """Stream tokens from Ollama, printing each one as it arrives."""
    with httpx.stream(
        "POST",
        "http://localhost:11434/api/generate",
        json={"model": model, "prompt": prompt, "stream": True},
        timeout=None,
    ) as response:
        response.raise_for_status()
        for line in response.iter_lines():
            if not line:
                continue
            chunk = json.loads(line)
            sys.stdout.write(chunk.get("response", ""))
            sys.stdout.flush()
            if chunk.get("done"):
                break
    print()  # final newline

if __name__ == "__main__":
    stream_response(
        "Explain the difference between RAG and fine-tuning in one paragraph"
    )

Next Steps

You now have a fully local LLM server with an OpenAI-compatible API. Natural next steps:

Add Open WebUI for a self-hosted ChatGPT interface with persistent conversation history
Build a RAG pipeline: connect your model to a pgvector database and query your own documents
Fine-tune a 7B model on your own data using Unsloth — see the companion guide for the end-to-end process
Explore multimodal models: ollama run llava adds vision capabilities for image analysis

Ollama Starter Pack

Modelfile collection, Docker Compose for Ollama + Open WebUI, and Python client templates. Ready to deploy.

Archive · 14.2 KB