Skip to content
workflows.diy
All writing

Ten Ollama tips for running local AI

After a model downloads, Ollama runs it on your own machine with no per-call cost and no internet connection.

Why run models locally

Cloud AI services raise three concerns: data privacy, recurring costs and the need for a constant internet connection. Ollama runs open-source large language models (LLMs) on your own computer, which avoids all three. For developers working in an AI-first way, that makes it practical to build AI applications privately and cheaply.

Running models locally with Ollama has these advantages:

  • Your data stays on your machine. That matters for sensitive or proprietary information.
  • Models cost almost nothing to run once they are downloaded, so experiments are cheap.
  • After download, models work without an internet connection. That suits development on the go or in restricted environments.
  • Local execution removes the network delay of remote servers, so responses come back faster.
  • You choose the model parameters, the workflows the models connect to, and how each model behaves.

Ollama handles the setup and management of these models. It supports reasoning models such as deepseek-r1, text models such as gemma3 and qwen3, multimodal models that handle text and images, such as llama3.2-vision, and embedding models for RAG, such as nomic-embed-text.

Here are ten things to know about Ollama.

1. Installation is straightforward and cross-platform

Go to ollama.com, download the application for Windows, macOS or Linux, and run the installer. It takes a few clicks. Ollama then runs as a background service, and you use it through its command-line interface (CLI). The models need enough RAM, and larger models need GPU VRAM.

2. Run a model with ollama run

The core command is:

ollama run <model_name>

For example, to run the Llama 3 8B instruction-tuned model, type ollama run llama3:8b-instruct. If the model is not on your system, Ollama downloads it automatically (this is called pulling). Once the model loads, you get an interactive shell. Type /bye to exit it.

3. Manage models with list, pull and rm

Model management uses three commands:

  • To see the models on your system, run ollama list.
  • To download a model without running it, run ollama pull <model_name>.
  • To free disk space, remove a model with ollama rm <model_name>.

You can download, switch between and manage as many models as your hardware and storage allow.

4. Model capability depends on size and hardware

The Ollama library includes general text models such as Llama 3 and Mistral, code models such as CodeLlama and DeepSeek-Coder, and multimodal models for image analysis, such as LLaVA.

Model size is measured in parameters, such as 3B, 7B or 70B. More parameters usually mean more complexity and possibly more accuracy, but they need more RAM and, for speed, more GPU VRAM. My NVIDIA RTX 4090 (desktop) and RTX 4070 (laptop) GPUs handle larger models well. I cover the hardware in The 30+ tools in my AI development stack, and I wrote a full post on Running Ollama on NVIDIA GPUs for local AI.

Smaller quantized models need less memory and run on weaker hardware, but they may be less capable. Learn the sizes, such as 7B, 13B and 70B, and the types, such as instruct-tuned or chat-focused, to pick the right model for your task and hardware.

5. Customize model behavior with Modelfiles

A Modelfile is a simple text file that customizes an Ollama model. In it you can:

  • Start FROM an existing base model.
  • Set PARAMETER values such as temperature, which controls creativity versus determinism. For example: PARAMETER temperature 0.7.
  • Set a SYSTEM message that gives the model a persona or standing instructions. For example: SYSTEM """You are a helpful AI assistant specializing in Python code.""" The message applies to every interaction.

Build your custom version with:

ollama create <your_custom_model_name> -f /path/to/your/Modelfile

You then run the custom model like any other Ollama model.

6. Ollama runs a local HTTP API

When Ollama is active, it runs a local HTTP server, usually on port 11434. The REST API lets any application that sends HTTP requests use your local models. That includes Python scripts, JavaScript frontends, n8n workflows and curl.

My Desktop AI Assistant uses this API, because its backend services talk to the local models. The main endpoints are /api/generate for single text completions and /api/chat for conversations.

7. Official Python and JavaScript libraries simplify API use

You can call the API with raw HTTP requests. The Ollama team also publishes official Python and JavaScript libraries that hide the low-level details.

For Python, install the library with pip install ollama. Common functions include:

  • ollama.chat(...) for conversations.
  • ollama.generate(...) for direct text generation.
  • ollama.list() to list your local models.
  • ollama.embeddings(...) to generate embeddings.

The official libraries are the recommended way to connect Ollama to Python or JavaScript applications. They are easier to use, offer more options and are easier to maintain.

8. Graphical interfaces work on top of Ollama

If you prefer a graphical interface, several third-party tools sit on top of Ollama. Msty.app, WebOllama and Enchanted give you a chat interface, manage model downloads, and can work with local documents for RAG (see point 9). They detect your local Ollama instance and its models, so the models appear in a chat interface like a cloud AI service, but run on your own machine.

9. Ollama works well for local RAG

Retrieval augmented generation (RAG) lets an LLM answer from your private documents, not only from its training data. Ollama is a good platform for building local RAG systems.

The pipeline I describe in Do million-token context windows replace RAG? and build in my n8n RAG chatbot workflow has five steps:

  1. Load your documents, such as PDFs and text files.
  2. Split them into chunks.
  3. Create an embedding for each chunk with an embedding model. Ollama provides models for this, such as nomic-embed-text or mxbai-embed-large.
  4. Store the embeddings in a local vector database, such as ChromaDB or Qdrant.
  5. Answer questions with a local Ollama LLM, giving it the relevant chunks from the vector database as context.

LangChain and LlamaIndex simplify building these pipelines with Ollama. A local pipeline keeps sensitive data on your machine, which is a core design principle of the RAG features in my Desktop AI Assistant.

10. Advanced features benefit from strong hardware

Ollama keeps adding features for developers and power users. Two are worth knowing:

  • Native thinking support. Models that support it return their internal thinking separately from the final content. Set {"options": {"think": true}} in an API call, or use /set think in the CLI, to get a separate thinking field where the model supports it. You no longer have to parse reasoning markers out of the main output, which makes the model's behavior easier to debug.
  • Structured outputs. Beyond format: json, which asks for valid JSON, Ollama is moving toward fuller structured output. That includes schema hints, such as format: <schema_name> with the schema defined in the Modelfile or system prompt. The aim is more consistent data, especially for constrained values such as enums, with a check that the output matches the structure.

Advanced features work best with larger models on strong hardware. Modern NVIDIA RTX GPUs help here, as I describe in my post on NVIDIA GPUs and Ollama.