Running Ollama on NVIDIA GPUs for local AI

The local AI-first approach

I run LLMs and vision models on my own hardware, with two NVIDIA RTX GPUs: an RTX 4090 in my desktop and an RTX 4070 in my laptop. Ollama runs the models on them. This pair is the core of my local AI work, as I describe in The 30+ tools in my AI development stack.

My development approach is AI-first. AI is the main engine for building software, and my role is that of an "AI orchestrator." The Desktop AI Assistant is one result. Google Gemini 2.5 Pro Preview built it entirely.

Why run AI locally?

Local processing has five practical advantages for my work.

• Sensitive data stays on my machine. That is non-negotiable for many projects.
• Apps run without a constant internet connection.
• Local models avoid API call costs, which makes experiments cheap. That matters when the AI is generating and testing code.
• Responses can arrive faster than a round trip to a cloud server, which helps interactive apps.
• Direct access to local models lets me design tightly integrated systems. Gemini builds the interfaces that call them.

This local-first approach supports the private, multimodal experience I describe in "The tech stack behind our desktop AI assistant."

NVIDIA RTX GPUs supply the compute

Running capable LLMs is compute-heavy. My NVIDIA RTX GPUs do that work, as I covered in The 30+ tools in my AI development stack.

• The CUDA cores and Tensor Cores are built for the parallel work inside deep learning models. That means faster inference and larger models.
• Modern LLMs are memory-hungry. The VRAM on an RTX 4090 lets me load larger models, or several smaller ones at once. Without enough VRAM, models swap to system RAM, which would cripple performance.
• NVIDIA's drivers and libraries, including CUDA and cuDNN, plus wide framework support make these GPUs the default choice for serious AI work.

The GPUs make these local projects practical. Gemini can build apps on models that were cloud-only until recently.

Ollama runs and manages local models

The GPUs provide the raw power. Ollama manages and runs open-source LLMs and vision models, such as Llama 3.2, DeepSeek, Gemma3, Mistral, Phi4 and LLaVA. The Ollama Library lists the available models. Setup that used to be complex now takes a few commands.

• Pulling and running a model takes one command each: ollama pull llama3.2 and ollama run llama3.2.
• Ollama exposes a local HTTP API. My FastAPI backend for the Desktop AI Assistant, which Gemini wrote, uses it to list models, generate responses, stream tokens and call vision models.
• Ollama runs on macOS, Linux and Windows.
• The range of supported models grows constantly.
• The local API works with automation platforms such as n8n.io. I have shared several n8n workflows that use it:
  • The Private and Local Ollama Self-Hosted AI Assistant workflow turns a local n8n instance into a chat interface for any local Ollama model. It processes messages on your machine and returns formatted replies, with no cloud dependencies. Setup means installing n8n and Ollama, downloading a model, adding Ollama API credentials if the defaults do not work, then importing and activating the workflow.
  • The Private and Local Ollama Self-Hosted + Dynamic LLM Router workflow routes each prompt to a suitable local model, such as qwen2.5-coder for code or llama3.2-vision for images. It keeps conversation memory and processes everything locally. Setup means ensuring Ollama is running, pulling the models, configuring the API in n8n and activating the workflow. You can change the router's decision framework and system prompts.
  • My Compare Local Ollama Vision Models for Image Analysis using Google Docs workflow downloads an image from Google Drive, runs it through several Ollama vision models and saves detailed markdown descriptions to a Google Doc. It suits image description and structured data extraction in real estate, marketing and research. Setup means running Ollama, pulling the vision models, adding Google Drive and Docs credentials in n8n and giving it the image file ID. You can change the image source, prompts and post-processing.
  These workflows show how local models power automated tasks.

For my AI-first workflow, the Ollama API is the central piece. Gemini can write the Python client code that connects Ollama to the backend of the Desktop AI Assistant. I can also connect Ollama directly to n8n for local automations.

The Desktop AI Assistant runs on this stack

The NVIDIA GPUs and Ollama work together in my Desktop AI Assistant.

• Chat and vision features, described in Anatomy, call Ollama models on my RTX 4090. Gemini wrote the Electron TypeScript services that call the FastAPI backend, and the backend calls Ollama for inference.
• For retrieval augmented generation (RAG), as described in Crawl4ai RAG in my Electron AI assistant, the app retrieves chunks from ChromaDB. An Ollama-hosted LLM then writes the answer, all locally. The same pattern works with cloud data through n8n. My AI Powered RAG Chatbot for Your Docs + Google Drive + Gemini + Qdrant workflow processes Google Drive documents for a RAG chatbot.
• The assistant speaks replies with local Kokoro TTS, and an Ollama model often produces the text. The full chain runs on local hardware: voice input, OpenAI Whisper, Ollama for text or vision processing, Kokoro TTS and voice output. The GPU and Ollama make this chain possible.

Here one AI (Gemini) builds software that calls other AI models (Ollama) on local hardware. That is the core of my AI-first, local-first approach.

Recent Ollama features and the GPUs that run them

Ollama keeps adding features. Two recent ones matter most for my work, and both benefit from a capable GPU.

1. Native thinking output

Models that support it can return their "thinking" process separately from their final content. Matt Williams covers this in his video Ollama now supports Thinking Natively. Set think: true in an API call, or run /set think in the CLI, and the response includes a separate thinking field.

• Parsing reasoning out of mixed output used to be complex. With a separate field, the code is cleaner. When I ask Gemini for an app that shows a model's thought process, the generated code is simpler and more reliable.
• Agents can use the reasoning to make decisions and handle errors. Larger GPUs run models that produce more detailed thinking without a big performance cost.

2. Structured outputs for reliable data flow

Ollama supports structured outputs with format: <schema>, not only format: json. The output is checked against the schema, which helps with constrained values such as enums. The goal is more consistent and reliable AI-generated data.

• My Desktop AI Assistant uses Pydantic to validate data from many sources, including AI models. Structured output from Ollama helps that data arrive correct from the source.
• When the output is consistently structured, Gemini writes less parsing and error-handling code.
• Larger models on capable GPUs can generate more complex structured data. Schema checks at the output stage help when building data pipelines.

These features need GPU power. Larger models on NVIDIA RTX GPUs can produce detailed reasoning or complex structured data in reasonable time.