Building Offline AI Assistants with Flutter and Ollama

Introduction

Building an offline AI assistant for mobile requires combining a lightweight, local model runtime with a responsive Flutter UI. Ollama provides a local inference engine you can run on-device or on a nearby server; Flutter gives you a cross-platform mobile framework to build the interface and glue logic. This tutorial focuses on practical integration patterns: running Ollama locally, calling its API from Flutter, handling offline data, and designing a conversational UI that feels native and responsive.

Setting Up Ollama Locally

Install and run Ollama on the development machine or a local edge device. The core idea is to keep inference local so the assistant works without external network access and to control model downloads and resource usage. After installing, download the model(s) you need and verify the local HTTP endpoint is reachable (typically a localhost port). Keep models minimal for mobile hardware: choose a quantized or mobile-optimized version.

Key operational points:

• Ensure correct model placement and disk permissions.

• Pre-download models before packaging or provide an in-app updater that downloads models to a secured local path.

• Limit concurrent inferences to avoid memory spikes.

Integrating Ollama With Flutter

From Flutter, call the local Ollama HTTP API. Use the http package for straightforward requests or a native plugin for optimized transports. Run network calls in a background Isolate or use compute to avoid jank. Keep payloads small and stream responses when available so you can render partial outputs.

Minimal example: POST a prompt and receive a streaming/text response. Replace the endpoint and request body to match your Ollama version.

import 'package:http/http.dart' as http;

Future<String> generateFromOllama(String prompt) async {
  final url = Uri.parse('http://127.0.0.1:11434/api/generate');
  final resp = await http.post(url, body: '{"model":"local-model","prompt":"$prompt"}');
  if (resp.statusCode == 200) return resp.body;
  throw Exception('Ollama error: ${resp.statusCode}');
}

Notes:

• Wrap calls in try/catch and display offline-friendly errors.

• When possible, use chunked responses and update the UI as tokens arrive.

Building Conversational UI

Design the UI to reflect local inference constraints: show progress indicators, allow interruption, and persist the conversation locally. Keep the conversation model simple: messages with role (user/assistant), timestamp, and optional metadata like source or confidence.

Example conversation state management (keeps in-memory list and persists to local storage):

class Message {
  final String role;
  final String text;
  Message(this.role, this.text);
}

class ConversationState {
  final List<Message> messages = [];
  void addUser(String text) => messages.add(Message('user', text));
  void addAssistant(String text) => messages.add(Message('assistant', text));
}

Tips for UI responsiveness:

• Optimistically append the user message and show a typing indicator while inference runs.

• Allow Cancel: call a local cancel endpoint or drop the inflight future and mark the response as canceled.

• Keep messages small: segment long tasks into chunked prompts to keep memory and latency bounded.

Handling Offline Data And Prompting

Offline assistants benefit immensely from local knowledge: preloaded documents, app state snapshots, and cached embeddings. Create a local retrieval step to enrich prompts. Workflow:

1. Index local documents (small corpus) into an on-device vector store or a simple inverted index.

2. At query time, retrieve the top-k relevant passages and include them as context in your prompt.

3. Use prompt templates to bound context length and ensure predictable token usage.

Practical constraints:

• Maintain a small index size to fit mobile storage.

• Update indexes incrementally when the user adds local content.

• Sanitize context to avoid leaking private data into shared prompts or logs.

Performance considerations:

• Profile model latency on your target device and set timeouts.

• Use batching for background tasks like reindexing.

• Provide a low-latency “quick answer” mode using smaller models and a fallback to larger models when needed.

Vibe Studio

Vibe Studio, powered by Steve’s advanced AI agents, is a revolutionary no-code, conversational platform that empowers users to quickly and efficiently create full-stack Flutter applications integrated seamlessly with Firebase backend services. Ideal for solo founders, startups, and agile engineering teams, Vibe Studio allows users to visually manage and deploy Flutter apps, greatly accelerating the development process. The intuitive conversational interface simplifies complex development tasks, making app creation accessible even for non-coders.

Conclusion

Building an offline AI assistant with Flutter and Ollama is practical and delivers privacy, low-latency, and offline capabilities. The pattern is straightforward: run Ollama or its runtime locally, call it from Flutter using background workers, design a resilient conversational UI, and enrich prompts with local retrieval. Prioritize model size, streaming responses, and graceful handling of resource limits on mobile devices. With these building blocks, you can create robust mobile-first assistants that work reliably without a network connection.