Building an On-Device AI Text Summarizer in Flutter (No Cloud, No APIs)

Running Gemma locally in Flutter to build a fully offline AI summarizer — no APIs, no cloud, no backend.

Why On-Device AI?

Most AI features today depend on cloud APIs — which means:

• Network dependency
• Latency
• Privacy concerns
• Recurring API costs

With on-device AI, we flip the model:

• Works offline
• User data never leaves the device
• Predictable latency
• No per-request cost

In this blog, I’ll walk through how I built a fully on-device text summarizer in Flutter using a locally bundled Gemma model, without any backend or API calls.

By the end, you’ll understand:

• How on-device inference works in Flutter
• How to bundle and load a large language model
• How to generate summaries safely within token limits
• What trade-offs to expect in real apps

What We’re Building

Before diving into the implementation details, here’s what the final app does:

• Accepts long text input
• Summarizes it locally using an on-device LLM
• Shows inference time
• Handles token limits and runtime errors gracefully
• No internet. No backend. Just Flutter + local inference.

The summarization runs directly on the device CPU, and the result is generated within a few seconds depending on the hardware.

Everything you see in this demo — from model loading to inference — happens locally inside the Flutter app.

1. Choosing the Model

For this demo, I’m using a quantized Gemma instruction-tuned model — specifically Gemma-3 from Google’s TF-Lite models catalog.

This model is available (along with other variants) on Kaggle:

👉 https://www.kaggle.com/models/google/gemma-3/tfLite

Why I Picked This Model

• Model: Gemma 3–1B parameters
• Quantization: INT4
• Format: .task

⚠️ Even quantized, models like this are hundreds of MB, so size trade-offs matter (we’ll cover that later).

2. Adding the Model to Your Flutter App

1. Place the model in assets

assets/
└── models/
    └── gemma3–1B-it-int4.task

2. Update pubspec.yaml

flutter:
  assets:
     - assets/models/gemma3–1B-it-int4.task

3. Initializing the Inference Engine

Before loading any model, the inference runtime must be initialized.

void main() async {
  WidgetsFlutterBinding.ensureInitialized();

  await FlutterGemma.initialize();

  runApp(const MyApp());
}

This should be done once, before your app starts using the model.

4. Installing and Loading the Model

In this app, I bundle the model directly inside the Flutter app as an asset. This keeps the setup simple, fully offline, and easy to reproduce.

Future<void> _installAndLoadModel() async {
  await FlutterGemma.installModel(
    modelType: ModelType.gemmaIt,
    fileType: ModelFileType.task,
  ).fromAsset(
    'assets/models/gemma3-1B-it-int4.task',
  ).install();

  await initModel();

  setState(() {
    _loadingModel = false;
  });
}

This step:

• Copies the model from assets
• Makes it available for runtime inference
• Happens only once per installation

💡 Model installation happens only once per app installation. Subsequent app launches reuse the already installed model.

4.1 Other Installation Options

While this example uses Flutter assets, flutter_gemma supports multiple installation sources such as:

• Downloading the model from a network URL
• Loading from native bundles
• Installing from external file paths

If you want to explore those options, the official documentation covers them in detail here:

👉 https://github.com/DenisovAV/flutter_gemma#installation-sources

This flexibility is especially useful for production apps where app size, model updates, or enterprise deployment requirements differ.

5. Creating the Runtime Model

Get Bolisetty Sujith’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Once installed, we create an active inference model:

Future<void> initModel() async {
  _model = await FlutterGemma.getActiveModel(
    maxTokens: 1024,
    preferredBackend: PreferredBackend.cpu,
  );
}

Key things here:

• maxTokens controls output size
• CPU backend keeps it universally compatible
• GPU / NNAPI backends may be added later for performance

6. Prompt Design for Summarization

Prompt quality matters even on device.

Here’s the prompt used:

final prompt = '''
Rewrite the following text concisely. 
- Write only the rewritten text as plain sentences. 
- Do not repeat the input or instructions. 
- Do NOT add explanations or commentary. 

Text:
${_controller.text}

Rewritten text:
''';

Why this works well:

• Explicit output constraints
• No ambiguity
• Prevents meta commentary
• Optimized for small instruction-tuned models

7. Token Limit Guard (Very Important)

Mobile models have hard context limits.

Before inference, we estimate token usage:

final inputTokens = await session.sizeInTokens(prompt);
const maxContext = 2048;
const outputReserve = 256;

if (inputTokens > maxContext - outputReserve) {
  await session.close();
  throw Exception('Input too long for on-device model');
}

This prevents:

• `OUT_OF_RANGE` errors
• App crashes
• Undefined inference behavior

8. Running the Inference Session

final InferenceModelSession session =
    await _model!.createSession();

await session.addQueryChunk(
  Message(
    text: prompt,
    isUser: true,
  ),
);

final response = await session.getResponse();
await session.close();

Each inference:

• Uses a short-lived session
• Cleans up resources immediately
• Avoids memory leaks

9. Measuring Performance

Inference time is measured using a Stopwatch:

final stopwatch = Stopwatch()..start();
// inference
stopwatch.stop();

_summaryTimeMs = stopwatch.elapsedMilliseconds;

This is useful for:

• Benchmarking devices
• UX transparency
• Model optimization later

10. Limitations of On-Device AI

1. App Size 📦

• Depending on the parameter count and quantization level, on-device models can add 300–500 MB to your app bundle — and larger variants may reach several gigabytes in size.
• Impacts Play Store downloads
• Not ideal for lightweight apps

2. Performance ⚡

• Slower than cloud GPUs
• Device-dependent speed
• Entry-level phones may struggle

3. Context Length 🧠

• Hard token limits
• Requires careful chunking for large documents

4. Model Quality 🤖

• Smaller models ≠ GPT-4
• Good for summaries, commands, extraction
• Not ideal for deep reasoning

🙏 Thank you for reading!

I hope this walkthrough of building an on-device AI text summarizer in Flutter inspired you to explore local inference in your own apps.

Running models directly on the device opens up powerful possibilities around privacy, offline capability, and performance — and we’re only getting started. Feel free to experiment, optimize, and extend this setup to fit your use case.

Got questions or ideas? Drop them below — I’d love to hear your thoughts. Until next time, happy building! 🚀

References

• Gemma 3 TF-Lite Models (Kaggle):
  https://www.kaggle.com/models/google/gemma-3/tfLite
• flutter_gemma Documentation:
  https://github.com/DenisovAV/flutter_gemma
• Full Source Code for This Project:
  👉 You can refer to the complete Flutter project implementation here:
  https://github.com/BolisettySujith/On-Device-AI-Summarizer-App

👨‍💻 A Small Intro About Me :

Hey, I’m Sujith Bolisetty — a passionate Mobile developer with 3+ years of experience in building scalable, production-ready mobile apps. I focus on writing clean architecture, optimizing performance, and delivering great user experiences. I enjoy working with mobile development, AI, UI/UX design, and also love creating content around the things I build. 🚀

Let’s connect:

💻 GitHub : https://github.com/BolisettySujith

🔗 LinkedIn : https://www.linkedin.com/in/bolisettysujith/

🐦 X (Twitter). : https://x.com/bolisettysujith