From Python to Flutter: User-Selected Object Recognition on a Mobile Device
How I moved my computer vision experiment from Python to Flutter and made it work on a real phone. In my previous experiment, I worked on one problem: How can a vision system remember one specific object selected by the user?
Imagine that the camera can see several people. The user selects one person, and after that, the system should try to recognize the same person in the following frames.
I first experimented with traditional tracking and visual similarity. These approaches worked in simple situations, but the problem became more difficult when objects moved close to each other or looked similar.
So I tried another approach.
Instead of using only a fixed similarity function, I use features from YOLO and train a small neural network for the object selected by the user.
I first built this idea in Python.
Then I wanted to answer another question:
Can I make the same idea work directly on a mobile phone?
This article is about the Flutter implementation.
From Python to a Mobile Device
My goal was to build the complete process on the phone:
1. Run object detection from the camera.
2. Let the user select one detected object.
3. Collect visual features for that object.
4. Collect negative examples from other detections.
5. Train a small target-specific neural network.
6. Continue processing the camera and recognize the selected target.
The important part is that the target is not known when the application is built.
The user defines the target at runtime.
The adaptation also happens locally on the phone. I do not need to send the training images to a server.
YOLO and TensorFlow Lite
For the mobile version, I use a YOLOv8-based model exported to TensorFlow Lite.
The model receives a 320 × 320 image.
Normal object detection gives me bounding boxes and classes, but I need more information for this experiment. I also need visual features for every detected object.
For this reason, my exported model exposes three intermediate YOLO feature maps: P3, P4 and P5.
For every accepted detection, I use ROI average pooling to extract information from these feature maps.
The feature sizes are:
- P3: 64 values
- P4: 128 values
- P5: 256 values
After concatenating them:
64 + 128 + 256 = 448
So every detected object gets a 448-dimensional descriptor, which I then L2-normalize.
This means YOLO has two jobs in my application:
1. Detect objects
2. Provide visual features for each detected object
These visual features are what I use to identify the object selected by the user.
Selecting a Target
When the user taps a detected object, the application starts the enrollment process.
During enrollment, I collect features from the selected object.
In the current implementation, I collect up to 30 positive frames.
The selected object becomes a positive training example. Other detections of the same class can become negative examples.
For example, if the user selects one person, other detected people can be used as negative samples.
During enrollment, I also use spatial consistency to continue following the selected object.
This is important because another nearby object should not suddenly become a positive training example.
Learning From Negative Examples
Positive examples alone are not enough.
The network also needs examples of objects that are not the selected target.
During enrollment, competing detections of the same class can be collected as negative samples.
The current implementation can prioritize up to three competing detections with higher detection confidence.
For example:
Selected person → Positive
Other person → Negative
Another person → Negative
Before training starts, the implementation also requires at least 10 negative samples.
This helps the small network learn the difference between the selected object and other possible candidates.
It becomes especially useful when several similar objects are visible at the same time.
The Target-Specific Neural Network
The next part is the small neural network.
I implemented this network directly in Dart.
Its architecture is:
448 → 256 → 128 → 32 → 1
The 448-dimensional YOLO descriptor is the input.
The 128-dimensional representation is L2-normalized before the final classification layers. The network also uses dropout during training.
The final output is converted into a probability representing how likely the detected object is to be the target selected by the user.
An important detail is that I do not retrain YOLO.
YOLO stays frozen. Only the small target-specific network is trained.
This makes runtime adaptation much more practical than trying to train the complete object detector on a mobile device.
Training Directly on the Phone
The current target network is trained for 240 optimization steps.
Training uses positive and negative examples collected during enrollment.
After training is complete, the new weights are immediately used by the application.
There is no server-side training step in this prototype.
This is probably the most interesting part of the project for me.
A common mobile ML application uses a model that was completely trained before the application was installed.
My experiment adds another small learning step.
The pretrained YOLO model provides general visual features. Then the application trains a much smaller model for the specific object selected by the user.
The user can define something new while the application is already running.
Keeping Flutter Responsive
Moving this project from Python to Flutter was not only about exporting YOLO to TensorFlow Lite.
A phone camera continuously produces frames.
For every frame, the application may need to perform:
Get Bolot Kalil uulu’s stories in your inbox
Join Medium for free to get updates from this writer.
- image conversion
- model inference
- detection decoding
- feature extraction
- target recognition
Doing all of this on Flutter’s main isolate could make the UI unresponsive.
So the current implementation uses Dart isolates for expensive operations.
Long-Lived Inference Isolate
TensorFlow Lite inference runs inside a dedicated long-lived worker isolate.
The worker owns its TensorFlow Lite interpreter and continues processing inference requests.
Communication with the worker uses SendPort and ReceivePort.
The isolate is not recreated for every camera frame.
This keeps the expensive inference pipeline away from the main Flutter UI isolate.
Training Outside the Main Isolate
Training the target-specific network requires forward passes, backpropagation and weight updates.
This work also runs outside the main UI isolate.
The current implementation uses Isolate.run() for training.
After training is finished, the resulting weights are returned and used by the application.
Why I Do Not Process Every Camera Frame
Only one camera frame is processed at a time.
If the inference worker is already busy, I do not continue adding frames to a growing processing queue. The new frame can be skipped.
There is a simple reason for this.
Imagine that the camera produces frames faster than the neural network can process them.
If every frame is stored in a queue, the application slowly starts processing older and older images.
For a real-time camera application, this is not useful.
I prefer to keep latency lower instead of trying to process every frame.
If the pipeline is busy, skip stale frames and continue with a newer frame.
This was an important difference between running a computer vision experiment on a video file and running it from a live phone camera.
Recognizing the Selected Target
After training, every candidate has information from two different systems.
YOLO tells me whether the detection is a valid object of the expected class.
The target-specific network tells me whether this object looks like the target selected by the user.
The application uses these values to decide which candidate should be accepted.
In simple terms:
YOLO: Is this a person?
Then:
Target network: Is this the specific person selected by the user?
That difference is the main idea behind the project.
Current Performance Limitations
The system works, but it is still an experimental prototype.
One important limitation is the performance of the small target-specific network.
The Target Network Is Currently Written in Dart
The forward pass, backpropagation and optimizer are currently implemented directly in Dart.
This was useful for building the prototype because I could control the training process and keep the implementation inside the Flutter project.
However, Dart is not necessarily the best option for performance-heavy numerical computation.
Even though the network is small, training still requires many matrix operations and gradient calculations.
One possible future optimization is moving this part to native C or C++.
Flutter and Dart could continue handling the application logic, while performance-critical numerical operations could run through FFI in native code.
I do not want to assume that moving code to C++ will automatically solve every performance problem.
The correct next step is profiling the current implementation, finding the real bottleneck, and then optimizing the parts that actually need it.
Computer Vision Limitations
Performance is not the only limitation.
Recognition can become more difficult when:
- the target is heavily occluded
- its appearance changes significantly
- another object looks very similar
- the viewing angle changes a lot
- the enrollment samples do not contain enough useful information
The current thresholds and training parameters are also experimental.
So I consider this a working prototype, not a finished production tracking system.
Demo
I recorded the current Flutter implementation running on a real mobile device.
The application detects objects, allows the user to select a target, collects examples, trains the small network and then tries to recognize that target in the following frames.
Source Code
The Flutter implementation and the full experiment are available on GitHub.
Flutter implementation
https://github.com/bolotkalil/user-selected-object-tracking/tree/main/mobile/flutter
Full project
https://github.com/bolotkalil/user-selected-object-tracking
What I Learned
For me, the most interesting result is that mobile AI does not have to mean only running a frozen pretrained model.
A large pretrained model can provide general visual information.
Then a much smaller model can adapt to something selected by the user at runtime.
In my project:
YOLO provides general visual features.
The small network learns the current target.
The first version of this project was a Python computer vision experiment.
The second version moved the same idea to Flutter and a real mobile device.
During this process, the problem changed.
It was no longer only:
Can this computer vision idea work?
It also became:
Can it work continuously on a phone without blocking the application?
Using TensorFlow Lite, background isolates, and skipping stale camera frames helped me build a responsive mobile prototype.
There is still a lot to improve, but now the basic pipeline works on the device.
The next step is to find the real bottlenecks and make the system faster and more robust.
