Week 1 Progress
FridgeAgent is pioneering new technology aimed at revolutionizing home food management. At its core, FridgeAgent utilizes ROS2, a robust robotics middleware, to process video recordings of fridge interiors and generate detailed inventories of fridge contents. This blog post dives into our initial strides, focusing on Node 1 and Node 2 of our ROS2 architecture, and our efforts to enhance detection reliability.
ROS2 Node Layout:
- Node 1 — Video Capture: This node captures video footage from a camera mounted above the fridge, currently triggered manually.
- Node 2 — Fridge Boundary Detection: Using an image segmentation model trained from scratch, this node identifies the boundary of an open fridge and publishes the corresponding polygon of pixels.
- Node 3 — Content Segmentation: Processes the fridge contents detected by Node 2 using a segmentation model and publishes the relevant pixel data.
- Node 4 — Inventory Management: Determines if new objects are added or removed from the fridge, updating the inventory count accordingly.
Demonstration:
Node 1:
Get Rick Koleta’s stories in your inbox
Join Medium for free to get updates from this writer.
Video Capture Here is an example frame from the recorded video:
Node 2: Fridge Boundary Detection Our current approach involves training an image segmentation model from scratch using a dataset of 300 manually labeled images collected and labeled via LabelMe. Below is an example of the model detecting the fridge boundary:
While effective under optimal lighting conditions, challenges remain in robust detection, as gaps in boundary detection can occur, as shown above.
Here is an example of the detection model under better lighting conditions
Next Steps:
To enhance detection reliability, our next step involves leveraging a pre-trained model and fine-tuning its weights with our dataset. This approach aims to improve boundary detection accuracy, particularly under varying lighting conditions and complex backgrounds.
Stay tuned as we continue to innovate and refine our technology to bring seamless food management solutions to your home with FridgeAgent.
