This project and article was a collaboration between Juan Ramirez, Javed Roshan and Steven Leung.

The residential door camera market has grown steadily in the past decade and is expected to reach $2.8B by 2025. The major players in the market like Ring (an Amazon subsidiary), Nest (a Google subsidiary), Arlo and Wyze recognize this opportunity and have been making their products more intelligent and capable with each new generation. AI-powered functionality such as face recognition and package recognition have become commonplace in these door camera systems, promising homeowners greater security through automated monitoring of what is happening outside the home.

While the production systems have steadily improved in their capabilities and accuracy, they do not offer the flexibility of a DIY system in terms of hardware (camera and compute) and custom model training. And of course, it’s just way more fun to DIY it.

Our smart door camera will use two models- one that can recognize known individuals through facial recognition and another that can identify whether an individual is a delivery (USPS, UPS, Amazon) or non-delivery personnel through full-body recognition. To achieve this, we will:

• Build and annotate our dataset for training.
• Fine-tune Yolov5 on our dataset using Amazon Web Services (AWS) Elastic cloud compute (EC2).
• Load both the Yolov5 fine-tuned model and dlib’s one-shot facial recognition model onto our edge device, an Nvidia Jetson Xavier NX, connected to a standard webcam.
• Set up an SMS notification system using Twilio to send out appropriate notifications as live streaming video is passed through the models.
• Package everything in a kubernetes-docker framework for ease of use.

While we had specific recognition objectives in this example, the beauty of a DIY solution is that you can teach your camera system to recognize whatever you would like by modifying the training data and objectives.

Training to Inference: Data Pipeline

Delivery Non-Delivery (DND) Recognition

As anyone who has worked on a new object detection problem knows, more than half the challenge is collecting and annotating enough images to allow a model to train effectively. To address this we leveraged youtube videos licensed under the creative commons license for training images. We identified these videos using two sets of search queries- the first set related to delivery personnel, for example, “Door Camera Delivery”, “Ring Delivery”, etc., and the second related to non-delivery personnel. By doing some further manual review and clean-up of the videos, we verified that individuals appearing in the first set of videos could be labeled as delivery personnel while individuals in the second set could be labeled non-delivery. Frames were extracted at a 1 frame per second (fps) rate from each video. These frames served as our training, validation, and test data.

To address the annotation challenge, we used deeplab segmentation to identify the pixel coordinates of personnel in each image. Through this method of annotation, we were able to annotate ~10k images in a semi-automated fashion.

We chose to use Yolo (You Only Look Once) v5 for this task as it is one of the most effective quick and easy pre-trained model solutions currently available. Yolov5 accepts annotations in a specific format so the coordinates from deeplab segmentation were converted for Yolov5 to accept in the model.

Facial Recognition

Dlib’s face-recognition python api provides one-shot capabilities for highly accurate face-recognition, performing at 99.38% accuracy on the Labeled Faces in the Wild dataset. We just had to load this library onto our edge device along with images of the faces we wanted the system to recognize.

Evaluation

We trained the model twice for 250 epochs each, first on a limited dataset and finally on our full dataset. The mAP@0.5 was 0.582 on our first run and 0.607 on our second. Each training run took around 22 hours on an AWS g4dn.xlarge EC2 instance.

Mean average precision was slightly higher for the delivery class than for the non-delivery class. A review of a sample of the errors reveals that the model mislabels delivery vs non-delivery in cases where individuals are further from the camera or in adverse lighting conditions. Additionally, the model also fails to identify a person at all in some cases. These errors are often related to individuals in unconventional poses or at the edge of the image frame.

The takeaway? Your training data matters. Many real-world problems won’t have data served up to you ready to go. This means that taking more than half of your project time on collecting an adequate volume of domain-specific, robust data is not unreasonable.

Measuring Success

In testing the model, we sought to simulate what would be considered the criteria for success in the real world. Since each individual at the door would be captured in multiple frames and thus, classified multiple times by the model, we grouped test image frames from each individual’s visit to the door into what we called “events”. While some of the frames in each event may be classified incorrectly, as long as the majority of the classifications being used to form a notification to the user were correct, the notification sent to the user would still be correct. Because of this, the event accuracy is actually more forgiving than the frame-by-frame accuracy.

Out of 1181 image files in the test dataset, 924 were classified correctly, 26 classified wrong and 249 were not identified as a person for a frame accuracy of 78%. However, when these frames were grouped into events and the majority classification of each event was determined, the model was correct 94% of the time, meaning the user would receive the correct notification.

Edge Inference and User Notifications

Inference is done on the Nvidia Jetson Xavier NX. The entire process is designed to run within docker containers in a kubernetes cluster. The architecture accommodates the dual model system. First, the live video stream is processed into frames stored locally. We use Eclipse Mosquitto message broker to publish messages to “Image Topic” in the format shown above, with the key information being the file path of the latest stored image.

The facial recognition and DND models both subscribe to the “Image Topic”. Upon message receipt, the models concurrently run inference on the image identified in the message. If the models recognize a person in the image, the output is published to the “Model Output” topic in the format shown. If a person is not recognized, no message is published.

The notification container is subscribed to the “Model Output” topic. Upon receipt of a message the notification adds the classification in the message to the cache. The cache is read every 20 seconds with the majority classifications received being output. We use Twilio services to send SMS notifications to the registered user.

Conclusion

While we hope this framework can get you going with your own custom smart home security system, this is only the beginning compared to what is possible. Some ideas we would love to explore:

• Continuous training of the model to continually enhance its predictive ability based on user feedback. This would involve pushing new image frames to the cloud as they come in and soliciting user feedback on the quality of the classifications.
• Transition the hardware of the system to devices more appropriate for the use case. Switch out the wired webcam for a wireless IP camera and the Xavier NX for a low-powered and cheaper GPU edge device.
• Incorporate more advanced functionality, such as deep learning video summarization that could provide the user with a full synopsis of everything that had occurred at their door for a given period of time.

To try Knock-knock and see all our code, head over to our public GitHub and navigate to the “To Run” section.