Zero-Shot Defect Detection with Gemini: Prompt Engineering Tips to Improve Accuracy
Zero-shot detection means that we provide zero image examples to the model. Instead, using only a well-defined prompt, we ask the model to identify and classify defects present in an image.
The model then returns the bounding box of the detected defect, along with the defect classification based on the instructions provided in the prompt.
This application can be very useful for speeding up the annotation process, especially when dealing with large volumes of images. It can reduce the amount of manual effort required and help annotators focus more on reviewing and validating the model’s predictions.
Get Chandana Rajeev’s stories in your inbox
Join Medium for free to get updates from this writer.
The quality of the results, however, depends heavily on how the prompt is designed. Through experimentation with the Gemini model, I found a few prompting techniques that helped improve detection accuracy.
Prompting Tips for Improving Detection Accuracy
I will share some of the prompting tips that helped improve the detection accuracy of the Gemini model:
- Use the Gemini API instead of the web chat when experimenting with prompts and extracting keywords. The API provides a more consistent environment for testing and integrating the model into an application.
- Start with a vague question to understand what the model observes first. For example, ask: “What do you see?” This helps identify the basic information the model is able to recognize from the image.
- The model may provide information such as product identification, shape, colour, structure, and other visual characteristics. These details can be used to create a product summary, which can then be included as part of the prompt.
- Provide a concise and straightforward definition of the defect you want the model to detect. Avoid using uncertain words such as “likely,” “mostly,” “sometimes,” etc., as these can make the defect definition ambiguous.
- Use prompt “Do you notice any anomaly to know the keywords that helps model to detect the defect?”
- Clearly specify the expected output format. For example, define exactly how the model should return the defect name, confidence, and bounding box coordinates. A structured output makes the results easier to process programmatically.
- Gemini returns bounding box coordinates in the format [ymin, xmin, ymax, xmax], with coordinates normalized to a range of 0–1000 and the origin at the top-left corner of the image. Here, ymin and ymax represent the top and bottom boundaries, while xmin and xmax represent the left and right boundaries of the defect. If your annotation tool expects [xmin, ymin, xmax, ymax], you need to reorder Gemini’s coordinates accordingly.
- Avoid negative prompts wherever possible. Instead of telling the model what not to detect, clearly describe what the target defect looks like and what characteristics it should have.
- Keep the defect definition and the image characteristics consistent. The terminology used in the prompt should match what is actually visible in the image. For example, if a black spot is present but the prompt describes it as rust while the model interprets it as grease, the model may classify the defect incorrectly.
- Be specific about the visual characteristics of the defect. Including details such as colour, shape, texture, location, and appearance can help the model distinguish between visually similar defects.
- Keep the prompt simple and focused. Adding too many unnecessary instructions can sometimes make it harder for the model to identify the actual objective.
Overall, prompt engineering plays an important role in zero-shot defect detection. A clear defect definition, consistent terminology, and well-structured output requirements can significantly improve the quality and reliability of the model’s predictions.
