Computer vision explained: How it works, what it does and where to start
Computer vision is the branch of artificial intelligence that extracts useful information from images and video and enables action based on it. It already operates in phones, factories, hospitals and warehouses.
A phone unlocking when it recognises your face and a car braking after its camera spots a pedestrian are both examples.
This guide explains the technology, its capabilities and limitations, and how to begin. Last updated October 2026.
In short:
- Computer vision recognises visual patterns; it does not understand scenes as people do.
- Representative data and testing in the actual environment matter more to project success than the model alone.
What is computer vision?
Computer vision turns pixels into useful results: “this X-ray may show a fracture,” “there are 14 pallets in bay 3,” or “this parcel label reads 9712 AB.”
Many systems use machine learning, learning patterns from examples instead of following manually written rules. Deep learning, based on multilayer neural networks, accounts for most recent advances. Other systems use traditional image-processing and geometric techniques, such as measuring parts under controlled lighting, or combine both approaches.
How does it differ from image processing and generation?
Image processing changes an image, for example by cropping or sharpening it, without trying to interpret its contents. Computer vision often uses those operations as preparation, but aims to determine what is present and where.
Image generation reverses the direction. Tools such as Midjourney create pixels from text prompts; computer vision analyses existing pixels.
Does it understand what it sees?
Not in the human sense. A model labelling a photograph “dog” has learned patterns associated with dogs in its training data. A driver recognises a stop sign partly covered in snow and knows to stop. A model may miss it if its training examples excluded snowy signs. Treat predictions as useful signals rather than comprehension.
How computer vision works
Whether inspecting welds or reading invoices, most systems follow five steps.
- Capture. A camera, scanner or drone records images or video. Position, resolution and lighting determine the quality available to every later step.
- Prepare. Images are resized, normalised and sometimes cropped to match the model’s requirements.
- Run the model. A trained model produces raw results, such as probabilities or bounding-box coordinates.
- Interpret. Software converts those results into a decision. A defect alert might require a confidence threshold or the same finding in three consecutive frames. Confidence is not a guarantee; test thresholds against real errors.
- Act. The decision enters a process, such as a reject arm removing a product or a specialist receiving a case for review.
Training and inference
Training is the learning stage. In supervised training, the most common approach, thousands of labelled examples, such as photographs with defects marked, teach the model to match its predictions to the labels. Other methods learn from unlabelled images or image-text pairs. Training requires substantial computing power.
Inference applies the trained model to new images. Its lower computing requirements allow models to run on phones or small devices beside production lines.
The models behind modern computer vision
Three approaches are particularly important. The first two are network architectures; the third is a broader system type built on them.
Convolutional neural networks (CNNs) drove advances in the 2010s. Small filters scan for edges, which are combined into shapes and objects. They remain common in production.
Vision transformers (ViTs) divide images into patches and learn relationships between them. In 2020, Google researchers showed that, with enough pretraining data, this approach can match or outperform leading CNNs.
Vision-language models connect pictures and words, usually using transformers. OpenAI’s CLIP learned from 400 million image-text pairs. The same principle allows ChatGPT, Gemini and Claude to answer questions about photographs.
Reliable results require representative data
Performance depends on the images used for learning and those encountered in practice. A model trained on daylight photographs may perform poorly during night shifts under orange factory lighting.
In a 2019 study, UC Berkeley researchers recreated an ImageNet test set using the original procedure. On slightly harder images, classifier accuracy fell by 11 to 14 percentage points. Training data must cover real variation. Test on images from different products, sessions or dates, rather than near-identical frames from one video.
What can computer vision do?
Nine capabilities cover most applications, often in combination.
Image classification
Classification assigns one or more labels to an entire image. Photo apps use it to group beaches or receipts. Insurers might classify damage photographs as “repairable” or “write-off.”
Object detection
Detection locates and labels individual objects using boxes. A traffic camera counting cars, bicycles and buses in each lane relies on it.
Image segmentation
Segmentation labels pixels, estimating each object’s boundary instead of drawing a rough box. Video-call background blur is one example. Meta’s SAM 3, released in November 2025, outlines every instance of a concept from prompts such as “yellow school bus.”
Object tracking
Tracking maintains an object’s identity across video frames. Sports broadcasts use it to calculate how far a player has run.
Optical character recognition (OCR)
OCR extracts printed or handwritten text. Parking garages use it to read number plates. Document-understanding systems combine the extracted text with layout and other signals to identify fields such as invoice totals.
Pose estimation
Pose estimation finds key points, including shoulders, elbows and knees, and connects them into a simple skeleton. Fitness apps use it to count repetitions and assess squats.
Anomaly detection
Anomaly detection learns normal appearance and flags deviations. It suits rare, varied defects because every fault need not appear in training. A model trained on thousands of sound castings can identify a previously unseen crack.
Depth estimation and 3D perception
Depth estimation determines distances within a scene, sometimes using one camera, to build a three-dimensional view. Robot vacuums use it to navigate furniture, while phones use it for portrait mode.
Multimodal image understanding
Models combining language and vision answer open questions, such as which switch is off in a photograph of a fuse box. They are flexible and easy to try, but less predictable than dedicated models for narrow, high-volume tasks.
Where computer vision is used
Some applications are established; others remain experimental. The table reflects the position in October 2026.
Healthcare illustrates both scale and caution. The US Food and Drug Administration listed more than 1,600 AI-enabled medical devices in September 2026. This includes all AI, not just computer vision, although imaging accounts for most entries. Each device has authorisation for a defined purpose. Most assist clinicians; a few screen independently within narrow limits, including the IDx-DR diabetic eye screening system, authorised in 2018.
Benefits and limitations
Computer vision offers speed, scale, consistency and automation. It can analyse thousands of images hourly using the same criteria at 3 a.m. and 3 p.m., leaving people to handle cases requiring judgement.
Get DataNorth AI’s stories in your inbox
Join Medium for free to get updates from this writer.
Performance nevertheless depends on conditions:
- Lighting and camera position: glare, shadows and displaced cameras change the input.
- Occlusion: partly concealed objects are harder to detect and count.
- Unfamiliar settings: a model trained in one warehouse may struggle in another.
- Dataset bias: under-represented groups or conditions receive poorer results.
- False positives and negatives: every system makes both. Decide which is more costly and tune accordingly.
Moving from a demo to dependable operation
Demos typically use clean, selected images. Deployment introduces dusty lenses, product changes, seasonal lighting and moved cameras. These shift the images away from training conditions. Run a pilot in the real environment for weeks and continue monitoring after launch.
Privacy, security and responsible use
Images of identifiable people can constitute personal data under the GDPR. Facial templates used for identification are biometric data subject to stricter rules. Camera projects need a legal basis, a defined purpose and notices for people being filmed. A data protection impact assessment is required where processing is likely to pose high risk, such as large-scale monitoring of public spaces.
Facial recognition and surveillance present the greatest risks. Since 2 February 2025, the EU AI Act has prohibited untargeted scraping to build facial databases, workplace and educational emotion recognition except for medical or safety reasons, and, with narrow exceptions, real-time remote biometric identification by police in public spaces. Certain other biometric applications are high-risk. Our EU AI Act guide explains the timeline.
Accuracy can vary between demographic groups. A 2019 US National Institute of Standards and Technology study found demographic differences in most tested face recognition algorithms, including higher false-positive rates for some groups. Protect camera feeds and stored footage as you would other sensitive information.
When human oversight matters
Human oversight is necessary when errors could harm someone or trigger an expensive decision, including medical findings, hiring, access control and law enforcement. One common design accepts predictions above a tested threshold and sends uncertain cases to a reviewer. Early reviewers also uncover patterns hidden by averages, such as failures on a particular product colour.
How to get started
Begin with a specific problem.
- Define the task and success criteria. “Use AI on our production line” is too vague. A hypothetical testable target is: “Detect scratches longer than 2 mm on at least 95 percent of panels, with at most one false alarm per 200 panels.”
- Check images, permissions and quality. Gather representative images across operating conditions, confirm usage rights and decide who will label them consistently.
- Choose an existing tool or adapt a model. Common tasks such as OCR may need only a ready-made tool. Specific products or defects usually call for fine-tuning a pretrained model. Training from scratch is a separate, much rarer choice.
- Pilot, evaluate and monitor. Test unseen images from the actual setting and compare performance against the existing process.
For a beginner project, Google’s free Teachable Machine can train a browser-based classifier without code. Use a webcam to distinguish recyclable packaging from general waste and observe how lighting and background affect predictions.
Business pilots can also stay small. Consider a hypothetical furniture manufacturer inspecting panels before packing. It fine-tunes a pretrained detector on about 2,000 labelled photographs from one camera station. For four weeks, the system runs alongside inspectors without rejecting products. The team then compares both against agreed criteria.
Five factors drive cost more than the model: collecting data, labelling it, computing for training and inference, integrating with machines or business systems, and maintaining the system, including retraining as conditions change. Labelling is often the largest effort.
What comes next?
Three developments combine capabilities already available with our forecasts.
Multimodal systems: assistants can already interpret photographs, charts and documents. We expect them to handle more work previously requiring dedicated models.
On-device processing: phones already perform face unlock locally. We expect more inspection and safety systems to run directly on cameras, improving speed and privacy.
Learning with fewer labels: Meta’s DINOv3, released in August 2025, learned visual features from 1.7 billion images without human labels, allowing adaptation with few labelled examples.
Choose one repetitive, measurable visual task in your organisation where errors are costly. DataNorth’s computer vision team can help define it and test a small pilot.
Frequently asked questions
Is computer vision the same as AI?
Computer vision is one field within AI, focused on images and video. AI also includes language, speech and numerical prediction. Many vision systems use machine learning, especially deep learning; others use traditional image-processing and geometric methods or a combination.
How does it differ from image recognition?
Image recognition identifies an image or object, such as a cat, car or product type. Computer vision also locates and measures objects, reads text, tracks movement, estimates depth and answers questions about scenes.
Can it process video in real time?
Yes. Driver assistance, sports tracking and production-line inspection already do. Feasibility depends on the required frame rate, model size and hardware. Compact models on dedicated chips process dozens of frames per second.
How much data is needed?
It depends on the task and starting point. Ready-made tools may require no training data, but still need representative evaluation examples. Fine-tuning often works with hundreds to a few thousand labelled images. Variety matters more than sheer volume.
Do you need programming skills?
No-code tools, ready-made applications and image-capable assistants support many simple tasks. Production integration with cameras and business systems, monitoring and retraining require technical skills, usually from developers or a specialist partner.
How accurate is it?
There is no meaningful universal figure. A model can perform reliably under controlled lighting and poorly outdoors. Request measurements from your own environment that include false positives and false negatives.
What does a project cost?
Costs range from subscriptions for existing tools to substantial custom-system investment. Labelling, integration and maintenance generally outweigh the model itself. A small pilot with explicit success criteria is the cheapest way to assess a larger investment.
What are the main privacy concerns?
Filming without a clear purpose, facial identification and excessive footage retention are central concerns. Images of people are personal data under the GDPR, and the EU AI Act prohibits several biometric uses. Collect only what is needed, blur faces where possible and assess risks first.
Turn images into decisions
Computer vision lets software recognise, locate, measure and read camera inputs, then act at scale. It works best with a narrow task, representative real-world data and people overseeing consequential decisions.
Originally published by Diederik Knol, DataNorth, on 26 October 2026: Computer Vision: How it works and what it does.
