AI video analytics software turns camera feeds into structured events, such as a person entering a restricted zone or a license plate passing a gate. Other common events are a forklift coming too close to a worker and a shelf going empty. A good result depends less on the model than on camera placement, tests on your own footage, false-alarm control and privacy law.
This guide explains the processing pipeline, which capabilities are mature, what drives hardware and cost, and how to judge a vendor or an in-house plan.
How AI video analytics works, step by step
Almost every video analytics system follows the same flow, whether it runs on a server in a warehouse or in a cloud region.
- Ingest. The system pulls streams from IP cameras, usually over RTSP. Many cameras follow the ONVIF standard, which supports discovery and configuration, and ONVIF Profile M adds a standard format for analytics metadata and events. An existing video management system (VMS) can often forward streams or events instead.
- Decode and sample. The system decodes H.264 or H.265 video, ideally on hardware decoders, then samples frames at the rate the task needs. Counting people at a door might need 5 frames per second, and reading plates on a fast road needs more.
- Inference. Models for detection, segmentation, classification, OCR or body pose run on each sampled frame or region.
- Tracking. A tracker links detections across frames into tracks, so the system knows that one object stays the same object over time. Rules for dwell time, line crossing and direction depend on tracks, not on single-frame detections.
- Rules and events. Zones, lines, schedules and thresholds turn tracks into events, for example "vehicle stopped in a no-parking zone for more than 120 seconds".
- Delivery. The system sends each event with a short clip or snapshot to operators, a VMS, a ticketing system or an API, and stores metadata for search and reporting.
Core capabilities and how mature they are
| Capability | Typical uses | Maturity | Main problems |
|---|---|---|---|
| Object detection (people, vehicles, safety gear, packages) | Intrusion, occupancy, safety compliance | Mature for common object classes | Small or distant objects, unusual angles, night and rain |
| Multi-object tracking | Counting, dwell time, line crossing, heatmaps | Mature in moderate crowds | Tracks swap identities in dense crowds and when objects block each other |
| License plate recognition (ANPR/LPR) | Gates, parking, tolling, logistics yards | Mature with suitable cameras | Needs a dedicated angle, fast shutter and infrared light. Plate formats differ by region. |
| OCR on containers, labels and meters | Logistics, utilities, manufacturing | Good on controlled views | Glare, dirt, motion blur |
| Logo and product recognition | Shelf monitoring, sponsorship measurement | Good with custom training | New packaging needs retraining |
| Defect inspection | Quality control on production lines | Strong in fixed setups | Rare defect types have few training examples |
| Pose and fall detection | Care homes, worker safety | Moderate | False alarms from sitting, bending and partial views |
| Anomaly detection | Flagging "something unusual happened" | Inconsistent | "Normal" is hard to define, so alerts are noisy |
| Natural-language video search with vision-language models | "Find the red van near dock 3 yesterday" | Usable, still changing fast | High compute cost. A person must verify results. |
| Facial recognition | Access control with consent | Technically mature | Heavily regulated, with bias and accuracy concerns. Some uses are banned or high-risk. |
Vision-language models can describe a clip or search footage from a plain-language question. Teams index footage with embeddings or captions, so they no longer train a new detector for every question, which helps investigations and audits. A vision-language model is slower and costs more per frame than a purpose-built detector, so production systems run small, fast detectors for real-time alerts and keep larger models for search and review.
Edge, cloud or hybrid
Inference means running the model on video frames, and where it runs is the biggest architecture decision.
- Edge. Inference runs on the camera or on a GPU box at the site. Latency and bandwidth stay low, and video can stay on the premises, which helps privacy, but you manage hardware, updates and failures at every site.
- Cloud. The cloud is easy to scale and update, and it suits heavy models and search across sites, but uploading many continuous HD streams costs a lot of bandwidth and raises data-transfer questions.
- Hybrid. Most deployments use this option: detection and tracking run at the edge, and only events, clips and metadata go to the cloud for dashboards, search and model improvement.
Bandwidth adds up fast. A 1080p H.264 stream at an assumed 4 Mbps is about 0.5 MB per second, so one camera produces about 1.8 GB per hour, or 43 GB per day. Fifty cameras that stream continuously to the cloud send over 2 TB a day before any analysis. A hybrid system that sends only events uploads a small fraction of that volume.
Compute grows with three factors: the number of streams, the frames analyzed per second and the model size. Analyzing 5 frames per second instead of 30 cuts inference work about sixfold, and cropping each frame to regions of interest cuts it further. Benchmark your chosen models on your target hardware early, because vendor throughput figures rarely match your own camera mix.
Camera systems often run next to other sensors. The guides to industrial IoT systems and smart pole infrastructure cover the device side, and the fleet management guide covers in-vehicle cameras and telematics.
Accuracy: test on your own footage
Published benchmark scores say little about your loading dock at 6 a.m. in fog. These practices give a useful measure:
- Collect a test set from your cameras. Cover day, night, weather, seasons and busy periods, and label the events you care about.
- Measure events, not frames. Operators need to know whether a real intrusion triggered an alert, which is recall, and how many alerts were correct, which is precision. A practical unit is false alarms per camera per day.
- Tune each camera. Zones, thresholds and minimum durations often matter more than the model.
- Limit alert volume. Operators mute a system that raises ten false alarms a shift. Many sites send fewer, high-confidence alerts and keep searchable metadata for everything else.
- Test again after changes. New cameras, lighting, layouts or models change results, and accuracy can fall so slowly over time that nobody notices.
Camera placement means choosing the height, angle, lighting and shutter speed, and enough pixels on the target. Moving or replacing a camera often solves a problem that looked like a model problem.
Privacy, biometrics and the law
Under the GDPR and similar laws, video of identifiable people is personal data, and biometric data used to identify people is a special category with stricter conditions. These laws lead to the following duties and design choices:
- EU AI Act bans. Several practices have been banned since February 2, 2025. The bans cover untargeted scraping of facial images to build recognition databases and emotion recognition in workplaces and schools, with narrow medical and safety exceptions. Real-time remote biometric identification in public spaces for law enforcement is also banned, apart from tightly limited exceptions. The full text of the EU AI Act is in Regulation (EU) 2024/1689.
- EU AI Act high-risk duties. Many other biometric identification systems count as high-risk, with duties that include documentation, testing and human oversight. A 2026 amendment moved the start of those duties to December 2, 2027.
- US state laws. Illinois' Biometric Information Privacy Act (BIPA) requires informed written consent before a business collects biometric identifiers. Statutory damages are $1,000 per negligent violation and $5,000 per intentional or reckless violation, and other states have their own biometric rules.
- Data minimization. Prefer analytics that count or detect without identifying anyone. Blur faces in stored footage where you can, keep retention short and restrict who can search video.
- Signs and assessments. Many jurisdictions require signs that announce camera monitoring, and regulators often expect a data protection impact assessment for large-scale monitoring.
Video used as evidence raises a separate question: proof that nobody changed it. If clips may serve as evidence in a dispute, record a hash of each exported clip and keep an access log. Some teams store those hashes on a blockchain to show that nobody changed the footage. The guide to blockchain for cybersecurity explains when a ledger helps and when a signed audit log is enough.
Buy or build video analytics software
Many modern cameras and VMS platforms include analytics, and vendors also sell add-ons for common needs such as intrusion, line crossing, counting and plate reading. Buy video analytics software when your use case is one of those. Build or customize when you need any of the following:
- Detection of objects specific to your business, such as a particular part, safety device or product
- Integration with operational systems such as ERP or incident tools
- Analytics across sites that no single product supports
Between buying and building, a third option combines a commercial platform with custom models. For a wider framework on AI project scoping and vendor checks, read the guide to choosing an AI development company.
Cost and timeline drivers
- Cameras and sites. Cost grows with the number of cameras and sites, and with the number of cameras that need an upgrade.
- Custom object classes. Each new object type needs collected and labeled examples, often thousands of images across conditions.
- Hardware. Edge GPU boxes for each site or cloud GPU hours, plus bandwidth.
- Integration. Connections to the VMS, access control, ERP or incident tools.
- Compliance work. Impact assessments, retention controls and audit logging.
The following durations are estimates based on the work involved. A pilot with existing detectors on a few cameras and a basic dashboard often takes a small team six to ten weeks. A multi-site rollout with custom models, edge fleet management and integrations usually runs six to twelve months.
Questions to ask an AI video analytics vendor
- Will you run a pilot on our cameras and report precision, recall and false alarms per camera per day?
- What runs at the edge, what leaves the site, and where is footage stored?
- How do you update models across sites, and how do you roll back a bad update?
- Which features process biometric data, and can we switch them off completely?
- Who owns the labeled data and the custom models we pay for?
- Which camera specifications does each analytic require?
Frequently asked questions
Can AI video analytics work with my existing cameras?
Often yes, if the cameras provide RTSP streams at a reasonable resolution. Plate reading, long-range views and night scenes may need specific camera models, angles and lighting. Plan a camera survey before you commit.
How accurate is AI video analytics?
Accuracy depends on the task and the conditions. Detection of people and vehicles in a well-lit, well-placed view is very reliable, but detection of rare events in crowded or dark scenes is much harder. A test on your own footage gives the only reliable figure.
Is facial recognition legal for my business?
The answer depends on the country, the state and the purpose. The EU bans or restricts facial recognition in several contexts, and Illinois requires written consent. Many deployments avoid identification and use analytics that do not process biometric data.
Should processing happen on the edge or in the cloud?
Real-time alerts and privacy-sensitive sites usually suit edge processing, while search across sites, heavy models and dashboards suit the cloud. Most production systems combine both.
How do we reduce false alarms?
Tune zones, schedules and minimum durations for each camera, and require a track to last a set time before an alert. Improve camera placement and lighting, retrain the model on the false alarms you collect, and track false alarms per camera per day.
What about vision-language models for video search?
Vision-language models let users search footage in plain language and summarize clips without a new detector for each question. These models cost more per frame, so they suit investigation and review more than real-time alerts on every stream.