Modern computer vision gets discussed almost entirely in terms of models. YOLO this, transformer that, a new architecture every few months. But a vision model does not do anything until it runs on silicon, and when the target is an embedded device rather than a data centre, the hardware stops being an implementation detail and becomes the thing that decides what is possible. A model that needs a 300-watt GPU is not a model you can put on a camera, a vehicle, or a machine. The embedded hardware behind a vision system is not the boring part underneath the interesting part. It is the constraint that shapes every model decision above it.
what edge AI actually is: computer vision running on-device, inside a real power and thermal budget, rather than in a rack somewhere.
The accelerator is the heart of the system
At the centre of any embedded vision system is the accelerator, the chip that runs the neural network. There is no single right choice, only a spectrum with genuinely different trade-offs.
The reason this choice cascades through the whole design is efficiency. Every watt spent on compute is a watt that has to be removed as heat and supplied by the power system, so the accelerator’s TOPS-per-watt sets the terms for the thermal design, the enclosure, the power budget, and ultimately the reliability of the device. Picking the accelerator is not step one of the software project. It is the foundational hardware decision the software then has to respect.
The rest of the compute stack matters more than it looks
The accelerator does not work alone. Around it sits a system-on-module or carrier board that provides the CPU, memory, storage, and interfaces, and each of these can become the bottleneck. Memory is the one teams underestimate most. On a small accelerator, performance is frequently limited not by raw compute but by memory bandwidth, how fast data can move between the camera, system memory, and the accelerator. A model that is arithmetically light but moves a lot of data will underperform its benchmark badly, which is why embedded vision hardware is chosen and evaluated on data movement, not just TOPS. Typical embedded vision workloads want several gigabytes of memory, commonly in the 4 to 8 gigabyte range, and the memory subsystem’s bandwidth is often the real ceiling.
Hardware and model are one decision
Here is the point most vision projects learn too late: the model and the hardware are not chosen independently. A model that runs in floating point at full size will not fit a 5-watt accelerator, and a hardware target chosen without knowing the model will either be over-provisioned and expensive or under-provisioned and slow. The two have to be co-designed.
Quantization and pruning shrink a model to fit the embedded envelope, and they are not afterthoughts, they are how a modern vision model becomes deployable at all. Quantization converts the model from 32-bit floating point to INT8, roughly quartering its size and letting the accelerator use lighter, faster integer math. Pruning removes weights or whole channels the model does not need, cutting both computation and the memory traffic that so often dominates on small parts. Done properly, through quantization-aware training rather than a naive post-hoc pass, these techniques recover most of the accuracy while making the model fit hardware it could never have run on otherwise. The result is that a smaller, cheaper, cooler accelerator can carry a workload that looked like it needed a bigger one, which improves every downstream property of the device.
The comparison that frames the choice
| Hardware class | Efficiency | Power | Best fit |
| Dedicated NPU (e.g. Hailo-8) | ~10 TOPS/W | ~2.5 W | Single optimised model, fanless devices |
| GPU module (e.g. Jetson Orin) | ~4.6 TOPS/W | 15-60 W | Multi-model pipelines, sensor fusion |
| FPGA module (e.g. Kria) | High for fixed pipelines | <10 W | Deterministic, custom datapaths |
The physical envelope is the final filter
Even a well-matched accelerator and model have to survive the physical environment, and this is where embedded vision hardware differs most from its data-centre cousin. A device on a vehicle or in a sealed cabinet cannot rely on active cooling, so passive thermal design has to carry away the heat the accelerator generates or the part will throttle under sustained load, quietly dropping the frame rate exactly when the system is busiest. Reliability, expressed as mean time between failures, is derived for the whole assembly under its real operating temperature, which ties longevity directly back to the thermal design. And component availability over a product’s multi-year life is a hardware selection criterion in its own right, because an accelerator that is designed out in two years forces a redesign. These physical constraints are the final filter that a vision system’s hardware has to pass, and they are decided at design time, not discovered in the field.
Benchmark on the target, not the datasheet
One habit separates teams that choose embedded vision hardware well from teams that get surprised in integration: they benchmark on the actual target, running the actual optimised model, before they commit. A datasheet TOPS figure is measured under ideal conditions that your workload will not reproduce. The number that matters is frames per second, at your resolution, with your model, in INT8, on the specific board, at the worst-case temperature it will see. That figure is routinely a fraction of what the headline spec implies, because memory bandwidth, preprocessing, and thermal throttling all take their cut.
The bottom line
The embedded hardware behind a modern computer vision system is not scaffolding around the model. It is the constraint that determines which models are even possible, and it has to be designed as one decision with the model rather than chosen after it. Lead with efficiency rather than peak compute, respect memory bandwidth as the real bottleneck it usually is, use quantization and pruning to make the model fit the silicon rather than demanding silicon that fits the model, and design the thermal and reliability envelope for the environment the device will actually live in. Get the hardware right and the impressive model has somewhere to run. Get it wrong and the best vision model in the world stays exactly where it was trained, on a machine that will never leave the lab.
