What is embedded vision?

With a quiet hum, the autonomous vehicle rolls through the narrow lanes of the parking garage. The lidar sensors scan the environment, detecting walls, pillars and free parking spaces. Suddenly, the car comes to a halt. A gap – perfectly sized for the car – is detected. The vehicle carefully begins to park, precisely reversing into the parking bay and detecting the side and rear walls. But then something unexpected happens: the vehicle’s vision sensors detect a small sign at the edge of the parking space. The image data is analyzed using an OCR algorithm. There is writing recognizable! It is analyzed, interpreted and the result is clear: “Private – towing at your own expense” is written on the sign. The car immediately stops the parking process, pulls forward out of the space and continues its search. A few meters further on, it finds a regular parking space – avoiding the towing. 

This scene illustrates the crucial importance of embedded vision in modern autonomous systems. While lidar sensors do an excellent job of spatial detection, it is only when they are combined with vision sensors that a deeper contextual analysis can be carried out through image recognition and text interpretation. But what exactly is embedded vision? 

A brief introduction to embedded vision 

Embedded vision is basically nothing more than a digital system with a camera. At a minimum, it consists of data-processing hardware and software and a camera module, which includes a sensor, lens and an interface that matches the hardware and software. The purpose is always to give a larger system the ability to see. Just like in a car. 

In the field of embedded vision, some terms are used that are sometimes used synonymously, such as machine vision or computer vision. However, there is a classic interpretation of machine vision and a comparatively narrow definition of computer vision. Machine vision, although sometimes used synonymously with embedded vision today, was previously interpreted as a system consisting of standalone standard cameras connected to a traditional desktop or rack computer with x86 architecture via a long cable such as Ethernet, USB or CoaXPress. Analog parts or converters such as frame grabbers were and are still common to some extent today. 

In modern thinking, however, machine vision or embedded vision is more of a highly integrated digital camera system that efficiently collects and processes data “at the scene”. In contrast to machine vision, embedded vision implies that vision capability is planned with right from the beginning of the embedded system design process, comprising a vision-capable architecture with specific features, an application-specific design and the opportunity to access and “tune” the image signal processor (ISP). The hardware is consolidated to the functionality that is really required by the application. Therefore, you could speak of embedded vision as a technically and economically consolidated and improved version of machine vision that improves functionality and resulting image material while reducing the bill of materials (BOM). In recent years, more powerful hardware has favored this development and also enables networked, highly integrated vision systems at the (network) edge. Here we speak of edge devices with vision capability. 

Computer vision, on the other hand, is more clearly defined as a subfield of artificial intelligence. Computer vision is used as AI-supported machine vision or processing of raw image data to learn certain skills such as object recognition. Two things are necessary for this: Seeing with cameras and understanding with AI. To stay with our example, these technologies make it possible to recognize signs in parking garages. However, this requires the use of very powerful (large) computers, such as those more commonly found in laboratories. 

Modern, powerful embedded hardware 

Raw data such as image data from a parking garage sign is now captured, analyzed and interpreted on site using machine vision and powerful hardware. The interpretation is the basis for the actuator in the digital control loop. Put simply, and without taking further raw data into account, the purpose is to pass the signal on to the wheel adjustment mechanism and the car’s acceleration unit so that they can act accordingly. A modern car consists of many electronic control units (ECUs) that are networked with each other and interpret data. Traditionally, ECUs are microcontrollers that perform only a single task.

However, for consolidation reasons, they are increasingly being replaced by multiprocessor system-on-chip (SoC) solutions, on which several applications can run simultaneously. This powerful hardware is usually a combination of performance and efficiency cores from different CPUs or microcontrollers, often based on ARM architecture, mounted on a single circuit board. The performance cores – the Cortex-A cores in the case of ARM – can execute compute-intensive applications. This is particularly interesting for processing image data. Cortex-R and Cortex-M cores, on the other hand, are used for safety-critical, real-time tasks.

As SoCs, like NXP Semiconductors i.MX 8M Plus and i.MX 95 application processors are often also designed for image processing and come with the required interfaces, such as MIPI CSI-2 for camera sensor connectivity, and dedicated hardware engines for the ISP, Vision Processing Units (VPUs), and Neural Processing Units (NPUs) for AI acceleration within the SoC, many new possibilities for embedded vision are opening up here. These SoCs are suitable for use in automotive, avionics, rail, industrial, and medical technology markets, as well as in many smart, compact devices in various industries. The possible applications are very diverse. 

Modern camera modules for high-quality raw image data 

On the sensor side, the development towards ever more potent camera modules in recent years is remarkable. These camera modules consist of at least a lens, matching mount, image sensor and circuit board with ISP and an interface for the adjacent hardware. Just a few years ago, sensors that deliver razor-sharp images with a resolution of 200 megapixels were still a long way off. The standard and cost-effective use of global shutter sensors, which deliver distortion- and artifact-free individual images, is now possible in the industrial sector with sensors such as Sony’s IMX900 and is in ever-increasing demand. Back-illuminated pixels-now improve the vision in the dark for many sensors. A lot is also happening in terms of connectivity: With the GMSL standard, distances of up to 14 meters can be bridged on the camera modules, and for communication between hardware and camera module, interfaces such as SLVS-EC or MIPI CSI-2 support high data rates, so that the large amounts of data that are generated by increasingly powerful sensors can also be forwarded and processed.

The software, including drivers, reference applications and layers for using certain camera modules on a roll-your-own Linux like Yocto, for example, is becoming more and more extensive and powerful. Camera modules can thus be used more and more flexibly, specifically and easily. 

Fields of application for embedded vision 

Nowadays, there are camera modules that can be conveniently integrated for almost any application. System integrators and camera manufacturers refine these modules into industry-specific embedded vision systems. For example, a system integrator can build camera systems that are specially tailored to industrial needs and certified according to the IEC 61508 safety standard, and can thus be used in areas where humans work with machines and safety has the highest priority. This is the case, for example, with robotic grippers that work alongside humans on a production line. In the automotive industry, the ISO 26262 and ISO PASS 8800 standards are leading the way when it comes to safety (ISO 26262) and the use of artificial intelligence (ISO PASS 8800). It is conceivable, for example, that cars could use their connectivity to form ad hoc networks over-the-air (OTA) to supply each other with safety data and thus help to prevent accidents. For example, cars driving ahead could use their vision to detect potholes and forward the information to the cars behind. Traffic signs could also be detected and processed earlier. If these cars not only share safety image data with surrounding cars, but also with a central computer, this is referred to as an edge-to-cloud application. This makes sense in particular for computer vision because the valuable – and nowadays essential: real – raw image data can be sent to a central computer (in the cloud) for further processing, where these central computers can improve machine learning of specific image data and thus also provide feedback to the participating edge devices to improve safety data recognition. 

Conclusion 

Embedded vision is the ability of a digital processing system to capture and process image data. Developments in recent years have made its use increasingly powerful, simple and cost-effective. The captured image data helps to make our world safer and more efficient. The use of computer vision maximizes the potential and lays the foundation for groundbreaking applications. FRAMOS offers suitable camera modules and an entire ecosystem to generate the best possible image data to achieve this goal.