AI-algorithms on IoT devices

→ LOGIN

Advances in “edge AI” much more significant for the future than 5G

When I attended the CeBIT conference in Hannover in 2016, it was full of supposedly “smart” devices that didn’t seem particularly smart to me. The “smart” trend inspired many different devices that were essentially just connected to the internet. In most cases, their connectivity meant some additional, often narrowly defined, benefit for the user.

I still remember some of them:

A pipe valve that let you monitor its position (open/closed) remotely
A device for measuring gas flow, secretly installed in the fuel tank to identify fuel thieves
An electrical outlet that can be switched on and off through wifi

What I remember most from the CeBIT trade fair, however, is the presentation by the IBM development lab, which changed the way I thought about how artificial intelligence could be distributed. IBM was presenting its Synapse project at the trade fair, an AI chip called “TrueNorth”, that could deliver computing power equivalent to an ant brain while consuming just 73mW of energy. The only obvious disadvantage was that the processor cost about $1 million at the time.

This example proved, however, that running AI algorithms on end devices (edge computing) will be possible. It was also clear that Moore’s law would drop the price within a few years and make computing power more accessible. The question was how quickly this would happen and how many other similar solutions would emerge on the market. At that time Neuron soundware had already begun pursuing that IoT strategy – develop its own AI-optimized IoT solution, meaning algorithms for audio machine monitoring would work directly on the IoT end devices.

The future of edge computing

Several months later, I created a graph showing the relationship between energy consumption and intelligence as a function of computing power that IoT HW can deliver.

In 2016 almost no intelligence could be achieved for a reasonable price with just a few mW.
Narrow AI, such as the capability to drive a car, would need HW with power consumption in the double or triple digits of watts. Analysis of camera input about ten times per second, for instance, requires about 4 TFLOPS (4 trillion floating-point operations per second – FLOPS, flops, or flop/s – is a measure of computer performance).

Put in terms of what we do at Neuron Soundware: We can use the same computing performance either to drive a car or to analyze the sound of machines in order to detect an upcoming mechanical failure. Either would require computing power equivalent to the brain of an ant. And IBM made me see this power coming within a single ultra-low-energy chip.

The rise of edge computing

Edge computing capability has been on the rise since then. It’s worth a look at several projects involving chips specialized in hardware acceleration for artificial intelligence.

In 2017, Movidius introduced the Neural Compute Stick, which could deliver 0.1 TFLOPS for under $100 and about 0.5W in power consumption. It is designed to improve the performance of smaller computing boards such as Raspberry Pi with about 10x computing power boost.

In 2018, Huawei introduced its Kirin 980 processor with 0.1W and almost 0.5 TFLOPS. Other vendors were not far behind. Google announced its Edge TPU units and Rockchip demonstrated its RK3399 chip equipped with a Neural Processing Unit. Both provide the performance of about 3 TFLOPS and cost around $100.

In 2019, specific microcomputers with hardware accelerators of AI technologies (specifically Neural Networks) become generally available for use in commercial projects. All key HW players have released edge optimized versions of the AI software stack, which further increases the performance. Generally available AI boards include Google’s Edge TPU, a purpose-built ASIC chip for AI. Nvidia’s Jetson Nano brings 128 CUDA cores into action for less than $100. ToyBrick’s RK3399 Pro is one of the first development boards with a specialized Neural Processing Unit, which slightly outperforms even Nvidia Jetson.

In 2020 and 2021, Tesla, a well-known manufacturer of electric cars, presented its self-driving computer with the performance of 36 TFLOPS, while Hailo AI offers the performance of 26 TFLOPS in a single chip for about $100 and consumption of just several watts.

This rapid advancement in IoT technology allowed us at Neuron Soundware to develop nBox – a specialized device for recording and processing sound that can not only record high-quality audio with up to 12 channels but also evaluate data locally using AI algorithms. Edge computing allows us to run only a few processes in the cloud (central platform) and process most of the data using AI algorithms in the end device.

Intel, for instance, confirmed the significance of edge computing when it acquired Movidius for an estimated $400 million, as well as Mobileye, a manufacturer of chips for self-driving cars, for over $15 billion. I was fascinated to see the Tesla Motors online presentation of its specialized AI, developed for their self-driving cars. It can process more than 2000 high-definition camera images per second, and Tesla believes this offers enough power for complete autonomy.

I see the four key advantages of edge computing as these:

Security: All processed data can be stored locally and kept secure as needed.
Speed: AI can process data in milliseconds, which means minimal latency.
Efficiency: Powerful, low-energy microcomputers are available at reasonable prices.
Offline: The AI algorithm can also be operated in areas with limited internet access.

The advantages of edge computing over 5G

Why go to the trouble of using specialized IoT HW, you ask? Why not wait for 5G networks and make use of the highly developed cloud infrastructure? There are several reasons why “waiting” may not be the best IoT strategy.

Imagine sitting in a self-driving car that suddenly loses its 5G signal. The car will not only be unable to sense what is around it but will literally lose its brain, its decision-making ability. Why take that risk when the computing power necessary for high-speed communication with low latency can cost practically the same as an additional neural processing unit? Using specialized AI would also consume less energy.
Mobile internet providers want to ensure return on their investment in developing and using 5G networks. While unlimited data plans are theoretically possible, they may not be commercially viable in the near future. Our nBox, for instance, has 12 acoustic sensors and can produce up to 1 TB of audio data per month. At the current price for a GB of data in LTE, transferring that volume of data to the cloud would not be financially sound.
Lastly, 5G coverage will be available primarily in cities, and many places will be outside 5G range. Edge IoT devices can be deployed immediately anywhere, with clear initial and operating costs that usually do not significantly increase total costs in IoT solutions.

AI edge computing can process huge volumes of data at or near the place where the data was created. The additional costs for hardware accelerators are marginal already. Computing power for neural networks is growing in leaps and bounds, ten times more powerful each year. This trend is not slowing down because data can be processed in parallel, thus overcoming the limits of conventional processor design.

The future at hand

Using edge computing in applications like self-driving cars, facial recognition, and predictive maintenance is just the beginning. Soon we will have enough computing power to build truly autonomous working machines. Eventually, they will be able to move around safely in cities and factories, becoming nearly as competent at their jobs as humans. It is incredible to think that someone could have foreseen this nearly 100 years ago. Last year, 2020, marked 100 years since the word “robot” was coined. In his science fiction play R.U.R., Karel Čapek envisioned humanoid robots that gained awareness and learned to feel emotions such as love. This vision, and the word “robot”, spread quickly around the world. Considering the rate at which computing power and improvements in the Internet of Things are growing, I can’t help but think that Čapek’s vision could become reality much sooner than we think.