Just last week, DxOMark began evaluating selfie cameras on phones. Now, popular benchmarking service AnTuTu has announced a new evaluation tool for AI devices.
With AI performance being a much-touted feature of processors and phones these days, we can say that the benchmarking tool has arrived just in time. AnTuTu says that the tool is still in beta.
In an official press release, the service revealed that there is no single standard for AI yet, as each chipset manufacturer understands it differently. Qualcomm uses its DSP to handle AI tasks, while Huawei, Samsung, and MediaTek have a dedicated AI unit called an NPU (APU for MediaTek). All chip manufacturers also have different SDKs. - NeuroPilot for MediaTek, SNPE for Qualcomm, and TensorRT for NVIDIA. Samsung has yet to announce its own.
Nevertheless, AnTuTu has managed to develop a unified benchmarking test standard, thanks to the cooperation of processor manufacturers. The test is said to be divided into two phases – image classification and object recognition.
Image classification is based on the Inception v3 neural network and the test includes 200 images. Object Recognition is based on the mobilenet SSD neural network and its test is a 600 frame video.
AnTuTu recommends that the benchmark test should be done using Wi-Fi, as the data package is approximately 160MB in size, even though the app itself is only 33MB. Downloading the data package to a device you intend to benchmark is a one-time thing, so you won't be able to repeat the test any other time. However, there will be a data package verification each time.
Devices are ranked based on a combination of speed and accuracy. Therefore, a device that is fast but not accurate will be ranked lower. AnTuTu states that there are other conditions that can result in a lower score for a device.
For example, a device running an older version of Android will score lower than one running a newer version in the benchmark test. This is because Google optimizes AI performance with each new version of Android that is released. Also, the fact that Samsung has not released its own SDK will result in lower scores for its devices. Also, there will be no significant differences between two processors running the same “AI Processor”. An example is the Snapdragon 710 and Snapdragon 845 which both use the Hexagon 685 DSP.
