With smartphones becoming a centerpiece in our lives, their ability to do more and more with incredible power and staggering efficiency is only going to grow more important. Through computational photography, smartphone cameras are able to achieve impressive results, but there are drawbacks. This method of creating images demands serious performance, and it can take time to render the final images. This can mean image previews don’t line up with the final result, and it can create a large gap in image quality between still photography and video, which can’t get the benefit of stacking multiple frames together for a wider dynamic range and cleaner imagery.

The demands of AI image processing are steep, and even phone SoC’s with ISPs and Neural Processing Units (NPUs) on board simply lack the power to perform this AI processing. This lack of power ultimately means shortcomings in image quality that wouldn’t be necessary with a sufficiently powerful NPU. In order to better handle the sheer amount of data coming from cameras, OPPO has built a dedicated and powerful NPU that can go well beyond the capabilities of current smartphones.

To demonstrate the huge leap a powerful NPU can offer, OPPO is introducing its first imaging NPU — MariSilicon X — which will launch in the Find X Series in early  2022. The company has spent 10 years working on its imaging technology, from lenses and sensors to in-house algorithms and now its own NPU. With the vertical integration of these features, OPPO’s MariSilicon X is set to offer new levels of photographic performance for smartphones.

With a cutting-edge Imaging NPU built from the ground up, the MariSilicon X is able to offer serious performance. The built-in NPU can hit 18 TOPS@INT8 of processing and achieves an efficiency of 11.6 TOPS/watt. Its processing speeds need memory that can keep up, so OPPO has included a Terabit memory subsystem and an extra 8.5GB/s of DDR memory bandwidth on MariSilicon X. The memory plays a role in this efficiency as do many AI operations being hardwired into the NPU’s design. With this high performance, the MariSilicon X can perform noise reduction on 4K video at 40fps, fast enough to provide this in real-time.

Currently the smartphone image processing pipeline involves capturing an image and compressing that image down from RAW into RGB and YUV — a process that sees much of the original image data lost. Phones then run their computational photography work on these compressed images, which severely limits the potential quality of the final product. The MariSilicon X’s NPU steps right into the middle of this process, as it has the speeds necessary to handle an image processing pipeline in the RAW domain. In other words, it can run its computational photography work on the original image while all of its data is still intact.

The MariSilicon X enables video capture at 4K with Ultra HDR in 20-bit and a 120dB dynamic range by combining multiple HDR frames in the RAW domain while recording. The effect is individual video frames that are on a level footing with still photography. The power of this processing and the speed it can achieve even allows for 4K AI Night Video — a first-time capability for Android — using RAW data with a final output that’s as vivid as the live preview.

MariSilicon X supports 20bit Ultra-HDR Video

When combined with specialized hardware, the MariSilicon X NPU can take things even further. The MariSilicon X can support OPPO’s RGBW Pro mode using an RGBW sensor, which then lets it process red, green, blue (RGB) and W signals separately and combine them into a single image with an 8.6dB improvement to dynamic range and 70% more texture.

With the limitations of the current imaging pipeline and the breakthrough potential of a dedicated NPU with enough power to process images in the RAW domain, the benefit of adding imaging NPUs to smartphones is evident. They’ll allow users to capture photos at a higher quality and turn night video into a reality.

MariSilicon X brings 4K AI Night Video on Android smartphones for the first time  where light intensities lower than 5 lux

You can learn more about the MariSilicon X and all of OPPO’s forthcoming innovations and technological offerings at OPPO Inno Day 2021.