Enhance! Why AI is getting so good at sharpening images

June 15, 2023

From our smartphones to our surveillance cameras, artificial intelligence is getting very good at turning blurry, grainy photos into high resolution ones.

The secret behind the improvements are a pair of technologies called AI super-resolution and AI upscaling, in which AI models are trained on huge data sets to either add hidden details to blurry imaging, or increase the resolution of a blurry photo.

“AI upscaling has improved significantly over the years, thanks to several key technological advancements,” said IEEE Member Yale Fox. “One such advancement is transfer learning, which allows developers to use pre-trained models for new tasks, greatly increasing the speed and accuracy of upscaling. Another key development has been the availability of large datasets of images, which are essential for training deep learning algorithms to recognize and generate high-quality images.”

Fox said that advances in graphics processing units (GPUs) have also improved this technology.

“The combination of GPUs and deep learning has also been critical in making AI upscaling faster and more efficient. GPUs can process large amounts of data simultaneously, allowing for faster training and inference.”

Many smartphones use AI upscaling to improve the quality and resolution of photos. But the technology has uses in other arenas as well, including the medical, satellite and entertainment sectors.

Satellite imagery: Recent years have seen an explosion in the number of satellites launched into orbit. It’s easier and less expensive to launch smaller, lighter satellites. But smaller satellites may not be able to carry expensive, high resolution cameras that larger ones can. As this article from IEEE Xplore® shows, AI super-resolution allows smaller satellites the ability to capture better images with smaller cameras.

Weather forecasts: Some weather satellites watch over big parts of the Earth. The pictures they take may only show details that are several square miles in size. That works for a lot of uses, but some folks want to know if it’s raining right outside their house, not over a huge area. An article from IEEE Access describes a system that teaches low-resolution weather satellites how to take better pictures. It does this by using old weather and climate data, which helps to improve the quality of the pictures from these large area weather satellites.

Video Meetings: Real time video, streamed over a laptop or even a phone, uses a lot of internet bandwidth. For decades, these services have relied on audio and video codecs to compress and decompress the data stream. Many of these codecs are decades old, developed at a time when the internet was much slower.

“State-of-the-art algorithms now only require a single high-resolution photograph of a person, and then the camera tracks how their facial muscles move to create a kind of deepfake image,” Fox said. “This represents a nearly 800x reduction in bandwidth and file size, which is crucial for applications such as video calls. That is such a massive increase, that it really does change the game a little bit.”

Video games and streaming video: As is the case with online video meetings, image upscaling is having dramatic impacts in video games. A report in IEEE Spectrum shows that, for some graphics-heavy video games, seven out of every eight pixels are being generated by AI, resulting in speeds that are 530% faster. There’s also significant research into ways to apply super-resolution technology to on-demand video streaming.

Medical Imaging: Super-resolution techniques have broad applications in the medical field. Many medical images, such as X-rays and MRI scans, have low resolutions due to various factors – including the length of time of a scan and the limitations of equipment. For example, improved resolution of tiny blood vessels around tumors can aid cancer identification and treatment. Super-resolution can also improve imaging of fetuses, which tend to move around and make imaging difficult.

So what would make these technologies even better?

Better training data, says IEEE Senior member Fabrizzio Soares.

“It is important to mention that these algorithms rely on previous knowledge from images, so the more samples are provided, the better accurate the model will be,” Soares said. “But building the most complete image database is a hard problem. Moreover, the more samples are provided, the bigger the model will be. So, it will require a larger effort to train and maintain the model.”