iPhones can now automatically recognize and label buttons and UI features for blind users

iPhones can now automatically recognize and label buttons and UI features for blind users

3 years ago
Anonymous $y15ULlV7sG

https://techcrunch.com/2020/12/03/iphones-can-now-automatically-recognize-and-label-buttons-and-ui-features-for-blind-users/

Apple has always gone out of its way to build features for users with disabilities, and Voiceover on iOS is an invaluable tool for anyone with a vision impairment — assuming every element of the interface has been manually labeled. But the company just unveiled a brand new feature that uses machine learning to identify and label every button, slider, and tab automatically.

Screen Recognition, available now in iOS 14, is a computer vision system that has been trained on thousands of images of apps in use, learning what a button looks like, what icons mean, and so on. Such systems are very flexible — depending on the data you give them, they can become expert at spotting cats, facial expressions, or as in this case the different parts of a user interface.