Watch how Google's AI catches shoulder surfers spying on your phone

Web giant's R&D boffins to tease privacy-protecting tech in NIPS show-and-tell

Video Google researchers claim to have developed an "electronic screen protector" that can alert you when nosy parkers are looking over your shoulder at your phone.

Essentially, it's machine-learning-powered software that can use the front-facing camera on a smartphone to pick out faces, identify anyone who isn't the owner, and if they are gazing at the screen, alert the user that there's someone behind them snooping on their texts or web browsing, and so on, all in real time.

The Googlers' work on this technique – which appears to be just an academic project at this stage rather than an upcoming feature – will be presented at the Neural Information Processing Systems (NIPS) conference next week in California. A video, embedded below, created by the small research team, shows how an online conversation on Google’s messaging service Hangouts is repeatedly interrupted when the front-facing camera on the smartphone detects strangers looking at the screen – perhaps on the subway, office, or elevator.

Youtube Video

A red bounding box is placed around the stranger’s face, and if that wasn’t enough of an alert, a stream of rainbow vomit appears to flow out the stranger’s mouth and a message flashes saying: “A STRANGER IS LOOKING ALERT!!!” If the team were British, no doubt the warning would read: "It's behind you!"

Technical details on how the code works are scarce. However, it looks as though it works by identifying faces that are not recognized as being an owner of the phone. That data is provided by a facial authentication system, and means that the software has already been trained to spot strangers.

A gaze tracker follows the direction of people’s eyes so it can detect when someone is looking directly at your phone's screen.

The researchers, Hee Jung Ryu and Florian Schroff, said the system is robust under different lighting conditions and head poses. The facial recognition and gaze tracker has to be fast, and takes two milliseconds to detect each person’s gaze, 47 milliseconds for each face recognition operation, and an average of 115 milliseconds per frame for face detection – not enough time for a human to steal much of a glance, let alone to read private messages.

“Because of the quick, robust, and accurate gaze detection mobile model we can now easily identify the face identity and gaze simultaneously in real time,” a description of the demo reads. “We enable this by transfer learning from one mobile model to a different, but related task.” ?

El Reg will be at NIPS next week – drop your friendly vulture an email or tweet if there's anything we should be looking out for, or if you want to say hello.

PS: Facebook has made an AI thing that automatically flags up potentially suicidal people on the social network, based on their posts, so they can be offered help from humans.


Biting the hand that feeds IT ? 1998–2017