On-Device Bystander Blurring for Live Video

The challenge

A retailer’s mobile app lets shop staff hold live video conversations with customers from the shop floor. Anyone else in the shop appears in those calls without having agreed to it. The requirement was to blur every person in frame who is not a member of staff — during a live call, on a phone, starting with Android and extending to iOS.

What we built

We built the recognition model and the Android integration. Staff are enrolled by the shop owner, who uploads a short video of each member of staff. The resulting model is distributed to each device scoped by store and staff identity, so a phone carries only what its own store needs.

• Enrolment from a short video. The shop owner uploads a short video of each staff member, rather than posed stills.
• Everyone else is blurred. People in frame who are not enrolled staff are obscured.
• During the call, not after. Blurring happens in the live video conversation, not in post-processing.
• On the device. Inference runs on the phone.
• Scoped distribution. The model is pushed to each app by store id and staff id, so a device only receives the model relevant to it.

Outcome

The recognition model and the Android integration were delivered. iOS was specified as the following phase.

Technology

• Computer vision
• On-device inference
• Staff identification model
• Android