Cornell's WatchHand Turns Any Smartwatch Into a Hand-Tracking Controller Using Sonar

Your next smartwatch might double as a gesture controller. Researchers at Cornell University and KAIST have developed WatchHand — the first AI-powered system that turns off-the-shelf smartwatches into real-time hand-tracking devices using nothing but built-in speakers and microphones.

How It Works

WatchHand emits inaudible sonar waves from the smartwatch speaker. These waves bounce off the wearer's hand and return to the microphone, creating an echo profile. A machine learning algorithm processes this data directly on the device to estimate hand pose in three dimensions — all without any additional hardware.

Test Results

• 40 participants across four studies totaling 36 hours of gesture data
• Tested across multiple smartwatch models, right and left hands, and noisy conditions
• Reliably tracks finger movement and wrist rotations in real time
• Currently works on Android smartwatches only — not Apple Watch
• Limitations: struggles during walking, less accurate with rapid hand movements

Why It Matters

This could transform how we interact with AR/VR headsets, smart glasses, and even smart home devices — all from a watch you already own. The technology could also enable assistive features for users with limited mobility. For now, this is a research project, but its path to commercial deployment is clear: Android watches like the Pixel Watch 3 and Galaxy Watch Ultra could gain gesture control via a software update. Check Pixel Watch 3 prices and Galaxy Watch Ultra prices.