Apple brought gesture support to its smartwatches not too long ago. The capabilities, however, are pretty limited. Right now, the smartwatch can only recognize a two-finger tap gesture and a wrist flick gesture. But it seems the smartwatch already has the right set of sensors to recognize and learn a much wider variety of hand gestures, and doing so wouldn’t require any monumental effort.
Researchers from Northwestern University and Carnegie Mellon University have developed a system called TransfHAR that uses the motion sensors inside the Apple Watch, the accelerometer and the gyroscope, to learn and identipfy activities based on the movement of your hands and wrist. Imagine your Apple Watch triggers an action by recognizing your hand motion, without any touch input or voice command. Yeah, that’s the core premise here.
Your watch can literally read your hand movements
The broad idea behind TransfHAR is to have the smartwatch learn repetitive movements and then remember them. As part of the research, 10 participants each picked seven activities, such as stirring coffee, drawing a heart in the air, applying hand sanitizer, opening a Gatorade bottle, tying a shoelace, filing their nails, and more.
Interestingly, one of the members of the team behind TransfHAR is a research scientist from Google, and the system’s core model is adapted from LSM-2, a wearable sensor model developed by Google researchers. That’s a sign this capability could eventually make its way to Wear OS smartwatches as well, instead of being restricted to the Apple Watch. After all, the technique relies on the same kind of accelerometer and gyroscope data that nearly every smartwatch already collects.
The biggest surprise to come out of this experiment was just how good the Apple Watch is at learning these motion patterns when the data is fed to a model. To get started, the researchers trained the model on roughly 2,760 hours of publicly available wrist movement data from 381 people. Most of that data covered broad activities like sitting, walking, and exercising. They then asked participants to wear an Apple Watch Series 10 and perform their chosen activities. And these are mundane, everyday activities I’m talking about.
Unbelievably fast learning, and high accuracy
Here’s the most astounding part. The watch was able to recognize seven new activities with 90% accuracy using just one minute of recorded movement data per activity. Even with far less data, the results hold up. The researchers point out that with the 6-axis encoder, a single 2.56-second clip per activity delivers a balanced accuracy of 75.2%. With five clips, or about 13 seconds of data, accuracy climbs to 86.7%. And with 20 clips, or about 51 seconds of data, it goes all the way up to 89.3% when taken as a balanced average across all the activities.
Everything also happens on the watch itself. Training the model on a new set of activities takes less than a second, and the watch can make a prediction in about 24 milliseconds, without needing a paired phone or a remote server. Users can also keep adding new activities over time. In one test, where the researchers pooled 70 activities into a single model, accuracy still stayed above 82%.
The potential here is huge. Now that we know Apple’s smartwatch can learn new gestures in less than a minute, users could train an app to recognize certain gestures and automatically kick off specific actions. For example, when the watch detects a coffee-stirring motion, it could play some relaxing music on a connected smartphone, pop up your daily caffeine tracking, pull up some coffee trivia from the internet, or do anything else you want that gesture to be linked to.
Likewise, let’s say the watch has been trained while you’re doing bicep curls. That motion can be linked to a custom action, so as soon as the smartwatch detects your biceps workout, it automatically starts logging it, plays a certain playlist, or connects to a particular health dashboard. The researchers also point to medical uses, such as passively measuring symptoms like tremors or hyperactivity, and factory systems that can track assembly steps.
The possibilities are bright, and seeingly endless
Linking these gestures to smart home products is one of the most promising outcomes of this research, allowing users to automate a whole bunch of tasks without saying a voice command or pressing a button. As soon as your Apple Watch detects that your hands are performing an activity, it can automatically trigger an action using a specific app or a shortcut you’ve already built.
![TransfHAR: Self-Supervised Wrist Representations for On-Demand Activity Recognition [UIST 26']](https://i.ytimg.com/vi/8bVF4_c3y6s/hqdefault.jpg)
The best part is that the training process isn’t too complicated. You don’t need to wear any custom sensor for the Apple Watch to learn the repetitive activities you engage in on a daily basis. The watch’s built-in 6-axis IMU, which is essentially the combination of the accelerometer and the gyroscope, provides enough data to learn from a few seconds of motion. Recording a full minute of each activity pushes accuracy past 90%.
It’s not perfect, though. The system struggles with activities that involve small, sustained hand movements, such as writing, scrolling on a phone, or applying lotion. These look very similar to the watch’s sensors, even though they’re completely different tasks. It also has a harder time with multi-step routines, like the individual steps of making a latte. Plus, the study was short and controlled, and the team didn’t measure the impact on battery life. But the broad premise is extremely exciting. I can’t wait to see what developers can do with this capability once the training tools are publicly available, or once someone figures out an easier way to do it.
