The prototype runs entirely on a Meta Quest 3, using its cameras to track the hands without controllers or extra wearable sensors. A short calibration records how far the user can open and close their hand, and I use that range to calculate how much to amplify the finger curl. A small physical closing movement can then become a fuller virtual grasp.

The tricky part is where to do this. Simply posing the visible fingers differently wouldn't change what the game thinks your hand is doing. I placed a custom amplification component in Meta's hand-data pipeline, before both the hand visuals and interaction logic receive it. That way, the hand you see and the hand that operates the tool agree. Your movement still drives the action; it just gets a little more mileage.

Of course, opening and closing your hand while staring at a counter isn't exactly the most exciting game idea. Instead, I made a little plant-watering environment: someone asks you to help their plant, you water it, and it grows and blossoms in front of you. The exercise becomes a small, visible achievement rather than just another repetition.

There are two watering tools, each encouraging a different movement. The garden hose needs you to close and hold your hand, while the spray bottle responds to repeated squeezes. Early testing taught me that the model itself matters quite a bit: a spray-bottle-shaped tool made people squeeze repeatedly even when the instructions asked them to hold. Giving each mode its own recognizable tool helped the interaction make sense at a glance.

The tool stays attached to the hand and the water is guided towards the plant, keeping the task focused on hand closure rather than fiddly picking-up or aiming. A little artistic assistance to go with the movement assistance.

To test the interaction, I ran an exploratory study with 11 healthy participants whose finger movement was restricted using medical tape. Each person tried the same two 60-second watering tasks with and without amplification, then rated how the interaction felt.

What interested me most was how that assistance felt. With amplification, participants rated their grip as more powerful, closing their hand as more comfortable, and their own performance as more competent. They felt less insecure, discouraged or irritated, and were more willing to keep practicing. For an exercise built around repetition, feeling that your effort actually gets you somewhere is quite an important part of the experience.

The questionnaire reflected this across several dimensions: perceived grip power rose from 3.36 to 6.73 on a seven-point scale, while competence rose from 3.41 to 6.07. Motivation and perceived rehabilitation value increased too, and burden/frustration dropped for every participant. The extra flowers were encouraging - everyone completed more plants, with the mean total rising from 3.45 to 8.36 - but the change from "this feels impossible" to "I can do this" was just as meaningful.

Without amplification, some participants couldn't get a watering tool to work at all. With amplification, all 11 were able to use both tools and successfully water plants in both exercise modes. For those participants, assistance didn't simply improve a score; it made the exercise possible in the first place.

There was a tradeoff: ratings related to agency and hand fidelity stayed high overall, but dipped slightly, particularly for how correctly the virtual fingers matched the physical hand. Some participants noticed glitchy or overactive movement. Making the action useful and keeping it believable need to go hand in hand, quite literally.

This was a proof of concept under simulated restriction, rather than a test of recovery in stroke patients. What it showed was an encouraging interaction principle: limited movement could become enough to succeed at a task, while that success still depended on the person's own attempt.

The project has since grown beyond the thesis: I adapted the exercise environment for AutoAssist, a collaborative stroke-rehabilitation research project funded by the German Federal Ministry of Education and Research (BMBF). With partners including ottobock and Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), my work on the VR environment is in preparation for an upcoming study at Uniklinik Erlangen with the exoskeleton setup.

The broader project explores assist-as-needed rehabilitation: combining ultrasound sensing, intent detection, AI and VR to adapt assistance and exercise difficulty as a patient progresses. The goal is more personalized therapy that keeps people engaged in their practice. It's an exciting context for the same question that started my thesis: how can we make the movement someone has available enough to do something meaningful?

There is also a less visible side to the environment: the app connects to a database to report information about its usage, and a live dashboard lets the therapist send commands back to the running app - including switching the balloon game's level, as shown in the video above. The exercise on the headset is one part of the project; being able to see what it is doing and interact with it from outside is another.

Bringing together interaction research, programming and technical art has made this a particularly rewarding project to work on. From helping a small plant bloom to steering a balloon around a monument, the common thread is making a little movement go further - and giving it something worth doing.