Radio Inspire

How To Learn Sign Language

M.Gesture: An Acceleration-Based Gesture Authoring System on Multiple Handheld and Wearable Devices


– I’m Ju-Wong Kim from Industrial Design department at KAIST. The co-authors are Han-Jung
Kim and Professor Tek-Jin Nam. Growing number of people are using more than two mobile devices, such as smartphone and activity trackers or smart watches. With the popularization
of wearable devices, many motion sensors are now tracking your body
movements in everyday life. So there’s no reason we
don’t use these sensors for full body gesture interactions. Despite the potential
of full body gesture, it’s very difficult to
define a unified gesture for all users. Body language is highly
dependent to culture, context, and individual preference. Different configurations of motion sensors also make it hard to define
one universal gesture. One way to overcome the
limitations is to make users to customize their own gestures. Customization is free from
contextual dependency, as well as individual preference. Custom gesture is also
known to be more memorable than a gesture defined by other person. Our research goal is to propose and test a gesture authoring system that end users can easily customize
accelerometer-based gestures using diverse combinations
of mobile devices. We chose to focus on an accelerometer as the main gesture sensor because it’s embedded
in many mobile devices, it’s very inexpensive and power efficient while sensing rich gesture information. However, an accelerometer is not accurate. It’s very hard to reconstruct
the original motion from only accelerometer values. Furthermore, the concept of acceleration is very unfamiliar to general users. If a user can’t understand the behavior of an accelerometer, then it’s very difficult
to define and perform accelerometer based gestures. Existing gesture authoring systems can be categorized by two approaches. Programming by demonstration
defines a gesture from user’s performed motions. It doesn’t require any pre-knowledge to define a gesture, but when an error occurs, it’s very hard to fix the problem or modify the already defined gesture. Another approach is
declaration based authoring, the gesture is defined
by high level language, and the gesture can easily read and modify the gesture definition. But users required to
learn the gesture language, in order to create a gesture. Several systems combine the advantages of demonstration based and
declarative approaches. For example, gesture
studio allow the users to demonstrate touch and strokes while their sequences are defined in a corrective manner. But because each method
defines its separate features, the limitations of each
approach still remain within individual features. This work is also
inspired by visual gesture definition systems such as Event Hurdle. Event Hurdle allows
graphical gesture authoring based on Hurdle scheme, but the interface of Event
Hurdle was too complex to be used on mobile devices, and its recognition was
inaccurate compared to proven by demonstrations. M.Gesture system is
based on the lessons from literature review and formative study. The goal of our research is to let users understand the behavior and constraints of accelerometers. It should be easy to learn and use, and the definition is to
be intuitively modifiable. Our formative study informed that our participants preferred the simple and subtle gesture motions. Most of their gestures were linear or two-dimensional. Participants also preferred
the single device gestures much more than multi-device gestures. And when they define two device gestures, about 90% of their
motions were synchronized. Please read our paper for
more detailed information about the formative study. We present M. Gesture, a gesture authoring application
using the accelerometer values of diverse combinations
of mobile devices. The accelerometer values are visualized with a physical metaphor of mass spring. A gesture is defined by
demonstration and declaration. The sense of value trajectory is visualized in 3D space. Then the user declares
the gesture trajectory using a graphical
gesture authoring scheme. M.Gesture visualizes sense of value using a metaphor of mass spring. The system shows a virtual
ball on top of a spring that is attached to a mobile device. If the oscillation can be ignored, the mass is affected by
inertia force and gravity, which means the position of the ball is solely determined by
the accelerometer values. The mass spring metaphor
allows users to understand the behavior of accelerometers, and they can predict how the sensor values will be changed when the
user enters a gesture motion. An intuitive representation could be the gesture’s direct path, but we did not use it because of its inaccuracy of the sensor. Another direct representation could be the acceleration value itself or a damped mass metaphor, but all the users who
experienced our system without the metaphor, thought its behavior was unpredictable. We then came up with the
mass spring metaphor. It was the result of the decision to make accelerometer
based gesture in a way that human can understand
while technically feasible. A gesture is defined by
demonstration and declaration. In terms of demonstration, a user can quickly and
easily define a gesture with a single demonstration. It calculates the distances
from input trajectory and defines trajectory by
dynamic time warping algorithm. Dynamic time warping is one of the most common algorithm for
comparing two trajectories. A hurdle based gesture
scheme allows the users to graphically specify the
range of trajectory path. A hurdle is a plane that detects intersection
of input trajectory. The sequence of hurdle intersection can be defined by user and it is represented
as a series of arrows. A hurdle plane defines the range of allowing trajectory. A short hurdle defines narrow and accurate gesture trajectories, whereas the wide hurdle defines brief and diverse trajectories. Our interface allows the users to easily create, modify, or remove
hurdles with a single stroke touch input. Because a user can visually
detect an intersection of input trajectory with the hurdle, she can intuitively determine where the hurdles should be located. To sum up, our hybrid gesture authoring allows users to easily
create the first gesture, and then visually change
the definition as they want. We conducted two studies
in order to answer three questions. Is it easy to understand the concept of Mass-Spring? Do users easily learn and use M.Gesture? Is it precise and fast compared to conventional algorithms? We conducted a lab based
task completion study with a post survey. We asked 20 participants to compose and correctly perform three gestures as illustrated in the slide. We designed the three task gestures subtle and simple as you
found in the formative study. The later tasks are designed to include more complex operations. From demonstration, hurdle declaration, and handling multi-devices. We gave the over
instruction about our system to participants, and before each task we explained the necessary functions and they had enough time to
practice those functions. The whole study took about 25
minutes per one participant. Every participant completed all tasks, except one failure in task three. The average time to compose
and test the gesture was less than 100 seconds. The biggest portion of
the task completion time was hurdle authoring and modification. 90% of gesturing authoring took less than two and a half minutes, and it can be improved
considering all participants were beginners. In the post survey, participants answered
the easiest of our system in five point scale. 100% answered the concept was easy or very easy to understand. The overall easiness was evaluated easy or very easy by 85%. Participants were
positive about M.Gesture’s interface and function in general. One participant said it
was good to visualize the noise of the accelerometer, which is an important
constraint of the sensor. Another participant said participant liked the hurdle scheme for allowing him to control
the range of trajectory. However, some usability
issues were also raised. For example, it was hard to
see the interface’s screen when a user performed a gesture with rotational motions. To evaluate the precision and processing
time of M. Gesture, we compared it with
its original algorithm, dynamic time warping. We defined 12 gestures from a reference, and recruited 20 participants
to perform all gestures. In the graph, the top
is dynamic time warping and the bottom is M.Gesture. As you can see, M.Gesture
was 4% point more accurate, I mean more precise. And 26% faster than the original dynamic time warping algorithm. The conventional dynamic time warping calculates all distances
from the input trajectory and defined trajectories, which uses intense processing resources. But M.Gesture system performs
the hurdle test first, then purges unnecessary candidate gestures before calculation. The hurdle test saves
the processing resource, and it’s an important
aspect in mobile computing. There are some discussion
points of our hybrid definition. As mentioned earlier, it made
the recognition much faster than the original algorithm. Second, a user can
flexibly define a gesture between program by
demonstration and declaration. She can quickly define a gesture with many more number of hurdles. Otherwise, she can spend
some time elaborating the gesture definition using
many number of hurdles. And lastly, our hurdle
scheme was simplified by the supplementary
information from demonstration. If a gesture must be defined
only with the declaration, then the gesture expression
must be very complex in order to specify
various gesture features. We also found some
limitations in future works of M.Gesture system. Because people constantly make
movements in everyday life, a guidance system is
required to make gesture designs more robust to
passive noise motions. There might be some
undiscovered problems since we did not conduct a
long term field study. Our system may support
advanced users in the future such as interaction
designers, programmers, or HCI researchers who want to easily create and modify gesture interactions. We propose M.Gesture system
that allows accelerometer based gesture customization
using physical metaphor of mass spring with a hybrid approach. M.Gesture was evaluated to be easy to understand, learn, and use. The performance of hybrid authoring was slightly more precise and much faster than
the original algorithm, dynamic time warping. Thank you for your attention, and it’s time for Q&A. (audience applauds) – [Leader] We’ll take
questions at the microphone. – [Gregory] Hi, Gregory,
from Georgia Tech. Did you get a chance to try seeing how creative people
could be with this tool? In the study, you reported here, you gave them gestures to do. Did you give them any chance to try and explore the gestures that they would up with on their own? – Actually, not. They were given only
designated three tasks. Yeah, myself, I sometimes
play with this system, and try to compose as
many gestures as possible. But, it’s kind of … Similar to what I found
in the formative study, people don’t want to use some very complex gestures like in a
three dimensional space. It’s completely unusable. – [Gregory] I was just wondering, also. You give only one example. So how do you know whether that gesture would be recognized in the future, if you have only one kind
of training example for it? – You mean the accuracy wise? – [Gregory] Yeah. – In this study, I measure the precision of our system, and we collected 1000 gesture motions and compared it with 12 gesture sets, and its overall precision was 88%. Thank you. – [Audience Member] One question. You showed that it is a 3D space that they’re performing the gesture in, and then most of your
hurdles are within 2D. – Yes. – [Audience Member] Have you seen anything with people working with this, and trying to do more in 3D space and their challenges, do you know? – That’s very important question. Because the gesture is defined in demonstration and declaration, in terms of demonstration, it’s full 3D. There’s no difficulties in distinguishing between gestures, but in the corrective manner. As you said, it’s designed in 2D space, some academic people ask me if they can compose in
three dimensional spaces. But, yes, actually, it’s the
constraint of our system, but as I said, most of users generally define the gesture like linear motions or two dimensional motions, so we thought 2D space
is enough for defining a usable gestures. (audience applauds)

Leave a Reply

Your email address will not be published. Required fields are marked *