Radio Inspire

How To Learn Sign Language

Touch+Finger: Extending Touch-based User Interface Capabilities with “Idle” Finger Gestures in …


– Thank you for the introduction. Good morning everyone. My name is Hyunchul Lim and I’m a research assistant at Human Centered Computing Lab at Seoul National University, South Korea. Today I’m so pleased to present
my work named Touch+Finger. This work is in
cooperation with my advisor Professor Bongwon Suh and
Professor Joonhwan Lee and three PhD students, Jungmin Chung, Changhoon
Oh, and SoHyun Park. At first, let me introduce
Touch+Finger briefly. Touch+Finger is a new
interaction technique that augments touch input with a multi-finger in-air gestures for reach and expressive interaction. Let me show you a quick video demonstrating the basics of Touch+Finger for a better understanding. As you see in the video, the main idea of Touch+Finger is to use idle fingers. What I mean is that when one primary finger is
engaged in touch interaction the rest of the secondary finger are in idle states. So we call them idle fingers. Since they are able to move a user can use these idle
fingers to add modality and expressiveness to the
primary touch interaction. For example, by making different
hands pose before a touch a user opens a file by tapping
with a basic hands pose. Or open a context menu of the file by tapping with a spread-over hands pose. Also, as we saw in the video before a user can use the side
of the finger to make an expressive brush stroke
in a painting application. So, why I have introduced the
amazing idea of Touch+Finger here I go through the
motivation and related works. Reasons to re-invent the
finger aware shortcut includes the input space on the keyboard by pressing same key with
a different hands pose for different shortcut commands. I will work closely with this work in that it also use the rest of the idle fingers. But our work focuses
on touch-based devices rather than a keyboard. Also many research has
been explored in increased input of vocabularies
by combining… oh sorry by combining touch and gestures. For example, touch
interaction can be enhanced by regulating additional information such as in-air gesture with a thumb or with a touch screen on a smartphone. Or the thumb based touch
gestures on sensitive touch sensor in a ring-like devices. Or the movement of the wrist by using ultra sensory microwaves. However, based off our
knowledge we are not aware of any existing work
that enhances the primary touch interactions by making full use of these idle fingers. So, our goal with this
research is to explore how the rest of the other fingers can be used as additional information, for enhancing the primary
touch interaction. To do so, we created a design
space, covering a variety of interaction techniques
with these idle fingers. We aimed to augment existing
two dimensional touch gestures, by using idle fingers. So, we have divided a
combination of touch gestures into two categories. Single-touch, and multi-touch. In this paper, for single
touch we used an index finger as a primary finger, and for multi-touch we used a thumb and an index finger as the primary fingers. In this case, the secondary
fingers are the thumb and/or the middle plus fingers. This is middle plus fingers. The middle plus finger
means that the motions of the middle, the ring, and the little fingers are together. Also with touch interaction,
we divide the touch of interaction into two dimensions, before touch and the during touch. Before touch is the
period that’s just before the primary finger touched the screen. In this case, the secondary idle finger can make a different hand pose before a touch event. During touch refers to the time when the primary finger is touching the screen. In this period, a secondary in-air gesture with idle finger enhanced the
primary touch interaction. By combining these two… two factors, we created the two-by-two design space of before single-touch,
before multi-touch, during single-touch,
and during multi-touch. So based on the design
space, we determined all possible motions
of the secondary finger in each sub-section. Now, let’s look at before
single touch gestures first. This is a basic hands pose, and this is a stick hands pose, and bend hands pose. This is a L-shaped hands pose, this is bend all hands pose. This is a spread all hands pose. This is all different
by using either fingers. The six different hand poses can be made with the secondary idle
finger before touch with the primary fingers. In the case of the different
multi-touch gestures, you just can make a limited
number of hand poses due to their anatomical limitations. This is a basic plus two hands pose, this is a bend plus two hands pose. This is a middle plus fingers bend. This is spread plus two fingers, sorry, spread plus two hand pose. So there are three different hand poses while performing multi-touch gestures with the thumb and the index fingers. Now we look at during
single-touch gestures. In this case, let me give you a video demonstrating each in-air gestures. This is tap. This is a flick gesture. It’s a bending. It’s swipe up. Swipe down. Bending plus. A flick plus. Bending all. Flick all. A user can perform nine in-air gestures while performing
single-touch touch gestures such as tap, and drag on the
screen with the index finger. Lastly, let’s look at during
the multi-touch gesture. In this case, the only secondary fingers are the middle plus fingers. So there are only two
in-air gestures available. This is flick plus two. This is a bending plus two gestures. In this case, only standard
multi-touch gesture can be performed while
performing secondary in-air gesture. This is because we found that, it is difficult for middle plus fingers to perform in the air
gestures while simultaneously performing a dynamic gesture
with a primary finger. So this figure shows all
of the touch plus finger gestures we explored in our design space. All of these gestures
are not user-friendly. So we conducted a user
study to investigate the easier of the performing the gesture in evaluation section. Now, we talk about a touch
plus finger prototype to implement the touch
plus finger gestures. Our prototype consists of three parts, a ring-like device with
an IMU sensor attached for detecting hands
pose or in-air gestures. A touch-based device,
in this case, a tablet, for getting touch input information and the external PC for data
processing and classification. Our prototype was simple,
robust, and reliable enough for initial exploring of touch plus finger gestures. This shows our classification model. The calculated IMU sensor
based and touch information are used to us for classification. We tested several basic classifiers such as decision tree and SPN And then, the decision tree classifier achieved the best results
in our pilot test. So decision tree was
applied to our prototype. With this prototype we evaluated the touch plus finger gestures. We measured two things. One thing is the recognition accuracy and the other one is the user rating of each gesture with five point scale and a semi-structured interview. For the (mumbles) we
recruited eight participants. They are asked to wear
our prototype and then perform 20 touch plus finger,
10 times for 10 seconds for steady and dynamic touch gestures with primary finger. We collected 2,080 gesture samples in total for classification. This procedure is a little
bit complicated, I think. Detailed explanation
is given in the paper. And we also asked the participants to rate it properly, how easy the gesture was to perform on a scale of one to five. The results. For gesture classification,
we conducted two validations. Overall touch plus finger gestures were classified very well, with high accuracy of over 99%. It means that the participants performed touch gestures in
an almost identical manner. Next, participants gave both positive and negative feedback to touch plus finger gestures. For before touch gestures, they were able to make all of the hand pose easily except
for L shape and the bend. This is because the participants thought that this gesture
are uncomfortable, requiring them to put in more effort to maintaining the poses. For the ring touch gestures, there are significant differences between static and dynamic
gestures with primary finger. While performing the static
gestures with primary finger, in this case a tap on the screen, participants were able to perform mostly in-air gesture without any problem. Except for bending, flick plus, flick all or flick plus two. Some participants thought that these gestures were quite stressful. On the other hand, while performing dynamic touch gestures
with primary finger, for example drag on the screen, participants report
that most in-air gesture were very difficult. This is because secondary in-air gestures draw much attention away from the primary touching interaction. So participants reported that
only a few in-air gestures such as tap and swipe up and down are easiest to perform. Now, let me show you a video demonstrating some of the examples of a prime touch plus finger technique. This is a free-form line
with a basic hand pose, and a straight line
with a stick hand pose. Also a curved line with
a spread all hand pose. So we can fluidly switch
between interaction modes in a drawing application. So here we talk about the discussion. We think that our prototype is enough for initial exploration
of a touch plus finger. But there is still room for improvement. We believe that our ring-like device can be more miniaturized so that it can be a stand-alone wearable device. Also, some participants pointed that wearing multiple rings may be impractical. We think that this could be solved in the future by exploring alternatives such as depth cameras
or capacitive sensors. And we found that there is a trade-off between discomfort of performing secondary in-air gestures and the high classification accuracy. Interestingly, we think that the fifth primary finger on the screen may result in discomfort of
performing in-air gestures. But it also restricts the possible secondary fingers movement allowing users to perform the gesture almost identically. It could help to achieve
the high accuracy. Lastly, in the picture work, it would be interesting to explore potential combination of
touch plus finger gestures on other devices such
as small screen devices, virtual headsets and laptop interfaces. So we conclude that by using idle fingers as additional (mumbles) for
primary touch interaction, we can enhance the touch interaction in the various ways we explored. Thank you for the listening and I’m happy to take your questions. (audience claps) – [Announcer] We have time
for one or two questions. – [Audience Member 1] – Hi, Dan Ashbrook, University of Copenhagen. With the two accelerometers, the two IMUS, can you ignore the angle of the hand? Or do you always have to
be in one hand position? – [Lim] Actually in our experiment, we did ignore the angle of
the (mumbles) variables. This is because we just conducted our usual studies in our
controlled environment. – [Audience Member 1] Right. – [Lim] So. – [Audience Member 1] So do you think that this would work while walking? – [Lim] Actually our work is focused on exploring the gestures and so I think that could be addressed
in the future work, I think. – [Audience Member 1] – Okay. – [Lim] Yeah, thank you. – [Audience Member 1] Thanks. – [Announcer] Any other questions? – [Audience Member 2]
So I have one question. Have you thought about instead of wearing the rings on the finger
that’s touching the screen, wear it on the other hand and whether that could improve
other types of techniques? Cause I could basically imagine touching the screen
whilst scratching my head and have that as a new
interaction technique. What do you think about that? – [Lim] I think that’s a good
question and interesting. Actually, I didn’t think about that. – [Audience Member 2] But
you think it’s possible? – [Lim] Yeah, yeah, I
think that it’s possible using multiple rings
wearing in the other hands, and for more expressive
interactions, maybe. I appreciate your
comment and in the future I will address it to you. – [Announcer] Great. Let’s thank our speaker one more time. (audience claps)

Leave a Reply

Your email address will not be published. Required fields are marked *