Eclipse Community Forums: Technology Project and PMC » Gesture / Voice recognition research?

Help

Home

Home » Eclipse Projects » Technology Project and PMC » Gesture / Voice recognition research?

Show: Today's Messages :: Show Polls :: Message Navigator

Gesture / Voice recognition research? [message #67369]

Mon, 20 June 2005 21:48

Erkki Lindpere

Messages: 90
Registered: July 2009

Member

Hello!

Are there any Eclipse-related research projects in gesture / voice
recognition? I know about the Voice Tools Project, but I'm thinking of
gesture and voice recognition *in* the IDE.

I am doing most of my Java developement work in IDEA (hoping to move to
Eclipse as WTP matures) and there's a really simple but nice Mouse
Gesture plugin available for it. It lets you bind orthogonal mouse
gesture sequences (Left, Right, Left, Up, Down etc.) to IDEA's menu
items. I find this somewhat useful and have bound some of the most
common tasks to gestures, so I can avoid remembering keystrokes or
navigating menus. I find it easier to draw the letter C for Commit, U
for Update, backwards C for rollback etc. than to remember the shortcut
keys. And somewhy I feel like it gives me this sense of power -- can't
really explain.

The IDEA plugin is based on a LGPL Mouse Gesture library (search for
Java Mouse Gestures to find it) that is very simple and easy to use. I
tried porting it from Swing to SWT and got the basic gesture recognition
working in Eclipse, but I'm not progressing very fast towards a usable
plugin, being a beginner in Java and not having touched SWT before. Is
anyone else trying to do something similar?

An extension to this idea would be hand gesture recognition, with help
from a webcam. You could move your hand a bit up from the keyboard and
make a gesture without having to reach for the mouse. There seem to be
some open source computer vision projects that already have code that
can do this, maybe they could be leveraged for an eclipse plugin?
One I found is HandVu, that makes use of Intel's OpenCV project:
http://www.cs.ucsb.edu/~matz/HGI/HandVu.html

Another related interesting area of interaction would be programming by
voice, but I don't have the slightest clue about what advances have been
made there.

I am not really knowledgeable in this field, I've only had a casual
interest in it so far, but it seems to me that about now would be a good
time to experiment with future interaction technologies in various
environments. I want to be using an interface similar to the one in the
movie "Minority Report" by 2010! :)

With best regards,
Erkki "Villane" Lindpere

Report message to a moderator

Re: Gesture / Voice recognition research? [message #67452 is a reply to message #67369]

Mon, 04 July 2005 00:10

Eugene Kuleshov

Messages: 504
Registered: July 2009

Senior Member

Erkki,

I have mouse gesture recognition framework for SWT. Unfortunately it
isn't easy to integrate gestures into Eclipse platform without special
support in core.

regards,
Eugene

Erkki Lindpere wrote:
> Hello!
>
> Are there any Eclipse-related research projects in gesture / voice
> recognition? I know about the Voice Tools Project, but I'm thinking of
> gesture and voice recognition *in* the IDE.
>
> I am doing most of my Java developement work in IDEA (hoping to move to
> Eclipse as WTP matures) and there's a really simple but nice Mouse
> Gesture plugin available for it. It lets you bind orthogonal mouse
> gesture sequences (Left, Right, Left, Up, Down etc.) to IDEA's menu
> items. I find this somewhat useful and have bound some of the most
> common tasks to gestures, so I can avoid remembering keystrokes or
> navigating menus. I find it easier to draw the letter C for Commit, U
> for Update, backwards C for rollback etc. than to remember the shortcut
> keys. And somewhy I feel like it gives me this sense of power -- can't
> really explain.
>
> The IDEA plugin is based on a LGPL Mouse Gesture library (search for
> Java Mouse Gestures to find it) that is very simple and easy to use. I
> tried porting it from Swing to SWT and got the basic gesture recognition
> working in Eclipse, but I'm not progressing very fast towards a usable
> plugin, being a beginner in Java and not having touched SWT before. Is
> anyone else trying to do something similar?
>
> An extension to this idea would be hand gesture recognition, with help
> from a webcam. You could move your hand a bit up from the keyboard and
> make a gesture without having to reach for the mouse. There seem to be
> some open source computer vision projects that already have code that
> can do this, maybe they could be leveraged for an eclipse plugin?
> One I found is HandVu, that makes use of Intel's OpenCV project:
> http://www.cs.ucsb.edu/~matz/HGI/HandVu.html
>
> Another related interesting area of interaction would be programming by
> voice, but I don't have the slightest clue about what advances have been
> made there.
>
> I am not really knowledgeable in this field, I've only had a casual
> interest in it so far, but it seems to me that about now would be a good
> time to experiment with future interaction technologies in various
> environments. I want to be using an interface similar to the one in the
> movie "Minority Report" by 2010! :)
>
> With best regards,
> Erkki "Villane" Lindpere

Report message to a moderator

Mouse gestures [message #67473 is a reply to message #67452]

Tue, 05 July 2005 15:34

Erkki Lindpere

Messages: 90
Registered: July 2009

Member

Eugene Kuleshov wrote:
> I have mouse gesture recognition framework for SWT. Unfortunately it
> isn't easy to integrate gestures into Eclipse platform without special
> support in core.

Is the difficulty related to the case below?

I found (assuming mouse gestures should be started by for example right
clicking anywhere, as in the IDEA plugin) is that if I register a mouse
button listener for the Shell, it will only work if I start the gesture
by clicking in the very edge of the window (and a few other places),
because if another component (that I click over) handles the event it
won't reach the shell. The mouse move event seems to work fine in the
whole window, but it has to start from the window's edge.

I haven't looked at how IDEA does this. Maybe event handling is
different enough in Swing?

The only solution I came up with would be to somehow register listeners
for every component, which sounds insane, so I haven't even tried.

Erkki

Report message to a moderator

Re: Mouse gestures [message #67495 is a reply to message #67473]

Sun, 10 July 2005 22:35

Eugene Kuleshov

Messages: 504
Registered: July 2009

Senior Member

Erkki Lindpere wrote:
> Eugene Kuleshov wrote:
>
>> I have mouse gesture recognition framework for SWT. Unfortunately it
>> isn't easy to integrate gestures into Eclipse platform without special
>> support in core.
>
> Is the difficulty related to the case below?
>
> I found (assuming mouse gestures should be started by for example right
> clicking anywhere, as in the IDEA plugin) is that if I register a mouse
> button listener for the Shell, it will only work if I start the gesture
> by clicking in the very edge of the window (and a few other places),
> because if another component (that I click over) handles the event it
> won't reach the shell. The mouse move event seems to work fine in the
> whole window, but it has to start from the window's edge.

IDEA UI is AWT-based (well, Swing built on top of AWT) and you can
register listener for a global even queue. SWT does not have such
luxury. The closest you can do is to register an even filter, but the
problem is that you can't register it early enough without hacking into
platform.

> I haven't looked at how IDEA does this. Maybe event handling is
> different enough in Swing?
>
> The only solution I came up with would be to somehow register listeners
> for every component, which sounds insane, so I haven't even tried.

I've managed to make it working in editors and views but not in edges
and menu areas.
Another issue is that I wasn't able to find how to get to the list of
all registered actions in order to assign gesture mappings.

regards,
Eugene

Report message to a moderator