hci – Page 2 – In Traction

January 20, 2008

SmartKom

At Ubicomp 2007, there was a book stand by Springer just outside the conference room. On the last day, the volunteer behind the stand told me that I could choose one of the books that were still lying there. I didn’t see anything interesting at first. Since a few people at our institute are working on multimodal systems, I picked the book SmartKom: Foundations of Multimodal Dialogue Systems.

During the holidays, I read the first part of the book and noticed the book was relevant for me after all. SmartKom was a large four-year project about multimodal dialogue systems. They developed a system that provides symmetric multimodality in a mixed-initiative dialogue system with an embodied conversational agent. There is also a follow-up project that should ends in 2007: SmartWeb. SmartWeb goes beyond SmartKom in supporting open-domain question answering using the entire (Semantic) Web as its knowledge base.

Symmetric multimodality means that every input mode (e.g. speech, gesture, facial expression) is also available for output, and vice versa. Multimodal interaction is one way to make interaction between humans and computers more intuitive. Human dialogue is not only based on speech but also on nonverbal communication such as gesture, gaze, facial expression, and body posture. One of the major characteristics of human-human interaction is the coordinated use of different modalities (e.g. allowing all modalities to refer to or depend upon each other). Symmetric multimodality combined with a mixed-initiative conversational agent results in more intuitive interaction. The SmartKom systems reduces recognition errors by modality fusion. By considering multiple input modalities together (e.g. speech, facial expression and gesture), the system can more correctly estimate the user’s intention.

SmartKom has been used in several application scenarios: in public telephone booths, home entertainment systems, mobile systems and in a car environment. The last part of the book discusses techniques to evaluate multimodal dialogue systems, which should be an interesting read.

January 14, 2008

Reality-Based Interaction

Kris pointed me to an interesting CHI 2008 paper: Reality-Based Interaction: A Framework for Post-WIMP Interfaces by R.J.K. Jacob, A. Girouard, L.M. Hirshfield, M.S. Horn, O. Shaer, E.S. Treacy, and J. Zigelbaum.

Abstract:

We are in the midst of an explosion of emerging human-computer interaction techniques that redefine our understanding of both computers and interaction. We propose the notion of Reality-Based Interaction (RBI) as a unifying concept that ties together a large subset of these emerging interaction styles. Based on this concept of RBI we provide a framework that can be used to understand, compare, and relate current paths of recent HCI research as well as to analyze specific interaction designs. We believe that viewing interaction through the lens of RBI offers both explanatory and generative power. It provides insights for design, uncovers gaps or opportunities for future research, and leads to the development of improved evaluation techniques.

The paper discusses amongst others the results of a CHI 2006 workshop on the next generation of HCI. The authors provide a framework for classifying, comparing and evaluating new interaction styles. The framework concentrates on four themes used in these emerging interaction styles:

Naïve Physics: people have common sense knowledge about the physical world.
Body Awareness & Skills: people have an awareness of their own physical bodies and possess skills for controlling and coordinating their bodies.
Environment Awareness & Skills: people have a sense of their surroundings and possess skills for negotiating, manipulating, and navigating within their environment.
Social Awareness & Skills: people are generally aware of others in their environment and have skills for interacting with them.

These four themes are clarified by the accompanying picture:

The workshop proceedings should be interesting as well, with an impressive list of participants (amongst others Hiroshi Ishii, Ben Shneiderman, Steven Feiner, George Fitzmaurice, Desney Tan, Brygg Ullmer and Andy Wilson).

This framework can be useful to evaluate the “intuitiveness” of new interaction methods by measuring the extent to which they use knowledge and skills from the real world.

January 13, 2008

Johnny Lee interviewed by Hacked Gadgets

I just read a (short) interview with Johnny Lee by Hacked Gadgets. I covered a few of Johnny’s Wii projects in my blog before (the finger tracking and interactive whiteboard hacks). His latest project uses the Wiimote to perform head tracking.

Apparantely, even when Johnny is procrastinating, he is doing interesting work

JL: I guess I just spend a lot of time on my hobbies that I really enjoy doing and it turns out that my hobbies end up being productive. Even the Wii remote work started as a way to procrastinate working on my thesis.

December 12, 2007

Missed a talk by Nicolas Nova in Brussels

I found out a bit too late that Nicolas Nova would be giving a talk at iMAL in Brussels yesterday. Luckily he always puts his slides online

The talk also explained his (seemingly random) blog title: “Pasta&Vinegar”. He states that the hybridization of digital and physical environments is explored both by academic researchers (pasta) and artists and designers (vinegar). In the talk at iMAL he talked about why vinegar is important for pasta

His slides contain lots of interesting and creative ideas, such as blogjects, augmenting animals (e.g. a dog with sensors that controls a WoW character) and a tooth implant that vibrates when you have an incoming call.

If you want to invent something that is to be used 10 years from now, who can you observe? Nicolas states that looking at new media, art and design can give us clues. He also explains that art and design can better convey desire of people for the future, and shows a typical diagram from an IT company that is not appealing to people and too much focused on the technology in the background. He finally refers to the use of technology in art. SIGGRAPH’s Emerging Technologies and Art Gallery are good examples of this and of combining pasta and vinegar.

December 10, 2007

Low-cost multi-touch surfaces using a Wiimote and IR light pens

Via Hack a day:

Johnny Lee’s back again with his Wiimote interactive whiteboard. Commercial versions of these things are expensive and heavy. His technique doesn’t even need a projector, just a computer, a Wiimote and a simple IR emitting pen. The pen is just a stylus with an infrared LED in the tip.

Johnny Lee is back again indeed I posted about his method to track your fingers using a Wiimote earlier. This time he uses a the Wiimote’s infrared camera to track light pens (pens that emit an infrared light at the tip) on a surface to create an interactive whiteboard. It’s really nice that he can use any surface. You could use a projector in combination with an ordinary projection screen, a wall or a desk. If you don’t have a projector, you could turn any LCD display into a tablet surface.

Since the Wiimote can track up to four different points, these surfaces are also multi-touch. This means you can have multi-touch interaction on any projected image. It would be interesting to combine this with a steerable projector system.

[youtube:http://www.youtube.com/watch?v=5s5EvhHy7eQ]

The source code is available. I will definitely keep an eye on his Wii projects page.