Keynote by Cecilia Mascolo at Percom2016: Technology and Experiecne in the Physcial World

Cecilia Mascolo presents the keynote at Percom2016. Her opening statement is: “Technology must enhance and not substitute the physical experience”.

Cecilia makes the point that continuous sensing with mobile devices can overcome many issue that are well known with traditional studies (especially the classical problem of psychologist studying psychology students in a dark lab). One of here early papers (EmotionSense, see [1]) shows how we can move studies into the real world. This is not without difficulties, especially when you try to understand emotions.

Putting research apps into android market changes the game, large numbers of users become within reach. Higher numbers of participants require a clear purpose of the applications (Nielse Henze provide in [2] a nice recipe of how to do this). Her experience is that user engagement through gamification really worked. Even if the duration of participation of individuals is limited to weeks or months this generates very useful information. A short introduction to social sensing by Cecilia can be found in [3].

Different sensors have different energy and privacy cost and also different types of contributions. Correlating the accelerometer and happiness is really interesting. Users who are more active (not just movement, “being out and about”) are happier. Clustering accelerometer data and correlating it with other high level data opens exciting questions, e.g. health. Similarly correlating happiness and location leads to more surprising results: less happy at home and work, more when out and active. Looking at people’s personally and demographics shows that gender, age, employment, etc. has a clear correlation with activity and usage of communication.

Physical space matters! Using active badges they looked at how the change of physical space can impact peoples interactions [4]. The sensing approach allowed to understand how changes in physical space changes the behavior on a really fine grained level.

References:
[1] Rachuri, K. K., Musolesi, M., Mascolo, C., Rentfrow, P. J., Longworth, C., & Aucinas, A. (2010, September). EmotionSense: a mobile phones based adaptive platform for experimental social psychology research. In Proceedings of the 12th ACM international conference on Ubiquitous computing (pp. 281-290). ACM. http://csce.uark.edu/~tingxiny/courses/5013sp14/reading/Rachuri2010EMP.pdf
[2] Henze, N., Shirazi, A. S., Schmidt, A., Pielot, M., & Michahelles, F. (2013). Empirical research through ubiquitous data collection. Computer, 46(6), 0074-76. http://doi.ieeecomputersociety.org/10.1109/MC.2013.202
[3] Mascolo, C. (2010). The power of mobile computing in a social era. IEEE Internet Computing, 14(6), 76. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.466.2037&rep=rep1&type=pdf
[4] Brown, C., Efstratiou, C., Leontiadis, I., Quercia, D., Mascolo, C., Scott, J., & Key, P. (2014, September). The architecture of innovation: Tracking face-to-face interactions with ubicomp technologies. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 811-822). ACM. http://arxiv.org/pdf/1406.6829.pdf

MobiSys 2012, Keynote by Paul Jones on Mobile Health Challenges

This year’s ACM MobiSys conference is in the Lake District in the UK. I really love this region in the UK. Already 15 years back when I studied in Manchester I often came up over the weekend to hike in the mountains here. The setting of the conference hotel is brilliant, overlooking Lake Windermere.
The opening keynote of MobiSys 2012 was presented by Dr. Paul Jones, the NHS Chief Technology Officer who talked about “Mobile Challenges in Health”. Health is very dear to people and the approach to health care around the world is very different.

The NHS is a unique intuition that is providing healthcare to everyone in the UK. It is taxation funded and with its 110 billion pounds per year budget it is one of the cheaper (and yet efficient) health care systems in the world. The UK spends about 7% of its national cross product on health care, whereas the US or Germany nearly spend double of this percentage. Beside the economic size the NHS is also one of the biggest employers in the world, similar in size to the US department of defense and the Chinese people’s army. The major difference to other larger employers is, that a most part of the staff in the NHS is highly educated (e.g. doctors) and is not easily taking orders

Paul started out with the statement: technology is critical to providing health care in the future. Doing healthcare as it is currently done will not work in the future. Carrying on will not work as the cost would not be payable by society. In general information technology in the health sector is helping to create more efficient systems. He had some examples that often very simple system help to make a difference. In one case he explained that changing a hospitals scheduling practice from paper based diaries to a computer based systems reduced waiting times massively (from several month to weeks, without additional personal). In another case laptops were provided to community nurses. This saved 6 hours per week and freed nearly an extra day of work per week as it reduced their need for travelling back to the office. Paul argued, that this is only a starting point and not the best we can do. Mobile computing has the potential to create better solutions than a laptop that are more fitting the real working environment of the users and patients. One further example he used is dealing with vital signs of a patient. Traditionally this is measured and when degrading a nurse is calling a junior doctor and they have to respond in a certain time. In reality nurses have to ask more often and doctors may be delayed. In this case they introduced a system and mobile device to page/call the doctors and document the call (instead of nurses calling the doctors). It improved the response times of doctors – and the main reason is that actions are tracked and performance is measured (and in the medical field nobody wants to be the worst).

Paul shared a set of challenges and problems with the audience – in the hope that researchers take inspiration and solve some of the problems ;-)

One major challenge is the fragmented nature of the way health care is provided. Each hospital has established processes and doctors have a way they want do certain procedures. These processes are different from each other – not a lot in many cases but different enough that the same software is not going to work. It is not each to streamline this, as doctors usually know best and many of them make a case why their solution is the only one that does the job properly. Hence general solutions are unlikely to work and solutions need to be customizable to specific needs.

Another interesting point was about records and paper. Paul argued that the amount of paper records in hospital is massive and they are less reliable and save as many think. It is common that a significant portion of the paper documentation is lost or misplaced. Here a digital solution (even if non-perfect) is most certainly better. From our own experience I agree on the observation, but I would think it is really hard to convince people about it.

The common element through the talk was, that it is key to create systems that fit the requirements. To achieve this it seems that having multidisciplinary teams that understand the user and patient needs is inevitable. Paul’s examples were based on his experience of seeing the user users and patient in context. He made firsthand the observation, that real world environments often do not permit the use of certain technologies or create sup-optimal solution. It is crucial that the needs to are understood by the people who design and implement the systems. It may be useful to go beyond the multidisciplinary team and make each developer spending one day in the environment they design for.

Some further problems he discussed are:

• How to move the data around to the places where it is needed? Patients are transferred (e.g. ambulance to ER, ER to surgeons, etc.) and hence data needs to be handed over. This handover has to work across time (from one visit to the next) and across departments and institutions
• Personal mobile devices (“bring your own device”) are a major issue. It seems easy for an individual to use them (e.g. a personal tablet to make notes) but on a system-level they create huge problems, from back-up to security. In the medical field another issue arises: the validity of data is guaranteed and hence the data gathered is not useful in the overall process.

A final and very interesting point was: if you are not seriously ill, being in a hospital is a bad idea. Paul argued, that the care you get at home or in the community is likely to be better and you are less likely to be exposed to additional risks. From this the main challenge for the MobiSys community arises: It will be crucial to provide mobile and distributed information systems that work in the context of home care and within the community.


PS: I like one of the side comments: Can we imagine doing a double blind study on a jumbo jet safety? This argument hinted, that some of the approaches to research in the medical field are not always most efficient to prove the validity of an approach.

Keynote at the Pervasive Displays Symposium: Kenton O'Hara

Kenton O’Hara, a senior researcher in the Socio-Digital-Systems group at Microsoft Research in Cambridge, presented the keynote at the pervasive displays symposium in Porto on the topic “Social context and interaction proxemics in pervasive displays“. He highlighted the importance of the spatial relationship between the users and the interactive displays and the different opportunities for interaction that are available when looking at the interaction context.

Using examples from the medical field (operating theater) he showed the issues that arise from the need of sterile interaction and hence avoiding touch interaction and moving towards a touchless interaction mode. A prototype, that uses a Microsoft Kinect sensor,  allows the surgeon to interact with information (e.g. an x-ray image) while working on the patient. It was interesting to see that gestural interaction in this context is not straightforward, as surgeons use tools (and hence have their hands not free) or gesture as a part of the communication in the team.

Another example is a public space game; there are many balls on a screen and a camera looking at the audience. Users can move the balls by body movement based on a simple edge detection video tracking mechanism and when two balls touch they form a bigger ball.  Kenten argues that “body-based interaction becomes a public spectacle” and interactions of an individum are clearly visible to others. This visibilility can lead to inhibition and may reduce the motivation of user to interact. For the success of this game the designing of the simplistic tracking algorithms is one major factor. By tracking edges/blobs the users can play together (e.g. holding hands, parents with the kids in their arm) and hence a wide range of interaction proxemics are supported. He presented some further examples of public display games on BBC large screens, also showing that the concept of interaction proxemics can be use to explain interaction .

TVs have change eating behavoir. More recent research in displays in the context of food consumptions have been in contrast mainly pragmatic (corrective, problem solving). Kenton argued that we look at the cultural values of meals and see shared eating as a social practice. Using the example of eating in front of the television (even as a family) he discusses the implications on communication and interaction (basically the communication is not happening). Looking at more recent technologies such as phones, laptops and tablets and their impact on social dynamics probably many of us realized that this is impacting many of us in our daily lives already (or who is not taking their phone to table?). It is very obvious that social relationships and culture changes with these technologies. He showed “4Photos” [1] a designed piece of technology to be put on the center of the table showing 4 photographs. Users can interact with it from all sides. It is designed in a way to stimulate rather than inhibit communication and to provide opportunities for conversation. It introduces interaction with technologies as a social gesture.

Interested in more? Kenton published a book on public displays in 2003 [2] and has a set of relevant publications in the space of the symposium.

References

[1] Martijn ten Bhömer, John Helmes, Kenton O'Hara, and Elise van den Hoven. 2010. 4Photos: a collaborative photo sharing experience. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (NordiCHI '10). ACM, New York, NY, USA, 52-61. DOI=10.1145/1868914.1868925 http://doi.acm.org/10.1145/1868914.1868925

[2] Kenton O’Hara, Mark Perry, Elizabeth Churchill, Dan Russell. Public and Situated Displays: Social and Interactional Aspects of Shared Display Technologies. Kluwer Academic, 2003

CHI2012 opening Keynote by Margaret Gould Stewart – Empowerment, Disruption, Magic

Margaret Gould Stewart, a highly regarded user experience designer currently leading UX design at YouTube, presented the opening keynote at CHI2012.  She started her talk with reminding us that humans are story tellers – they always have been and probably always will. What is not constant is the medium – as technologies change so do means for storytelling and sharing.

The topic started out with talking about video connects the world. It extended to a larger view – changing the world through experience design (in the context of video). I often wonder what designers are and she added another quite interesting explanation: designers are humanist. By putting up the definition for humanism she made her point clear that this could apply to good people in design, essentially it is down to caring for humans in their works.

To show the power of video in connecting people she used the following example: the film “Life in a Day” and as it said in the credits “a movie filmed by you”. I have not seen it yet, but the trailer made me curious to look at this one (see the film on YouTube).

By asking the question: what are the things that make sites like YouTube have impact? she introduced 3 principles. Sites have to be:

• Empowering
• Disruptive
• Magical

She outlined what these 3 principles mean for user experience design.

For empowering she had very strong examples: how photo sharing, video sharing, and social networks changed what we see of natural disaster and the effect on people. It also changed way we see it and how we can respond to it. The concrete example was the information coverage on the Hurricane Katrina 2005 (pre-video-sharing age) and the recent flood in Asia. Empowering = helping people to share their stories.

Disruption is in this context the change in use of media and especially how it changes how we perceive the ubiquitous technology of TV. The capabilities of video sharing platforms has, are very different than those of TV – at the same time it is disrupting TV massively. She had a further example of how such technology can disrupt: The Khan Academy (basically sharing educational videos) is challenging the education system. As a further step she had an example where a teacher encourages students to make their own instructional videos as means for them to learn. Disruption = finding new ways that are challenging / overthrowing the old approach.

Magic is what makes technology exciting. There is a quote by Arthur C. Clarke “Any sufficiently advanced technology is indistinguishable from magic”. The term “magic” has a long tradition in human computer interaction. Alan Kay talked about it with regard to graphical user interfaces. We had some years back a paper  a paper on Magic beyond the screen [1]. In the talk Margaret Gould Stewart used as another example Instagram, as software that provides magical capabilities for the person using it. Another example of magic she discussed is the GPS based “moving dot” on a map that makes navigation in mobile maps easy. Even without navigational skills people can “magically” find their way. Her advice is “do not get in the way of magic” – focus on the experience not technology in the back ground. In short she summarized:  “Magic disrupts the notion of reality”.


She combined the principles in one example in the design of YouTube. She discussed the page design using an analogy to a plate.  A great plate makes all food presented on it look more attractive and the design goal of the YouTube page is to be such a plate for video. It should make look all videos look better.

Another example used to highlight how to empower, disrupt, and create magic is the http://www.thejohnnycashproject.com/. Each participant can manipulate one frame of the video (within given limits) and the outcome of the whole video is amazing. Cannot be described, you have to watch it.

Related to the example above an interesting question comes up: How much control is required and what type of control is applied. Here one example is twitter, which limits how much you can write but it does not limit what you post (limiting the form but not the content). She made an interesting argument about control. If you believe that democracy works and is good you can assume that people in general will make the right decisions. One further indicator is, that positive things go viral much more often than negative things. One of the takeaway messages is to believe in people an empower them.

To sum up, there are three questions to be asked when designing an experience:

• How to empower people?
• How to disrupt
• How to create magic?

A final and important point is that there are things that cannot be explained and she argued that we should value this.

[1]  Albrecht Schmidt, Dagmar Kern, Sara Streng, and Paul Holleis. 2008. Magic Beyond the Screen. IEEE MultiMedia 15, 4 (October 2008), 8-13. DOI=10.1109/MMUL.2008.93 http://dx.doi.org/10.1109/MMUL.2008.93

Keynote at Percom 2012: Andy Hopper from Cambridge on Computing for the Future of the Planet

In his Keynote "Computing for the Future of the Planet" Andy Hopper brought up 4 topics and touched shortly on each of them: (1) Optimal digital infrastructure – green computing, (2) Sense and optimize – computing for green, (3) Predict and react – assured computing, and (4) Digital alternatives to physical activities.

In the beginning of his talk he discussed an interesting (and after he said it very obvious) option of Green Computing: move computing towards the energy source as it is easier to transmit data than to transmit power. Thinking about this I could imagine that Google’s server farms are move to a sunny dessert and then the calculations are done while the sun is shining… and using the cold of night to cool down… This could be extended to storage: storing data is easier than storing energy – this should open some opportunities.

As a sample of an embedded sensing systems Andy Hopper presented a shoe with built-in force sensing (FSR) that allows to measure contact time and this helps to work out speed. There initial research was targeted towards athletes, see Rob Harle's page for details. It is however easy to imagine the potential this has if regular shoes can sense movement in everyday use. He hinted to think about the options if one could go to doctor and analyze the change in walking pattern over the last year.

In various examples Andy showed how Ubisense is used in commercial applications, production, and training. It seems that medium resolution tracking (e.g. below 1 meter accuracy) can be reliably achieved with such an off the shelf systems, even in harsh environments. He mentioned that the university installations of the system at an early product stage were helpful to improve the product and grow the company. This is interesting advices, and could be a strategy for other pervasive computing products, too. For close observers of the slides there were some interesting inside in the different production methods between BMW and Austin Martin and the required quality ;-)

Power usage is a central topic in his labs work and he showed several examples of how to monitor power usage in different scenarios. On example is monitoring power usage on the phone, implemented as an App that looks at how power is consumed and how re-charging is done. This data is then collected and shared – at current over 8000 people are participating. For more details see Daniel T. Wagner’ page. A further example is the global personal energy meter. He envisions that infrastructure, e.g. trains and building, are broadcasting information about the use of energy and that they provide information about one individuals share of this.

With an increasing proliferation of mobile phones the users’ privacy becomes a major issue. He showed in his talk an example, where privacy is provided by faking data. In this approach fake data, e.g. for calendar events, location data, and address book, is provided to apps on the phone. By these means you can alter what an application sees (e.g. location accuracy).

For more details and papers see the website of the digital technology group: http://www.cl.cam.ac.uk/research/dtg/www/

Opening talk at the Social Media for Insurances Symposium

I was invited to Leipzig to talk about social networks and in the context of insurance companies (http://www.versicherungsforen.net/social-media). The main focus of the talk was to show what people currently do in social networks and to speculate why they do it (and  I used a picture of the seven deadly sins as an illustrations…) Additionally I discussed some prototypes of activity recognition and their potential once integrated into social media.

My talk was entitled “500 Freunde (auf Facebook): Wozu noch eine Versicherung?“ – „500 friends (on Facebook) – Is there still need for insurance?“ and discussed how ubiquitous capture and social media may shape the next community [1]. The slides in are in German.

The event was very interesting and I would expect that there is a great potential out there for insurance companies to tap into. Looking back at the original idea of insurance (e.g. old fire insurance communities) or sharing the risk of hail in farming communities can give interesting inspiration for peer-2-peer insurance models. It will be exciting to see if there a new products and services that come out of the “big players” or if new players will come to the game. To me the central issue to address is how to make insurance products more visible – and I think a user centered design approach could be very interesting…

In the future I would expect that finding the right value mix (privacy, price, safety, etc.) will be essential as we argued for other services in [2]. Some years back we wrote in an article about RFID [3] “privacy is sacred but cheap” and the more services we see the more I am convinced that this is more than a slogan. If you can create a service that is of immediate value to the user I would I expect that privacy will be a lesser concern to most? On the other hand if you reduce privacy without any value in exchange there is always an outcry…

[1] “500 Freunde (auf Facebook): Wozu noch eine Versicherung?“ - Ermöglichen allgegenwärtige Aufzeichnungs-technologien und digitale soziale Netze die nächste Solidargemeinschaft? Slides as PDF (in German)
[2] Albrecht Schmidt, Marc Langheinrich, Kristian Kersting, "Perception beyond the Here and Now," Computer, vol. 44, no. 2, pp. 86-88, Feb. 2011, doi:10.1109/MC.2011.54 (final version at IEEE, free draft version)
[3] Schmidt, A.; Spiekermann, S.; Gershman, A.; Michahelles, F., "Real-World Challenges of Pervasive Computing", Pervasive Computing, IEEE , vol.5, no.3pp. 91- 93, c3, July-Sept. 2006. 10.1109/MPRV.2006.57

Bryan Reimer: Opening keynote at Auto-UI 2011 in Salzburg

Bryan started his keynote talk the automotive user interface conference (auto-ui.org) in Salzburg with reminding us that having controversial discussions about the HMI in the car is not new. Quoting a newspaper article from the 1930s on the introduction of the radio in the car and its impact on the driver he picked an interesting example, that can be seen as the root of many issues we have now with infotainment systems in the car.

The central question he raised is: how to create user interface that fit human users? He made an important point: humans are not “designed” to drive at high speed in complex environments; perception has evolved for walking and running in natural environment. Additionally to the basic limitations of human cognition, there is a great variety of capabilities of drivers, their skills and cognitive ability (e.g. influence of age). A implication of the global change is demographics is that the average capabilities of a drivers will be reduced – basically as many older people will be drivers…

Over the last 100 years cars have changes significantly! Looking more closely Bryan argues that much of the chance happened in the last 10 years. There has been little change from the 1950s to the 1990s with regard to the car user interface.

It is apparent that secondary tasks are becoming more important to the user. Users will interact more while driving because the can. It is however not obvious that they are capable of it.

Even given these developments it is apparent that driving has become safer. Passive safety has been improved massively and this made driving much safer. There seems to be a drawback to this as well, as people may take greater risks as they feel safer. The next step is really to avoid accidence in the first place. Bryan argues that the interaction between driver, environment, and vehicles is very important in that. He suggests that we should make more of an effort to create systems that fit the drivers.

The Yerkes-Dodson Law helps to understand how to design systems that keep peoples attention in the optimal performance. He made an important point: there are certain issues that cannot be solved, e.g. if someone is tired we can do only very little – the driver will need to rest. We should make sure that we take these things into account when designing systems.

Visual distraction is an obvious factor and much discussed in the papers at the conference – but Bryan argued that “eyes on the road” is not equal to “mind on the road”. I think this is really a very important point. Ensuring that people keep their eyes on the road, seeing things is not enough. The big resulting question is how to keep or get people focused on the street and environment. It seems there is some more research to do…

The variety of interfaces and interaction metaphors build into cars opens more choices but at the same time creates problems, as people need to learn and understand them. A simple question such as: How do you switch the car off? may be hard to answer (Bryan had the example of a car with a push button starter, where you cannot remove the key). I think there are simple questions that can be learned from industry and production machines… add an emergency stop button and make it mandatory ;-)

If you are interested more about Bryan’s work look at his webpage or his page at the MIT agelab or one of his recent publications [1] in the IEEE Pervasive Computing Magazine’s special issue on automotive computing, see [2] for an introduction to the special issue.

Sorry for the poor quality photos ... back row and an iPhone...

[1] Joseph F. Coughlin, Bryan Reimer, and Bruce Mehler. 2011. Monitoring, Managing, and Motivating Driver Safety and Well-Being. IEEE Pervasive Computing 10, 3 (July 2011), 14-21. DOI=10.1109/MPRV.2011.54 http://dx.doi.org/10.1109/MPRV.2011.54

[2] Albrecht Schmidt, Joseph Paradiso, and Brian Noble. 2011. Automotive Pervasive Computing. IEEE Pervasive Computing 10, 3 (July 2011), 12-13. DOI=10.1109/MPRV.2011.45 http://dx.doi.org/10.1109/MPRV.2011.45

Closing Keynote at AMI2011, Beyond Ubicomp – Computing is Changing the Way we Live

On Friday afternoon I had the privilege to present the closing keynote at AMI2011 in Amsterdam with the title ‘Beyond Ubicomp – Computing is Changing the Way we Live’. The conference featured research in Ambient Intelligence ranging from networking and system architecture to interfaces and ethnography. It brought an interesting set of people together and it was good to see many students and young researchers presenting their work.

In my closing keynote at talked about my experience of the last 13 years in this field and about a vision of the future. My vision is based on a basic technology assessment - basically looking what technologies will (in my view) definitely come over the next 20 years and looking at the implications of this. I stared out with a short reference to Mark Weiser's now 20 year old article [1]. The upcoming issue of IEEE Pervasive Magazine will have a in-depth analysis of the last 20 years since Weiser' article - we have also an article in there on how interaction evolved.

The vision part of the talk looked "Perception beyond there here and now" [2] from 3 different angles:

• Paradigm Shift in Communication
  Here I argue that the default communication in the future will be public communication and only if something is secret we will try to use non public channel. First indicators of this are a switch from email to twitter and facebook. I used a cake baking example to highlight the positive points of this shift.
• Steep Increase in media capture
  The second angle is just observing and extrapolating the increase in capture of media information. If you go already now on youtube you will information about many things (backing a cake, repairing a bike, etc.). The implication of this increase in media capture will be virtually unlimited access to experience other people share
• Transformation of experienced perception
  The final angle is that this creates a new way of perceiving the world. We will extent perception beyond the here and now and this is bringing a completely new way of creating and accessing information. I used the example of enquiring about buying an international train ticket at the station in Amsterdam. If you can look there through other people’s eyes the question becomes trivial.

My overall argument is that we are in for a major transformation of our knowledge and information culture. I would expect that this shift is as radical as the shift from an oral tradition to the written societies - but the transition will be much quicker and in the context of a globalized and competitive world.

The main conclusion from this is: Ethics and values are the central design material of this century.

Looking at twitter it seems it got across to some in the audience ;-) If your are interested, too have a look at the slides from the keynote.

[1] Mark Weiser. The computer for the 21st century. Scientific American, Vol. 265, No. 3. (1991)
[2] Albrecht Schmidt, Marc Langheinrich, and Kritian Kersting. 2011. Perception beyond the Here and Now. Computer 44, 2 (February 2011), 86-88. DOI=10.1109/MC.2011.54 http://dx.doi.org/10.1109/MC.2011.54

ESSPRITS 2011 workshop, keynote

I was invited to present a keynote at the joint German-Canadian workshop on embedded systems, signal processing, and IT security (ESSPRITS 2011). The workshop is interdisciplinary and technically focused on embedded systems.

My talk was entitled “Challenges in a World of Ubiquitous Computing” and it looked at how the world changed over the last 20 years as ubiquitous computing has become reality. Computing technologies have become an integral part of our life and they shape more and more how we perceive the world and how we interact with each other. The talk highlights some technology trends that enable a new generation of computing systems. It presents a new vision suggesting new forms of perception and interaction without temporal and spatial boundaries.

During the lectures it was great to get a feel of what research is happening on a signal processing level. With regard to security I got the feeling that we need in the future a closer cooperation between people doing cryptography and those doing user interfaces. It seems it is currently still at the level that HCI people understand little about the algorithms, methods and concerns in IT-security and that people in security optimize for security that can be proved, but may not fit humans (and may not really increase security in the real world).

Keynote at EICS 2011

 I was invited to present a keynote at EICS 2011 in Pisa. In the talk "Engineering Interactive Ubiquitous Computing Systems" I motivated why user interface engineering approaches are well suited for creating user interfaces in the context of embedded and ubiquitous computing systems. Looking at desktop applications and mobile devices I think the quality and ease of use is high - compared to 20 years back or compared to embedded and ubiquitous computing systems. I think a lot of user interface research, and in particular engineering approaches for interactive systems, could have a great impact on real world systems beyond the desktop or phone.


As one example of an engineering process for embedded user interfaces I shared our experience with developing Gazemarks [1]. Gazemarks is a technology based on eye-gaze tracking that reduced the time required for attention switching. It eases tasks that require the user to move attention repeatedly between 2 or more displays or between the real world and a set of digital displays. Application domains could be looking at the street and at the satnav while driving or switching attention between a screen in an operating theatre and the patient.

When investigating development from an embedded user interface to interactive ubiquitous computing systems further issues come up. As we investigate the PDnet project [2] with public displays we see that the concerns of the stakeholders play a much bigger role than in traditional systems and that finding an appropriate business model is very close to the user interface development process. 

In the final part of the talk I shared a future vision of how technology may change the way we live. In the not so distant future we could imagine that the traditional boundaries of perception (mainly temporal and spatial) will fall [3]. This would create an entirely new experience where "Perception beyond the here a now" change fundamentally the way we see and experience the world. The slides  of the keynote are available in as PDF.

From the research we did over the last 15 years I picked some lessons learned:

• Novelty may be about the values/ethics
• Implement it and try it out!
• 20% who like the UI/system are a large market
• Humans are smart and adaptive
• Design for creative users

[1] Dagmar Kern, Paul Marshall, and Albrecht Schmidt. 2010. Gazemarks: gaze-based visual placeholders to ease attention switching. In Proceedings of the 28th international conference on Human factors in computing systems (CHI '10). ACM, New York, NY, USA, 2093-2102. DOI=10.1145/1753326.1753646 http://doi.acm.org/10.1145/1753326.1753646

[2] Pdnet project homepage: http://pd-net.org/

[3] Albrecht Schmidt, Marc Langheinrich, and Kritian Kersting. 2011. Perception beyond the Here and Now. Computer 44, 2 (February 2011), 86-88. DOI=10.1109/MC.2011.54 http://dx.doi.org/10.1109/MC.2011.54

CHI 2011 in Vancouver, Keynote and Papers

In the opening keynote Howard Rheingold proclaimed that we are in a time for learners and he outlined the possibilities that arise from the interactive media that is available to us. In particular he highlighted the fact that people share and link content and to him this is at the heart of learning. Learning as a joined process where contributions by students - in different forms of media - become major a resource was one example.

I best liked his analogy on how little innovation there is in teaching. "If you take a warrior from 1000 years ago on a battlefield today - they will die - quickly. If you take a surgeon from a 1000 years ago and put them in a modern hospital - they will be lost. If you take a professor from 1000 years ago and put them in a University today he will exactly know what to do. " I am not sure about the 1000 years but it by 100 years the story works just as well. In essence he argued that there is a lot of potential for new approaches for teaching and learning.

After initially agreeing I gave it some more thoughts and perhaps the little change in learning and teaching shows that learning is very fundamental and technology is overrated in this domain? What is more effective than a teachers discussing in an exciting topic face to face with a small set of students - perhaps even while on a walk? Reminds me about things I read about the Greek teachers and there practices several thousand years ago … and it makes me looking forward to our summer school in the Italian Alps (http://www.ferienakademie.de/).

I found the SIGCHI Lifetime Achievement Award lectures very exciting and educational. Especially the talk by Larry Tesler provided deep insight into how innovation works in user interfaces - beyond the academic environment. He talked about the "invention" of cut and paste - very enjoyable!

This year we had a number of papers describing our research in CHI:

•  Elba reported on the field study in Panama using mobile phones to enhance teaching and learning [1]
• Ali presented work on how to increase the connectedness between people by simple means of iconic communication in the context of a sports game [2]
• Tanja showed how touch and gestural input on a steering wheel can reduce the visual distraction for a driver [3], and
• Gilbert (from LMU Munich) presented work on interaction with cylindrical screens [4].

The most inspiring and at the same time the most controversial paper for me was the possessed hand by Jun Rekimoto et al. [5]. He reported their results in using electro stimulation in order to move fingers of a hand.

Bill Buxton showed throughout the conference his collection of input and output devices (Buxton Collection). Seeing the collection physically is really exciting, but for all who did not have the chance there is a comprehensive online version with photos and details available at micosoft research: http://research.microsoft.com/en-us/um/people/bibuxton/buxtoncollection/

[1] Elba del Carmen Valderrama Bahamondez, Christian Winkler, and Albrecht Schmidt. 2011. Utilizing multimedia capabilities of mobile phones to support teaching in schools in rural panama. In Proceedings of the 2011 annual conference on Human factors in computing systems (CHI '11). ACM, New York, NY, USA, 935-944. DOI=10.1145/1978942.1979081 http://doi.acm.org/10.1145/1978942.1979081

[2] Alireza Sahami Shirazi, Michael Rohs, Robert Schleicher, Sven Kratz, Alexander Müller, and Albrecht Schmidt. 2011. Real-time nonverbal opinion sharing through mobile phones during sports events. In Proceedings of the 2011 annual conference on Human factors in computing systems (CHI '11). ACM, New York, NY, USA, 307-310. DOI=10.1145/1978942.1978985 http://doi.acm.org/10.1145/1978942.1978985

[3] Tanja Döring, Dagmar Kern, Paul Marshall, Max Pfeiffer, Johannes Schöning, Volker Gruhn, and Albrecht Schmidt. 2011. Gestural interaction on the steering wheel: reducing the visual demand. In Proceedings of the 2011 annual conference on Human factors in computing systems (CHI '11). ACM, New York, NY, USA, 483-492. DOI=10.1145/1978942.1979010 http://doi.acm.org/10.1145/1978942.1979010

[4] Gilbert Beyer, Florian Alt, Jörg Müller, Albrecht Schmidt, Karsten Isakovic, Stefan Klose, Manuel Schiewe, and Ivo Haulsen. 2011. Audience behavior around large interactive cylindrical screens. In Proceedings of the 2011 annual conference on Human factors in computing systems (CHI '11). ACM, New York, NY, USA, 1021-1030. DOI=10.1145/1978942.1979095 http://doi.acm.org/10.1145/1978942.1979095

[5] Emi Tamaki, Takashi Miyaki, and Jun Rekimoto. 2011. PossessedHand: techniques for controlling human hands using electrical muscles stimuli. In Proceedings of the 2011 annual conference on Human factors in computing systems (CHI '11). ACM, New York, NY, USA, 543-552. DOI=10.1145/1978942.1979018 http://doi.acm.org/10.1145/1978942.1979018

Percom 2011 in Seattle, keynote

This year's Percom conference was held in Seattle and offered an exciting and diverse program. Have a look the program to see it for yourself. The two keynotes were both looking at the implications of pervasive computing and communication - especially when thinking about the data is collected and how the data may be used.

Alex Pentland from MIT talked about their work on reality mining. The work looks at how one can capture interactions between people and between people and their environment and how such information can be exploited. One example he gave was on looking at the effect of face to face communication on the performance on workers. The basics insights of this work are thrilling and thinking it through it becomes obvious that we are at the start of new era of mankind. The arguments he made that we can contain and control such information I did not find convincing and I think it may be dangerous to tell decision makers in politics that we can provide solutions. I see no way (that is not restricting people's freedom massively or which reduces productivity massively) that would allow to control the information that will become available through pervasive computing… and all the solutions I have heard either will plainly not work or would require a global agreement over data protection laws…

The keynote on the second day was by Derek McAuley from Nottingham University. One of his topics was on product history and how the availability of product history has the potential to increase the value of products. I think this is a very powerful concept and we will in the near future see this commercially exploited.
Furthermore Dereck discussed interesting issues that come up with crowd sourcing and participatory sensing. One central point is where the data is hold and who controls the data collected. Especially in the context of cloud services this becomes transparent and important at the same time. With regard to the implementation is does not matter; however from a legal perspective it may make a serious difference whether you cloud service runs in German, the US, or on a ship somewhere in the Atlantic. An example he gave are navigation systems in cars which have a back channel. The cars sent back information about their speed and whereabouts and the information is used to predict the state of the road, which is then used to improve the navigation. He raised the questions what happens if this information is held somewhere were legislation has no control? I think this is going to happen and there is no real approach against it…
He made a case that end-users (individuals) should be able to bring together information about them and make use of it. On principle I like this idea to put the individual into control and allow them to exploit this data. For me this is however not a solution for data protection, as a certain part of individuals will sell their data - and in a free country there is probably very little society can do against it.

In summary - we are heading towards an exciting future!

PS: Percom 2012 will be in Lugano with Silvia and Marc chairing the conference. And I have the honor to serve as program chair. See the web page for more information (will be available soon) or the photo of the call for papers here.

TEI2011 Keynote: Bounce Back by Gilian Crampton Smith

Gilian Crampton Smith is one of the people that massively influenced the field of interaction design from the very beginning. She is currently working in Venice: www.interaction-venice.com. Nearly 20 years back in London Durrell Bishop was one of her students. She pointed out that the central insight of the early work was "We know how to control objects by physical means". This insight has since become a basic motivation for tangible interfaces and embodied interaction. Closely linked to this is the complexity of interaction. To me one great point with designing tangible user interfaces is that you are forced to come up with a simple solution. Overall the argument was that embodied interaction works because it draws on knowledge we have. An example is that two physical things cannot be in exactly the same place and another one is that things stay where they are if there is no force moving them. There are clear limitations to the interaction with physical objects that give indications how to use it; she referenced a paper on an exploration of physical manipulation [1].

As one example Gilian showed the marble answering machine concept video from 1992. To me this video is a great example that shows the power of a concept video. Using a simple animation in this concept video enables viewers to understand the idea and interactions very quickly. I like the simplicity and clarity of this video - it would be very useful for teaching. Hopefully it will made publicly available; see below the bad-quality copy I took with my mobile. A good quality version ist at http://vimeo.com/19930744

One interesting point was a reflection on the shortcomings of the traditional view of interaction being based on input and output. Gilian argued that output should be separated in feedback and results. Input is categorized in according the interaction required, such as simple (e.g. text, minimal movement), medium (e.g. GUI), maximum (e.g. musical instrument, movement of the whole body). The feedback is related to the sense human have, and includes modalities such as visual, auditory, tangible, kinesthetic, proprioception. Results are related to feedback but are oriented on the outcome. Results may be visual (e.g. symbolic, words, icon, films); auditory (e.g. sounds, words or music) or physical such as touch (e.g. massage machine) or movement.

Gilian went on to the question: "Why aren't tangible, embedded and embodied interfaces out there in our everyday world?" For me the most important answer she gave is that it is hard to do them. It is very difficult to created sensor based interaction in a way that users understand it and that it feels natural to use. Counter examples where people got it wrong are those many lights that go on when you are already half way up the stairs or the automatic doors where people wave their hands in order to open them. As much of the interaction technology in such systems is not visible people have a hard time figuring out how things work. Based on their (incomplete) experience base on a few interactions they will create their mental modal … which is often wrong and potentially leads to frustration. A further point she made is that creating physical interactive objects and things is difficult and involves a lot of different skills. In combination with the function it is difficult to get the aesthetics right. One very good example is how hard it is to the aesthetics of feel is right, as it includes a well balanced design taking into account touch, weight, balance, movement, and sound. Overall this requires a great passion for detail to get the quality of engineering right; Gilians examples for this was the AUDI advert where an Engineer opens and closes the car door and listens carefully. I think I have seen in 8 years back in the UK as part of the ad-campain "Vorsprung durch Technik". If anyone has a link to a copy please post it.

When I was in 2002 (or around this time) visiting her at Ivrea near Torino I was very much impressed by the creativity and inventiveness. In her talk she explained one approach to teaching I really like. The students get a device (e.g. an alarm clock, answering machine) without manual with the task to figure out how it works. While exploring the functionality they film themselves. Based on this they reflect on the design and then go on to do a completely new design for a device with a similar functionality (with the constraint that it cannot have buttons).

I like a reference to something Bruce Steering wrote. In short it basically says research is like crime; you need Means, Motivation and Opportunity.

Some further example she showed:

• A communication device based candle: ceramic liaison. It is a bi-directional 1-bit communication device, that is esthetically pleasing. The state of the candle (real wax with fire) on either side is reflected by lighting up some ceramics on the other side.
• A full body experience interaction device for controlling games: Collawobble. Using two bouncing balls a packman game can be controlled. One user controls with the bouncing X and the other Y.

update:
Ellen Do posted 2 clips that were shown in the keynote

• http://vimeo.com/11802060
  L'età del fuoco - The age of fire
• www.youtube.com/watch?v=LE9t98Gox60
  Jaques Tati - Mon Oncle (1958) - Kitchen Scene - The struggle of a simple man with a "highly modern" kitchen and a bouncing jug

and suggested a further one: http://www.youtube.com/watch?v=oS6pwQqSY70

the marble answering machine video on http://vimeo.com/19930744

[1] Andrew Manches, Claire O'Malley, and Steve Benford. 2009. Physical manipulation: evaluating the potential for tangible designs. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction (TEI '09). ACM, New York, NY, USA, 77-84. DOI=10.1145/1517664.1517688 http://doi.acm.org/10.1145/1517664.1517688

Keynote: Steve Benford talking on "Designing Trajectories Through Entertainment Experiences"

On Tuesday morning Steve Benford presented the entertainment interfaces keynote. He is interested in how to use computer technology to support performances. Steve works a lot with artist group, where the University is involved in implementing, running and studying the experiences. The studies are typically done by means of ethnography. The goal of this research is to uncover the basic mechanisms that make these performances work and potentially transfer the findings to human computer interaction in more general.

I particularly liked the example of "Day of the figurines". Steve showed the video of experiences they created and discussed the observations and findings in detail. He related this work to the notion of trajectories [1], [2]. He made the point that historic trajectory are especially well suited to support spectators.

Some years back I worked with Steve in the Equator and we even have a jointed publication [3] :-) When looking for these references I came across another interesting paper - related to thrill and excitement, which he discussed in the final part of the talk [4].

PS: we had a great party on Monday night but the attendance was extremly good :-)

[1] Benford, S. and Giannachi, G. 2008. Temporal trajectories in shared interactive narratives. In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI '08. ACM, New York, NY, 73-82. DOI= http://doi.acm.org/10.1145/1357054.1357067

[2] Benford, S., Giannachi, G., Koleva, B., and Rodden, T. 2009. From interaction to trajectories: designing coherent journeys through user experiences. In Proceedings of the 27th international Conference on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI '09. ACM, New York, NY, 709-718. DOI= http://doi.acm.org/10.1145/1518701.1518812

[3] Benford, S., Schnädelbach, H., Koleva, B., Anastasi, R., Greenhalgh, C., Rodden, T., Green, J., Ghali, A., Pridmore, T., Gaver, B., Boucher, A., Walker, B., Pennington, S., Schmidt, A., Gellersen, H., and Steed, A. 2005. Expected, sensed, and desired: A framework for designing sensing-based interaction. ACM Trans. Comput.-Hum. Interact. 12, 1 (Mar. 2005), 3-30. DOI= http://doi.acm.org/10.1145/1057237.1057239

[4] Schnädelbach, H., Rennick Egglestone, S., Reeves, S., Benford, S., Walker, B., and Wright, M. 2008. Performing thrill: designing telemetry systems and spectator interfaces for amusement rides. In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI '08. ACM, New York, NY, 1167-1176. DOI= http://doi.acm.org/10.1145/1357054.1357238

Opening Keynote of Mensch&Computer 2010 by Ed H. Chi

Ed H. Chi from PARC presented the opening keynote for Mensch&Computer 2010. In the motivation of the talk he showed a document on "Applied Information processing psychology" from 1971 - probably very few had seen this before. It makes an argument for an experimental science that is related to augmented cognition. The basic idea is very similar to Vannevar Bush's Memex - to extend the human cognitive power by machines (and especially computer technology). It is apparent that these ideas became the backdrop of the many innovations that happened at PARC in the early days.

Ed stressed that there is still a lot of potential for the application of psychological phenomena and models to human computer interaction research. As an example he used the idea that speech output in a navigation system could use your name in an important situation making use of the attenuation theory of attention (the cocktail party effect). By hearing your name you are more likely to listen - even if you are yourself in a conversation. The effect may be stronger if the voice is your mother's voice ;-)

The main part of the talk centered on model driven research in HCI. Using the ScentHighlights [1] examples he outlined the process. I liked very much the broad view Ed has on models and the various uses of models he suggested, e.g. generative models that generate ideas; or behavioral models that lead to additional functionalities (as example he used: people are sharing search results in google, hence sharing should be a basic function in a search tool). Taking the example of Wikipedia he showed how models can be used to predict interaction and growth. I found the question on the growth of knowledge very exciting. I think it is defiantly not finite ;-) otherwise research is a bad career choice. Looking at the Wikipedia example it is easy to imagine that the carrying capacity is a linear function and hence one could use a predictive function where a logistic growth curve is overlayed with a linear function.

Random link from the talk: http://mrtaggy.com/

Ed discussed yahoo's social pattern library:
http://developer.yahoo.com/ypatterns/social/people/reputation/
This pattern library is pretty interesting. I found the reputation pattern pretty comprehensive. It seems that this library is now comprehensive enough for using it for real and in teaching.

[1] Chi, E. H., Hong, L., Gumbrecht, M., and Card, S. K. 2005. ScentHighlights: highlighting conceptually-related sentences during reading. In Proceedings of the 10th international Conference on intelligent User interfaces (San Diego, California, USA, January 10 - 13, 2005). IUI '05. ACM, New York, NY, 272-274. DOI= http://doi.acm.org/10.1145/1040830.1040895