Feature: Striking a Collaborative Chord

Hamamatsu, a Japanese city of Music, played host to NIME 2004 in the sunny month of June this year. The New Interfaces for Musical Expression conference series started as a workshop at CHI 2001 (Seattle, USA), an origin which has shaped the focus of subsequent conferences. The series is primarily aimed at exploring the use and design of new user interfaces that are used to create music, and has even spawned University courses.

Given participants' wide range of interests and backgrounds it is very easy to be distracted by the plethora of novel interaction devices which take on a whole range of shapes and sizes from small devices that you 'strangle' to embedded systems which play music based on people's movement in a space. The key to such conferences seems to be to try to take a step back and understand how it all fits together in terms of exploring the space of musical interaction and user interfaces in general.

The conference itself was hosted at SUAC (Shizuoka University of Art and Culture), a campus which afforded many glimpses into Japanese culture such as the rolling alpine meadow on top of the main building with a range of bizarrely sized benches, some so large that one felt like a 5 year old child when seated, and a spotlessly clean campus which is something you don't tend to find in many Universities.

The conference was impeccably organised apart from a few of the usual problems associated with connecting computers to data projectors - maybe there is a conference series on such issues, and surely there must be blogs about the trials and tribulations of connecting an array of laptops to a data projector. In fact, the conference organisation even included a 'navigator' to guide you on your ten minute walk from the railway station to the conference venue.

The conference this year included two keynotes - Robert Moog (yes, he of the Moog Synthesizer fame) and Toshio Iwai. Moog's talk was half about technical developments and half about his views on how people interact with electronic musical instruments. The technical developments focussed on development of the Moog synthesizer in the 1950s (the first electronic synthesizer which has since had a lasting impact on all forms of music) and the theremin (a strange instrument developed in the 1920s which if played by moving one's hands near the instrument; watch any 1950s sci-fi movie and you're sure to hear a theremin making eerie noises).

Moog's view on how performers interact with their instruments was rather more esoteric - arguing that your mind merges with your instrument as you play it. Such a view is inspired by Rupert Sheldrake, author of books as 'Dogs that Know When Their Owners are Coming Home, and Other Unexplained Powers of Animals'.

I certainly don't feel that my mind merges with my computer when I'm writing, but maybe we as user experience designers should be aiming for such levels of intimacy.

Iwai's keynote was inspirational as he stepped back from technological considerations to discuss the relationship between music and its visual representations - exploring the nature of punched cards in simple player-piano style instruments. He demonstrated quite neatly the multiple interpretations and use of visual representations by playing a roll of paper with happy birthday encoded as holes backwards through a small player piano. This produced a melancholy tune conjured up ideas about other transformations such as folding the paper, playing it upside down, and so on, which can be achieved without any musical knowledge. Iwai also treated us to a performance of music using children's games controllers which, for me, really summed up what new forms of interaction should be about - not necessarily technologically astounding, but something which can be used to really engage with the music and the audience.

Developing new interfaces for music expression using computers allows us to explore two aspects of interaction not typified by conventional musical instruments - non-determinacy, and embodied collaboration. With conventional instruments such as a guitar, we can be pretty sure of the sound that will be produced when we play it in certain ways, whereas with a computer based instrument we could develop musical instruments which take our input as a suggestion and then play music that some algorithm feels is appropriate.

We were treated to a performance by a violinist and one of LEMUR's 'guitarbots' which consisted of four independent single-stringed slide guitar units controlled algorithmically to accompany the violinist. Similarly, Katsuhusa Ishida and colleagues presented work on algorithms to automatically correct musical melodies as they are played in order to allow novice players to improvise musical pieces. However, such 'correcting' systems raise questions about learning - if we are automatically corrected in our playing of pieces, how do we learn, and what should the boundaries of correction be.

Related to these issues, Sergi Jorda presented a paper on learning curves for new interfaces which highlights one of the interesting aspects of designing expressive interfaces - that there is a difference between efficiency of use and learnability, and these need careful consideration given the target user group.

The second aspect of using computers in the development of musical interfaces is that collaboration can be embodied into the instrument. With a conventional instrument such as a flute, a player collaborates with other players through visual and auditory cues shared within a physical space. When we develop new interfaces we have the chance to construct instruments where the collaboration takes place within the instrument itself. For example, Sidney Fels and collaborators demonstrated the Tooka - a two player musical instrument where sound is controlled by pressure in a tube between the two players' mouths, and a set of buttons for each player. The configuration is such that the single output sound is a product of the two players' inputs - the joint production takes place within the instrument whilst visual interaction helps co-ordinate the activity. The instrument itself is a bendy tube that two players blow into from each end which looks pretty strange at first sight, as was confirmed when Linda Kaastra and Sachiyo Takahashi treated us to a performance one evening.

From a more technological perspective, Ge Wang and Perry Cook gave a paper on 'on-the-fly programming' as a way of creating music in collaboration with other people. In this situation collaborators program signal generators, filters, and mixers which can be combined in real time to create music. The key is that data, or signals, can be shared between different people's code so providing a space for collaborative real time music generation by programming.

There were many, many interfaces which were highly strange, and focussed on more deterministic and individual control of music. For example, Ajar Kapur and his collaborators presented the eSitar which augments a conventional sitar with pickups to monitor the position of fingers and plucking. Such information can then be used to create sitar style sounds as well as graphical accompaniment to the music. It would be interesting to see how such instruments could be developed to change the nature of the instrument itself, say by mapping it to different sound styles or by developing a two or four player sitar. Similarly strange was the use of sound in karate training as outlined by Masami Takahata and colleagues. In their work karate students wear sensors which are tracked by computer and used to generate sound based on the acceleration of various body parts in order to provide real-time, non-verbal feedback about their moves. Maybe this could be used in driving instruction to encourage the development of a rhythmical driving style.

There were a whole host of other interfaces and devices at the NIME conference which I haven't the space to outline but which highlight the wide range of possibilities there are to supporting creativity with computers. The conference series itself has been running for three years now and to me it had the feeling of growing maturity which I hope will drive it to explore expressive interaction in a range of new, exciting, and rigorous ways.

Nick Bryan-Kinns
Queen Mary, Univesity of London

Site Visit Interviews: from Good to Great
Source: User Focus, 29 August 2008
 
For those of you for whom the Basic Introduction to User Interviews wasn't quite enough.

Six Metrics for Managing UI Design
Source: Russell Wilson, 28 August 2008
 
A proposal of six metrics to be used for managing a user interface design department.

Don't Judge a Form by its Cover
Source: Formulate Information Design, 27 August 2008
 
The saying "don't judge a book by its cover" reminds us that looks are deceptive. It turns out that this idiom applies to forms too.

Beijing Olympics - special State of the eNation report
Source: www.abilitynet.org.uk, 26 August 2008
 
In this special report AbilityNet asked disabled users to try out the Beijing Olympics website in our interaction lab.

It's Who You Know (Or Don't)
Source: Stanford Magazine, 23 August 2008
 
Online social networks are powerful and ineffectual all at once.

Winning Considerations for Interactive Content
Source: UXMatters, 22 August 2008
 
Rich options for interactively presenting content also come with a challenge.

Microsoft sees end of Windows era
Source: BBC, 20 August 2008
 
Microsoft has kicked off a research project to create software that will take over when it retires Windows.

News you can Use
Source: Gerry McGovern, 18 August 2008
 
When the homepage is dominated by news you are not necessarily communicating more.

Feeling Through your Computer
Source: Discoveries and Breakthroughs Inside Science, 16 August 2008
 
A newly designed device lets computer users feel the texture and movement of what they are seeing in front of them.

User interviews - A basic Introduction
Source: Webcredible, 15 August 2008
 
It's surprising how few people have a real understanding of who's using their site.