Saturday 23 December 2006, by jmc
All the versions of this article:
Isis Truck did research in the MODECO Group in Reims (France) and received her PhD in computer science from the University of Reims in 2002. She joined the LIASD (Laboratoire d’Intelligence Artificielle de Saint-Denis) at the University Paris 8 in 2003 as an assistant professor (senior lecturer). Her research interests are in the field of knowledge representation under uncertainty, especially by way of fuzzy logic and many-valued logic. She is also interested in music and computer science and participates to projects in this field.
Research related to the visualisation: With Alain Bonardi, we are developing a project with a virtual assistant for staging and directing. It concerns a software tool that visualize the performer’s intents and emotions from a life show. The inputs of the software are audio and video files (or streams) coming from the show. For us, intents correspond to the conscious part of emotions, i.e. emotions correspond to the way the perfomer plays, and the intents to how the performer intended to play. Of course, the intents the performer wanted to show won’t always (never?) be those being really felt by people during the show. Thus the virtual assistant captures the emotions systematically and tries to compute the similarities (or dissimilarities) with the intents (that can be called the instrutions). The visualization phase allows us to observe the scene from a visual (body movements) and sound (voice) point of view. We also offer the director an interface to render her/his instructions for each temporal block. The aim is then to compare the instructions with the intentions observed by the assistant. On this screen capture, each line shows an emotion. On each line, one function symbolizes the emotion quantity the director wants to get and the other symbolizes the computed emotion depending on time. The number to the right is the correspondence percentage between both functions (work in progress).
Concerning the problem of sound visualization, it is obvious that methods from fuzzy logic are completely appropriate to represent sound in a pertinent and intelligent way. The fuzzy subset theory that contains the fuzzy logic was imagined by Lofti Zadeh in 1965. The idea was to mathematically model specifically human concepts that are typically impossible to model through classical formulas. The recent march proposed by Zadeh itself is to model mathematically the human perception. The perception of sounds is thus included. He notably mentions the computational theory of perceptions, the protoform theory (that refers to the notion of data summaries), etc. One of his last article is available here.
More concretely, the representation of sound with various tools is a question that has been already addressed by many researchers. For instance, the use of color - where the perception is of great importance - has been used many times to represent the sound waves (see Golan Levin, Kostas Giannakis, Vincent Lesbros, etc.). Color is quite relevant and convenient for such a representation because it depends on the human perception, like the sound waves. It is thus possible to use it to translate visually the sound perception. Once again, the fuzzy logic may be of good interest in modeling both the color perception and the sound perception.
About the aforementioned notion of emotion, Lola Canamero studies the adaptive intelligent behavior, which she is exploring with artificial autonomous creatures. She is focusing on motivated behavior and social interactions, with particular emphasis on the role(s) that emotional phenomena play in them. One can read, for example:
Cañamero, L.D. 2003. Designing Emotions for Activity Selection in Autonomous Agents. In R. Trappl, P. Petta, S. Payr, eds., Emotions in Humans and Artifacts, pp. 115-148. Cambridge, MA: The MIT Press.
About the virtual assistant, other studies exist in the literature in the same field. As an example, Friberg’s team has developed a collaborative game in which two players use the voice or body gestures for input control. (see http://www.speech.kth.se/music/projects/Ghostgame/)
I. TRUCK, A. BONARDI, F. ROUSSEAUX & H. AKDAG, "Vers une distanciation anticipation - cible du jeu d’acteurs : le cas de l’Opéra Interactif Alma Sola", Journées CogniEst "Retour sur l’Anticipation", Haguenau, November 2006.
A. BONARDI, I. TRUCK & H. AKDAG "Building Fuzzy Rules in an Emotion Detector", 11th International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems (IPMU’2006), pp 540-546, Paris, July 2006.
A. BONARDI, I. TRUCK & H. AKDAG "Towards a Virtual Assistant for Performers and Stage Directors", 6th International Conference on New Interfaces for Musical Expression (NIME’06), pp 326-329 (Poster), Paris, June 2006.
A. BONARDI & I. TRUCK "First steps towards a digital assistant for performers and stage directors", 3rd International Conference on SOUND AND MUSIC COMPUTING (SMC’06), pp 91-96, Marseille, May 2006.