User:Ke CHEN/Proposed/Affective computing
Affective Computing is computing that relates to, arises from, or deliberately influences emotion and other affective phenomena. Research in Affective Computing contributes to Artificial Intelligence, Pattern Recognition, Machine Learning, Human-Computer Interaction, Social Robotics, Autonomous Agents, Cognitive and Affective Sciences, Affective Neuroscience, Neuroeconomics, Health-behavior Change, and many other areas where technology is used to detect, recognize, measure, model, simulate, communicate, elicit, handle, or otherwise understand and influence emotion. Here is one illustration of its possible future use:
Imagine your robot entering the kitchen as you prepare breakfast for guests. The robot looks happy to see you and greets you with a cheery "Good morning." You mumble something it does not understand. It notices your face, vocal tone, smoke above the stove, and your slamming of a pot into the sink, and infers that you do not appear to be having a good morning. Immediately, it adjusts its internal state to "subdued," which has the effect of lowering its vocal pitch and amplitude settings, eliminating cheery behavioral displays, and suppressing unnecessary conversation. Suppose you exclaim unprintable curses that are out of character for you, yank your hand from the hot stove, rush to run your fingers under cold water, and mutter "... ruined the sauce." While the robot's speech recognition may not have high confidence that it accurately recognized all of your words, its assessment of your affect and actions indicates a high probability that you are upset and possibly hurt. At this moment it might turn its head with a look of concern, search its empathetic phrases and select, "Burn-Ouch-Are you OK?" and wait to see if you are, before selecting the semantically closest helpful response, "Shall I list sauce recipes for you?" As it goes about its work helping you, it watches for signs of your affective state changing - positive or negative. It looks for things it does that are associated with improvements in the positive nature of your state, as well as things that might frustrate or annoy you. As it finds these, it updates its internal learning system, indicating which of its behaviors you prefer, so it becomes more effective in how it works with you.
In the scenario above, the robot shows the ability to interpret emotion of a person from both direct outward observation as well as from its reasoning about the situation. It is also capable of adjusting its response both subtly, such as suppressing conversation and subduing its tone of voice, and through overt actions, such as choosing and displaying an empathetic expression. Additionally, affective information is used by the robot to build better models of how to behave successfully in social-emotional situations, and affect-like mechanisms operating within the robot can adjust its own machine learning algorithms, biasing what is and is not remembered or learned, as well as updating default biases used for making decisions and choosing actions.
While it is easiest to see how to apply affective computing to anthropomorphic technologies such as software agents or robots, it is also the case that people tend to interact in affective and social ways with many non-anthropomorphic technologies. Thus, affective computing is not limited to technologies with faces, voices, or other human-like characteristics, nor is it is always focused on improving the artificial intelligence or emotional intelligence of technology. For example, affective computing can be used in the design process to improve non-computational products by helping recognize customer experiences that cause stress, frustration, or other undesirable states, e.g., designing new product labels that elicit fewer brow furrows or frowns, or designing automobile interfaces that improve driver safety. Affective technology can also detect positive customer experiences, and help companies get a better sense of what brings these desirable results about. There are also opportunities to use affective technology to assist people with special needs, such as people on the autism spectrum who sometimes face extraordinary challenges recognizing, predicting, and responding to emotional information. Methods for modeling emotions developed in affective computing also contribute to our understanding of these complex phenomena, and affective computing thus serves to support basic research in emotion. In many cases, the goal of affective technology is to provide systematic measures or tools that help people gain understanding about emotion and better achieve its communication or expression, not to make technology more emotionally intelligent.
Research scope and challenges
Research in Affective Computing can be organized into four areas, although these are not mutually exclusive: (1) technology for displaying human or computer affective information or mediating the expression or communication of emotion, e.g., modulate pitch, font, word choice, physical movements, or graphics to indicate an affective quality; (2) technology for sensing, recognizing, modeling and predicting emotional and affective states, e.g., he looks and sounds angry now and if I say this, it might make him angrier; (3) methods for computers to respond intelligently and respectfully to handle perceived affective information, e.g., the strategy of acting subdued around a person who is upset; and (4) computational mechanisms that simulate internal emotions or implement their regulatory and biasing functions, e.g., change the memory search strategy and phrase retrieval strategy based on current internal affective state.
Research in Affective Computing combines engineering and computer science with psychology, cognitive science, neuroscience, sociology, linguistics, education, medicine, psychophysiology, value-centered design, ethics, and more, in order to enable advances in basic understanding of affect and its role in biological agents, and across a broad range of human experience. For example, the dominant psychology theories in the affective sciences do not currently include a state labeled “subdued;” however, a study comparing a computer using subdued and enthused speech with drivers showed that choosing subdued speech led to significantly better driver performance and safety when drivers were previously exposed to upsetting stimuli. In fact there are several affective states that arise commonly in human-technology interaction and are generally missing from affect theories, e.g. frustration and boredom. With the construction of interactive systems that can record and process affective aspects of interaction through use of technology, researchers in Affective Computing can monitor complex dynamic states of activity that happen naturally in everyday experience, including state duration and frequency, psychophysical, behavioral, and social-communicative characteristics. Thus, affective technology provides information that can contribute to improved theories in these other sciences, to more fully explain and predict measurable human experience. Revised theories can then be implemented in technology and tested under natural usage, and this process iterated until theory achieves descriptive and predictive accuracy.
Affective Computing faces many decades of challenges before researchers might succeed in building comprehensive computational models of emotion; nonetheless, there are already useful spin-offs of its application in the commercial world. For example, over 400 Million US dollars were spent in 2006 on call center software that automatically detects if customers sound upset so that those calls can be flagged to learn how to handle them better. While the affective pattern analysis techniques do not detect states such as “upset” perfectly, they can still be useful in helping reduce the huge number of cases an employee would have to attend to in seeking to provide appropriate customer service.
Affective Computing researchers often situate their studies in real contexts where emotion is combined with other factors. For example, an intelligent tutoring system might consist of a computer avatar interacting with a learner who is making lots of mistakes and starts smiling at the tutor. In such a case, the computer should be able to discern that those smiles probably do not express happiness, even if the learner’s orbicularis oculi and zygomaticus muscles are contracted, as seen in the case where children smiled more after failure than after success. Rather, the smiles may occur because of social factors, which are known to be present during human-computer interaction, even though the person knows that the agent is just a piece of software. The smart computer tutor looks jointly at affective expressions such as facial cues together with non-affective information such as performance, personality, social context, and past history, in order to decide what the student is likely to be feeling and what pedagogical move to make next. Some systems might recognize complex cognitive-affective states by analyzing joint head and facial movements and tracking how they change over time such as when a look of interest morphs into one of concentration, then into confusion, and perhaps from there into frustration or anger and other mixed feelings. One of the challenges in Affective Computing research is how to deal robustly with naturally-occurring affective information, which is usually not in either a pure or static form.
Displaying, communicating, or mediating expression of affect
Technology can easily give the appearance of having emotion without having the components that traditionally accompany emotion (Fig.<ref>F3</ref>). For years, Apple computers have displayed a smile when booting successfully, even though the computer has no accompanying feelings of happiness. Artists can masterfully craft robotic dogs, animated characters, and other technologies to look as if they have emotions. However, the hardest challenge in real-time interaction is figuring out when to display which affect. Without understanding social display rules and other important cues about the interaction context, technology is quite likely to irritate people with its emotional displays. For example, Microsoft Window’s operating system plays a triumphant tune when it boots up, and this is suitable when all is well. However, if a person has just experienced the misfortune of having to reboot because of a system crash and the first response they hear from the computer is triumphant music, then this is annoying. In contrast, people often smile socially after a mistake, so the Apple computer’s smile after the misfortune of a reboot is socially acceptable.
The difficulty of accurately communicating emotion through text-based online interaction has led to the development of emoticons and other means for people to add affective intent. Increasingly, artists and interaction designers augment chat, instant messaging, and other technologies with new means of communicating emotion, mapping emotion to colors, shapes, lay-out, dynamic fonts, and more.
With affect communicating technology, people who rely on text-to-speech devices could be given the choice to have some of their affective parameters (say typing pressure, heart rate, skin conductance, or some combination of these) automatically modulate their synthetic speech output. For example, physiological arousal might be used to modulate pitch or loudness with a single on/off switch controlled by the typist, as opposed to having to annotate each word and phrase directly, which could be arduous. Sometimes affective technologies can be used to help people who are non-speaking and non-typing to express or reveal their feelings, even when they are unable to use words to describe them. The technology can sense physiological or other parameters and map these to an output that the non-speaking person can use to communicate states such as "I'm very calm" or "I'm overloaded". The ability to use technology to communicate even some simple affective state changes can help reduce misunderstandings that might otherwise arise when a person with limited language tries to communicate with others.
Note that there are some people who prefer non-affective interaction with computers and who do not wish for computers to communicate social-emotional signals during interaction. Affective computing technology, unlike most people, can honor this preference by simply switching off this type of communication. Moreover, it can offer not only an on/off option, but also several in-between options. For example: a person might interact with another person through the computer, choosing on their end to not have emotional communication, but choosing on the output end for the computer to make them appear friendly and social, e.g. greeting with a smile and occasionally nodding in synchrony with what the other person says. Or they might choose to have all the emotions they truly display on their end to be visibly muted on the other end (e.g. during a video-conference negotiation or poker game.) The ability to communicate affect, mediated by technology, increases human communication possibilities.
Sensing, recognizing, modeling, and predicting affective state
Emotion researchers have traditionally used questionnaires, human observation, and physiological sensing to gather data for assessing emotional state. Affective Computing expands these options, enabling new kinds of real-time, automatic, mobile, and sometimes less obtrusive measurements, giving technology the ability to read affective cues from complex patterns that include tone of voice, language, facial expressions, posture, gestures, autonomic nervous system measures, and whatever combinations of modalities that people are comfortable with having sensed.
Advances in affective technologies allow for more natural sensing and communication of emotion outside the laboratory. Tools of pattern analysis and machine learning are typically used to discover possibly nonlinear combinations of the sensor data that enable recognition of complex dynamic affective states. These techniques can also construct statistical models to aid not only in recognizing the current state, but also in predicting the next state or states, much like speech models can be used to predict likelihoods of certain words following others.
Advances in artificial intelligence, common-sense reasoning about emotion, and user models are also important in building up scripts and other modeling mechanisms that help people and machines predict what emotion is likely to occur in a situation. For example if you give a person a present that they do not like, they will probably smile and say "thank you" in order to be polite; however, if they really like the present, then they may show even more enthusiasm and delight, perhaps repeatedly saying thanks, smiling, talking about the present, and even continuing to smile afterward. Computers need to be taught such typical social customs, which may vary with context and culture. They need to be taught the likely antecedents to emotion, and given the ability to test in their own environments when the antecedents are present and whether and when the commonly expected emotional response occurs. Networked computers can also be enabled to combine their observations, constructing empirical models of what emotional displays and outcomes really do arise in interaction during daily life.
This area of research -- sensing and possibly recording or transmitting people's emotion -- raises many questions concerning an individual’s privacy and the use and possible misuse of digital affective information. Great care must be taken to respect people’s wishes about what is and is not sensed, clearly communicating whether collected information could be associated with their identity and what benefits and harms they may receive from sharing this information. These concerns are especially important because in some cases sensors can obtain affective information without people being aware that sensors are even being used, for example when thermal imaging is used by remote cameras for deceit detection. The concerns and opportunities for abuse escalate when the technology is made high-cost and given to people with money, power, and influence, as opposed to when it is made low-cost and distributed first to individuals for their individual exploration and control. Affective computing was originally conceived to make technology more respectful of people’s feelings; consequently, it is in keeping with the origins of the field that designers of affective technologies are strongly urged to first respect the feelings of people using the technology. Such respect should be shown in every stage of the design, development, and evaluation processes. Researchers are urged to not fall prey to the common scientific tendency to make something “just because it can be done” but rather to work together, accepting criticism and seeking public input, to steadily discern what should be done in allowing technology to improve human experience.
Responding intelligently and respectfully to perceived emotions
When a person reveals affective information to a recipient, the latter can choose ways to respond that may be helpful or harmful. For example, if a person lets a computer know that its action is very frustrating then the computer could recognize its gaffe and take steps to avoid it in the future. It could also issue an acknowledgement of the frustration it has caused, and perhaps even apologize, in order to help alleviate the person’s immediate frustration. Sometimes it might be appropriate for a computer to display an empathetic or caring response. While it may sound absurd for a computer to express feelings when it does not have them, it is possible for a computer to empathize and appear caring without pretending to feel anything a person feels (Fig.<ref>F2</ref>). Studies suggest that computer-provided empathy can reduce frustration and stress and can impact perceptions of caring, which could help in health-care technologies, among others.
The idea of having a computer show empathy grew out of findings that people interact with computers similarly to how they interact with other people; consequently, the theory of human-human interaction can be applied to make predictions about what will work in human-computer interaction. For example, if we know that Bob does not like it when Alice sings triumphant songs after Bob experiences misfortune, then we can predict that Bob won’t like it when a computer plays triumphant songs after Bob experiences misfortune. While one cannot expect people to treat computers exactly the same as other people, many general principles about human-human interaction carry over to the case of human-computer interaction. These principles are often used in affective computing research to shape the design of affective communication.
Simulating emotions or implementing their regulatory and biasing functions
The most controversial area of Affective Computing arises when researchers talk about giving machines emotions, because the topic quickly moves to whether machines can be conscious and have feelings in any sense resembling the human experience. To date, there is no evidence that machines can have conscious experiences like those that people have, and yet there is no compelling proof that they cannot ever be given such an ability. Meanwhile, there is ample evidence that computer scientists can implement computational functions that imitate some aspects of biological emotions.
Emotion-like mechanisms inside a machine can perform functions that may or may not appear emotional to an outside observer. An emotion model within the Hasbro/iRobot toy doll My Real Baby evaluates inputs and causes the doll’s facial expressions and vocalizations to change, making the doll appear to have emotions. Another example is where a cognitive appraisal model within a software agent is used to trigger an emotional verbalization. In other cases, Affective Computing researchers implement less visible functions of emotion, such as ways in which affective states can improve machine decision-making, attention regulation, learning, and more. Affective computing often tries to implement findings from affective science and affective neuroscience, and such implementations can act to evaluate current hypotheses as well as to provoke and articulate new ones.
References
Breazeal, C. and Picard, R. (2006) In Neuroergonomics: The Brain at Work (Eds, Parasuraman, R. and Rizzo, M.) Oxford University Press, Oxford.
Burleson, W., Picard, R. W., Perlin, K. and Lippincott, J. (2004) In Workshop on Empathetic Agents, International Conference on Automonous Agents and Multiagent Systems Columbia University, New York, NY.
D'Mello, S. K., Craig, S. D., Sullins, J. and Graesser, A. C. (2006) International Journal of Artificial Intelligence in Education, 16, 3-28.
Gadanho, S. C. (2003) Journal of Machine Learning Research, 4, 385-412.
Healey, J. and Picard, R. W. (2005) IEEE Trans. on Intelligent Transportation Systems, 6, 156-166.
el Kaliouby, R., Picard, R. W. and Baron-Cohen, S. (2006) Progress in Convergence (Eds, Bainbridge, W. S. and Roco, M. C.) Annals of the New York Academy of Sciences 1093: 228-248, doi:10.1196/annals.1382.016.
el Kaliouby, R. and Robinson, P. (2005) In Real-Time Vision for Human-Computer Interaction, Springer-Verlag, pp. 181-200.
Klein, J., Moon, Y. and Picard, R. W. (2002) Interacting with Computers, 14, 119-140.
Marsella, S., Gratch, J. and Rickel, J. (2004) In Life-like Characters Tools, Affective Functions and Applications(Eds, Prendinger, H. and Ishizuka, M.) Springer, New York, pp. 46.
Nass, C., Jonsson, I.-M., Harris, H., Reaves, B., Endo, J., Brave, S. and Takayama, L. (2004) In CHI ACM, Portland, Oregon.
Ortony, A., Clore, G. L., and Collins, A. (1988) The Cognitive Structure of Emotions, Cambridge University Press, Cambridge, England.
Pantic, M. and Rothkrantz, L. J. M. (2003) Proc. of the IEEE, 91, 1370-1390.
Picard, R. W. (1997) Affective Computing, MIT Press, Cambridge, MA.
Picard, R. W., Vyzas, E. and Healey, J. (2001) IEEE Transactions Pattern Analysis and Machine Intelligence, 23.
Prendinger, H., Mori, J. and Ishizuka, M. (2005) Int'l J of Human-Computer Studies, 62, 231-245.
Reeves, B. and Nass, C. (1996) The Media Equation, Cambridge University Press, New York.
Rosis, F. d., Novielli, N., Carofiglio, V., Cavalluzzi, A. and Carolis, B. D. (2006) International Journal of Biomedical Informatics, 39, 514-531.
Schneider, K. and Josephs, I. (1991) J. Nonverbal Behavior, 15, 185-98.
Trappl, R., Petta, P. and Payr, S. (Eds.) (2002) Emotions in Humans and Artifacts, MIT Press, Cambridge.
Tsiamyrtzis, P., Dowdall, J., Shastri, D., Pavlidis, I., Frank, M. G. and Ekman, P. (2006) International Journal of Computer Vision, 71, 197-214.
UPI (2006) United Press International.
Recommended reading
Carberry, S., F. de Rosis (2007) Special Issue on Affective Modeling and Adaptation, User Modeling and User-Adapted Interaction: The Journal of Personalization Research.
Douglas-Cowie, E., R. Cowie, et al. (2003). Special Issue on Speech and Emotion. Speech Communication. 40.
<review> Fellous,J-M. & Arbib, M. (2005). Who Needs Emotions? NY: Oxford University Press. </review>
Isbister, K. and K. Höök (2007) Special Issue on Evaluating Affective Interactions, International Journal of Human-Computer Studies.
Paiva, R. Prada, and R. W. Picard (Eds.), Affective Computing and Intelligent Interaction 2007, Lecture Notes in Computer Science 4738, 2007. Springer-Verlag, Berlin Heidelberg 2007
Tao, J., T. Tan, and R. W. Picard (Eds.), Affective Computing and Intelligent Interaction 2005, Lecture Notes in Computer Science 3784, 2005. Springer-Verlag, Berlin Heidelberg 2005.
See also
HUMAINE portal of affective computing related research: http://emotion-research.net/
