View source for User:Ke CHEN/Proposed/Affective computing

<b>Affective Computing</b> is computing that relates to, arises from, or deliberately influences emotion and other affective phenomena.  Research in Affective Computing contributes to Artificial Intelligence, Pattern Recognition, Machine Learning, Human-Computer Interaction, Social Robotics, Autonomous Agents, <review> Cognitive and Affective Sciences, </review> Affective Neuroscience, Neuroeconomics, Health-behavior Change, and many other areas where technology is used to detect, recognize, measure,  <review> model,  simulate, </review>communicate, elicit, handle, or otherwise understand and influence emotion.  Here is one illustration of its possible future use: 

<i>  
Imagine your robot entering the kitchen as you prepare breakfast for guests.  The robot looks happy to see you and greets you with a cheery "Good morning."  You mumble something it does not understand. It notices your face, vocal tone, smoke above the stove, and your slamming of a pot into the sink, and infers that you do not appear to be having a good morning.  Immediately, it adjusts its internal state to "subdued," which has the effect of lowering its vocal pitch and amplitude settings, eliminating cheery behavioral displays, and suppressing unnecessary conversation. Suppose you exclaim, "Ow!<!--@#$% not appropriate for an encyclopedia-->!" yanking your hand from the hot stove, rushing to run your fingers under cold water, adding "I can't believe I ruined the sauce." While the robot's speech recognition may not have high confidence that it accurately recognized all of your words, its assessment of your affect and actions indicates a high probability that you are upset and maybe hurt.  At this moment it might turn its head with a look of concern, search its empathetic phrases and select, "Burn-Ouch-Are you OK?" and wait to see if you are, before selecting the semantically closest helpful response, "Shall I list sauce recipes for you?"  As it goes about its work helping you, it watches for signs of your affective state changing - positive or negative.  It looks for things it does that are associated with improvements in the positive nature of your state, as well as things that might frustrate or annoy you.  As it finds these, it updates its internal learning system indicating which of its behaviors you prefer, so it becomes more effective in how it works with you.  
</i>

In the scenario above, the robot shows the ability to interpret emotion of a person from both direct outward observation as well as from its reasoning about the situation. It is also capable of adjusting its response both subtly, such as suppressing conversation and subduing its tone of voice, and through overt actions, such as choosing and displaying an empathetic expression.  Additionally, affective information is used internally to adjust its machine learning algorithms, biasing its future choices.  

While it is easiest to see how to apply affective computing to anthropomorphic technologies such as software agents  or robots, it is also the case that people tend to interact in affective and social ways with many non-anthropomorphic technologies.  Thus, affective computing is not limited to technologies with faces, voices, or other human-like characteristics, nor is it is always focused on improving the artificial intelligence or emotional intelligence of technology.   For example, affective computing can be used in the design process to improve non-computational products by helping recognize experiences that cause stress, frustration, or other undesirable states, e.g., designing new product labels that elicit fewer brow furrows or frowns, or designing automobile interfaces that improve driver safety.  There are also opportunities to use affective technology to assist people with special needs, such as people on the autism spectrum who can face extraordinary challenges recognizing, predicting, and responding to emotional information.  <review>Methods for modeling emotions developed in affective computing also contribute to our understanding of these complex phenomena, and affective computing thus serves to support basic research in emotion.  </review> In many cases, the goal of affective technology is to provide systematic measures or tools that help people gain understanding about emotion and better achieve its communication or expression, not to make technology more emotionally intelligent.

[[Image:research_areas.jpg|thumb|400px|right|F1|Overview of several affective computing research areas]]
 
==Research scope and challenges==

Research in Affective Computing can be organized into four areas, although these are not mutually exclusive:  (1) technology for displaying human or computer affective information or mediating the expression or communication of emotion, e.g., modulate pitch, font, word choice, physical movements, or graphics to indicate an affective quality; (2) technology for sensing, recognizing, modeling and predicting emotional and affective states, e.g., he looks and sounds angry now and if I say this, it might make him angrier; (3) methods for computers to respond intelligently and respectfully to handle perceived affective information, e.g., the strategy of acting subdued around a person who is upset;  and (4) computational mechanisms that simulate internal emotions or implement their regulatory and biasing functions, e.g., change the memory search strategy and phrase retrieval strategy based on current affective state.  
 
Research in Affective Computing combines engineering and computer science with psychology, cognitive science, neuroscience, sociology, linguistics, education, medicine, psychophysiology, value-centered design, ethics, and more, in order to enable advances in basic understanding of affect and its role in <review> biological agents, and across </review> a broad range of human experience.  For example, the dominant psychology theories in the affective sciences do not currently include a state labeled “subdued;” however, a study comparing a computer using subdued and enthused speech with drivers showed that choosing subdued speech led to significantly better driver performance and safety when drivers were previously exposed to upsetting stimuli. In fact there are several affective states that arise commonly in human-technology interaction and are generally missing from affect theories, e.g. frustration and boredom.  Through the construction of interactive systems that can record and process affective aspects of interaction through use of technology, researchers in Affective Computing can monitor complex dynamic states of activity that happen naturally in everyday experience, including state duration and frequency, psychophysical, behavioral, and social-communicative characteristics.   Thus, affective technology provides information that can contribute to improved theories in these other sciences, to more fully explain and predict measurable human experience.  Revised theories can then be implemented in technology and tested under natural usage, and this process iterated until theory achieves descriptive and predictive accuracy. 
 
Affective Computing faces many decades of challenges before researchers might succeed in building comprehensive computational models of emotion; nonetheless, there are already useful spin-offs of its application in the commercial world.  For example, over 400 Million US dollars were spent in 2006 on call center software that automatically detects if customers sound upset so that those calls can receive special attention <review>attention is misleadingly linked to a scholarpedia article on cognition and attention, while here the word attention is used in a broader sense. Therefore, the link should be removed</review>.  While the affective pattern analysis techniques do not detect states such as “upset” perfectly, they can still be useful in helping reduce the huge number of cases a person would have to attend to in order to provide the same level of service.
 
Affective Computing researchers often situate their studies in real contexts where emotion is combined with other factors.  For example, an intelligent tutoring system might consist of a computer avatar interacting with a learner who is making lots of mistakes and starts smiling at the tutor.  In such a case, the computer should be able to discern that those smiles probably do not express happiness, even if the learner’s orbicularis oculi and zygomaticus muscles are contracted, as seen in the case where children smiled more after failure than after success.  Rather, the smiles may occur because of social factors, which are known to be present during human-computer interaction, even though the person knows that the agent is just a piece of software.  The smart computer tutor looks jointly at affective expressions such as facial cues together with non-affective information such as performance, personality, social context, and past history, in order to decide what the student is likely to be feeling and what pedagogical move to make next. Some systems might recognize complex cognitive-affective states by analyzing joint head and facial movements and tracking how they change over time such as when a look of interest morphs into one of concentration, then into confusion, and perhaps from there into frustration or anger and other mixed feelings.  One of the challenges in Affective Computing research is how to deal robustly with naturally-occurring affective information, which is usually not in either a pure or static form.

==Displaying, communicating, or mediating expression of affect==

[[Image:casey.jpg|thumb|500px|right|F3|It is easy to give machines the appearance of having emotion, but it is hard to give them the ability to know what emotion to express when.  The software agent shown here, created by Ken Perlin and John Lippincott, has a variety of continuously changing expressive parameters that, when hooked to sensors of human non-verbal communication, enable the agent to adjust its expression to each person (Burleson et al., 2004).]]

Technology can easily give the appearance of having emotion without having the components that traditionally accompany emotion (Fig.<ref>F3</ref>). For years, Apple computers have displayed a smile when booting successfully, even though the computer has no accompanying feelings of happiness.  Artists can masterfully craft robotic dogs, animated characters, and other technologies to look like they have emotions; however, the hardest challenge in real-time interaction is figuring out when to display which affect.  Without understanding social display rules and other important cues about the interaction context, technology is quite likely to irritate people with its emotional displays.  For example, Microsoft Window’s operating system plays a triumphant tune when it boots up, and this is suitable when all is well.  However, if a person has just experienced the misfortune of having to reboot because of a system crash and the first response they hear from the computer is triumphant music, then this is annoying.  In contrast, people often smile socially after a mistake, so the Apple computer’s smile after the misfortune of a reboot is socially acceptable.  

The difficulty of accurately communicating emotion through text-based online interaction has led to the development of emoticons and other means for people to add affective intent. Increasingly, artists and interaction designers augment chat, instant messaging, and other technologies with new means of communicating emotion, mapping emotion to colors, shapes, lay-out, dynamic fonts, and more.  Sometimes affective technologies can be used to help people who are nonverbal to express or reveal their feelings, even when they are unable to use words to describe them.
These are some of many attempts to use computers to help people better communicate affective information.

==Sensing, recognizing, modeling, and predicting affective state==
 
Emotion researchers have traditionally used questionnaires, human observation, and physiological sensing to gather data for assessing emotional state.  Affective Computing expands these options, enabling new kinds of real-time, automatic, mobile, and sometimes less obtrusive measurements, giving technology the ability to read affective cues from complex patterns that include tone of voice, language, facial expressions, posture, gestures, autonomic nervous system measures, and whatever combinations of modalities that people are comfortable with having sensed.  Advances in affective technologies allow for more natural sensing and communication of emotion outside the laboratory.  Tools of pattern analysis and machine learning are typically used to discover possibly nonlinear combinations of the sensor data that enable recognition of complex dynamic affective states. These techniques can also construct statistical models to aid not only in recognizing the current state, but also in predicting the next state or states, much like speech models can be used to predict likelihoods of certain words following others.  
 
This area of research raises many questions concerning an individual’s privacy and the use of digital affective information.  Great care must be taken to respect people’s wishes about what is and is not sensed, clearly communicating whether collected information could be associated with their identity and what benefits and harms they may receive from sharing this information.  These concerns are especially important because in some cases sensors can obtain affective information without people being aware that sensors are even being used, for example when thermal imaging is used for deceit detection.  Affective computing was originally conceived to make technology more respectful of people’s feelings; consequently, it is in keeping with the origins of the field that designers of affective technologies are strongly urged to respect users’ feelings in every stage of the design and development process, not falling prey to the tendency to make something “just because it can be done” but rather working together to discern what should be done in allowing technology to improve human experience. 

==Responding intelligently and respectfully to perceived emotions==
 
[[Image:empatheticdialogue.jpg|thumb|400px|right|F2|Computers sans anthropomorphic faces and voices or the pronoun “I”, can adapt responses to human emotion that subsequently influence the affective state of those with whom they interact.  The example shown here (Klein et al., 2002) did not pretend to have real feelings, yet conveyed the impression of active listening, empathy, and sympathy in an effort to help frustrated computer users feel better]]

When a person reveals affective information to a recipient, the latter can choose ways to respond that may be helpful or harmful.  For example, if a person lets a computer know that its action is very frustrating then the computer could recognize its gaffe and take steps to avoid it in the future.  It could also issue an acknowledgement of the frustration it has caused, and perhaps even apologize, in order to help alleviate the person’s immediate frustration.  Sometimes it might be appropriate for a computer to display an empathetic or caring response.  While it may sound absurd for a computer to express feelings when it does not have them, it is possible for a computer to empathize and appear caring without pretending to feel anything a person feels (Fig.<ref>F2</ref>).  Studies suggest that computer-provided empathy can reduce frustration and stress and can impact perceptions of caring, which could help in health-care technologies, among others.

The idea of having a computer show empathy grew out of findings that people interact with computers similarly to how they interact with other people; consequently, the theory of human-human interaction can be applied to make predictions about what will work in human-computer interaction.  For example, if we know that Bob does not like it when Alice sings triumphant songs after Bob experiences misfortune, then we can predict that Bob won’t like it when a computer plays triumphant songs after Bob experiences misfortune.  While one cannot expect people to treat computers exactly the same as other people, many general principles about human-human interaction carry over to the case of human-computer interaction.  These principles are often used in affective computing research to shape the design of affective communication.

==Simulating emotions or implementing their regulatory and biasing functions==

The most controversial area of Affective Computing arises when researchers talk about giving machines emotions, because the topic quickly moves to whether machines can be conscious and have feelings in any sense resembling the human experience.  To date, there is no evidence that machines can have conscious experiences like those that people have, and no compelling proof that they cannot, while there is ample evidence that computer scientists can implement computational functions that imitate many aspects of biological emotions.  
 
Emotion-like mechanisms can perform functions that may or may not appear emotional to an outside observer.  An emotion model within the Hasbro/iRobot toy doll My Real Baby evaluates inputs and causes the doll’s facial expressions and vocalizations to change, making the doll appear to have emotions.  Another example is where a cognitive appraisal model (such as that of Ortony, Clore and Collins, 1988) within a software agent is used to trigger an emotional verbalization. In other cases, Affective Computing researchers implement less visible functions of emotion, such as ways in which affective states can improve machine decision-making, attention regulation, learning, and more.   Affective computing often tries to implement findings from affective <review> science and affective </review> neuroscience, and such implementations can act to evaluate current hypotheses as well as to provoke and articulate new ones.
 
==References==

Breazeal, C. and Picard, R. (2006) In Neuroergonomics: The Brain at Work (Eds, Parasuraman, R. and Rizzo, M.) Oxford University Press, Oxford.

Burleson, W., Picard, R. W., Perlin, K. and Lippincott, J. (2004) In Workshop on Empathetic Agents, International Conference on Automonous Agents and Multiagent Systems Columbia University, New York, NY.

D'Mello, S. K., Craig, S. D., Sullins, J. and Graesser, A. C. (2006) International Journal of Artificial Intelligence in Education, 16, 3-28.

Gadanho, S. C. (2003) Journal of Machine Learning Research, 4, 385-412.

Healey, J. and Picard, R. W. (2005) IEEE Trans. on Intelligent Transportation Systems, 6, 156-166.

el Kaliouby, R., Picard, R. W. and Baron-Cohen, S. (2006) Progress in Convergence (Eds, Bainbridge, W. S. and Roco, M. C.) Annals of the New York Academy of Sciences 1093: 228-248, doi:10.1196/annals.1382.016.  

el Kaliouby, R. and Robinson, P. (2005) In Real-Time Vision for Human-Computer Interaction, Springer-Verlag, pp. 181-200.

Klein, J., Moon, Y. and Picard, R. W. (2002) Interacting with Computers, 14, 119-140.

Marsella, S., Gratch, J. and Rickel, J. (2004) In Life-like Characters Tools, Affective Functions and Applications(Eds, Prendinger, H. and Ishizuka, M.) Springer, New York, pp. 46.

Nass, C., Jonsson, I.-M., Harris, H., Reaves, B., Endo, J., Brave, S. and Takayama, L. (2004) In CHI ACM, Portland, Oregon.

Ortony, A., Clore, G. L., and Collins, A. (1988) The Cognitive Structure of Emotions, Cambridge University Press, Cambridge, England.

Pantic, M. and Rothkrantz, L. J. M. (2003) Proc. of the IEEE, 91, 1370-1390.

Picard, R. W. (1997) Affective Computing, MIT Press, Cambridge, MA.

Picard, R. W., Vyzas, E. and Healey, J. (2001) IEEE Transactions Pattern Analysis and Machine Intelligence, 23.

Prendinger, H., Mori, J. and Ishizuka, M. (2005) Int'l J of Human-Computer Studies, 62, 231-245.

Reeves, B. and Nass, C. (1996) The Media Equation, Cambridge University Press, New York.

Rosis, F. d., Novielli, N., Carofiglio, V., Cavalluzzi, A. and Carolis, B. D. (2006) International Journal of Biomedical Informatics, 39, 514-531.

Schneider, K. and Josephs, I. (1991) J. Nonverbal Behavior, 15, 185-98.

Trappl, R., Petta, P. and Payr, S. (Eds.) (2002) Emotions in Humans and Artifacts, MIT Press, Cambridge.

Tsiamyrtzis, P., Dowdall, J., Shastri, D., Pavlidis, I., Frank, M. G. and Ekman, P. (2006) International Journal of Computer Vision, 71, 197-214.

UPI (2006) United Press International.
 
==Recommended reading==


Carberry, S.,  F. de Rosis (2007) Special Issue on Affective Modeling and Adaptation, User Modeling and User-Adapted Interaction:  The Journal of Personalization Research.
 
Douglas-Cowie, E., R. Cowie, et al. (2003). Special Issue on Speech and Emotion. Speech Communication. 40.

<review> Fellous,J-M. &  Arbib, M. (2005).  Who Needs Emotions? NY: Oxford University Press. </review>
 
Isbister, K. and K. Höök (2007) Special Issue on Evaluating Affective Interactions, International Journal of Human-Computer Studies.
 
Paiva, R. Prada, and R. W. Picard (Eds.), Affective Computing and Intelligent Interaction 2007, Lecture Notes in Computer Science 4738, 2007. Springer-Verlag, Berlin Heidelberg 2007
 
Tao, J., T. Tan, and R. W. Picard (Eds.), Affective Computing and Intelligent Interaction 2005, Lecture Notes in Computer Science 3784, 2005. Springer-Verlag, Berlin Heidelberg 2005.
 
==See also==

HUMAINE portal of affective computing related research:  http://emotion-research.net/

[[Category:Pattern Recognition]]
[[Category:Robotics]]