Sign up for news

So with Steve Gaynor’s wager, GDC, and the discussion after each, I’ve been thinking about the tools we have at our disposal as designers to create emotional response. Not just what tools are in the toolbox, so to speak, but the biases designers have for and against using some of those tools. Some biases seem natural, and others baffle me. It seems difficult to discuss what sort of techniques to use when, if we can’t even agree on what should be in the toolbox to begin with.

When I wrote in response to Gaynor’s bet, about realism in human faces & behavior being key to the mass appeal of games that convey emotionally complex themes (more mass than today anyway), I hadn’t thought that much about the relationship between those two factors in that way until then. But in writing about it, I convinced myself, you might say.

Interestingly, there were a number of points and questions brought up about this specific topic in Ken Perlin’s roundtable at GDC on Interactive Actors that Express Emotion. Sadly, I couldn’t stick around to talk more and get people’s names, so I can’t credit the people with the corresponding questions, but I suck so oh well.

Is it possible? 

The question was raised if it was even possible for us to create a believable virtual human being. While it’s an interesting philosophical question about how much can we truly understand about our own nature, it constrasts with the unfailing progress our medium has made in the past 30+ years. Having gotten to the point we are at now, there is no compelling evidence that we couldn’t continue in this direction. With one key caveat, to follow.

Do we want to? 

Do we even want to create a realistic human being that you can interact with? When I read or hear Ray Kurzweil talk, he predicts that we will be able to create complete virtual beings, for all intents and purposes function as people. Their intelligence is computationally comparable, that visually and behaviorally they are comparable. To be honest, that kinda creeps me the fuck out. At least the notion that people might turn to virtual characters in full on replacement of human relationships – that is not a future I particularly want to encourage.  (The notion that this will somehow happen simply because the processing power is available also seems naively optimistic, to understate the point).

In other words, creating virtual humans is for suckers. Creating virtual actors, however, is the proverbial hot shit. An actor has to maintain a constrained fiction for a short amount of time, and one of an actor’s main goals is communication (of motivation and intent). This paradigm shift (which has more AI implications that I won’t go into here) is what specifically makes believability achievable, entertaining, and affecting. And ideally avoids the future where we all have virtual girlfriends/boyfriends. (Ok, realistically we’re probably headed there eventually anyway, but diverting the on rushing river is perhaps more feasible than stopping it).

Do we need to? 

It’s not like emotionally affecting media is a new thing, obviously the novel has existed for a long time. Although you could make the argument that reading is certainly less mainstream film, the argument is mainly a financial one (which I prefer not to have as the sole supporting leg in an argument on emotional affect). But that’s beside the point in any case – nothing in games limits you to using or not using that in this regard. Planescape: Torment is easily one of my favorite games. The amount of text in that game is, well, a lot.

That’s what I love about games, we have access to all the tools of all the other media. You just have to be careful to not overuse any. If the only way you have to convey a key plot point is a cutscene, you can do that. The guiding principle here is that you can’t rely to heavily on techniques from other media for fear players get frustrated, but as Planescape: Torment shows, it’s definitely a guideline and not a rule.

The tool of human performance to convey an emotional response is an incredibly powerful one. When that tool is completely in the toolbox, where it is easy to create those performances, then, something that is the opposite of shit hits the fan. That is still years off, of course. 

The strengths (or not) of the medium

Some say focusing on this realism doesn’t play to the strengths of the medium. Now, before I start on this point, let me say, Jason Mitchell’s talk on the visual style of Team Fortress 2 was probably one of the most inspiring sessions at GDC (now downloadable), with its very multi-disciplinary approach. He went from analyzing the artistic styles of American illustrators to the shader lighting equations used to recreate aspects of those styles.

Wandering into San Francisco’s Cartoon Art Museum the next day, I was equally inspired. The art being shown there there, from the work of Mary Blair to feminist political cartoons, just sent my mind creatively reeling in all the possible directions you could pursue for non-photorealistic visual styles. That is certainly one of the most open areas for exploration in games right now.

But to say realism doesn’t play to the strengths of the medium is incorrect. It doesn’t play well to the strengths of the technology, today. The strengths and weaknesses of the medium apply no stylistic constraints. Interactivity & simulation have no inherent art direction. As beautiful as the visual style of Team Fortress 2 may be, I can’t see those characters conveying more complex emotions such as regret or passion. Norman Rockwell’s characters (referenced in Mitchell’s talk) are certainly capable of conveying a specific set of emotions, but not others. TF2’s character are the perfect tool for the purposes of that specific game, and other games need other tools.

I don’t understand why there’s a natural distaste for this particular tool compared to any other – the more the better, in my book. And when it comes down to making the game you want, any tools you have, or even dirty tricks, are fair game in my book, if they create the desired response. When we have so rarely achieved the range in emotional responses that games are capable of, can we really afford to be so picky?

Diminishing returns 

Do we run the risk of hitting diminishing returns, as Jesper Juul asks? Certainly a lot of work goes into effects like ambient occlusion crease shading, and even more into all the animation systems needed to create believable motion. Five years ago, the same amount of effort certainly brought more drastic results. 

Does that bely lack of forward progress? If we’re traversing the slope of the uncanny valley, the distance we travel down the slope is longer than our horizontal movement. That doesn’t mean the work isn’t moving towards achieving the same goal, it just means it is difficult to pinpoint the impact of these effects while looking at them individually.

On most games I’ve worked on, we were building our own engine at the same time, so rendering features would go in slowly. Each individual one might be alright looking, noticeable but not wowing. Inevitably, as development goes on, someone takes a current screenshot and sends it out with an old one. Looking back at early playable demos with relatively untextured, poorly lit, unshadowed levels to a near-finished look always drives the point home. It can be easy to underestimate the holistic impact caused by such changes, without that perspective.

Holism, formalism and sensual…ism?

The lack of holism is problematic for me in this sort of discussion. I enjoyed Clint Hocking’s talk at GDC on immersive fidelity (also downloadable). In part he discussed the differences between formal immersion (stemming from game systems) and sensual immersion (from audiovisual stimulation). However he left it as an open question at the end as to how we can best combine techniques in these two areas to greater effect.

While it’s a very useful point to make, I can’t help but wonder if the underlying assumption is somewhat flawed. By looking at these aspects individually, they already begin separated. Granted, you have to break them down like that just to begin to understand the complex factors involved in each type of immersion. Once you do, can you continue to treat them separately and still successfully combine them? Or do you have to synthesize that understanding into a new perspective that treats boths aspects holistically?

A while back (an exceptionally long while back in internet time), Tadhg Kelly pointed out what is also a pet peeve for me: we all make fun of game designers that really want to be making movies, but those that really want to be making board games are just as confused. So treating techniques as belonging in the formal category vs. sensual may not end up being helpful. That breakdown may help simplify how you think about making one mechanic or performance more immersive, but relying on it exclusively seems to unnecessarily constrain how you think about the combination. Unfortunately dealing with the combination is a much more intuitive affair I can’t necessarily yet convey, I’ll just have to be critical here with contributing a more useful insight. Again, Isuckohwell. 

One of the points the roundtable came to generally agreeing upon (and I didn’t want to follow up on for fear of belaboring a point) was that creating a meaningful relationship with a character should primarily be driven through mechanical impact – how your relationships with characters change how characters allow, disallow or alter game mechanics. Which is just a bit weird. Now, to be clear, it is the most underused and overlooked aspect of creating emotional relationships to game characters today, and as such, is definitely the area to explore for biggest immediate effect (and affect).

Yet to dismiss the sensual components of our physiological responses to characters is like throwing the baby out with the bathwater – before you’ve even washed the baby. Completely ignoring those components is one of those painful lessons we might all be better off if we didn’t have to go through, the kind that often results in fantastic mistakes much the way Hocking points out Trespasser. That game using a IK driven player animation system to attempt to create immersion with one-to-one correspondence of player input to avatar actions – but by failing to simplify how you interact with the physics simulation at the same time, your interactions end up being much more complex than they should have been (and not immersive).

If you say either formal immersion or sensual immersion has more of an impact than the other in general principle, is that attitude really going to bring about meaningful change to our medium? In any individual game the balance may be shifted by design, naturally, but I don’t think actively prioritizing one over the over can lead to a useful basis for constructive analysis. The real strengths of the medium come when we are no longer watching movies while playing board games.

Taking these points to a silly extreme

These biases aren’t limited to designers, though. I’ve been slogging my way through The Transmission of Affect, by Teresa Brennan. She rails against the psychological bias on transmission of affect through visual means, and also its cause – our society’s reliance on the visual sense. She argues (so far anyway, I’m about a third of the way in) that the predominant factors in the transmission of emotions between people involve other biological processes, hormones, pheromones, and so on.

Which is unfortunate for two reasons. One, because I bought the book hoping at least some of it would analyze the pyschological processes of transmission of affect through media – especially given the mediated nature of the image on the cover, it’s misleading to say the least. Second, it’s ignorant at the same time. While those physiological processes may be primarily responsible when face to face, it’s obvious media can in fact encourage emotional response through purely audiovisual means. Although it does open the question of what other tools will we have when we have access to smell-o-vision…But god help me if I ever have to debug that.

Once more, with feeling

In all this discussion of creating emotional impact, we do run into the issue Leigh Alexander raises, like actors do in creating their performances, especially early on in studying acting, that of over-compensating. This point, which is raised in various guises on this theme, I’m always a bit conflicted on. The same argument is often taken to an extreme against games that have serious themes – because they have typically exhibited such ineffectual, over-wrought attempts at delivering their messages, it must be that the expending of effort itself is flawed.  While she doesn’t make this point, some often extend the argument to say we shouldn’t even make the attempt.

The clumsiness is not related to effort of course, but overly applying conscious or analytical thought processes, part of the natural progression of learning any skill set. Like acting students perhaps more effort is required, but in different areas, to advance enough to convey what one really intends. When you start out you have to apply conscious thought processes, and your performance is impacted negatively. As the skills become automatic, you perform the same processes instinctively, freeing yourself from the burden of over-analyzing yourself as you learn.

It seemed like the Portal  team tried quite hard for their game to have an emotional effect on players (or, more accurately, you might say they explored very hard). Not from playing the game, but from the stories of what they did during the game’s development. Most products of design or craftsmanship appear the most effortless when they have been slaved over incessantly.

The root of the argument, though, is key – keeping the sincerity of feeling in your attempts. Occasionally you read op-eds that call for designers to make games not just for themselves, in order to broaden the market. This sort of thinking is confused – the more useful advice is that whatever project you find yourself on, you have to find something in it that truly appeals to you, even if the game on the surface wouldn’t strike you as interesting. At the simplest level, you need to feel what you want the player to feel – if you don’t, you’re in trouble.

As usual, I wrote way too fucking much. Hope you made it to the end.

13 Responses to The Toolbox of Affect

  • Society’s reliance on the visual sense? Good luck to Teresa in changing that bit of, eh, “culture”. I think she might find that a sizeable chunk of brain has caused it to be our dominant sense for a very, very long time. Is she really that extreme? You make it sound like she might think we could all do what sniffer dogs do if we just tried a bit harder and built a different sensory hegemony…

    You’re absolutely right about creating virtual people being for suckers – in a commercial sense, at least. Talking to AI students was what enabled me to make the distinction between performance and simulation for the AI and the Uncanny Valley article. We’re so far away from synthesising human-equivalent intelligence that trying it really does belong in academia.

  • Patrick says:

    Maybe the evolutionary mirrors that game contexts provide can stimulate the hormonal releases normally triggered by olfactory receptors in face to face interaction.

  • Borut says:

    Well, part of her schtick is really based on compensating for what she thinks is far too much research on the visual side. Which, ok, I get, I can be accused of the same thing from time to time – arguing for the other side too much simply to help created a balanced perspective. But she doesn’t really seem to cover much if any of that type of research in her book, if even to give it lip service, or explain how it might be activated via other mechanisms (like you suggest, Patrick).

    And I buy that type of AI is perhaps best sought after by academia, but at the same time, there’s a dearth of research into the techniques required to really go after virtual actors. A few key exceptions, to be sure, but it just seems like developers run into a wall when trying to look for useful research (well, ok, I do, maybe it’s just me). :) After a certain point, it just doesn’t seem to be there, because academia just isn’t going after these problems. And they’re just as worthy of proper research, I’d say.

  • steve says:

    I think that photorealistic characters will eventually populate our virtual worlds. I think it will become the standard, and will be the turning point towards people broadly accepting what games become… people don’t seem to dig large doses of abstraction in their media. Actual photo footage of people connects best it seems.

    I imagine that this will come when characters and spaces don’t have to be constructed by hand anymore. I’m picturing a process like radiosity rendering.. fire this device into a room in real life, and it bounces a bunch of beams or particles all around it, reading the color and shape of the space (or actor) and automatically digitizing it. Basically I’m picturing a camera that generates fully 3D images, which can then be associated with physical properties within the simulation. Who knows if this even makes sense or will be possible, but I’m picturing a day when ‘level design’ will consist of building a movie set and then capturing it with this 3D virtualization device; same with character modeling, but with an actor in a booth. Animation fully procedural based on the properties of the actor and the environment. We’ll see.

  • Borut says:

    Yeah, they’ve already got tech to do that for peoples’ faces (capturing all the details, creating geometry, normal maps, spec maps, etc), next would be being to do something easier than existing mocap to capture performances (which yeah, has got to include procedural tools), then full environments. Though it will be somewhat ironic if making realistic games eventually then involves more of existing movie making skill sets when it comes to set creation, decoration, etc.

  • Aloosh says:

    “people don’t seem to dig large doses of abstraction in their media. Actual photo footage of people connects best it seems.”

    What about the effects the characters from animated films such as “Toy Story”, “The Incredibles”, even the lamps from “Luxo Jr.” etc. have on us? These are wildly popular, contain abstracted humans as characters (or not even human in Luxo Jr.’s case), and are highly emotive. You can certainly feel their emotions quite strongly.

    Do we even need to wait for highly realistic looking characters before we can express greater emotions in video games? The characters in the Team Fortress 2 shorts seem quite expressive.

    And Scott McCloud addresses the issue of abstracted representations of humans can be more acceptable to us than realistic ones – Amplification Through Simplification.

    From “Understanding Comics”: “When we abstract an image through cartooning, we’re not so much eliminating details as we are focusing on specific details. By stripping down an image to its essential ‘meaning,’ an artist can amplify that meaning in a way that realistic art can’t.” And, “…when you look at a photo or realistic drawing of a face you see it as the face of another. But when you enter the world of the cartoon you see yourself. We don’t just observe the cartoon, we become it.”

    Sorry, ran out of time to take this idea further.

  • Borut says:

    Aloosh – it’s about not about individual games, it’s about games as a whole appealing to a wide range of people. It’s not that any single game can’t create an an emotional response without realism, it’s certainly not *required* to create emotion.

    And Understanding Comics is all well and good, I’ve read, I get it. But to Steve’s point, ask any random bus full of people if they read books, comics, play games, or watch movies and almost all of them will say they watch movies but many fewer will say the others. My current theory on this (the short version) is while the abstraction let’s you identify with the character (becoming it), adding in the details the image has had removed is more work than perceiving the details in a actual human, in part because so much of our brain is devoted to doing just that. And therefore if it takes more work or skills to enjoy the medium that way, that seems to be a limiting factor (the question is whether this is still more work than learning to play a game, and we’re screwed anyway, or if this is fact the limiting factor – we won’t know until we do it, of course).

  • steve says:

    Yeah, I’m totally on board with the power of abstraction (I like to cite McCloud myself.) But while certain examples do connect strongly with people, I think there’s a reason that animated film makes up so very little of the total film consumption by the audience, relatively speaking. I don’t think it’s because animated movies are more expensive to make; I think it’s because people generally connect more at a core level with that other 95% of films that are composed of photographic footage.

  • It was a good read. It wasn’t too long by any English Major’s standards.

    As I was reading the post, I tried to connect what was being discussed to my own games that I’m currently working on. Personally, I stick to “classic” (eastern) design principles and philosophies. Because of this and my other bits of style, I generally never consider emotional responses in games. Sure, I want my products to be entertaining and satisfy the player, but talk of virtual actors and simulated humans extends far beyond my aims.

    However, I am doing some interesting new things with images and music in my latest project. By using realist characters with “true to life” facial expression coupled with music created from the player to create the language through with the player is to continue to communicate with, I hope a new connection between audio and visual emotional stimulation can be created.

    Ah, wishful thinking in the early stages.

  • Borut says:

    Yeah, there will always be a wide spectrum of games that are part of the medium, this is only one aspect (although one that needs pushing/expansion, I think).

    I appreciate what you mean about eastern design sensibilities in that sense, although even if there’s not a focus on emotional response, there is a focus on player response at least in the classic risk vs. reward balancing that is practically iconic about Japanese games (look at Mario, Street Fighter, etc.). But that response will inevitably be emotional in some capacities, and part the general communication process w/the player. Stephane Bura has an interesting breakdown of the sort of emotional responses some of those types of decisions make here: (couldn’t quite work it in to the post itself, but relevant anyway).

    That project sounds pretty intriguing – good luck with it, & I hope to see it sometime.

  • Nick says:

    I think one of the issues we have is that we even refer to what we create as ‘videogames’. Are there cutscenes in Monopoly? Should there be? What about Football and text descriptions? Applying conventions from other mediums seems absurd when you consider them being applied to a game, but there should be a clear distinction between what we are creating and what a game is. This sounds roundabout, but bear with me.

    Some of the things that go into videogames make no sense in the traditional game paradigm; therefore ‘videogame’ is a misnomer — we’re not creating a game on a screen, not all the time; many new videogames are interactive experiences that rely on game mechanics to communicate a certain feeling, but they also use a vast array of other tools such as the impact of visuals, sound, writing and cinematics.

    So I would deem most independent or simpler games, those games made by purists who rely only on game mechanics, whose designs could potentially be translated to paper or real life, to be videogames. The games that larger developers are producing now are turning into something more than that, and I think that will end up bending those conventions.

    For example, I believe the definition of game writing will become more concise and develop into its own format, different from what is used in film and television — because it needs to in order to support an interactive experience, as opposed to just a borrowed convention stacked onto a game.

    And I currently live in a world of bullet points and concise summaries that forget subtleties; a wall of text is a breath of fresh air.