Design Fiction

You are on chapter // 04 Fiction follows Fact

Minority Report and the diegetic prototype

There’s a scene in the film Minority Report, which also happens to be a wonderful prototype of a ubiquitous computing future, in which Tom Cruise’s character Inspector John Anderton manipulates a database of sound and images that are from the near future. In this scene, which just about everyone in the world knows about, Cruise’s character makes orchestra con- ductor-like gestures, summoning and juxtaposing fuzzy snippets of what is almost about to happen. It’s all happening in a mad-dash effort to piece to- gether a puzzle. The puzzle is, of course, unlocking the mystery of a murder we know will take place, unless the clues of its location and perpetrator are discovered.

Unfortunately, there is no easy way to share this segment of the film, which I show when I present this material as a lecture. If you have the film, pop it in now and just watch until around time code mark 4:22.

The example I bring up here is, of course, the gesture interface that Anderton uses to piece together the clue fragments for the future murder he is investigating. As a film element, it has a well-balanced mix of visual dynamics that will keep today’s science fiction film audience riveted, and legible interaction rituals that allow the audience to follow the gestures closely to develop an understanding of precisely what is going on — what is being manipulated and how bits of clue are juxtaposed and re-arranged as one might do so with a puzzle. Special attention is placed on the precision of the gestures that Anderton uses in order to manipulate the fragments of video and sound — zooming in on a bit of imagery with hand-over-hand gesture; deleting a few things by moving them with a forceful and dismissive sweep into this interface’s version of today’s user interface trash can.

This sequence, which begins at the very start of the film and continues until 4:22, presents a compelling extrapolation into a near future world. It does much more than demonstrate some bit of technology, relying less on the object and more on its situatedness in the world of human social life. The sequence tells a story, helping to move us from our present and what we know about the world, into the year 2054, a possible near future. The ex- trapolation from today into 2054 happens in just under five minutes, and it does this without fetishizing the device or the technology too much. Rather, more convincingly, we are led through a bit of convincing human drama, something particularly timeless as murder, law and order, and justice.

There’s more than the clue-construction device that Anderton uses — whatever its called. It would be a simple matter to show a few still images from this sequence as an index to the small bit of argument I’m presenting. But, it is precisely this longer bit of story that I want to highlight, and not just the instrumental technology. Not the story itself — the pre-murder. Rather, I want to highlight what the story does so as to fill out the meaning of the clue-construction device, to make it something legible despite its foreignness. It’s a device used to edit sound and images somehow extracted from the future. It’s as if the story is sharing with the audience, who may be reasonably wondering — how do you edit and manipulate fragments of sound and images from the future? How does police evidence gathering work in the year 2054, when evidence is things that have not yet happened — but will? Do they travel into the future through some device and collect things that they bring back? Do detectives still use little baggies and tweezers to collect scraps of bone fragment, sending them to clever forensic scientists back at the lab?

No, of course not. Or not in this possible future. In the speculative near future Department of Precrime, evidence is a story, pieced together through these extracted fragments from the near future.

This is a bit of visual storytelling that is done carefully in science fiction. In the best instances, it does not happen by merely pointing to a prop and saying to the audience didactically — this thing will show us the future. A story is told, with images because it’s a film, that provides a point of entry for the further development of the plot. It usually happens rather early, so as to lay the groundwork and make it possible for the audience to imagine that whatever unusual, unexpected thing has happened in the future to create the drama, it makes sense because it’s been explained, often through extrapola- tions of science speculation from today.

Whereas “design” might typically highlight the object itself, outside of its dramatic context — perhaps the special interface gloves and screen floating in 3D CAD space — by introducing the drama of this moment I mean to reveal the advantages of attaching “fiction” to the design, filling the object with a meaning and a context that it would never really have sitting by itself on a photographer’s silk pillow, demonstrating its vague, latent power absent its engagement by people and their practices. We can put the designed thing in a story and move it to the background as if it were mundane and quite ordinary — because it is, or would be. The attention is on the people and their dramatic tension, as it should be.

Perhaps 4:22 is a bit long — I don’t think so. It allows the design fiction to tell a story that is broader than the instrument itself. This is what design fiction is about. You don’t fetishize the instrument; rather you emphasize the rituals and the drama — the social elements that stories are always about The audience wants to know what this thing is in the context of a story in which people — people in the year 2054 — routinely (lets assume so) operate machines to do their work using gestures such as this. Were the instrument itself shown alone, it would appear meaningless particularly if there were no story to surround it. This is a bit like the product design pre- sentations that show the object in a featureless, white CAD backdrop, as if people use their blenders and motorcycles in featureless 3D landscapes.

What is it about this sequence that should be highlighted? Is it just the gestural interface? Not exactly. If that were the only matter that concerned us, showing stills might suffice rather than watching over four minutes of film. Stills by themselves would remove the dramatic context. We’d have to fill in some gaps, explaining what this guy is doing, perhaps by gesturing ourselves, or using descriptions such as “its like..” and then adding in the story ourselves, explaining that he is doing a future form of detective work, manipulating these images from the future. And the questions would be raised, such as — “Images and sound from the future? What are you talking about? How do you get images and sound from the future?”.. “Well, there are these evolved humans, they’re sort of these biological technical devices floating in a kind of nutrient rich slurry, and they can see into moments of the future where crime occurs and then send those images from their mind into the ap- paratus that use manipulate using gestures”.. “C’mon..I don’t get it..”

And so on.

Finally it becomes apparent that its just easier to show a broad sequence and introduce the story with a little bit of action, some didactic anchorage that explains what’s going on without being as heavy-handed as a documen- tary. And then it becomes apparent that the capability to tell stories — even visual stories — about what you’re imagining offers a richer way of mate- rializing these ideas, and circulating them. Providing a broader context by moving the instrument into the background, and bringing people and their stories into the foreground provides a more effective, compelling fiction. This bit of design fiction extrapolates and “designs” a future fictional world with some speculative technological instruments, but mostly through a visual story by Steven Spielberg and Philip K. Dick.

I tell you about this four minute sequence in the film not because I want to discuss the story per se. I want to unknot a small tangle of activity that is precisely what I mean by property swapping — the action by which the science of the film swaps back and forth between fact and fiction.

Science fact and science fiction jump through each other’s hoops. What
I want to do is follow just a small bit of this tangle of conversations and objects and ideas between Philip K. Dick’s short story “The Minority Report” through to the Steven Spielberg production of a film based on the story.

In between are the activities of scientists in their labs, conversations with film directors and props makers and experts on the future, back through to special effects artisans working in their shops with their film production software. Following just a few of these linkages shows how easily science-fact and science-fiction swap ideas, properties and objects.

Science-fact and science-fiction are entangled in the Minority Report drama, which isn’t a bad thing. In fact, it should happen more. Science- fiction has way more imagination than science-fact and almost certainly circulates knowledge and ideas more effectively than all the science journals and science journalism in the world.

In the production of Minority Report, the idea for such a gestural inter- face came from somewhere and at least in part from the film’s technical consultant, John Underkoffler. Underkoffler was a member of the Tangible Media Group at M.I.T., and had participated along with a panel of luminar- ies in providing some speculations as to what the future of Minority Report might be experienced based on their insights and their extrapolations of the current trends in the technology world. What was needed were some projections to help trace a vector from the present to the future of 2054, when the film takes place.

From a project at the Tangible Media Group called “The Luminous Room” were a number of “immersive” computing concepts that were drawn from some of the principles of Ubicomp. The principles are related to the idea that computers might become more directly integrated into the archi- tecture of the environments that people occupy. Rather than manipulat-
ing them with a keyboard and mouse, people might use gestures for direct input.

Translating laboratory principles into a dramatic film allows for the lab ideas to circulate in a bold fashion, beyond what would be accepted in the typical, conservative research-academic-industrial context. There is a larger military-industrial-light-and-magic complex in effect here, which is pre- cisely the larger, messy tangle through which fact and fiction become indis- tinguishable through a blend of science and entertainment. The action is a kind of science fact-fiction work that effectively tries out some ideas within a film’s narrative. It’s sort of like prototyping — sketching out possibilities by building things, wrapping them around a story and letting them play out as they might.

More formally, this is what David A. Kirby calls the “diegetic prototype.” [David A. Kirby, “Future is Now: Diegetic Prototypes and the Role of Popular Films in Generating Real-World Technological Development” forthcoming in Social Studies of Science, a journal.] It’s a kind of technosci- entific prototyping activity knotted to science fiction film production that emphasizes the circulation of knowledge and ideas. It is like a concept pro- totype, only with the added design fiction property of a story into which the prototype can play its part in a way different from a plain old demonstra- tion. The prototype enlivens the narrative, moving the story forward while at the same time subtly working through the details of itself.

The diegetic prototype refers to the way that a science fiction film provides an opportunity for a technical consultant to speculate within the fictional reality of the film, considering their work as more than a props maker or effects artist creating appearances. The diegetic prototype inserts itself into the film’s drama which activates the designed object, making it a necessary component of the story. The film itself becomes an opportunity to explore an idea, share it publicly and realize it, at least in part and with the consis- tency necessary for film production rather than laboratory production.

“..scientists and engineers can also create realistic filmic images of “tech- nological possibilities” with the intention of reducing anxiety and stimu- lating desire in audiences to see potential technologies become realities. For scientists and engineers, the best way to jump start technical development is to produce a working prototype. Working prototypes, however, are time consuming, expensive and require initial funds. I argue in this essay that for technical advisors cinematic depictions of future technologies are actu- ally “diegetic prototypes” that demonstrate to large public audiences a tech- nology’s need, benevolence, and viability. Diegetic prototypes have a major rhetorical advantage even over true prototypes: in the diegesis these technol- ogies exist as “real” objects that function properly and which people actually use.” [Kirby]

In Kirby forthcoming essay he describes Underkoffler’s role as a technical consultant where he “..treats his diegetic prototypes as if he were designing not only physical prototypes but real world objects that are a part of “every- day life” in the diegesis [of the film.]” [Kirby]

In the particular case of Minority Report, Underkoffler participated in a three-day pre-production conference convened by Spielberg, in which the director brought together smart, forward thinking people to speculate about life in the year 2054. Underkoffler saw this as an opportunity to channel technical knowledge in a new way, using film as a way to articulate his imag- ination, trying out some ideas with the backdrop of the film’s production. Laboratory production and film production get to swap properties.

The film becomes an opportunity to create a vision of the future but, perhaps more important, to share that vision to a large public audience. In specific cases, such as the evocative “gesture interface” concepts Underkoffler introduced into the film’s story and its production design, ideas gather a kind of knowledge-mass. They become culturally legible and gain weight and currency. We “get” the idea of using conductor-like gestures to interact with our information technology because it is given to us through the film, it’s pre-science, the discussions that evolve in media and with friends, the formation of companies to further develop the ideas, bolstered on the cul- tural literacy with touch and gesture interactions, and so on. To gain cultur- al legibility takes more than a scientist demonstrating an idea in a laborato- ry. What is needed is a broader, context — such as one that great storytellers and great filmmakers can put together into a popular film, with an engaging narrative and some cool gear.

The follow-on to this science fiction film introduction of gesture interfaces to a large public audience are more gesture interfaces, each one staking out Minority Report as a point of conception, either explicitly or implicitly. It’s as if Minority Report serves as the conditions of possibility for more and further explorations of the possibility for gesture interaction — whether touch- based gestures, as in the Apple iPhone and other techniques, or free-space and tracking gesture interactions, like the Nintendo Wii, for example. This is not precisely the case: we are not interested in claims as to priority, owner- ship and who did what first. What is much more interesting is the brocade of activity that weaves in and through the fictional/factual special effects props of Minority Report.

The entanglements amongst science fact and science fiction is perhaps best summarized with the networked culture digital looming device — Google. Searching across Google’s database for “Minority Report interface” reveals most plainly the property swapping gymnastics of this bit of science-fact- science-fiction. Search results present us with stills from the film right next to things that look like DIY garage science projects, to demonstrations of touch panels at industry trade shows like TED and CeBIT, to reviews of the iPhone interface using Minority Report as a point of reference, to promises that the Minority Report interface is just around the corner — wherever that corner may be. One can quickly trace the dynamics as the science fiction of the film swerves into a series of science fact developments that are inextrica- bly knitted together, whether the facts or the fiction care. It goes on and on, and through it all, just a step back from the specific “results” is what I mean by the entanglements where science fact and science fiction swap properties.

This isn’t to say that Minority Report serves as the canonical origin story for gesture interaction, but it certainly is a powerful, gravity-like force providing a reference point through which science fact and science fiction swap prop- erties and become partners in their own exploration of possible futures. The film is what gives some sense to a curious speculation that says, in the future people will be flapping their arms around to interact with computers.

In fact, it ends up that somehow this idea makes enough sense for it to spread outward, beyond the film itself into other experiments, and (inevi- tably) commercial endeavors. A mentioned, the Nintendo Wii comes to mind, but there are others, depending on which communities and networks of ideas one circulates amongst. It even becomes one of those rare specu- lations that can gain some time on a national news broadcast. So long as it can provide an anchor for an audience to this popular film, it becomes “legible” as an opportunity for a bit of light news. It is this example as seen in a short bit of CNN news candy that perhaps speaks most directly to the possibilities of the design fiction principle that we should create inextricably tangled weaves of fact and fiction.

Over a few years we see a variety of variations on a theme, which is not to say that Minority Report started it all in any kind of essential, definitive way. What is intriguing is to consider the circuits by which ideas and their mate- rialization and their circulation back to the world of re-considered ideas rely on larger cultural imaginaries to provide necessary contexts and meanings. It’s not just “science”, it’s also the ways that the science is framed and given meaning. We need our metaphors — they provide anchors for thought and reflection and motivation for creating new things. Design fiction is a way to work on and refine these object-ideas, particularly as we consider them to be important transition points towards new, more habitable kinds social worlds.

Remember, this is a kind of knowledge-making work. It’s good, playful work to think about how such a gesture interface might operate, and how it expands upon or even disrupts existing interaction protocols. For the film- makers who are more attentive to the story, what the design-fiction produc- tion team cobbles together is valuable to the extent that it helps move the story along without drawing too much attention to itself. (In a minimal case their work may even perform the role of the “MacGuffin” famously deployed by Alfred Hitchcock. The object need not have the same depth of possibility and consideration that the Minority Report interface device has.
It need only be there, as an element that moves a story forward without any extensive, fleshed-out details of operation. It could in fact be the vaguely specified but full-of-meaning device often used in technothrillers, often having the power to entirely re-imagine the future for the worse, and usually simply called — “the device.”)

Minority Report shows how science fiction is shaped and informed by science fact. It also shows how science fact learns from and finds inspiration through the science of fiction. Practitioners (film production people, scien- tists) who stake a claim along either side of the science genres come together to participate in the action of turning ideas about the future into a visual story containing ideas about the future.

There are other frameworks for the production and circulation of ideas, things other than a high-budget film with fancy effects tools, toys and cre- ative talent. It just as well may happen from the other side of the imaginary fact-fiction continuum. The initial impulses may very well be biased toward the creation of the more fact-based kind of science — the kind that may enjoy a good science fiction film but take that as entertainment more than a reference to anything real and possible to create in the laboratory. This is perhaps the more familiar road to materialization. You take an idea based more on its ability to become an object containing a value proposition that yields a more or less good chance of producing a positive balance called profit. We’re less interested in those, because those kinds of ideas necessarily filter out provocation way more often than not.

In this case, with the bias on doing “real” science — no one’s really being fooled into thinking that things like gesture interfaces of the sort one sees in Minority Report are something you can get and play with in your living room. Except — hold on. Yes, you can. The design fiction of the film probed the possibility spaces of this new, curious kind of interaction ritual, activating further consideration and design work and refining the concept into something that holds itself together for another context, one other than film production. We begin to see living room entertainment with game con- soles, or smaller-scale gestures with MP3 players and portable telephones,
or coffee tables with screens built into them. It’s as if on some level Minority Report holds some stake in creating the conditions of possibility for such things, producing the cultural imaginary that makes such things have some legible context. Minority Report is not on its own in doing this, certainly. It’s not a game of who was first, although there are plenty of people who spend their time prioritizing who-was-first, arguing as if they were still on the schoolyard playground. There’s far too great a muddle of activity to delineate a clear order of things and, besides, that’s not as interesting as enjoying the confusion.

We just looked at the science fiction side of things, where props collapse and knot together in the activity of prototyping, thereby circulating ideas and encouraging further material production outside the context proper to the fiction. Now lets look at things from the side of the science of facts, and investigate how matters-of-fact knit into matters-of-fiction. In this case, a science indebted to fiction called Ubiquitous Computing, or Ubicomp.

Ubicomp has a curious relationship to science fact and science fiction. It is a science fact, the kind practiced by thoughtful experts with advanced degrees and corporate budgets to build peculiar devices that explore the future ways in which we may interact and communicate with one another, or with our kitchens or with the most quotidian of things like door knobs and vacuum cleaners. Because of the way Ubicomp focuses its attention on the everyday and routine, it is as much a science that tells stories about what a possible future world may look like and how we live within it, as it is a science that creates technical specifications and patentable intellectual prop- erty. As we’ll see next, Ubicomp seeks to refashion the entirety of the rela- tionships amongst people, what they do, how they do it — all through new kinds of networked, computational devices.

In the section that follows, I will introduce Ubicomp as an example of a design fiction — a hybrid of science fact, design and science fiction. In con- trast to this section where fiction collaborates with fact, the next section ex- plores the way that science fact collaborates similarly, revealing the way that science fact is also a kind of science fiction.

For things that become Minority Report-like, Google effortlessly shows the fabric that is their interwoven references and swirls amongst the core concept. Air quotes, “reality” and “-like” are the tell-tale signs of similarity, indexical- ity and self-reference. This is the texture of imbricated associations amongst fact and fiction. The cultural legibility of “Minority Report Interface” owes as much to the science fiction of the search term as to its science fact. The film had production design and technical design assis- tance from engineers who had been — and go on to involve themselves with — the “reality” of this sort of interface. It’s not a chicken-or-egg problem, and the questions are not about “primacy” or who-did-what-first. Those are silly, except for bothersome precedence hooligans of the intellectual property world. What matters is the mutual activities, the back-and-forth and the evo- lution of curious new ideas into larger cultural imaginaries.

CNN Reports on the “fact” of the Minority Report fiction in a segment of reporting on John Underkoffler’s intriguing, continued material speculation on the Minority Report interface he conjured for the film’s production. This time it is through technology he and his colleagues are constructing, which they call “g-speak.”

This CNN segment captures the swapping back and forth between fact and fiction with no irony whatsoever, which is reason enough to take seriously the possibilities of blurring the boundaries in productive ways.

In this short report we see the way an entanglement of ideas can slip and slide amongst fact and fiction so effortlessly as to effectively blur the boundaries. The film becomes a means to circulate the idea of an evolved computer-human interface (which may or may not be “better” than today’s — no matter, so long as it has disrupted convention. We here are not only focusing on things that become commercial hits and make investors their fortunes. We are tracing the knots and entanglements by which ideas become their material counterparts, and material fold back onto continuing imaginaries.)

In 2002, Minority Report was released, which we may describe as the diegetic prototype for the gestural interface concept. In this segment, CNN reports on the real-world prototype in the year 2005. The following years trace a knot of interpretations and reflections as the idea of a gesture-based interaction circulates and gains “idea mass” — the “Google” of “Minority Report Interface” indicates the breadth of interpretations and the notion that moving ideas to their materialization can happen through the lens of fiction.