The Tangible Mind

When we design for the IoT we tend to think we design networked objects. But that’s a misinterpretation. We rather design ourselves. This article summarizes my talks at node15, thingscon15 and IoTCon15.

Our mind needs embodiment

Due to consideration of the psychology of perception, every designer knows that cognition is not limited to our heads. However, we are used to separate the mind (“Geist”), usually interpreted as something “higher”, from the body. Hence designers see the mind as the stage for emotions. The body is in contrast considered as an instrument that has to be supported by the right ergonomically conditions to be able to run a product.

The rediscovered “philosophy of embodiment” sees the mind per se as something expanded into the body. Here are some proofs for the theory giving impulses for designing networked objects.

My body means “me”

The body is one of the first object we discover in our world. As babies we learn that existence has a body and that we only exist, because we have a corpus. Surely everybody knows the picture of a baby, clapping one’s hand again and again. This is an act of self-assuring. In this growth stage we need the experience of the body to experience our selves. As long as we live we never lose this connection between our ego and our body.

When we are a bit older, we still need our body as affirmation of our self the most topical way to do that is the so-called “selfie”.

Fig. 1 | A young woman takes a selfie for showing her presence.

If you thought, “ok, selfie, but isn’t that just a silly nowadays fashion?” Not a bit. Please find here a selfie of the year 1900.

Sefie in the style of 1900 with a box camera and a mirror

Fig 2: Selfie in 1900.

Gestures structure information

Not only the existence of the body is important. In addition to all of that, the gestures of our body help us structuring the information we want to keep in mind. Same time they help to memorize the information connected better, deeper and quicker. This turns the body and it’s movement into a part of the cognitive activity of a human being. When we use things by the help of interfaces – we move, and the question of pressing a button or swiping at a screen makes a difference for us.

We can’t think without objects

Our body is a valuable precondition for our mental abilities. As we live in a material (in contrast to an abstract) world, we are coined by conceptualized thinking. Most of us are barely able to think in any abstract way. The structure of our thinking is basically metaphoric, because it is influenced by (bodily experienced) schemata like inside/outside of something, way/target or any sensory perception. We explain abstract phenomena to ourselves by translating them to the material world. This can be found in our language in many cases. Think about the terms “integration“ or “exclusion”. The original blueprint to understand the concepts meant is the idea of a container with an inside and an outside. Or think about “time passes by“. Of course “time” is not sitting in any means of transportation and overtake you on the fast lane. But with the idea of being passed by, the notion of time becomes clear.

Things store knowledge

We use things like artifacts, environment, and media as supporter of our knowledge. They can even contain whole stories, like my favorite chair, my car or my wedding dress. Try to find any object in your personal possession that doesn’t contain any information. I would wonder, if you found any.

Fig 3 | Our objects contain stories – and even if they are new to you they generate stories for you.

Interaction enables thinking

To be able to conceptualize we need the interaction of our body with the material world. Whereas we learnt that it’s most likely unhealthy to utter every object that appears in our field of vision, we still have a strange impulse to touch it. Especially when the touch promises a positive experience most of us can’t resist. When we buy new things, we often prove them by touch. Many people still prefer printed books and magazines, because they like to touch them.

Fig 4 | Look at this fur – alluring to touch, right?

No wonder that we try to approach to any ideas about the future with technology by interactive solutions, because interaction means thinking, and thinking means processing our mind by the help of the body.

Our embodied mind is expanded to the world

The preceding paragraphs explained the embodiment of our minds. The mind is strongly addicted not only to our body, but as well to the interaction between our body and the objects of the world surrounding us. Those objects even are used as memory storage for our mind, to source out knowledge. This lead me to the point in the philosophy of embodiment, that our mind is not only embodied, but as well already expanded to the world.

Fig 5 | By embodiment the mind expands to the world.

The journey from virtual to real

Now then, we have an embodied mind. But aren’t we living pretty much in a virtualized world? We did, I would say. But we are back from this journey.

Fig 6 | Stages of the journey from the digital, back to the real world.

The journey we did with interactive media, is one of the reasons for me to write this article, as we are back from a three decades long journey of attempts in the digital world. As a good news for our embodied mind, with the first steps into the Internet of Things, we give our mind again a “natural” environment to expand itself. Here are the stages:

1. Presenting the code was not the best idea

As we know the concept of the command line interfaces didn’t work too well. No reason to write more about it.

2. Depicting the world was not sufficient

Do you remember the picturesque websites in the mid 90ies? The metaphor-driven concepts tried to explain the new digital world by translating them into known looks. I particularly remember the horrific banking homepages, designed as malls with information desks, customer service cabins, galleries, etc.

Banking mall of the German Dresdner Bank, 1997

Fig 7 | Banking mall of the German Dresdner Bank, 1996

3. Simulating the world does not really work

The idea that we will be able to simulate a self designed world perfectly, is still valid. Have a look at the up-to-date concepts for Virtual Reality Interfaces like Okulus Rift. Following the theory of embodiment it is clear that it is not enough to visit a virtual world, without a physical connection to your body. So as long as there will not be any real “touch”-illusion, the virtual world for al long time in our evolution, will not be a real place to expand your minds.

4. Processual mental models are still a good idea

As interaction needed to be translated, the next step back to our real world has been processual mental models, like in “The Legible City” from Jeffrey Shaw. Here the user rides a real bicycle as interaction tool through a simulated representation of a city (here Amsterdam), constituted by three-dimensional letters, forming words and sentences along the sides of the streets. Do you notice that bodily processes get a revival?

5. Ubiquitous computing is often too abstract

The idea of Mark Weiser of “machines that fit the human environment, instead of forcing humans to enter theirs” has been realized in the concepts of ubiquitous computing. Unfortunately in many ubicomp-concepts, the computer with all its functions vanishes too perfectly into thin air. Remember? We need objects to be able to explain the world. The computer itself might not have been the perfect object for our nature, but if it is taken away there is just nothing.

6. Tangible Media takes our mind home

With tangible user interfaces (TUI) such as the reactable, we have been pretty close to the idea of the natural environment for an expanded mind. Here interaction is combined with the sensual contact to real objects. (Don’t ask me, why they could not resist developing an iPad and an iPhone version.)

7. Internet of Things: “You have reached your target”

Welcome back to our common world, shaped by objects. But how can we take advantage of the effect that the digital world finally enters a mind-friendly environment? Are we really on a good way?

Back home. Yeah! And now?

Lets’s come back to the original idea of the expanded mind supported by objects. The whole concept sounds fantastic in a literal sense – doesn’t it? Imagine a mind that is expanded to the world, connected to our body and the objects of the world. It must be a networked part of the world then. For those who saw James Cameron’s movie “Avatar” this idea must sound familiar. Drifted too far away? Be just a bit patient with me.

Fig 8 | Pandora as model for the Internet of Things

From HCI to the expanded me

Let’s have a look how this model can change the general design process for IoT-Design.

For a long time, designers and developers created the Human/Computer-Interface. From time to time the perspectives shifted a bit. Usability became standard, followed by User Experience Design. The general relationship stayed the same: here the user – there the computer.

The Internet of Things so far brought the aspect that the objects themselves have to interact with each other, and beyond that independently from the user. So the next term in the repertoire of a designer will be “inter-usability”. But when many processes run beyond the control of the use – can we still call him user? Isn’t the right term “beneficiary”?

The internet of Things leads the user back to the world of ordinary everyday objects. Same time people grew accustomed to the use of computers. Do we really want to turn someone, who drinks his morning coffee into the “user” of his mug? Do we really want all our ordinary objects, to get computer interfaces that blink and beep to catch our attention? Or have an App – or a Master App? No, please, no!

With the Internet of Things, design is facing a paradigm shift. As described above, humans have another relationship to objects than to software. They live in a world of objects, linked by and to our mind. What should happen now, when the objects themselves get smart and are networked with each other? Isn’t it now exactly the right time to think about that?

How about turning the user into a human being again, with a natural relationship to objects, using the given sensory interfaces of a human being?

Fig 9 | Shift of perspective for designers

Here is one strategy to welcome the mind back home.

A call for sensory interaction

The sensors available for the Internet of Things are already so developed, that experts talk about the digital nervous system of the IoT.

Fig 10 | The digital nervous system of the IoT

In contrast, the communication back channel is still a poor one. Due to the incomprehensible idea, every object of the IoT must be connected to my smart phone, most of the communication means used in the IoT are just – visual. With a bit of luck there is sound added (applause).

Objects in the IoT mostly are connected to a smartphone with the corresponding limited sensory abilities.

Fig 11 | Customary communication means of the IoT – Smartphones.

As we learnt, the mind of the human being depends on sensory experiences. Why don’t we rediscover them and use them for the interaction with the elements of the IoT?

Fig 12 | Advanced sensory channels of a human being.

Example

As an example I would like to show you a project of four students of mine. Felix Gillmann, Clauido Giovanniello, Rafael Mäuer and Sven Szota did this project in their fourth semester in the Interactive Media Design Course. Meanwhile the concept has been presented several times – at last at the ThingsCon15 in Berlin.

Fig 13 | Juno, a project by Felix Gillmann, Clauido Giovanniello, Rafael Mäuer, Sven Szota

JUNO is a muti-sensoric concept for tele-presence relationships. It addresses the problem of couples, living in long-distance-relationships. Although there are many ways to communicate with each other, the emotional closeness – the simple “togetherness” – has not been solved by one of them; the presence of the partner is simply not given when separated through distance. This often results in jealously, anxiety and sadness with which many couples can not cope with. Juno tries to solve this problem by transporting the presence of the partner, appealing to different senses to strengthen emotional closeness.

Juno consists of two internet-connected devices that can communicate only with each counterpart bidirectional through a peer-to-peer-Network and hardware-based-encryption. Each part will be stationed at a partner’s room or flat, sensing his or her presence and forwarding it to it’s counterpart in another city or even in another country. This presence is represented by the counterpart in using light, real and generated sounds and noises and real warmth.

Fig 14 | Juno’s sensoric channels

Juno separates space into four zones, which is inspired by Edward T. Hall’s social distance zones: public zone, social zone, personal zone and intimate zone. Depending on the zone a partner is located, he perceives different impressions of his lover’s presence and simultaneously transmits more information about himself.

Public zone: Inactive
Social zone: Movements are translated into light effects („shadow“ of the partner). Quiet and hollow noises from the partner’s space are perceived
Personal zone: light becomes stronger and sounds become clearer and louder. In this zone it is possible to „eavesdrop“ into the parter’s room. Furthermore emotional sounds which are generated based on the temperature, the pulse and the movements of the beloved one are hearable; even a heart beating is perceptible. Additionally the emitted light colors accordingly: orange when the pulse of the partner was relatively high or normal, purple when it was rather low and quiet.
Intimate zone: By touching Juno the intimate zone is entered. The partner can actually feel the warmth of the other and talk to each other if they touch Juno at the same time. While touching Juno, temperature and heart rate are measured and transmitted to the counterpart.

Fig 15 | Juno’s interaction zones

The IoT design model

To summarize the article: We are facing a paradigm shift at the beginning of the IoT era. Design processes and targets have to be adapted to the new field of work for designers. Special attention deserve the human factors that are connected to the interaction mit everyday objects. Therefore I suggest to adapt the known UX-Design-process like described in the following graph (new items in red color).

Fig 16 | IoT Design Model

As you can see, I plead for renaming the user as “human” to regain the option of a deeper view to the human’s psychological background. Instead of User Experience Research I would prefer Human Experience Research. The term Conceptual Model is not new for IoT-Design-processes, but I like to highlight it, because I see that this task will be underestimated. The suitable conceptual model is essential to give the user the feeling that he can halfway understand what is happening in his IoT-world. As I explained, we should lay a new accent on Sensory Design to face the new IoT-Situation properly. We don’t design Apps anymore. Last but not least, the IoT forces designers and developers to leave the “one-product-mono-cell” and therefore they should be aware that Organism-Integration will become a part of the design process. The “Pandora”-vision might be a super sign for that movement.

Media

Fig 7: Banking mall of the German Dresdner Bank, 1996 / http://web.archive.org/web/19970112174121/http://www.dresdner-bank.de/, retrieved 08/2015

Fig 8: Pandora as model for the Internet of Things / Graphics © Andrea Krajewski / Background picture: © http://de.123rf.com/profile_subbotina

Fig 16: IoT design model / Andrea Krajewski