Abstract

The author examines concepts of real time and real-time in relation to notions of perception and processes of sonification. He explores these relationships in three case studies, and suggests that sonification can offer a form of reconciliation between ontology and phenomenology, and between ourselves and the flux we are part of.

Keywords: sonification, real time, real-time, perception, attention, ontology, phenomenology

 

No man is an island, entire of itself.

John Donne, Meditation XVII, 1624

I can indeed say that my representations follow one another; but this is only to say that we are conscious of them as a time sequence, that is, in conformity with inner sense.

Immanuel Kant, Critique of Pure Reason, 1781

You can never directly know the outer world. Instead, you are conscious of the results of some of the computations performed by your nervous system on one or more representations of the world. In a similar way, you can’t know your innermost thoughts. Instead, you are aware only of the sensory representations associated with these mental activities.

Christof Koch, The Quest for Consciousness, 2004

 

1: Introduction

To start, some basic thoughts about the key objects of scrutiny: sonification, perception and time.

1.1 Sonification

Sonification is, after all, through its representations, connecting your understanding to something else (the data) by means of your capacity to connect with and understand (represent to yourself) sound.
And often this sonification is of something that has happened or is happening in a traversing of time; in this essay I will be focused on situations where our sense is that the sonification that we are hearing is being generated from the originating data synchronously.
Whether or not the data is, most surely the sonification itself is traversing time, sound being the sensory domain that is most clearly time dependent, and thus Sonification is a time based practice, whether in mechanical, utilitarian dispositions, such as in medical use, or in more aesthetics based application in sound art. In all, the representations and our representations of them in consciousness follow one another, and that is how we make sense of them.
And because it is sound, and we are trying to make sense of it, it is very likely that we will have some ‘musical’ sense of it. That is, musical in the way of being a sequence of aural events that makes conceptual sense as it traverses a span of time. This can be said to be true even of the mechanical sonifications heard in medical situations, that they may have this non-musical ‘musicality’.
The rise of interest in sonification is supported by a growing understanding of how our apprehension of the world, as well as our perception of it, is multi-sensorial. This reflects a move away from a culture dominated by sight. Henri Bergson (1911) said, speaking about our consciousness of time when we apprehend a melody:

Doubtless we have a tendency to divide it and to represent it to ourselves as a linking together of distinct notes instead of the uninterrupted continuity of the melody. But why? Simply because our auditive perception has assumed the habit of saturating itself with visual images.

Bergson was talking about the phenomenology of time – our phenomenal consciousness of time as a continuum (which in fact he exampled through musical melody, as did James and Husserl), as opposed to an ontology of time, in which any instant is distinct, separable and measurable. He presents a model where sight cuts time into bits, while sound binds it together.
In an overall, and very important sense, Bergson talked about our perception of time through our perception of change (e.g. the changing pitches in a melody), and how this was not a perception of discrete events, but a perception of flow and sequence, the arch of relationships unfolding in time, which, in music, enables us to apprehend a melody or phrase as a whole.

1.2 Sound and vision

The doctor in the operating theatre does not listen to the sonifications of vital functions. Their conscious attention is focused on the relationship between hands, eye and flesh. The moment there is a change in the sound, however, it abruptly commands their conscious attention.
For most of our evolutionary history we were all hunter-gatherers, either on the lookout for food, or for that which might feed on us. This has had important implications for the way our senses work together. The eyes are in constant movement (saccades) as they scan the visual field for information; in doing so they focus on one thing at a time, while keeping the entire visual field in the background. We are not aware of this process: when we are examining a picture we may deliberately scan its surface; when we are looking at it, our eyes are unconsciously saccading over the surface, putting together a composite image of the whole). However, while our eyes are engaged in this process, and in particular when the visual attention is focused on a particular object, our hearing is continually attending to, assessing and decoding sound coming from every direction. Of course if an arresting sound event occurs, the aural attention focuses on it and usually brings the visual apparatus round to that region of the aural field. However, in our normal alert state, we are either looking for or looking at a particular thing, while listening to everything. There seem to be three implications of this difference: firstly, we can attend to distinct events with our sight and our hearing simultaneously (Spelke, Hirst & Neisser, 1976; Müller et al, 2003); secondly, we are much better at handling, and keeping discrete, multiple strands of information with our hearing than with our sight: we can listen to the totality and understand it (Bregman, 1990), or we can focus on a single strand (someone talking to us) (Colin Cherry 1953), or several unrelated (someone talking, the traffic behind us, music coming out of a shop door to our left) (Song, Skoe, Banai, & Kraus, 2010) or related (the parts in a fugue) (Bigand, McAdams & Forêt, 2000). Thirdly, considering seeing and hearing in a ‘Bergsonian’ way, our sight puts us at a distance from what it attends to, it separates us; our hearing puts us at the centre of what it attends to, we are inside.

1.3 Real-time and real time

The term(s) comes from computer science, and here are some definitions/statements:

1. A system is said to be real-time if the total correctness of an operation depends not only upon its logical correctness, but also upon the time in which it is performed. http://en.wikipedia.org/wiki/Real-time_computing
2. Occurring immediately. The term is used to describe a number of different computer features. For example, real-time operating systems are systems that respond to input immediately [my italics].
http://www.webopedia.com/TERM/R/real_time.html

however,

3. Real-time applications have operational deadlines between some triggering event and the application's response to that event. To meet these operational deadlines, programmers use real-time operating systems (RTOS) on which the maximum response time can be calculated or measured reliably for the given application and environment. A typical RTOS uses priorities. The highest priority task wanting the CPU always gets the CPU within a fixed amount of time after the event waking the task has taken place. On such an RTOS the latency [my italics] of a task only depends on the tasks running at equal or higher priorities, all other tasks can be ignored. https://rt.wiki.kernel.org/index.php/Frequently_Asked_Questions

also

4. Real time can also refer to events simulated by a computer at the same speed that they would occur in real life. In graphics animation, for example, a real-time program would display objects moving across the screen at the same speed that they would actually move [my italics]. http://www.webopedia.com/TERM/R/real_time.html

and

5. Real time is a level of computer responsiveness that a user senses as sufficiently immediate or that enables the computer to keep up with some external process (for example, to present visualizations of the weather as it constantly changes). Real-time is an adjective pertaining to computers or processes that operate in real time. Real time describes a human rather than a machine [my italics] sense of time.
http://searchcio-midmarket.techtarget.com/sDefinition/0,,sid183_gci214344,00.html

The term, in its computer science origins, is ontological. It is concerned with time as it can be recognised as a thing, quantified and subdivided. Creating and using ontologies of time is a necessary part of computer science, as exampled in this from the DAML (DARPA Agent Markup Language) project for the Semantic Web (Hobbs and Pan 2004):

The time ontology links to other things in the world through four predicates: atTime, during, holds, and timeSpan. We assume that another ontology provides for the description of events, either a general ontology of event structure abstractly conceived, or specific, domain-dependent ontologies for specific domains.

This ontological basis is clear from definitions (1), (2) and (3). However, as we scrutinise these statements, things start to get a bit fuzzy. Although (2) uses “immediately” to describe real-time iterations, (3) talks about latency (delay). This throws into question what ‘immediately’ could possibly mean. Does it describe an acceptable (within the computer process as defined in a RTOS) level of latency? Or does it describe the human experience of a very short latency? If it does then it has strayed into the world of phenomenology.
In (4) we start to go seriously adrift in relation to ontology: we can have no idea what “the same speed” might mean, or via which (scaling) parameters this ‘same speed’ might be measured. Does the statement imply that, again, we are in the realm of perception?
In (5) we have definitely moved into another conceptual domain, where the user’s sensory apparatus (? or, in fact, their interpretation?) is invoked, measured with the word ‘sufficiently’, whatever that might delineate, and where, in contra-distinction to ‘real-time’, presumably as defined in (1), (2) and (3), we are asked to understand ‘real time’ as a human rather than a machine sense (?) of time, whatever either of those might be.
Furthering the confusion at the level of definition is the way the term has been appropriated, as exampled by (5), to mean ‘something being represented or iterated in the computer environment at the (approximately) same time that it is happening in the external world’. I should make it clear that it is this quotidian ‘folk’ use of the term that I will be looking at.

(I write the above not to make nit-picking criticisms of use of language, but because something that emerges from examining these texts is an apparent confusion between ontology and phenomenology. The fact that real-time is a concept that lies within the taxonomy of computer usage can lead to it appearing to be an ontological term in situations where in fact it is being used as a phenomenological one. This might be important when considering Sonification, where the data would reside in the realm of ontology, while our apprehension and understanding of the sonic representation of it is within the realm of phenomenology. In this way sonification provides a bridge between ontology and phenomenology.)

So (1) through (3) are definitely in the realm of ontology: they are about what time actually is in a particular instance – something that is measured by the computer’s clock. (4) is ambiguous: it seems to be ontology: ‘same speed’ is surely measurable in centimetres per second; however, in order to arrive at a semblance of ‘same speed’ the simulation relies on the viewer’s capacity to decode the (small) image of, for example, a horse travelling at not many centimetres per second across a small screen, and to extrapolate that back to a mental image of a full size horse moving at metres per second at a certain distance from the eye. This is phenomenology, and, most definitely and clearly, so is (5).
It is, in fact, (5) that most people are talking about when they use, more or less loosely, the terms ‘real-time’, or ‘realtime’ or ‘real time’ and they mean something like: “the computer response was fast enough (the latency was low enough) that it felt in synch with the real world”.
If we take the phenomenological statement: “It feels as if the results of the sonification process I’m listening to are in synch with the data they represent” as a starting point for exploration then we need to uncover what ‘in synch’ means for different persons in different situations and engagements, what levels and sorts of consciousness might be engaged, and how the various strands above might relate to each other in a phenomenological timespan where some data are being sonified in what is perceived as real time. While doing so I will attempt to keep sight of the fact that ontological time that is measured and quantified.

2: Three case studies

2.1 Christina Kubisch: Electrical Walks1

Christina Kubisch uses induction loops in headphones worn by participants to make audible the electromagnetic waves that surround us in both urban and rural locations. There is no question about the synchronicity of the experience, as the sound is produced directly by the waves. Indeed, it is not a representation as one would usually think of it, more a mediation – inasmuch as the electronic oscillations that are experienced as sounds are produced by the interaction of the waves with the induction loops, they are not directly the sound of the waves themselves, nor are they Kubisch’s rendering of data into sound.
A most important aspect of these walks is that Kubisch stands outside the process of the listener’s relationship with where they go and what they hear: she provides the means, gives some guidance, and in some way locates or bounds the physical area that is being explored. As she herself says:2

The basic idea of these sound spaces is to provide the viewer/listener access to his own individual spaces of time and motion. The musical sequences are experiencable in ever-new variations through the listener's motion. The visitor becomes a "mixer" who can put his piece together individually and determine the time frame for himself.

The relationship of her work to dérive is clear and has been noted by other (Cox, 2006; Young, in Neset and Dzuverovic, 2005). What is of interest to me is what real time means here, and how it is, in fact, constructed.
The electromagnetic landscape that an electrical walk reveals is pre-existent from the piece’s point of view. The listener is invited to explore, not intervene. The electromagnetic vibrations will be proceeding in ontological time, whether they are heard or not. However, the hearing of them, and more importantly the listening to them, is played out in phenomenological time. That is, the hearing, in time, of these vibrations, is something that is constructed by the explorative and aesthetic decisions of the listener, how they choose to conduct themselves on the walk, and is a function of how their hearing elicits listening, and how their attention is engaged, held and diverted in a landscape that contains both the vibrations and all the other components which are apprehensible with the unaided senses. Further, the experience of time – whether it seems to pass quickly or slowly, whether its passing is recognised or not – is a function of this attention which the exploration and hearing bring about. This phenomenological time, the ‘melody’ (continuum) that Bergson speaks about (1911), is in constant flux, stretching and compressing, yet it at all times remains completely in synch with the ontological time of the vibrations, and the whole experience is charged with this real time sense, that one is hearing these electrical events as they happen, and that they are part of the ‘landscape’. This sense of real time and location is central to the experience and irreducible – that one is revealing these vibrations in this moment in this place by performing these actions. As Kubisch says: 3

The perception of everyday reality changes when one listens to the electrical fields; what is accustomed appears in a different context. Nothing looks the way it sounds. And nothing sounds the way it looks.

And one might note that part of that “everyday reality”, the perception of which changes, is the time. I would suggest that the apprehension of phenomenological time is, in part, a function of attention. I will return to this in section 3.

2.2 Peter Sinclair: Road Music

In British law, there is a motoring offence called ‘Driving without due care and attention’. The neurobiologist Christof Koch, talking about attention, cites an example of inattentional blindness in a test where 25% of pilots on a flight simulator did not notice a small aircraft unexpectedly superimposed on the runway (Koch 2004). Simons and Chabris (1999) report how subjects having to track two balls in a game do not see someone in a gorilla suit walking through the players (it’s true, I’ve done the test). Aristotle himself says: 4

There is a further question about sensation, whether it is possible to perceive two things in one and the same indivisible time or not, if we assume that the stronger always overrides the weaker stimulus; which is why we do not see things presented to our eyes, if we happen to be engrossed in thought, or in a state of fear, or listening to a loud noise.

In spite of this rather off-putting context, Peter Sinclair’s Road Music5 invites you to do exactly that – attend to two things at once. In most sonifications, we are not presented with the unfolding source data at the same time that we hear their representation in sound. In Road Music the sounds are generated in real time with minimal latency, from various aspects of the driving experience: elements of the car’s motion and elements of the visual scene that the car is moving through. In some sense, therefore, the driver is the performer in this piece: her/his decisions and actions affect the route and the motion of the car, and thence the output of the system. The resulting sound is clearly musical, both in intention and effect, which supports its eliciting of your attention at exactly the time that British Law says you should be attending to the road (“Keep your mind on your driving, Keep your hands on the wheel, Keep your snoopy eyes on the road ahead”6 ). But, having experienced Road Music both as a driver and a passenger, I can affirm that it is possible to attend to both the road and the sonification, pace Aristotle.
So why is this so? Koch (2004) notes two sorts of mental function that are relevant. One he describes as ‘zombie agents’ – these are specialised sensory-motor processes that take care of very complex sequences of action while consciousness gets on with something else: most of us can recognise having driven from A to B and on arrival at B having no memory at all of having done so, because consciousness was taken up with, say, thinking about a complex problem. Yet we clearly have driven perfectly competently, and in fact executed a series of controls of motion that have played out in a span of ontological time of which we had no phenomenological experience.7 He also refers to ‘gist perception’ where we are conscious of components of the visual scene with only marginal perception of them. Gist perception in hearing has also recently been researched (Harding, Cooke and König, 2007)8.
However, I don’t think that evoking these functions is sufficient to explain the multilaterality of the experience in Road Music, though they may have bearing on it.9 It seemed to me in my own experiences of the work that my attention was fully on both the music and the visual and physical scene, and certainly my consciousness at the time was that these attentions, of different sorts, were both proceeding uninterrupted. If there was an alternation of attention then it was too rapid for me to notice. This coherence of attentions is, I think, due to two factors: the real-time relationship of the two and the fact that this relationship is evident and perceptible; and the fact that the audio is experienced as musical (at the very minimum in the sense that I use in 1.1 paragraph 4), and, acting in the Bergsonian sense, draws the other attention into its ‘melody’, creating a sort of flux ubermusik, i.e. that the totality of the phenomenal experience is drawn and held together by the ‘melodic’ continuum of the music; one could even perhaps describe Road Music as a sort of Gesamtkunstwerk10 , though I’m not sure if Sinclair would be pleased. This accord is in contradistinction to the kind of situation for which the UK law was predicated, one where the attention to the scene needed to drive the car is in conflict with something else that demands or seizes the driver’s attention.11 The coherence of the experience is one of the most engaging aspects of Road Music, and it is dependent on the sense of real time synchrony that the relatively low latency of the computer process delivering the sonification permits, the sense that this is all happening together now.

2.3 Stuart Jones: Meter

Meter dates from 1970. Performers are asked to track one environmental variable (e.g. temperature, light level, atmospheric pressure, humidity, noise level) and one own body variable (e.g. heart rate, breathing rate, temperature, blood pressure) and use them as the ‘score’ for performance; they are also asked to start by tracking the parameters using meters of whatever sort, but to aim to be tracking using their own unaided senses by the end; the piece ends when all performers consider they have achieved this. There are no other instructions – instrumentation and the way of re-presenting the data (the word ‘sonifying’ did not exist then) is up to individual performers. The performers used a variety of means to gather the data, which were usually ‘lo-tech’ – barometers, thermometers both atmospheric and clinical, fairly crude heart monitors, stethoscopes or contact microphones strapped to the chest, stopwatches and other timing devices. In some instances performers would have to put down their musical instruments in order to gather data. The data gathering was part of the performance, as was the move from scientific instruments to senses as the data gathering technique. All in all, I would say that it was the process of gathering the data and representing it that interested me, rather than the sheer accuracy of the data gleaned.
The piece differs from Alvin Lucier’s Music for Solo Performer (1965) in that performers are expressly asked not to attempt to control their body variables, merely to observe them and use the data. I was interested in different ways of generating material for use in performance and in different approaches to performance, and in this piece I was particularly interested in the performers’ sensitivity to the flux they were in and how their own actions changed that flux: these performers, creating this, in this place, at this time, in these circumstances, under these conditions; and how these factors affect one another.
As they do. The piece attempts to make this explicit, inasmuch as variations in the data chart (they can do no more) the variation of factors that are synchronous. It is also explicit that the performance (the product) is part of this, that the piece is reflexive in that the performers’ heart rates etc are likely to be affected by the performance itself as well as other factors.
The piece has some historical interest in its relation to various artistic currents of the time, in particular Fluxus. It was also at the least unusual in its use of sonication as a basis for performance. It operated in a sort of real time, though with, of course, very high latency (humans are much slower than computers at converting data into sounds). However, it is fundamentally rooted in a notion of real time as an explicit concurrence between what is happening and what is experienced, where the relationship between ontological time and phenomenological time is fundamental to the experience, where what would be classified as part of ontology (the variation of measured temperature in measured time, the heart rate in bpm) is clearly incorporated into the ‘Bergsonian Melody’.
The audience were outside of this. They were intellectually aware of what was going on as the programme note consisted of the score, but not privy to the actions of the performers except as spectators. My interest was with performance, and whether the kind of concentration the piece was intended to elicit would work to generate a viable spectacle – that the engagement of the performers and the resultant sounds could be musically compelling. In this respect the piece differs from the works of Kubisch and Sinclair, where the ‘audience’ is the performer. However, inasmuch as all three examples are performed in some way, and that they all in some way depend on particular and high level attention, I would now like to consider real time in relation to these factors.

3. Attention, real time and performance

In section 1.2, I suggested that humans can simultaneously attend to both the visual and aural scenes, and discriminate several simultaneous audio strands. In their different ways, and to different degrees of explicitness, these three examples of real time sonification seem to evoke and support this capacity to attend to simultaneous but distinct sequences of events. I would suggest that they support it, in the case of Electrical Walks by the understood categorical relationship between the headphone sounds and the location; in the case of Road Music by the apprehensible relationship between the musical sonification and the parameters that cause it; in the case of Meter by the obligation to actively read data and render it into sounds.
Although the ‘zombie agents’ and gist perception referred to above in 2.2 (paragraph 3) can help us understand how sequences in consciousness can be supported by underlying unconscious activity, or how attention can be diverted or enlarged to draw elements at its edge into full consciousness, they do not account for the kind of attention that these pieces seem to demand: it is clear that to succeed to their full potential, all three require very high level and ‘divided’ or ‘dispersed’ attention in order that simultaneous sequences and their relationship may be fully available to conscious discrimination.
The capacity to divide attention does exist. Experiments have shown that people can simultaneously focus their visual attention ( very briefly – Müller et al, 2003) or their aural attention (Shinn-Cunningham & Ihlefeld, 2004) on two objects, or attend to a visual and an aural object (though with some identification deficit – Bonnel & Hafter, 1998). At a high cognitive level, in a classic experiment people learned how to simultaneously read and write different texts (Spelke, Hirst and Neisser, 1976); it has also been shown that presenting material in a mix of visual and aural modes can reduce cognitive load (Mousavi et al, 1995). However, dividing high level attention and sustaining the resultant consciousness does not seem to be an everyday habit. Far from it – no doubt we’ve all heard “Pay attention!” at some point in our youth (Broadbent et seq).
In sum, I would say that real time sonification requires of us to attend to separate streams of information which we perceive as having a real time relationship (the data; its representation) in such a way that we can correlate them and contextualise them not only in understanding but in ‘here and now’ experience; I will call this ‘dispersed attention’. I would suggest that this dispersed attention demands of us a cognitive state that requires expectation, alertness and engagement.
John Cage was someone who understood this expectation-alertness-engagement very well, recognising it as a condition that is necessary in the span of attention that makes his work into art (4’33”: the composer and performer frame; the listener attends; the art is made). I well remember him performing Cheap Imitation, and using his (then) very real pain from arthritis and gout, and his (always) nervousness as a performer, to generate an excruciating ten minute pantomime of distress and disability that screwed the audience to an almost unbearable pitch of sympathetic anxiety which he released with a smile as he started to play, and which left them in an exalted state of engagement that ensured the attention the piece required.
It is my own belief that Cage’s success as a composer stems, in part, from his understanding that attention-perception-consciousness is not unitary, but that it is (a) divisible and (b) the divided parts are sustainable within an apprehended whole. It is easy enough to understand this if one listens to contrapuntal music such as Javanese Gamelan or Bach. We can perfectly well distinguish each part, listen to all of them simultaneously and hear them together as a whole. If we couldn’t, the makers of such music would be wasting their time; in fact they would be unable to make such music except as an abstract, mechanical exercise. Cage himself was an advocate of inclusion, asserting that to enjoy this we do not have to exclude that; that we can include all we are perceiving in the act (the creative act) of enjoyment. He thus extends outwards from the enclosed world of Bach to bring in to the accounted everything that is there at the time to be perceived (this is quite different from the projects of Wagner and Scriabin, which, although multisensory, were restricted to those events that the composer had prescribed). Cage achieved this by the seemingly simple trick of giving the listener permission to include. I still remember my own experience of the European Premiere of HPSCHD12 (John Cage and Lejaren Hiller, 1967-69), in which I participated. As audience one could be with the piece in any way one wished13 : one could focus on one thing, take in the totality, ignore it all, have a conversation while still paying attention etc etc.
I would like to underline that it seems to me that the separate and to consciousness distinct parts (in real time sonification, the data on one hand and the representation on the other) are brought together in consciousness as parts of an apprehended whole that is continuous and indivisible within the span of the attention which holds it together. In this sense it adheres to Bergson’s concept of duration.
In HPSCHD the ‘attender’ is within the space of the piece but outside the action of it. No matter how engaged they may be, their position is fundamentally passive. On the other hand, the case studies in section 2 all have an element of performance: rather than passively listening to the sonification, participants engage with it on another level by in some way performing it. This is not a question of intensity of engagement but of experiential and psychological position of engagement: the performer is de facto inside the action of the work and in various ways responsible for its unfolding. In real time data sonification that involves performance, this unfolding is not just of the sonification itself, but of the data also (this is particularly clearly the case in Road Music). The ‘performer’ is both witness to and part of the scene – the unfolding of which their actions contribute to – which is played out in the data and thence played, in real time, in the sonification. This performativity places the attender at the centre of a complete scene, as both originator and validator of their own experience – which is, of course, the assertion of phenomenology (Merlau-Ponty, 1945, preface). I contend that this is only possible because of the quality and positioning of the attention we bring to this span of time, a state of attention which is evoked by our involvement as performer. I would also contend that the faculty to attend in this way is completely natural to us who evolved to hunt and be hunted simultaneously. If it seems in any way unfamiliar, it is probably because everyday modern (domesticated) life not only no longer requires it of us, but also favours the sense of sight above the sum of senses as decoder of the scene (the data).

4: Conclusion

If we conceive of a span of time when we are attending to all that is presented to us by the sum of our senses, it is hard to conceive of this without including the ontology of the scene (indeed, part of Bergson’s project was to reconcile the phenomenological experience of duration with, as argued by Deleuze and Guattari (1991), an ontological conception of it as ‘the variable essence of things’). We can only conceive of or read the ontology, but perhaps one of the things that real time sonification of data does is to offer a form of ‘reconciliation’ of its own: (some of) what we perceive forms at the same time a data set, ontological, measured, synchronously represented to our perception. The ontology remains unperceived, but perhaps appreciated in understanding, as we ourselves reconcile in consciousness the originating phenomena and their representation.
There seems to be a ‘recipe’ of requirements for the sort of consciousness I have been describing in this essay: the understanding that we need to attend in a certain way, and the willingness to do so; alertness; active engagement; multi-sensory perception; dispersed attention; and in the case of real time sonification, apprehension of the relationship between the data and the representation. I suggest that this capacity of consciousness derives from our human origins, not out of any romantic prelapsarian nostalgia, but because it seems clear that it exists as an innate faculty, that is valuable but now underused. I suggest also that engaging in performative real time sonification can arouse it and the connection that it delivers.14
If phenomenology, from Bergson and Husserl to Heidegger and Merleau-Ponty, has a metaphysical ‘mission’ it is to alert us to the scope and capabilities of our consciousness, and I would like to end with three quotes. The first is from Merleau-Ponty, in the preface to Phenomenology of Perception (1945):

“Phenomenology's task was to reveal the mystery of the world and of reason. […] It is as painstaking as the works of Balzac, Proust, Valery, or Cezanne - by reason of the same kind of attentiveness and wonder, the same demand for awareness, the same will to seize the meaning of the world or of history as that meaning comes into being [my italics].”

The second is from Bruce Chetwin’s book The Songlines (1987), and is part of an account of a journey with an Aboriginal who wanted to visit a part of his Dreaming (Native Cat) to which he had never been:

This lesser stream was the route of the Tjilpa [Native Cat] Men, and we were joining it at right angles. As Arkady turned the wheel to the left, Limpy bounced back into action. Again he shoved his head through both windows. His eyes rolled wildly over the rocks, the cliffs, the palms, the water. His lips moved at the speed of a ventriloquist’s and, through them, came a rustle: the sound of wind through branches. Arkady knew at once what was happening. Limpy had learnt his Native Cat couplets for walking pace, at four miles an hour, and we were travelling at twenty-five. Arkady shifted into bottom gear, and we crawled along no faster than a walker. Instantly, Limpy matched his tempo to the new speed. He was smiling. His head swayed to and fro. The sound became a lovely melodious swishing; and you knew that, as far as he was concerned, he was the Native Cat.

And the third, once more Bergson (1911):

There is simply the continuous melody of our inner life - a melody which is going on and will go on, indivisible, from the beginning to the end of our conscious existence.

 

Notes

1Kubisch’s work is probably more properly described as audification, given that the sound is produced by the direct rendering of electromagnetic waves into sound waves, rather than by introducing a mediating process to represent them. However, this does not affect my argument. I am not aware of Kubisch herself using either term when describing her work.

2 on her website

3 on her website

4 from On Sense and Sensible Objects

5 See Peter Sinclair’s website http://nujus.net/~petesinc/wiki/

6 from Seven Little Girls Sitting in the Back Seat by Lee Pockriss and Bob Hilliard

7 Musicians are dependant on ‘zombie agents’. The reason they have to practice so much is that, in order for them to be able execute the extremely complex physical tasks involved, these tasks have to be learned to the point of automation and unconsciousness. This is not just so that the mind can focus on the creative act which directs this unconscious activity, it is actually a cognitive necessity: Castiello, Paulignan and Jeannerod (1991) estimated that more than 250 msec intervened between action and conscious percept. Just think how many notes Jimi Hendrix or Franz Liszt routinely played in a quarter of a second!

8 Their paper is also of interest in relation to my remarks in 1.22 above. See also the POP project (Horaud et al 2006-2008)

9 One could perhaps imagine several people in the car: one is engrossed in both the scene and the music; one is conscious of neither because they’re thinking about chess; one is listening intently to the music and oblivious to the scene; one is vice versa; one has the music in attention and the scene in gist consciousness; and so on. Hmmm, there’s a lot of people in this car…

10 Wagner’s term for his music dramas (operas) which were predicated on an enmeshed text, music and staging.

11 See Strayer and Johnston (2001) for an account of experiments combining simulated driving and mobile phone use.

12 in 1972 at the Philharmonie, Berlin. A full performance of HPSCHD entails 7 harpsichords, 52 tape recorders, 6400 slides and 40 movies. In this performance only one harpsichord (playing the originating Mozart material) was in the auditorium, everything else was dispersed throughout the multi level foyer, with virtually every square inch of ceiling and wall projected on. One could imagine bedlam but in fact the experience was gloriously coherent. The subsequent British Premiere at the Roundhouse as part of the Proms was a disastrous travesty.

13 Cage himself said after the premiere in America: "When I produce a happening, I try my best to remove intention in order that what is done will not oblige the listener in any one way. I don't think we're really interested in the validity of compositions any more. We're interested in the experiences of things." (quoted in Time Magazine, Friday, May. 30, 1969)

14 One might say that the Songlines of the indigenous Australians are the oldest existing form of sonification. It would seem that, to the indigenous Australians, the songline traces a vast repository of spatial, temporal and mythic data, represented in its singing. For the most part, the song acts as a mapping and storing of knowledge, but it seems also to be in some way a representation of the duration of life, and can be sung in situ, in real time (Chetwin 1987).

 

Acknowledgment:

The excerpt from The Songlines by Bruce Chatwin, published by Jonathan Cape, is reprinted by permission of The Random House Group Ltd.

 

Bergson H (1911) La Perception du Changement (The Perception of Change), collected in La Pensée et le mouvant (1934), translated as The creative mind: an introduction to metaphysics. Citadel Press, New York

Bigand E, McAdams S, Forêt S (2000) “Divided attention in music”. Int J Psychol 35:270–278

Bonnel A-M, Hafter ER (1998) “Divided attention between simultaneous auditory and visual signals”. Percept Psychophys 60(2): 179–190

Castiello U et al (1991) “Temporal dissociation of motor responses and subjective awareness. A study in normal subjects”. Brain 114: 2639–2655

Cherry EC (1953) “Some experiments on the recognition of speech, with one and with two ears”. J Acoust Soc Am 25:975

Chetwin B (1987) The Songlines. Jonathan Cape, London

Cox C (2006) “Invisible cities: an interview with Christina Kubisch”. In: Electricity, Cabinet Magazine, Issue 21, spring

Deleuze G, Guattari F (1991) Qu’est-ce que la philosophie? Paris: Editions de Minuit. Trans: Tomlinson J, Burchell III G. What is Philosophy? Columbia University Press, New York

Harding S, Cooke M, König P (2007) “Auditory gist perception: an alternative to attentional selection of auditory streams?” In: Lecture notes in computer science 4840:399–416. doi: 10.1007/ 978-3-540-77343-6_26

Hobbs JR, Pan F (2004) “An ontology of time for the semantic web”. ACM TALIP 3(1):66–85

Horaud R et al (2006–2008) POP project. http://perception.inrialpes.fr/POP/. Accessed 14 Sep 2010

Koch C (2004) The Quest for Consciousness. Roberts and Company Publishers, Englewood, Colorado

Kubisch C http://www.christinakubisch.de/english/install_induktion.htm. Accessed 14 Sep 2010

Merleau-Ponty M (1945) Phénoménologie de la perception. Paris: Gallimard (1962) Phenomenology of perception (trans: Smith C). Routledge & Kegan Paul, London

Mousavi SY, Low R, Sweller J (1995) “Reducing cognitive load by mixing auditory and visual presentation modes”. J Educ Psychol 87(2):319–334

Müller MM, Malinowski P, Gruber T, Hillyard SA (2003) “Sustained division of the attentional spotlight”. Nature 424:309–312

Shinn-Cunningham B, Ihlefeld A (2004) “Selective and divided attention: extracting information from simultaneous sound sources”. In: Proceedings of ICAD 04-tenth meeting of the international conference on auditory display, Sydney, Australia, July 6–9

Simons DJ, Chabris CF (1999) “Gorillas in our midst: sustained inattentional blindness for dynamic events”. Perception 28:1059–1074

Song J, Skoe E, Banai K, Kraus N (2010) “Perception of speech in noise: neural correlates”. J Cogn Neurosci. doi:10.1162/jocn. 2010.21556

Strayer DL and Johnston WA (2001) “Driven to distraction: Dual-task studies of simulated driving and conversing on a cellular phone,” Psychol. Sci. 12:462-466

Time Magazine Archive http://www.time.com/time/magazine/article/0,9171,840155,00.html (accessed 14/09/10)

Young R (2005) “Christina Kubisch Electrical Walks. Dérivative Sounds.” In: Her Noise. Neset, A.H. and Dzuverovic, L., eds., pp. 35-36. Newcastle upon Tyne: Forma arts and media