Motion Capture & Gesture Control

A variety of applications have further extended the physical control of the 4DSOUND system with motion capturing technology - mapping various controller-data of the way the body moves in space to OSC messages for the spatial control of sound. 

Multimedia-artist Rumex created an integrated spatial sound control by the movements of a dancer in her work ‘Sonic Lure’. She used Kinect video motion capture to translate gestural qualities of the body to move sounds in space and RiOT sensors for capturing the orientation of the limbs and body to control rotations in the sound field. Composers and live-electronic performers Mathieu Chamagne and Herve Birolini played on 4DSOUND using two LeapMotion sensors and a motion-sensitive Wii-controller to perform the spatial behaviour of twenty-four sound sources with their hands and fingers.

 

Real Time Position Tracking

4DSOUND has developed a real-time position tracking system based on UWB-frequency (Ultra Wide Band) with Ubisense sensors and tags. The sensors map the physical space and trace up to five tags that can be carried by people moving in the space. 4DSOUND maps the real-time position data of the moving bodies to spatial position information that can then be interpreted inside the 4DSOUND system to move or transform spatial sound images.

Screen Shot 2016-03-29 at 16.36.49.png

The position tracking system has been used in various projects created in 4DSOUND. In the opera Nikola, singers carry their voices with them as a sonic aura they moved through the space, providing for a natural sounding and localisable amplification of their voices. Performance artist and composer Marco Donnarumma used the real-time position tracking in his 4DSOUND work ‘0:Infinity’ to capture three visitors in a complex interwoven system of sound processing based on their spatial positions, distance in relation to each other and behavioural derivatives, such as how much attraction visitors developed to each other over time. The position tracking system has also been used to create an interactive spatial sound remake of the well-know videogame Pong.

 

Bio-Physical Feedback Systems

4DSOUND is focused on investigating the relationship between the external environment and our internal physical and physiological space, and how our ability to listen opens up new levels of awareness about those spaces. As a result, a variety of new interfaces and control applications have been developed that establish a direct connection between the inaudible states and transformations of the inner-body and the audible space around us.

the Xth Sense is a bio-wearable instrument that captures inaudible sounds from the internal body, like the heartbeat, muscle tension and blood flow. Then, the pitch, texture and rhythm of those sounds can be performed by free movement in space. 4DSOUND combined with Xth Sense creates a musical instrument that allows to sonically explore the spatial dimension of one’s inner body, by being able to moving inside it and interact with it. Xth Sense allows twelve derived parameters from the inner-body to be mapped in a variety of ways to attributes of the sounding space. 4DSOUND and XTH are currently developing a template for integrated spatial sound control with Xth Sense.

 

In the work ‘Body Echoes’, techno-artist Lucy was able to capture the inner movements in the body of yoga-teacher Amanda Morelli through a system of custom-built Arduino controllers and close-up microphone capturing of Morelli’s breath and heartbeat, and translated the raw data from the body into corresponding sound images moving the energy of the audible breath in space. 

4DSOUND and Lisa Park developed communication between commercial EEG brainwave headsets and the 4DSOUND system through a custom-built smart-phone app that translates the real-time brainwave data into OSC messages for the spatial control of sound.