of 4

Abstrações: an audiovisual installation based on motion recognizing

All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Abstrações: an audiovisual installation based on motion recognizing
   Abstrações: an audiovisual installation based on motion recognizing    Clayton Rosa Mamedes, Denise H. L. Garcia Department of Music State University of Campinas Campinas, Brazil José Fornari, Jônatas Manzolli Interdisciplinary Nucleus of Sound Communication State University of Campinas Campinas, Brazil   Abstract  — This paper aims to present an overview on  Abstrações , an audiovisual installation controlled by video motion parameters. This article focuses on an implemented state machine model, the interaction between video motion recognition, the audio and video processing and the states of its implementation. Keywords– sound interactive design; multimedia installation; dataflow I.   I  NTRODUCTION    Abstrações  is an audiovisual installation based on motion recognition through video analysis within a delimited space, which provides data for audio synthesis and video  processing. Its design works on four interactive states that run continuously, in permanent or portable fashion. Its design aims to create a ludic and immersive ambient, where visitors can interact and play with sounds and geometric figures. It has four interactive states exploring different audiovisual correlation with video motion recognition, defined by audiovisual real-time parameters.  Abstrações  is here described as a state machine model that controls parameters for audio synthesis and video processing. These ones refer to the  beginning and the end of sound and video events, simulate the acoustic space controlling a quadraphonic sound diffusion system, and change the configuration of video particles. All states of the installation are controlled by parameters retrieved from the movement of visitors. The mapping described above is obtained through the usage of a video camera that constantly retrieves data from the mapped ambient, defined by a video display and a quadraphonic system where each loudspeaker is placed in the vertices of a rectangle, inside this ambient. Video data retrieved  by this camera is analyzed with GEM/PureData objects, which report horizontal and vertical (  x ( t  ),  y ( t  )) axes of the moving visitors' center of gravity. Both are based on a bi-dimensional mapping resulting from the video difference between successive frames, where a resulting difference matrix  D  is obtained by the following expression:  D =  A k  −  A k  − n ()  (1) Equation 1 is a general formulation of a video-processing algorithm to detect movement that is explained in [1]. The video processing is based on the generation of  particles. Each state is associated to a geometric shape and other parameters, such as: RGB color, particle size and particle orbit behavior. The audio is generated by a frequency-modulation (FM) synthesis algorithm based on Moore’s model [2], which changes its sonority in each state. The next sections will describe the state machine model (i.e. a formal generalization of the installation), followed by a  brief overview of each state. II.   S TATE M ACHINE M ODEL    Abstrações  is here described as a state machine regulated  by a general formulation as follow: S  i  x t  () ,  y t  () , P  j , V  k  ()  (2) for i = 1, ..., 4; j = 1, ..., 9; k = 1, ..., 7. The horizontal parameter  x ( t  ) an the vertical parameter  y ( t  ) are acquired in real-time from the video motion data. These two values are the main control  parameters and they are retrieved directly from the visitor’s motion behavior. Table I shows 9 parameters used to control audio and Table II describes the other 7 video control  parameters.  A.    Audio Parameters TABLE I. P ARAMETERS OF A UDIO C ONTROL   State P 1  P 2  P 3  P 4   1 [1 - 8] [0.25 – 0.75] [0 - 1] [5000 – 10000] 2 [1 - 8] 8 Eq. 3 [600 – 1400] 3 [1 - 8] 2.8 Eq. 4 [800 – 3300] 4 [110 - 610] 3.9 Eq. 5 [400 – 650] Where: P 1  = Index of a list of frequencies, if state ≤  3, or a random frequency within a specified range P 2  = Harmonicity index P 3  = Modulation index P 4  = Tone duration P 5  = Gain of the Front Left signal controlled by (6) P 6  = Gain of the Front Right signal controlled by (7)  P 7  = Gain of the Rear Left signal controlled by (8) P 8  = Gain of the Rear Right signal controlled by (9) All these parameters are mapped using the following 3 equations:  f  1  t  () =  31 − t  ()  (3)  f  2  t  () = 6 5 − t  ()  (4)  f  3  t  () = 0.6 2 + t  () 31 − t  () if if    0 ≤  t  ≤ 0.50.5 <  t  ≤ 1  (5) The next equations are related to the quadraphonic sound diffusion system: g 1  x  () = 1 −  x   (6) g 2  x  () =  x   (7) g 3  y () = 1 −  y  (8) g 4  y () =  y  (9) where: 0 ≤  x  ≤ 1  and 0 ≤  y ≤ 1  These 4 equations are correlated to the motion capture system in such way that there is a univocal correspondence  between the visitor’s bi-dimensional motion in the space and the quadraphonic sound diffusion system used in  Abstrações  (see Fig.1). Figure 1. Diagram of the correlation between motion gravity center and the quadraphonic sound diffusion system.  B.   Video Parameters TABLE II. P ARAMETERS OF V IDEO C ONTROL   State V 1  V 2  V 3  V 4   1 1 [0 – 0.2] none none 2 2 none [0 – 16] none 3 3 none none [1.7 – 2.5] 4 4 none none none State V 5  V 6  V 7   1 none none 85 2 none none 80 3 none none 96 4 [0 – 2.1] RGB Color 85 Where: V 1  = Geometric Shape V 2  = Geometric Orbit V 3  = Ratio of the sphere V 4  = Size of each particle V 5  = Amplitude of the particle oscillation V 6  = RGB Color [100...255, 100...255, 100...255] V 7  = Index of blurring effect [80, 85, 96] C.   Transition Diagram When a visitor starts to stroll inside  Abstrações  a chain of sequential states will initiate (showed by the dotted lines in Fig. 2). Each state is activated by the motion behavior of the visitors, according to constrains that will be described in section 3. Fig. 2 shows the general architecture of  Abstrações , where is possible to observe the real-time dynamics of the installation. Figure 2. Dynamic map of the installation III.   S TATES    A.   State 1 State 1 generates particles based on a geometric figure formed by circles. This is the initial state that produces sound and video during moments without information about motion, such as when there are no visitors inside  Abstrações . The automation of this stage generates random audiovisual information. Beside the stochastic srcin of events, some degrees of freedom are left apart, to be controlled by the visitors, in order to change some specific parameters of the installation. The video generation of this state uses basic circle-shaped  particles and applies over it two multiplicative indexes (  x ( t  ),  y ( t  )) obtained from horizontal and vertical axes, as described in (2). These values are correlated to the intensity of changes in the video processing. The average of motion detection in the last five video frames analyzed is used as a multiplying factor, intensifying the video signal in relation to the growth of visitors’ movement. Motion is also used to control the coordinates of center and orbit of the particles. For  instance, when visitors are concentrated in the central area of the ambient, the particles’ movements are intensified so to run out of the initial position located at the center of the screen. This is done through the increasing of orbit scope parameter. Figure 3. Visual result of the first stage, presenting the initial particles being  pushed away from the center-position, through orbit control parameterization.  B.   State 2 State 2 has a more direct correspondence between visitors’ motion and audiovisual synthesis. The video of this stage generates particles in a three-dimensional model of spheres. These shapes are actually rendered as polyhedrons, hence, when they are strongly enlarged, it is possible to observe the  junction of their vertices as shown in Fig. 4. The spaces  between vertices are filled with random levels of gray. The visitors’ motion controls the dimensions of the spheres through a multiplicative index, which is based on a quadratic function that transforms the detected motion from horizontal axis in a  parameter which continuously alters the size of the sphere into a conjunction of trapezoid polygons. That way, the visual aspect can change abruptly, resulting in strong changes of the visual pattern, where small movements can mean great changes. As in the previous state, a multiplier factor is also applied to the final particles’ video chain, intensifying the luminosity of the screen in relation to the average of detected motion, for the last five frames. Figure 4. Visual result of the second stage, presenting an augmented sphere that turns into a polyhedron, thus turning visible its vertices and trapezoid  polygons. Audio interaction in this stage also works with horizontal and vertical axes, from retrieved motion data. Our system enables the horizontal axis to control audio amplitude and duration of musical events. Values from vertical axis are associated with onset information of sound events and their frequency. The way we implemented it, the clearest sound results (in terms of sound frequency and onsets) are associated with the vertical axis. C.   State 3 State 3 keeps a direct relation between visitors’ motion and the events generated by audio and video started in the previous state. Video here is formed by on  particles in the shape of cones. As in the previous state, there is a noticeable increasing in the size of the figures, but the signal processing is implemented in a new way. While in the state 2, the process changes the size of the geometric figure being rendered, here the system make changes in the size of the particles at the top of the processing chain. In the previous state, the system simply enlarged a predetermined figure. In this state, the system manipulates the height for the cones. In practical terms, there are no polygons, as in the previous stage. Once that we are dealing with a cone-shaped particles in this state, a simple yet useful and computationally low-expensive strategy is here used, which is to invert the proportions of the cones by sending small parameters of their height. Once that the system allows these shapes to oscillate their parameters in a minute range, the resulting visual aspect seems like a white circle with short variations around its srcinal position, showing different distributions of lines. By processing this signal with blurring effect and defining to them a mid-to-long lifespans, a moon-shaped figure is created, located in the first half of the horizontal axis. From the middle region to the second half of the horizontal axis, a black hole is opened in these circles, which inverts the figures, placing the sides of white cone over the background. Once that this process is linear, visitors  playing with the installation in this state can create and explore several topographies of these objects. Figure 5. Visual result of the third stage, presenting the initially centered-located cones with their heights scattering in different directions. The audio of this stage is based on a model of FM synthesis that emulates the sound of bells. The system is divided into 2 main parts. The computational implementation of the first structure of this state reacts to visitors’ motion, which is very similar to the one used in previous state. The major differences are in the configuration of the parameters of sound synthesis, which change its timbre by modifying the intensity envelope shape, the harmonicity index and the modulation index. The  polyphonic section in this state is based in randomly generated  melodies, creating a counterpoint to stimulate visitors’ interaction.  D.   State 4 This state aims to reach the highest level of audio and video flux of information for the system. We have implemented this model to generate complex audio and video signals, while keeping low consumption of CPU processing. This audio model implements a sound synthesis that emulates a woodblock instrument. We have adapted Andy Farnell’s [3] model for the synthesis of fire sound, based on strongly filtered white-noise and a trigger that generates random crackling sounds, similar of the ones heard when materials are burned. This model was customized to generate several bursts of woodblock onsets, making a sound texture formed by dense randomly percussive attacks. This is implemented by the low- pass filtering of white-noise audio signal, which is then mapped through an envelope follower that sends pulses of notes. These  pulses manage all parameters used in the woodblock synthesis. This structure is random and automatic, ensuring that our last state will have ever-dense sound information. The visitors’ motion controls a layer based on the polyphonic structure from state 2. We have implemented clusters of notes that are filtered. The filter parameters were associated to the horizontal axis, which divides in two halves the mapped area. Each side has very distinct filter parameters, which intends to make their acoustic signature highly noticeable. The aim is to induce the visitors to make large gestures in order to change the filter  parameters, playing with each side of the recognizable area. The video model implemented in this last state uses an object to synthesize the physical behavior of waveforms. This object uses a mass-spring model that generates right lines with linked vertices, creating a three-dimensional object. Considering our aim of having as much information as  possible, this object help us reach this goal without the need of increasing the number of particles or visual events generated. With the objective of intensifying the resulting visual feature, this last state is the only one that uses color information. We have defined a generator of random colors for each frame that is mixed together with the previous ones through the use of a small-sized buffer for blurring effect. A lower limit in the random generator parameter results in an always-shining image  being projected, thus avoiding an undesired darkening effect once that this one removes abrupt changes. Following the same two-halves model for the audio synthesis, we mapped the horizontal axis into a height control for the vertices of the figure. That way, the visitors’ location is graphically retrieved  by a positioning control algorithm. Figure 6. Visual result of the fourth stage, presenting the mass-spring object with their vertices appointing to the center. IV.   C ONCLUSION    Abstrações  is an audiovisual installation based on the detection of motion behavior from the visitors. We developed a robust system to implement our goals in a high-level dataflow environment, as briefly described in this article. In computer simulations that we have applied during the implementational stages it was possible to foresee the complexity of the sound material and video processing real-time behavior.  Abstrações  showed how a single mapping from a video camera can be effectively used to control a complex chain of audio synthesis and video processing. This is a simple and thrift, yet creative way of generating immersive behavior in the context of artistic installations. V.   A CKNOWLEDMENTS  This project was developed as part of the discipline “Interactive Computational Music Recital” supervised by Dr. José Fornari and is also part of the main author (Clayton Mamedes) research, entitled “Sound Design applied to multimedia installations”, currently in progress, advised by Prof. Denise Garcia and Prof. Jônatas Manzolli, both from the Department of Music, at State University of Campinas (Unicamp), located in Brazil. This research has the financial support of São Paulo Research Foundation (Fapesp). The work was entirely developed in the open-source computer environment PureData (www.puredata.info). R  EFERENCES   [1]   The programming codes are available in: <http://puredata.info/docs/developer/GettingPdSource> [2]   Moore, F. Richard. Elements of Computer Music, Prentice Hall, New Jersey, 1990. [3]   Farnell, A. Designing Sound, MIT Press, England, 2010.
Related Search
Related Docs
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks