Description & Motivation
When was the last time you went to a music concert in an huge auditorium and realized that your seat location in the hall actually was making a significant difference to the sound you were listening to?
When was the last time you tried to feel what each instrument's importance is when you listen to a song?
What was the last time you listened to your favorite song and remember exactly what each piece of instrument brings the "vibe" to the whole music experience?
If you are unsure about your answer, then it's the time!
According to the statistics, The Average American Listens to Four Hours of Music Each Day. This was our motivation to bring this experience in Virtual Reality!
After this experience, the user will have a significant insight into the following data:
1. Music and Acoustics Experience inside and outside a concert hall.
2. Every single instrument's contribution to a song and overall musical experience.
This immersive VR experience can be deployed to Google Cardboard and/or Oculus Rift. It has been developed using Unity3D Gaming Engine. This project was developed as a class project for the Virtual Reality (CIS6930) class at University of Florida. The team consists of 3 graduate students:
Izhar Shaikh (https://github.com/its-izhar)
Salim Chaouqi (https://github.com/Salimchaouqi)
Kannabiran Maheswaran (https://github.com/KannaNoob)
Video Link
"Insight Objective" of the Virtual Reality Experience
We had two major insights that we wanted to user to take from this VR experience:
1. The first objective was to let user know what does a concert hall bring to the sound of a musical performance.
This is of course determined by the acoustics of the concert hall. An insight we wanted people to have is that in front of the stage in a concert hall is the location where all the sound reflections re-converge creating a very muddy, and very echoey sound.
2. The Second objective was to allow the user to see what each musical instrument brings to the sound of the song.
This is determined by the change in the sound of the song as the user enables/disables a particular instrument (similar to "a conductor" in an orchestra). A very interesting part you will notice is that in the song 'Never going to give you' up by Rick Astetly, the electric guitar part has much less of a significant roll then in the song 'In the end' by Linkin park, which has an essential electric guitar part. An interesting fact we would like users to notice is that 'Never gonna give you up' could be considered a Pop song, whereas 'In the end' is a Rock song, which sparks an interesting correlation depending on genre.
Benefits of experiencing this in VR
We feel that when experiencing music in VR, it allows you to be more enveloped and focused on what you are listening too similar to what a conductors focus would be. It allows you to ignore any outside distractions and focus on the task at hand. Experiencing this in VR also gives you a sense of actually being inside a "real" concert hall as an attendant of a music concert, which is quite of a unique experience in itself.
Technical Description
The acoustics of any hall, as perceived by a member of the audience, consist of three factors: volume, equalization, and reverberation (i.e., reflected sound).
In a concert hall, the sound one hears consists of 1) directly radiated sound and 2) reflected sound.
1) Directly radiated sound, referred to as primary sound, is the sound that reaches the ear directly from the source.
2) Reflected sound is the sound that reaches the ear after being reflected off the various surfaces of the hall (walls, ceiling, etc.)
- Most of our research was spent in understanding how the acoustics of a concert hall work, and how to create as close as possible digital representation in the short time span we had for this project.
- The reverb zones are placed strategically to emulate a real concert. When standing in front of the musicians the music sounds unpleasant due to the fact that sound waves travel longer before converging and the reflection causes a muddled sound in the convergence area. The best spots to be at in a concert as seen in the video accurately reflects how sound waves travel and reflect off walls to produce the music that an artist would like you listen to. This was done with the help of Reverb Zones and 3D audio controlled by a custom curve.
- The scripts use RayCasting to determine which object was selected and a function to toggle between the songs and the specific instruments are called.
For turning off the concert hall we simply used a tag zones, and set the concert hall, reverb zones to that tag and setActive to false when the user wanted to disable the concert hall.
How the experience leveraged concepts found in the VR Book sections Chapter 19?
Hardware: For hardware to be least significant factor adding to the Simulator Sickness, we made sure the HMD is free of flickers, used a light weighted phone (LG G2) with a Plastic Google Cardboard with Good lenses. In addition, we used lightweight OTG cable with PS3 controller for navigation to nullify counterbalance and finally we used a bluetooth-paired audio headset.
System Calibration: Since the HMD and Joystick were tightly attached, Pupillary Distance (PD) was proper.
Latency reduction: Scripts are written in a manner that would reduce the end-to-end delays as much as possible. We repeatedly tested the system for FPS lags, so dropped frames and increased latency won't occur.
General Design: Deleted directional light to reduce shadows as much as possible, this added an average of 40 FPS increase! we wanted to make sure the system ran with as few FPS lags as possible considering we had a many large sound files. We implemented this is by having a minimum amount of graphical models as possible to where the user can still understand the scene and have low FPS!
Interaction design: We used PS3 controller with OTG cable, allowed user to select what instruments are on and off as well as to enable/disable the concert hall.
Usage:
- Safety: Enough room for user to move and be comfortable.
- Hygiene: Wiped all instruments with Lysol wipes.
- New users: Encouraged them to proceed with slow head motions.
- Sickness: Made sure the user is not sick and we respect their decision if they want to stop the experiment or not try it all.
- Adaption/Re-adaption: If the user felt sick, we advised them not to drive car for next 30 minutes.
- Measuring Sickness: Used the post exposure simulator sickness questionnaire.
Measurement of Simulator Sickness & Analysis
We did this experiment with 3 participants randomly chosen from University of Florida. The following procedure was followed:
- We asked all of the participants about their previous knowledge of the learning objective (i.e. the music reverb zones, instrument sounds and their understandings, acoustics in concert hall etc.) as the requirement of project 3.
- We completed the experiment with all of them and asked them to fill up the Post-exposure Simulator Sickness Questionnaire.
- We accumulated the results and calculated the overall results and verified our learning objective.
After our survey, we found that no user felt any severe simulator sickness. All of the user had better understanding of the learning objective and it was found that the users now had a much deeper insight into the data presented in VR.
Here are the results from the Post-Exposure Simulator Sickness:
User 1: This user did not report any sign of heath getting affected by the VR Experience. This user did not have previous knowledge of the topic being presented in VR. After the experience, he explained that it was a great learning experience!
User 2: This user reported slight General discomfort, Nausea, Fullness of the head and Stomach awareness. This user spent almost all of his time in VR experience in the reverb zones, making him hear the sounds with reverberations, which ultimately caused him a general discomfort.
User 3: This user reported that she absolutely loved the option to turn off the stage and felt like she was outside the world listening to music in another dimension, in fact this user spent over 12 minutes enjoying the VR experience, and had lost track of time. Due to the fact that this user spend more than 12 minutes in the VR experience, carrying the weight of the HMD made her feel general discomfort.
Overall Results of Simulator Sickness
Every single person to try the VR experience reported significant surprises in not only how the acoustics of a concert hall bring different taste to the musical performance, but also what each instrument brings to the sound of a song. Overall, the simulator sickness was negligible and all user's had an immense learning experience!
Source Code and Executable
Download Source Code + .APK File: https://drive.google.com/folderview?id=0Bxjovy3eWt8MR1VtUkNSRXdHNGc&usp=sharing
GitHub Repo: https://github.com/its-izhar/Music-Theory-In-Virtual-Reality
References
Bass Guitar Icon https://upload.wikimedia.org/wikipedia/commons/c/cc/70's_Fender_Jazz_Bass.jpg
Electric Guitar Icon https://www.cordasonline.com/uploads/category%20icon/1362416969Bass%201%20copy.png
Music Files https://mega.nz/#F!v8oHCLza!SqPtonX0xbfz5eXqKFdZ6g
The Dome concert venue https://3dwarehouse.sketchup.com/model.html?id=161f2e83756d0176de101a75dc75aed5
Guitar & Amplifier from Unity Asset Store https://www.assetstore.unity3d.com/en/#!/content/12177
Church Drum Set https://3dwarehouse.sketchup.com/model.html?id=u58404b67-9d9f-492d-989f-936234d75760
Digital Keyboard https://3dwarehouse.sketchup.com/model.html?id=uadca5049-eb80-41b5-bc71-bc328bd528e6
Electric Guitar https://3dwarehouse.sketchup.com/model.html?id=e266d5bfb1c733c0399e268abcba227e
Linkin Park Icon https://talenthouse-res.cloudinary.com/image/upload/v1/invites-97/nozfham1an2fboppxesk.jpg
Rick Astley 'Never Gonna Give You Up' Album Cover Icon https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcR5YgGHt-bbAv_1gI3iT6sZFL1t57QXSUL2wKvtyLuZfr6PNqStsQ
Concert Hall Icon https://perfectlypitched.files.wordpress.com/2010/07/perfectly-pitched-concert-hall.jpg
Cymbals Icon http://images.fanpop.com/images/image_uploads/Drums-drum-77597_1000_865.jpg
Snare Drum Icon http://cdn.mos.musicradar.com/images/Rhythm/Issue%20129/RHY129_gear_ahead_snare-460-80.jpg
Kick Drum Icon https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRqg5sAKsYGC9AOaWFGl64mDLBWptYDQ3OOWRqISK2tmh4yZPmR
Vocals Icon https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcTuVdngrIv6vcsM4t-51T5_Cvfu20DiQzet3Xj1DKO57h4vCwlO
Synthesizer Icon http://www.synthtopia.com/wp-content/uploads/2010/09/ultranova-synthesizer.jpg
