The Magic Mirror project is an interactive Kinect installation built for Tony DiTerlizzi's "Never Abandon Imagination" exhibit at the Norman Rockwell museum by the Academy of Art University. It features three characters (a troll, a dwarf, and a group of ghosts) that users can interact with via motion detection. The exhibit opened on November 11, 2017.
I came into this project after the majority of the base foundation was completed (ie. concept ideation and definition, character models, textures, and rigs, environment models and textures, basic audio, and base interaction with the character and kinect).
There was an initial test run at the San Diego Comic-Con where a number of issues were discovered:
With the project being heavily reliant on visual and auditory cues, I wanted to address these problems from both fronts while working within the framework of the coded interaction.
Characters have three states of interaction: green, yellow, and red. The green state is the default neutral state where the character is seen strolling around outside and executing random actions in between (ie. The troll picking his nose). The yellow state is triggered once a user is detected moving. Once the yellow state is triggered, any additional movement causes the red state to trigger (ie. When the troll walks up to the entrance and talks to the user).
Concept art by Abigail Lashbrook.
Based off of how the character interaction was developed, I created a list of solutions that I wanted to achieve and be able to test.
For the visual cue that triggers once a user is detected, I worked with Ryan Gee, our VFX artist, in creating and testing out different effects. Instead of an additive effect, I hypothesized that a subtractive effect such as the clearing of smoke would improve the user's perception of being detected by the magic mirror.
We conducted a series of tests with different visual effects to find out which would be most effective. Based off the results, we concluded upon the smokescreen-like effect that would clear from the screen once a user is detected.
After realizing the importance of audio cues in creating a more immersive experience, I worked with the supervising sound editor to create a document of the required audio assets that were needed. Because the user interaction is limited, I wanted every characters' reactions to showcase their personalities.
However due to the time constraints, we were unable to obtain voice lines from the sound designers in regards to the yellow state. My solution was to program in vox audio in place of the voice lines to attempt to achieve this solution.
One of the core issues of the troll character was the time that it took for the character to walk up to the entrance of the dungeon. In the initial state, the troll would slowly lumber towards the user. This caused confusion among users since they did not realize that their actions were causing the reaction. Instead, they thought that what they were seeing was a video loop being played.
Because we needed to maintain the integrity of the characters within Tony DiTerlizzi's universe, we couldn't increase the speed in movement or have the character jump towards the entrance of the dungeon. My solution was to program a "Scooby-Dooby Doors" effect where the character would walk out of view and then appear closer to the entrance of the dungeon. This solution allowed us to stay true to DiTerlizzi's universe, while increasing the speed of the interactions.
My initial proposal to the client was to use a projector to extend the digital world into physical space and display where the user had to stand in order to interact with the Kinect. In order to maintain the theme, I planned on having it extend the flooring of the virtual dungeon. My fallback plan if this was refused by the Norman Rockwell museum was to have a sticker of some sort on the floor to represent where the user should stand.
The initial day and night lighting setup was composed of two different sets of lights, post processors, and background images. They were set up in different test scenes, so I had to figure out how to combine the two within the main scene.
For the lights, I created a 'master lighting' prefab that pulled the intensity and color values from the other two sets of lights. By using Mathf.MoveTowards, I transitioned between the day and night lights at a determined speed.
With the two post processors, I worked with a colleague to determine which one to use instead of trying to figure out how to switch between the two seamlessly.
Lastly with the background sky, I created two planes with each of the background images on them and transitioned between the two by altering the opacity levels on one of the planes.
The current lighting was adjusted for the TV at the Norman Rockwell museum and to our client's preference.
The character switching system involved two parts: moving the character outside of the frame and the actual system of switching of characters.
For the green state, all the characters utilized a waypoint system in which they randomly moved between the left and right side of the screen. I extended this waypoint system to encompass a waypoint outside of the view of the user that triggers a character switch.
The character switching system involved a list of characters that simply removed a character once it had been randomly picked. Once the list was empty, it repopulates and resets.
The initial location of the installation within the Norman Rockwell museum was originally set to be located in the lobby next to elevator.
We argued that the location was not ideal for the type of installation due to the importance of the motion detection and that foot traffic would interfere with the experience.
"Initial setup location" - Image provided by the Norman Rockwell museum.
Despite the complications in location, I created and presented a setup documentation to the client. It involved the construction of a fake wall to better frame the installation and hide the "innards" of the piece.
With the final location we opted to use a soundbar instead of hiding speakers within the fake wall or around the setup. In addition, stickers of troll footprints were placed on the floor to show the area of interactivity.
"Final installation" - Image provided by David Goodwine
After opening night, we quickly realized there a repeating issue of too many people trying to interact with the installation due to its location. The original specifications was designed and tested for a single user and the Kinect became quickly overwhelmed when a group arrived.
To account for this we created additional Kinect avatar objects in order to detect multiple users and the museum installed stanchions to better regulate the number of users interacting with the installation.
In the future, testing in a live environment throughout the development process would allow for better discovery of user experience issues such as the one stated above.