AI Responsive Storyteller
"Team Campfire is a project at Carnegie Mellon’s Entertainment Technology Center. Our team of four is passionate about interactive storytelling, and exploring what this means in an audio-only space. We will examine narrative-based design through a combination of traditional story development, artificial intelligence and machine learning. Our goal is to produce engaging experiences that create a rapport between the user and the AI agent. The result creates an interactive, audiobook experience with an AI agent that responds to questions intelligently, tapping into the listeners’ own creativity to fill in the visual details. Our project roadmap includes three phases: interaction experimentation, prototyping and developing a blueprint for more stories in this format. We’re developing our project on a smart home platform, like the Amazon Echo."
• Byunghwan Lee, Programmer
• Rikky Roy Koganti, Programmer
• Sarabeth Boak, Writer & Producer
• Seth Glickman, Sound Designer
January 2017 - May 2017
AR-Assisted Music Learning
Music Everywhere is an augmented reality piano learning tool that focuses on teaching improvisation in blues, rock, jazz and classical styles. Leveraging a bidirectional MIDI-over-Bluetooth connection with a Microsoft HoloLens AR device, students at the piano can engage with interactive lessons, watch virtual hand demonstrations, see and hear example improvisations, and play their own solos and accompaniment along with AR-projected virtual bands and musicians.
Technologies used: Microsoft HoloLens, Oculus Rift (capture for virtual hand projections), Leap Motion (capture for virtual hand projections), Roland FP-30 digital piano, Bluetooth, Unity, custom-developed software tools
• Shantanu Das, Programmer
• Byunghwan Lee, Programmer
• Fu Yen Hsiao, UX/UI Designer
• Seth Glickman, Instructional Designer
Improvisation is a key component of performance in many musical genres. To study improvisation is to learn by actively, and repeatedly, thinking in musical notes, scales, dynamics, phrasing, articulation, context and theory. It is often thought to increase musical awareness and understanding in beginner, intermediate and advanced students. Yet, instruction of this topic is left out of most musical curricula.
Augmented reality provides an ideal environment for this type of hands-on learning. Demonstration, guidance and user feedback can all be displayed and facilitated directly on the piano keys themselves. By spawning virtual musicians, the AR-enabled student can then practice new ideas and techniques covered in interactive lessons at their own pace, for as long as they need.
The Music Everywhere learning system begins with augmented reality piano keyboard overlays that are projected into the student’s holographic field of view. These indicators specify piano keys the student should play (along with the proper fingering). In one mode, the interface indicates distinctions of timing and duration, while in another mode it displays ranges of notes that work well within the progression of chords being played.
Students are guided through interactive lessons to learn the concepts and theory to form their own improvisational ideas. They listen and play along with examples of other pianists improvising and strengthen the skills necessary to develop new motifs and phrases, to be played with the accompaniment of the virtual bands. The gradual pace of instruction allows for a scaffolded overall learning experience. Finally, rehearsal sessions enable students to practice their technique, while showcases further illustrate more advanced concepts.
Virtual bands produce an environment of engagement and variety to reinforce learned harmonic and rhythmic occurrences commonly found in performance experience with live musicians. Musical passages are designed to both highlight solo sections in the various genres, and also guide students what to play in other song sections.
The core components of the Music Everywhere system are the custom-built score reading, hand recording and editing, and lesson building tools which allow for streamlined content development.
Projection Mapping in Game Design
A team of four designed and created an interactive installation for two guests to work together and complete.
Technologies used: Arduino, Makey Makey, Unity, ViewSonic PJD5126 projector, Ableton Live
• Kaalen Kirrene, Hardware
• Melanie Kim, Programming
• Adrienne Cassel, Art, animation and sound design
• Seth Glickman, Music and sound design
Special thanks to: Jesse Stiles for his technical assistance, instruction, advice and guidance.
Live Improvisation Performance for Music and Video
Being too far from New York, too regularly, I definitely miss improvising and performing with other musicians, in general, and with my band, specifically. I was fortunate to meet Fu Yen Hsiao, and he invited me to try band-style interaction, only with audio and video playing together.
The following is a 1-minute sample of this idea. We've had the excellent help of programmer Byunghwan Lee.