Cover image for website.png


Greg Chen, Youngryun Cho, Wei-Chieh Wang

While at

Carnegie Mellon University


Dina El-Zanfaly, Kyuha Shim


Oct to Dec 2021
Duration: 5 weeks


A mixed-reality museum experience rooted in interactive experiential learning and participatory social interactions.

01. Intro


What I Did

Put together the team;
Conducted research and ideation with teammates;
Led the multisensory interaction design;
Prototyped all the MR interactions;

Today, many museums are also facing the challenge of transitioning from rigid institutions to experiential and flexible spaces. Museums are turning to virtual reality, apps, and interactive experiences to keep tech-savvy visitors engaged.

The Carnegie Museum of Natural History (CMNH), in 2029 is interested to have new ways for the visitors to
experience the information in the exhibits in an interactive way and to emphasize the Anthropocene lens throughout the journey.


02. Problem Space

How might we provide engaging learning experiences in the Carnegie museum of natural history and enable visitors to advocate for a sustainable future?


03. Outcome

Project BEYOND

Beyond is a mixed-reality museum experience rooted in interactive experiential learning and participatory social interactions.

Proof of concept video 


We built mobile app and mixed reality prototypes to test the experiences and interactions. 

Mobile app prototype

Mixed-reality headset Varjo-XR3 prototype


Experience Roadmap



Guide visitors to various exhibition halls.



Provide immersive experience of exhibitions.



Browse exhibits and obtain information through experiential learning.



Add exhibits of interest into pocket and take home.



​Share stories with friends and inspire new visitors. 

Get inspirations

Discover what people encountered in the museum that you might never have imagined.


Get the way to galleries

Lightweight augmented visuals guide visitors to the galleries they want to visit. 


Pick and choose.gif

Grab any item you like 

Choose and drag the exhibit you want to learn about in the diorama.


Browse information more intuitively

Manipulate the exhibit freely with your hands to learn the details in a more intuitive way.


Learn like playing a game

Learn deeper stories behind the scenes and collect exhibits as rewards.


Collect items

Drag exhibits of interest onto the AR watch to collect.


Take your learnings with you

View all exhibits collected in the museum on your mobile phone.


Create your own AR filters

Turn your collections to AR filters and share with your friends on Instagram & Tiktok.


Have fun and share

​Share your stories with other people and inspire new visitors through the AR filters you created. 



User Journey


04. Design Process

CHNH background.png


On-site Observation

We visited CMNH several times to find out pain points from the visitors' perspective, observe visitors, and study how dioramas in the galleries were communicating deeper information.


Through our visits, we found the following general painpoints for visitors:

Need Navigation

Visitors are easy to get lost in the museum and often cannot find their way to the gallery they want to visit.

Lack of Interactivity

Museum experience is passive, and lacks interactions with visitors. It makes it hard for visitors to engage in and learn form museum experience.

Weak Post-Museum Experience

Museum can't offer further exploration beyond the scientific information after leaving the museum.

Lack of Ways to Share Thoughts

Museum curators want to allow people to express their feelings, ask questions, make their own opinions, have conversations even from political, social perspective.

Speaking with Nicole Heller, Associate Curator of Anthropocene Studies for Carnegie Museum of Natural History.

Inviting visitors beyond the first floor

How might we help or trigger visitors to explore the museum's massive collections?

Adding little touches than reinventing

It seems irrelevant, but human activity and other species and nature are interdependent.

Highlighting the interdependent relationship between human and nature

Instead of renovating the whole exhibition hall, adding up-to-date scientific discoveries would be quicker and easier.

Fostering communications among people

Exchanging thoughts and feelings would make visitors' experience richer, and help curators learn what influences they are making.


Meeting with the "Client"

We had the opportunity to speak with Nicole Heller, an associate curator of CMNH. She gave us valuable insights into the current museum experience from a curatorial and exhibition design perspective.


The Hall of Botany

This hall caught our attention as the emphasis on the renovation aligned with the museum's vision and our insights. It demonstrates the interconnectedness of the natural world, which is really important to understand the cascading effect human actions have on natural ecosystems.

This refresh emphasizes the Hall of Botany’s relevance in the context of environmental change and the importance of plants in our daily lives.

Observation & Insights

However, we found that the Hall of Botany was one of the most empty halls despite the renovation in May 2021.

The hall of botany - panorama.

So we spent hours in there to learn why people just walk around and head out. After a while, we color-coded each diorama into two categories, PURPLE for the usefulness of plants, and GREEN for the interdependent relationship.

floor plan.jpg

We also found that the information was not well organized and was not well delivered to visitors:

Text is too heavy

It is too long for young people to read topics like climate change.

Detached Visual Experience

The diorama and its text narrative are not well connected. Visitors need to switch back and forth.

Unintuitive Information

"Can you find it?" games are barely noticeable.

Missing Context

Big topics are lacking context information to be easily understandable.


Opportunity Area

After speaking with Nicole and a few times of museum visits, we summarized our thoughts and came up with three main questions that we aim to tackle in our project.


How might we provide relevant and engaging learning experiences to its visitors?


What experiences we can provide to trigger visitors' reflections on themselves and the relationship between humans & nature?


How might we create a more memorable experience for museum visitors to share with others?


Design Principles

With our research insights in mind, we developed 6 design principles to help us better define our experience.


Make the experience immersive. Actively engage visitors with content, then invite them to express themselves.


Invite community, either online or physical, to advocate a sustainable future.


Recontextualize artifacts with multisensory engagement.


Develop a clear, big-picture narrative that ties each exhibit together and showcases human connections to natural history.


Celebrate and augment existing artifacts, rather than create a separate digital experience.


Develop a clear, big-picture narrative that ties each exhibit together and showcases human connections to natural history.


Concept Generation

We then started generating rough concept ideas around our principles, framework, and technology.

Observation & Insights

Further narrowing down, we developed 4 key stages in the visiting experience that reflect our design principles: Wayfinding, Information browsing, Experiential learning, and Social interaction.


Information Browsing


Concept Generation

Experiential Learning

Social Interaction

We discussed leveraging AR smart glass technology. Smart glasses have the ability to create a more immersive, multi-sensory experience while still allowing users to navigate through space.

Why Mixed Reality

After the technology research, we decided to leverage mixed reality technology, which has the ability to create a more immersive, multi-sensory experience while still allowing users to navigate through space.

Techonology system diagram


User Journey Storyboard

As a starting point, we sketched the user journey and interactions in storyboards. This process helped us imagine not only how users interact with virtual objects, but also about onboarding, signage, systems, customer services, and technologies.


Journey Map

The next step was contextualizing our idea in the Carnegie Museum of Natural History. We designated spots for each step of the journey based on the museum's map.


User Journey Storyboard

After several rounds of refining the visiting experience and storyboarding, we came up with the technology touchpoints and features for the visiting experience:


Brand Design & Visual Styles

We took the natural green of plants from the botany hall to be the primary color, supplemented by bright and warm orange as the accent color.

3D assets are low poly style, which is related to our theme - beyond and connection. We chose the diamond shape as the main visual element.


Multisensory Interactions

Wearable Interface

How can we leverage body-anchored interfaces to create convenience while not obtrusive?

Voice Assistant

How does a digital system interact with voice input to reflect status and reduce users' confusion?

Hand Gesture Interaction

What gesture can be leveraged to make interactions natural and experience more interesting?


Wrist UI

We redesigned the admission ticket in the form of wristband, which is more convenient to carry. It can also be used as the anchor point of the wearable interface.

Current admission ticket of CMNH.

Redesigned wristband ticket.

Voice Assistant - Motion & Status




The right panel is automatically expanded to display functionalities, users can tap on UI buttons or make voice commands.

Waiting for Collection

When the user is pinching items to collect, display the vacant spot to indicate where items should be put.

When the user is speaking, all panels are collapsed, animated voice UI indicates that the system is listening.

When the system is asking questions or giving answers.

Item Collected

When the user releases items above the vacant spot, it is filled with a diamond, the top panel is then expanded to display all collections. The user can tap on collections to put them to different spots of the body for other visitors to see through their MR glasses.


Hand Gesture Interactions

While wearing MR glasses, hands naturally serve as the pointing device in 3D environment. Therefore, we need to redesign the functionalities of standard pointing devices, such as pointing, clicking, dragging, etc.

Based on our storyboard, we summarized the following functions that require hand gestures interactions:


at exhibits


exhibits over from diorama





Interaction Design Principles


Interactions should be easy enough for users to understand and pick up by tapping into natural affordances.

Avoid massive arm movements

Just as the emergence of the mouse replaces the lightpen that requires keeping an arm in the air for a long time. Gesture interactions should not make users too tired, so I try to use micro gestures when designing.

Consistency and standards

Interactions should be consistent and under the same framework to reduce the users'cognitive load.

I. Pointing

Fitts’ law criteria for evaluating a pointing device: Accuracy, Speed.

Index finger pointing

After testing in unity, I found that the accuracy was a big issue as the fingertip was easily blocked by the palm from the perspective of the head-mounted CV module (leap motion). In addition, it conflicts with the tap gesture if the ray shoots from the fingertip. So we gave up this method.

Palm upward pointing

Then I studied this method, but we finally thought that the gesture carries a provocative meaning at the socio-cultural level, or like begging, so we gave up.

Palm pointing

Finally, I tried to make the ray shoot from the center of the palm. This method meets our needs in accuracy and speed.

II. Summoning

Inspired by the interaction technique "slide to confirm" - the command is activated by continuously pressing from one endpoint to another. 
Put the slide bar into 3D space to get a XYZ axis. When the pinch gesture moves a certain distance on the z-axis, the drag function is activated.

III. Manipulation

For interactable objects, we referred to MRTK developer documentation to realize object manipulation.

Gesture Chart

Palm hovering

Raise waist

Pinch and collect

Pinch and drag


3D Interfaces

Standing in the activation zone in front of dioramas to start interacting with exhibits. Users can either tap the diamond next to dioramas to directly play the game related to the diorama, or choose an exhibit to learn its information.

Active Zone Indicator

Show active zone to initiate interactions with dioramas

The difficulty here was how to represent button features as 3D objects. Using only colors and shapes was not clear enough. But we didn't want to make buttons look like a screen UI element (rectangular flat button with labels.) So we put labels under the 3d diamond button using the same style as the wrist UI components to look like a lable of the button components.

Color/ Shape/ Saturation test for different use and variants.


Mobile App Design

We sorted out key screens of mobile app. Users can see other people's thoughts and filters which are presented on the sharing wall in the museum lobby. We included users' visiting history, collection, and filter tabs.

Users can access the information they learned in the museum through history and collection tabs. Also, there are two routes to create AR filters through the collection and filter tabs.
The AR filter creation consists of simple steps: 1. resize, 2. place at a few preset anchor points, 3. record&share.

05. Reflection



A successful design came from continuously unblocking the path through iterations.

Along the way, I learned about paradigms, affordances, and signifiers for emerging AR and VR technology (haptic feedback, proximity, gesture control).

In this new and speculative field, research, trial, error, and critique were critical for me to form design decisions and figure out different ways to prototype our ideas. Every fulfilled objective should rightfully generate even more questions, and I doggedly chased those answers through multiple rounds of hypothesis, testing, conclusions, and next steps.

Next Step



Due to the time limit, I tested most of interactions with a VR headset as the Hololens II development environment was unfriendly for debugging. I wish I could test the usability more on real users.



The concept can apply to any dioramas in any museum, but we might need more research on how to apply to dioramas of different subjects.


Group Experience

Beyond experience is crafted for individuals. However, museums often host group visitors such as field trips, our challenge would be how to provide the same level of engagement to groups.