design brief


BABL is a design challenge project for Microsoft Design Expo 2017 with the prompt Intentional Design for Positive Cultural Impact in Mixed Reality. It asks us to consider the context and implications of the technology we’re designing for, while simultaneously demonstrating its clear value and differentiation.



Mixed reality creates a space for new multisensory experiences that traverse across space and time. Our team explored the mixed reality references from Microsoft Hololens website and brainstormed many fields that we thought mixed reality could be intentionally and responsibly applied to. From numerous domains, we picked embodied learning as the most promising area for mixed reality.

For 4 months, we did 3 main design researches - exploratory, generative, evaluative research - and presented the research takeaways to Microsoft UX design lab.




DOMAIN: Language learning


When picking up the target problem space, we considered 3 main points, which are 1) we should be able to find access points to start research, 2) the topic should have a clear, open problem space and 3) the topic should be interesting and rich enough for all of us to work for the whole semester.


After the long discussion, we all felt that language learning is the most promising domain for embodied learning application because there is a big demand and also has a clear problem in learning experience.





Foreign language is very hard to learn because it takes a lot of time and effort, but hard to master. Most of all, expressions that can be found in textbooks are not really used by native speakers. We focused on this disconnection between textbook language and real spoken language and decided to tackle the problem space.


"How might we help language learning experience easier and more efficient?"



territory map: reframing Language learning



We discussed how we learn our mother tongue and then reconstruct the whole process as a territory map. At the end, we felt that full immersive language learning experience comes from true cultural learning and understanding. We believe that language learning is not just about the four key skills we identified, but that it can lead to a deeper understanding of cultures that are different than our own. From this came the ‘clear need’ of learning a new language and understanding a new culture can be intimidating process and how through our design we could address that.


"Language is a byproduct of social & cultural interaction"






Our exploratory research includes fly-on-the-wall observation, participation, expert interview, student interview and online diary study. We visited a couple of language classes, preschools and language professors and explored how students learn foreign languages and what their pain points are. 




Firstly, through the research, we found out that language and culture are deeply intertwined, as we assumed. Understanding culture really Helps people to connect to new environment/community and to be part of it. And finally, this understanding Motivates people to learn further.

We could hear this repetitively from several interviewees that the most wonderful moment is when they are able to connect to people deeply, understand them, and build relationships with them as part of them. So, one of our design implications might be using meaningful interactions to help people to build relationship easily.

In speaking with our experts and from our secondary research, we learned about the importance of critical thinking in all these pedagogical structures. Being able to actively draw connections and understand both a new culture and your own from a different point of view is key to successfully engaging with and integrating into a new lived-in language and culture.

Finally, one implication of this finding is that we want to help learners express, not translate. Conversational fluency is not about thinking in one’s native tongue and translating it into another language as much as it is about building thoughts from the ground up in this new language.



After the whole exploration, we summarized all findings using Rose (Promising Space) - Bud (Opportunity Space) - Thorn (Pain Point) method and  journey mapping to figure out what might works well in the mixed reality learning environment and what might not. The ultimate purpose of mixed reality experience is to alleviate the painful moment when learners start losing their motivation and interest in language learning, so they constantly enjoy the learning experience. At the end of the exploratory research phase, we defined our design principles as below.





Over the couple of weeks, we had 3 workshops which had 2 activities with 12 participants who were language professors and learners at CMU before we jumped on to the future envisioning. We wanted to see the overall language learning process of each participant and what kind of solutions they had in mind. For the smooth and easy prototyping sessions, we prepared some props to express ideas concrete. 

The ideation workshop consisted of two parts - First, we asked them to create a map or timeline of their own learning experience and add their feelings on each step. After then, we asked everyone to envision the 'magic technology' for an ideal learning experience by sketching scenes or using props 



The first exercise gave us a glimpse into the pain points that current language learners struggle with. First, traditional classroom learning doesn’t always sustain interest and motivation. It can quickly become a chore rather than a desire. Second, many learners are afraid of being judged and are embarrassed when they make mistakes. Especially in certain familial or cultural contexts, fear of failing to reach expectations for language proficiency can prevent people from feeling comfortable enough to try. Lastly, there often is a lack of “real” practice space outside the classroom. As we heard in our exploratory research, practicing the language in context is key to learning and retaining language skills but that context can be hard to come by.

During the second exercise, their ideas circled around themes like sharing cultural experiences, getting instant feedback, practicing, prompts, and coaches for non-verbal behaviors. Some fun ideas included a karaoke mic that can translate songs and capture pronunciation, a virtual assistant for practicing conversations, a projector that can simulate a new environment, and magic goggles that can alter the world around us.



We also visited Duolingo office and had the same workshop with the content designers. We asked them to pick out some happy moments when learning second languages. Some common ones were 1. when they became more integrated into the new environment, 2. when relationships were created with locals, and 3. when they were able to be successful in the new language. 

Besides the workshop, we also have a deep conversation about building more natural and live spoken language database. Because there are so many variations exist in each expression, it would be impossible to make a perfect database. Also, they advised us to keep in mind that even if it's a same language, there are many different accent and expression variations by region. If a learner will live Ireland for a long time, it would be less impressive to local people if he/she speaks in American English.

While synthesizing key findings, three main concept ideas began to emerge: one, providing passive exposure to the language and culture in everyday life; two, an immersive virtual environment for contextual practice; and three, a companion of sorts that would help learners apply what they’ve learned to a real-world context.





After narrowing down the concept, we made decisions about how a user could interact with our system and what that interface looks like. We started by playing with the Hololens to get an idea of what interactions exist now and what we’d have to consider.

Because “tour mode” is such a prominent part of our concept, we bodystormed with Hololens and experience how the tour mode look like. One of us gave a “tour” of our building and “introduced” us to local people while we were all on Skype and Facetime. One of the local people said, she thought it was weird for her to say hi to us without seeing us, but when she was able to see who she was talking to, it was a lot smoother.

Screen Shot 2017-10-24 at 5.24.04 PM.png

We also roleplayed the learner/host with our 3 key scenarios and personas and checked how each of them worked well. We found that the system can bring big value when it connects distant learners to local people before landing in a new place. On the other hand, reading prompts and translations while having a conversation didn’t work well especially when the prompt gets longer.

From the bodystorming, we felt that our solution should focus more on connecting people and encourage users to speak it out. We think bodystorming and enactment are very useful activity for mixed reality because it helps expect how the user experience would like.





One another interesting method we used was rapid paper prototyping. We made various paper props and played with them and see what kind of technology is needed for our solution and what kind of features are needed for immersive experience. Lo-fi prototyping definitely helped a lot when we figured out the feasibility of a scenario. And it was also super helpful for our evaluative workshops.





Before jumping to the hi-fi prototyping, we conducted evaluative researches including concept speed dating and evaluative workshop. To test our concepts, we used paper prototypes and sketches to walk our participants through the experience of using the system. We asked for feedback, noted their instant reactions to each scene, any hesitations they had about it, and probed on social boundaries.


What we learnt from the UI evaluation workshop was different learners have very different needs in terms of what helps them learn best. Some participants learnt through grammar tables, others liked to practice in a real social context. Also, People have set expectations for giving and receiving feedback on their language use. Something we observed was people want to be as engaged in the moment as possible with as little intervention from the system. Interventions and helping elements like grammar tables, caption etc.. might be helpful in an individual practice mode, but this would interrupt a conversation which isn’t good for building relationships.



The next thing we did was narrowing down from three different concepts to a final concept and creating 12 storyboards. We asked 6 potential users about which part resonates you the most and which part do we need to reconsider.

From concept speed dating, we learned a couple of key takeaways.


  • People didn’t think simple passive exposure would help that much. Rather, they wanted to maintain their interest in language learning by exposing themselves into more immersive environment where forcing them to speak the language.

  • When it comes to immersive language environment, they were curious about the sustainability of the solution in terms of business perspective. For example, How to recruit native speakers? Do learners have to pay for hiring a tutor? etc.

  • And for the same reason, we should be able to provide strong motivation for both learners and tutors to keep participating in the immersive learning environment.





In this making phase, we’ve mostly been making decisions about how a user could interact with our system and what that interface looks like. We started by playing with the Hololens to get an idea of what interactions exist now and what we’d have to consider and then mapped out the system and made a list of all the features we wanted to include.

We referred to the gesture interaction guide from Microsoft Hololens Developer Portal and mix-and-matched interaction combinations for features including doodling, bookmarking scenes and targeting objects, etc.   


We then moved into paper prototyping for MR, eventually deciding on a tool “tray” that can be expanded and retracted into the bottom/side of your peripheral vision. The tray would contain tools like doodle, pin, stop sharing view, switch perspective, and end call.


We also considered the screen UX flow. We each created a wireframe flow for the sign-on and matching process and then decided together on a singular flow.

babl flow.jpg


At the end, we designed 2 sets of UI elements for practice mode and tour mode and branded our product as "babl", using the color combination of white and orange. Since we are designing the UIs overlaying in the real space, we focused on improving the readability of each icons. 



We also made a simple prototype with Unity and checked how the overall look and feel of our UI elements look like in the real world with Hololens. We learned that the system should be able to adjust the UI colors according to the background color and brightness since the contrast became lower in the light outdoor environment.  However, because of its limited feature and view field, we had to made an elaborated prototypes with After Effects.


And finally we shot the concept video that explains how BABL works. We used blue screen to combine the videos of two different spaces. It actually worked really well so I didn't have to use rotoscoping to illustrate the MR moment!






BABL started from the bold idea of bridging between learners and local hosts at distance. While making the concept video, we found it really awkward to ask people to let us shoot them in public. Some people actually feel intimidated to being seen in the video. Besides technical limitation and social awkwardness, we felt that establishing social norm is very important to apply mixed reality technology in the real world. In the future, we might experience the privacy issue triggered by mixed reality, just like Google Glass became a controversial technology because of its camera recording feature.