AR lens studio experiments
The technology required to support fully augmented experiences has grown at a rapid pace in just a few years. As we explore the potential with our clients, we are fortunate to have access some very powerful consumer-level tools that if used correctly can be as powerful as pro-level. One of our favorite tools for quickly building concepts is Snapchat’s Lens Studio.
What’s up, dawg
Along with Meta’s Spark AR, these seemingly simple software tools gave us some of the fastest access to body tracking, face tracking, segmentation, and cloth simulation with a full set of UI tools, and robust coding IDE, allowing for a wide range of triggers, motion controls, and user experiences. And, the software gets frequently updated with regular feature additions and improvements to the existing ones.
Most of the models we use in these experiences are created in DCC applications such as Maya, Zbrush, Substance Painter. For a few applications, we use Houdini and Blender, both having some of the best tools for rigging and creating exports for animated GLTF files. The files are imported directly into Lens Studio, where the models are linked to the provided tracking frameworks in Lens Studio. For example, when we need to do facial tracking, we used Maya’s blend shape tools, and connect them to be activated triggers on the face tracker in Lens Studio.
Much of the underlying methods are in a “black box” in tools such as Lens Studio and Meta’s tools. Once the client approves these initial “proof of concepts”, work begins on developing a fully-functional tool. We use threeJS, MediaPipe, and many of the libraries available to identify and track body parts, finger and facial features. This takes a great deal more resources to integrate into our larger experiences, but allow for more flexibility and scale than these consumer level tools. However, for smaller experiments, these tools prove extremely valuable for the entry level to our production pipeline.
While the research into AR, particularly body segmentation is active and improving with each iteration, there are still some challenges with the technology. A complete VFX or game studio motion tracking system requires dozens of infrared cameras, placed around the capture space, and the subject needs to be covered in reflective balls. The camera data is triangulated, and the locations of the spheres are reconstructed in virtual space, In other systems, special suits are worn, with embedded sensors that use accelerometers and gyros to track the wearer’s movement. The data is processed, and often “cleaned” by artists, removing noisy data, and adding missing motion for the gaps in the data. This is particularly challenging for the optical motion capture suits, where markers become occluded by props, other actors, or even the floor.
Machine Learning motion tracking libraries and datasets such as COCO (Microsoft Common Objects in Context), MPII Human Pose, JHMDB (Joint-annotated Human Motion Data Base), LSP (Leeds Sports Pose), and others will become more reliable, easier to set up, and read data into game engines such as Unity. Reliable gesture recognition continues to be a challenge in developing interactive experiences, and the research into Machine Learning will become increasingly important to our work. Currently, a base knowledge of Python is required to access the functions in the datasets.
Author: Chris Tedin