The release of Azure Kinect promised significant advancements in motion tracking and AI-powered capabilities, making it an exciting tool for our R&D project. As a studio rooted in animation and interactive design, we were drawn to its potential for real-time visualisation and gamified environments. The project aimed to investigate gesture-based interactions, mocap applications, and visualisation techniques within Unity, with the goal of pushing the boundaries of user engagement and artistic expression.
While the Azure Kinect showed promise on paper, its practical limitations quickly became evident. Our exploration of gesture recognition and motion capture highlighted challenges in achieving the precision and responsiveness required for seamless interactivity. During the lockdown, we worked remotely with limited hardware, streaming demos and sharing captured footage. These constraints naturally shifted our focus toward visualisation opportunities, particularly in creating immersive particle-based systems using Unity’s VFX graph.
The project evolved into a deep dive into visual storytelling through particle systems. By targeting specific body joints, we developed innovative ways to interact with digital environments, from holographic representations of the human form to abstract, emergent animations. These explorations demonstrated the creative potential of combining depth sensors with real-time graphics, offering both tangible and ethereal user experiences. Although gesture and mocap applications felt less promising, the visualisation work informed key projects like Holonic Fields and Wiggle Room.
This project underscored the importance of adapting technology to fit creative goals. Gesture-based inputs often felt cumbersome, but passive, intuitive interactions—such as recognising overall movement patterns—opened up new possibilities for user engagement. We learned to design systems that prioritise fluidity and accessibility, embracing the limitations of the tools while finding ways to push their boundaries creatively.
The R&D initiative with Azure Kinect highlighted the value of exploring emerging technologies, even when the results deviate from initial expectations. The insights gained not only enhanced our technical capabilities but also deepened our understanding of creating meaningful interactions within immersive environments.
