Behind the Build: Providing Real Solutions for Virtual Worlds

Video games are more than just a pastime; they're the intersection of art, storytelling, and cutting-edge technology, creating immersive worlds that captivate and challenge players. I'm Tommy, a Senior Technical Account Manager here at MSquared, and my role is to act as the crucial bridge between our valued customers and our innovative technical teams. I thrive on delivering tailored solutions to complex customer problems, ensuring seamless integration and optimal performance (and, yes, occasionally chasing up an invoice or two to keep things running smoothly!).

Throughout my time at MSquared, I've had the privilege of being involved in every major event we've delivered, experiencing firsthand the evolution and impact of our work. My journey includes managing the "tech war rooms" for our groundbreaking first major experience with Yuga Labs, where every detail had to be perfectly orchestrated. I also played a key role in capacity management and onboarding new pixel streaming providers for our significant event with MLB, ensuring that tens of thousands of fans could experience the action without a hitch. Most recently, I had the incredible opportunity to program manage MSquared’s breakthrough event with the BBC, a project that pushed the boundaries of what's possible. So, you could say I definitely have some stories to tell!

Not unlike the intricate puzzles that live within my favorite videogame franchises, often the most effective solution to delivering "a thing" isn't immediately obvious. The path forward can be abstract, requiring significant out-of-the-box thinking, a willingness to challenge assumptions, or even the humility to backtrack and meticulously re-examine every step to ensure no crucial clue has been missed. This methodical and creative approach is what we at MSquared bring to every challenge, ensuring that we not only solve problems but also innovate and exceed expectations for our customers. I’d like to walk you through a few of these puzzles and what we did to solve them.

The Experience

To say the BBC event was important would be an understatement. It’s not often you get to work with an iconic organization such as this (I mean the BBC are as synonymous with the UK as Jaffa Cakes). So when two thirds of the way through a 30 month project my contact at the BBC said “hey, let's put on a digital concert to showcase what we can build together” I knew that this was game time!

Our collaboration with the BBC on their digital concert event was a big deal, showcasing the full spectrum of MSquared's capabilities. The core objective was to create an immersive, interactive digital concert hall where users could experience a live performance in a completely new way.

Within a few weeks we had already delivered several key components to ensure users would have the best time possible, this included:

Digital Concert Hall: Utilizing the ease of our technology, where essentially “if you can build it in Unreal you can build it on MSquared” we meticulously designed and built a bespoke virtual concert hall, focusing on creating an atmosphere that blended the grandeur of a traditional venue with the flexibility and innovation of a digital space and of course allowing for thousands of users to share the same virtual world.
Avatar Implementation: To personalize the experience, we integrated with MSquared’s robust avatar system, allowing users to customize their digital representations within the concert hall. This fostered a sense of community and individual presence, enabling attendees to interact and engage with each other in the virtual space.
Live Performance Integration: The critical element of the event was the seamless integration of a live stream of a physical musical performance directly into our digital concert hall. This required low-latency streaming solutions, robust encoding, and meticulous synchronization to ensure the virtual audience experienced the live event in real-time, free from delays or disruptions. It was a good thing that our software supports so many integrations as implementing the video screens was super easy, leaning on our millicast integration.

With the above implemented, I was confident that we could deliver something cool, something that would attract younger audiences and those who traditionally go to classical music concerts. The concert itself also made sense, “The Sounds of Videogames” a concert series playing everything from God of War to Tetris.

Then came a request. At face value, relatively innocuous but one which resulted in…

The Motion Capture Conundrum

“Wouldn’t it be cool if we mocapped the conductor?” said my contact at the BBC. I mean it makes sense. This is meant to be an immersive experience and beyond strapping a VR headset to your face having an avatar mirroring the expressions of the conductor in real-time can’t be more immersive.

My mind cast back to a year before where an MSquared tech demo spike produced an integration with mocap technology. I could have easily said “no”, “this isn’t possible” but where’s the fun in that?

Integrating motion capture technology, especially in a live, real-time environment for a digital concert, presented a unique set of challenges. While our previous tech demo showed the possibility of integration, scaling that to a live event with thousands of concurrent users was a different beast entirely. The primary difficulties we faced included:

Latency and Real-time Processing: Capturing a conductor's subtle movements and translating them into an avatar's actions in real-time, without any noticeable delay for the audience, was paramount. This required extremely low-latency data transmission and processing, pushing the limits of our existing infrastructure. We needed to ensure that the data from the motion capture cameras was quickly and accurately interpreted by our systems and then rendered for every user.
Data Volume and Network Bandwidth: Motion capture data can be incredibly rich and therefore heavy. Streaming this constant flow of detailed positional and rotational data for even a single avatar, let alone ensuring it was consistent across all users, placed a significant strain on network bandwidth. We had to optimize the data stream to be as efficient as possible without compromising fidelity.
Accuracy and Fidelity: The goal wasn't just to make an avatar move, but to accurately reflect the nuances of the conductor's performance. Small hand gestures, shifts in posture, and the subtle energy of their movements all needed to be translated faithfully. Achieving this level of fidelity required precise calibration and robust tracking to avoid any jarring or unnatural movements from the virtual conductor.
Testing: This was for a live event. This was for a live concert event where the conductor's time was primarily dedicated to ... conducting. How on earth were we going to test this adequately enough without detracting from the performance itself?

On paper the solution to the above puzzle seems daunting. In reality the solution was simple, MSquared’s capable and enthusiastic engineering team. You see MSquared is not only its incredible technology but also its incredible people and once I posed the challenge of navigating some serious “here be dragons” areas of our platform I was inundated with volunteers. Our engine team got to work with re-enabling the mocap integration into our new version of the platform, our rendering team got to work and established the best way a mocapped entity could be replicated to thousands of users and our principal engineers offered to be on site with the orchestra to install, implement and do some serious coding to bring the whole thing together. The result?

Looking back, we picked up a few key things that really shape how we tackle new projects:

Teamwork Makes the Dream Work: Seriously, the BBC gig and all our big projects only clicked because our tech folks, project managers, and especially our clients, worked so well together. Keeping lines of communication open and being on the same page is super important.
Don't Shy Away from a Challenge: What sometimes looks like a massive tech hurdle at first usually turns into a chance to do something awesome and new. By pushing ourselves with a "yeah, we can figure this out" attitude, we keep making our platform and what we can do even better.
Stay Flexible and Adaptable: The virtual world moves fast, so we've gotta be nimble. Being able to quickly switch gears, rethink things, and roll with new requests – even last-minute ones – is huge for delivering killer experiences.
Our People Are Our Superpower: That motion capture challenge showed how much the MSquared engineering team's smarts, hard work, and passion mean to us. Our tech is powerful, but it's really the amazing folks behind it who make the magic happen.

These takeaways really drive home our commitment to building solid solutions for virtual worlds, and we're always pushing to innovate and go above and beyond for our partners.