Exploring the transformative capabilities of games in healthcare | Playable Futures

This Playable Futures series explores how the concepts, technologies, and creativity from video games are shaping other sectors. All previous articles can be accessed here.

The intersection of video games and training or simulation has been evolving for many years. In the field of healthcare, this has led to the development of numerous training applications to familiarize professionals with various procedures and equipment. Modern game development tools and technologies, which are increasingly adaptable and user-friendly, present new possibilities for significant advancements within the healthcare industry.

At Speech Graphics, a firm known for its expertise in facial animation, the team has been aiding healthcare providers through its Rapport platform. The horizon looks promising as game technology and developers begin to offer more diverse and innovative solutions to support patients, healthcare professionals, and broader medical systems. Founded in 2010, Speech Graphics, an offshoot of the University of Edinburgh's School of Informatics, has established its credibility within the gaming world, particularly through its facial animation technology.

The Rapport platform allows companies to create and animate characters by using AI to align recorded dialogue with facial expressions that reflect the original emotions of the speech. This technology, while beneficial for various industries, has shown significant promise in healthcare. Gregor Hofer, the co-founder and CEO of Speech Graphics, highlights that the cultural integration of games into everyday life contributes to their rising impact in the medical field.

"We’re certainly observing an increasing role for games in healthcare, and it’s an exciting prospect," explains Hofer. “Part of this stems from the cultural ubiquity of games. Nowadays, the most popular films and TV shows are often game-based. People increasingly encounter cultural elements like music and avatars through gaming. The presence of sophisticated, game-like animated characters in various media makes the potential of gaming technology more visible to everyone."

"Games have long been utilized in areas like simulation and training," Hofer continues. "However, advancements in game engines and tools, which are now more powerful and user-friendly, have led to a surge in their application—from Unreal Engine becoming a standard in film production to our own Rapport platform being used in healthcare."

"Digital brings so much potential to the healthcare world, but physical presence will always be a requirement... That's why bringing those elements together through gaming technology is so powerful"

The core technology behind Speech Graphics is designed for gaming, yet the Rapport platform finds uses in other entertainment fields and sectors like advertising. This raises an important question: how does a facial animation technology known for conveying emotional realism fit into a sector centered around the complexity of the human body? Exploring this question highlights the manifold ways in which game technologies could support healthcare professionals in the future.

Firstly, it's crucial to understand what Speech Graphics does. "Our core real-time animation technology analyzes speech and generates corresponding facial expressions on digital characters," Hofer explains. "This system is based on a model of facial muscles required to express speech at different intensities and emotions. Our technology doesn't just automate lip-syncing based on audio; it also considers the entire face's movements to accurately convey emotions."

"Effective communication involves more than just voice and lip movement," Hofer elaborates. "Our technology replicates those additional facial expressions and movements that play a role in expressing emotions, automating the entire facial performance from an audio input. This efficiency is crucial for animating extensive dialogue."

Despite the technological complexity, the capability of Speech Graphics' tools is proven, particularly in the triple-A gaming sector. Moreover, understanding and conveying human emotion is integral to medical practice.

"Our technology's first clear application was in training," recalls Hofer. "Rapport utilizes our technology to efficiently create training content, akin to games, making it more engaging and impactful. We developed medical training tools allowing nurses, doctors, and medical professionals to interact with simulated patients or staff. Effective communication is vital for understanding patient needs or working efficiently within medical teams."

"We're constantly developing the realism and potential of these training tools," Hofer continues. "We started working on systems that help patients more directly. My co-founder Michael Berger collaborated with UC San Francisco and UC Berkeley to develop a system enabling people with severe paralysis to speak and express emotions through a digital avatar. The university teams created a device that interfaces between the patient's brain and our technology."

In an August 2023 publication in Nature, this electrode-based device demonstrated its ability to intercept brain signals meant for facial and vocal muscles. These signals are redirected to a specially adapted form of Speech Graphics' technology, which, after weeks of machine learning training, translates brain signals into expressive communication through a digital avatar, capable of producing speech at 80 words per minute.

"It's incredible to contribute to projects that enhance patients' lives and advance healthcare," Hofer reflects. He adds that such research not only elevates patient care but also generates practical benefits for Speech Graphics. "Our training initiatives generate revenue and, with projects like the medical avatar system, there’s potential for commercial viability. These ambitious endeavors validate our technology's quality and potential, reinforcing our reputation in the gaming sector and beyond."

"Other games companies should see if they can become involved with research and other forward-looking projects around healthcare"

"Other game companies should consider engaging in research and exploratory projects within healthcare," Hofer advises. "Ambitious research projects not only validate technology but also teach valuable lessons applicable to commercial products. Rapport leverages our facial animation tech with character building, AI, and text-to-speech, which benefits medical training and other sectors like retail experiences. The dividends extend in many directions."

However, integrating game technology into healthcare presents its own set of challenges. Game tech needs to be adjusted for comprehension by medical professionals and to integrate with existing healthcare technologies, which might include research-based devices. Addressing privacy and data protection concerns is critical, especially when collaborating with patients and research teams. Additionally, global healthcare regulations vary significantly, with large organizations like the UK's NHS often slow to adopt new technologies.

Despite these hurdles, Hofer insists they are manageable and worthwhile. "Training is a significant business area for us, but the potential applications of game technology in healthcare extend far beyond that. Innovations like VR experiences for pain management are particularly intriguing. We might see more applications of game tech creating experiences that use audio and visual elements to complement traditional treatments."

"Elsewhere, efforts are underway to use VR experiences for patient comfort, movement, and rehabilitation," Hofer continues. "There are instances where game technology creates immersive scenarios to mentally prepare patients for intimidating procedures." These concepts have existed in some form previously. For example, over a decade ago, Blitz Games' TruSim division worked on medical training tools, including a Kinect-based game for custom physiotherapy for recovering soldiers. But with today's burgeoning intersection of game industry expertise and healthcare, opportunities continue to grow exponentially.

With advances in scanning technology and real-time capabilities of game engines, detailed 3D models could soon be displayed on medical devices or through VR headsets, enhancing surgical procedures' precision and understanding. Although medical scans have been used before, the upcoming level of detail, interactivity, and fidelity is unprecedented.

Specialists in breast cancer are making promising strides in this area. Lumpectomies, for instance, currently face a 16-25% failure rate in removing entire tumors. Utilizing scans and AR headsets powered by gaming technology could significantly improve the accuracy of such surgeries, leading to better outcomes and faster recovery times.

Additionally, real-time telepresence in surgeries is gaining traction. Specialists from around the world could use AR or VR to virtually gather in one space, interacting with live patient scans facilitated by game engines. This approach could enhance expertise sharing during complex procedures, potentially leading to significant efficiency gains and cost savings for healthcare systems.

"It's incredible to be a part of improving patients' lives and helping advance healthcare"

Transcranial magnetic stimulation (TMS) is another area ripe for improvement through gaming technology. Currently an imperfect treatment, TMS uses magnetic fields to influence brain activity. Researchers at Stanford University are investigating how game tech could enhance the precision of TMS delivery. They are also exploring the use of lightweight AR devices to provide surgeons with real-time data, precise information, and scan overlays, potentially transforming surgical procedures. Advances in motion-capture and VR tracking systems could also revolutionize remote physiotherapy, making it far more effective.

The mass adoption and trial of game technology, both by developers and millions of players globally, make it an attractive prospect for healthcare innovators. "The term 'metaverse' has various interpretations, but I believe the future of game technology in healthcare lies in blending digital and physical realms," says Hofer. "We will see this integration in training, diagnosis, therapy, healthcare management, and care delivery. Gaming technology's ability to bridge digital and physical elements offers considerable potential for the healthcare sector."

Games have the potential to leave a powerful legacy extending beyond their traditional boundaries. Enhancing patient care, advancing medical practices, and increasing the efficacy of healthcare organizations like the NHS would establish an unmatched legacy for gaming technology.