As we’ve mentioned in previous posts, augmented reality (AR) and virtual reality (VR) are being used to maintain business production, but what about actually fighting the coronavirus? Scientists are using mixed reality to study coronavirus proteins. Doctors are using mixed reality to visualize organ damage and alert the public of coronavirus misconceptions. Newspapers, social media, and search engines have even jumped on board to educate their communities on social distancing and COVID-19 statistics, facts, and guidelines. Here’s a look at the novel and virtuous projects developed by hospitals, academic institutions, and various media platforms.
AR COVID-19 Infected Lungs by George Washington Hospital and Surgical Theater
Much of the news we originally received from the media painted the coronavirus as a disease that just targets those 65 years and older. As new data emerged, scientists and journalists have discovered that that is not the case. Many young people are still admitted to the hospital for respiratory failure and sustain dramatic lung damage post-virus. Even with these findings disseminated throughout the public, many young people still feel invincible and are not taking the necessary precautions to avoid infection. That’s why Dr. Keith Mortman, a thoracic surgical chief from the George Washington University Hospital paired up with Surgical Theater, an AR/VR, healthcare-services startup. Surgical Theater was “the first to combine cutting-edge fighter jet flight simulation technology with a patient’s own anatomy scans, using medical imaging such as MRI, CT and DTI, to create a 360-degree virtual reality reconstruction of the patient’s own anatomy and pathology.” Together, Dr. Mortman and Surgical Theater developed a 3D, AR pair of lungs infected with COVID-19 to alert and sober up the younger population that was not taking the pandemic as seriously as they should.
The translucent blue represents normal lung while the yellow areas depict inflammation from infection. The YouTube video is shocking and intricately rendered, but we wish we could engage with the content in it’s true AR form via a headset, or an app and our phone camera.
AR City Block Teaching Social Distancing by USA Today
The USA Today’s app has provided the AR game “Flatten the Curve” to educate and simulate social distancing. As the user holds their phone camera up, a city street appears on their screen overlaid onto the real environment. One can click on the icons hovering above a grocery store, a pharmacy, a friend’s apartment, etc. to receive a trivia question describing a situation, asking what the appropriate behavioral response is given the current health pandemic and quarantine.
Check out the app here. The AR feature itself worked seamlessly for us—no glitches or crashes—and the graphics were fun albeit basic. However for the informed citizen, the education provided is intuitive and will not supply deep data or current news. Moreover, the AR “game” is a bit hidden on the USA Today app. One won’t likely find the tool unless they’re already looking for it, or one naturally clicks all buttons that say augmented reality.
VR Laboratory Studying COVID-19 by CSIRO and Nanome
Scientists are collaborating all around the world to research the coronavirus and discover a vaccination, some of which use VR laboratories instead of video calling to share information. One VR lab, (a gray dome in this case), that popped up was live streamed to educate listeners and was hosted by Dr. Michael Kuiper and Dr. Michael Bishop. Dr. Kuiper is a biomolecular modeler based in Australia. He works for Data 61, a digital research network belonging to CSIRO, Australia’s federal science agency. Dr. Bishop is a drug discovery specialist in the United States who works for Nanome, a software company that creates VR environments to experiment, design, and educate within the science and technology arena. “We want to understand the function of these proteins in a realistic environment,” said Dr. Kuiper. “In real life, proteins wiggle and jiggle. By loading protein models into Nanome software and running simulations, we can interrogate the binding sites, right down to the amino acid residues and hydrogen bonds, and come up with a model that can potentially be used to identify a therapeutic target.”
No commentary necessary on our end; we bow down to their innovation and mastery that is leading to scientific heroism.
AR Lenses and Geofilters Disseminating COVID-19 Facts and Encouragement by Snapchat
Snapchat has over 200 million daily users whereas The New York Times has somewhere between 4 and 5 million subscribers...meaning Snapchats’s reach is over 40 times that of one of the top newspapers in the United States., Snapchat has enormous power in terms of disseminating accurate information and debunking fake news, and therefore teamed up with the World Health Organization (WHO) to spread coronavirus awareness, not infection. They’ve developed lenses and geofilters—their most actively utilized features—to encourage social distancing and reiterate effective safety precautions. Users can bust myths about the virus and are reminded to wash hands, not touch other people, keep six feet away from others, and to stay at home. One feature even projects an AR safe space: a green circle marking a 6-foot radius around the user in all directions.
Check out the app here. The AR features themselves aren’t particularly informative, but they are addicting—a Snapchat user opens the app an average of 30 times a day. The repetitive instruction of basic information really drives home the point so messages become habits. As entertaining as the AR is, the in-depth information is presented in supplemental texts provided by Snapchat. One has to actually type coronavirus in the search bar and scroll through the categories (e.g. Editions, Publishers, etc.) to find pertinent information. We do appreciate that they’ve created an entirely new category called “Here For You” that places an emphasis on mental as well as physical health and offers relevant, journalistic, COVID-19 material.
An AR Globe Mapping COVID-19 by Yahoo
Yahoo has evolved from a search engine in the 1990s to a content creation machine in the 2000s and beyond. Sitting on a gold mine of knowledge, they have opportunities to educate the public with buried information people may be unaware of or too lazy to dig for. Using data they’ve sourced and consolidated from John Hopkins, Yahoo developed an AR experience to publish the running tallies of coronavirus infections around the world. As a user holds their phone camera up, a 3D, spinning globe with dots marking virus outbreaks appears on their screen overlaid onto the real environment. On either side of the globe, purple boxes hover with up-to-date totals of current coronavirus cases, deaths, and recoveries for each continent.
Check out the link here to view the 3D map on the desktop and access the barcode to view the 3D map on a phone for a true AR experience. The data is useful, but the product itself is more of a novelty project. There are plenty of other online resources that post the coronavirus cases by continent/country/city in plain text without all the bells and whistles. The more clicking and “work” on the user’s end, the more likely the user will abandon the idea for a simpler way to receive information. This 3D globe is an instance when AR just isn’t practical, not to mention the website crashed everytime we moved our phones too quickly. We never actually experienced the globe in AR in our real world environment due to glitches on Yahoo’s end.
A Virtual Nurse to Assist Military by USC Institute of Creative Technologies
When nurses are in short demand and under threat of infection, avatars can be a potential alternative. Sharon Mozgai is the associate director of medical VR at the USC Institute of Creative Technologies and is designing one such VR nurse. The first iteration of this virtual human agent was designed for the military as a “Battle Buddy” and has the potential to be adapted into a “hospital worker” to support medical staff and provide comfort to patients. This virtual human agent “can interpret data from wearable devices, analyze language responses, administer psychological assessment questions and respond accordingly. It could be used to screen sick patients, monitor patient recovery or help hospital staff answer calls or patient questions.”— USC News
We geek out on mixed reality labs and are blown away by the equipment and technology they’ve developed. If only we could have a warehouse filled to the ceiling with AR/VR gadgets…
Saluting the Virus Fighters
While many of us shelter-in-place, we are reminded here about the work being done to stop the spread of COVID-19. From George Washington Hospital to USA Today, we are humbled and impressed by the altruistic efforts institutions are making to keep our communities safe. Just as we are beyond thankful for our frontline workers, we salute those who are working remotely studying the coronavirus and educating the public on best health practices during this pandemic.
2) Nanome, 2020