* Note: I give a small introduction for those unaware of AR/VR. If you are here to find out about options to start recording experiences for VR jump to the last paragraph ;)
As Moore’s law continues to predict the tremendous advancements in technology, our imagination and creativity is becoming the essential catalyst in converging them to target particular goals. In order for people to understand the potential of Moore’s law I recommend for people to imagine where they would be in 30 linear steps or in 30 exponential steps. Here is a slide I made.
One of the beauties of Innovation is that exciting ideas arise when we mix different fields. A perfect example of such is Pumps & Pipes. P&P is an association of Medical, Energy, Aerospace, Academic, Community professionals and leaders that brings together professional groups who may not otherwise have the opportunity to interact for the transfer of knowledge and technology knowhow.The ability to see beyond a particular profession and understand that we can incorporate methods, processes, theories of different ones into our own is what ignites the innovation spark.
In case you have 5 minutes take a look at this video since it applies to any profession.
When Google Glass came out, it was called an explorer program for a reason. It was meant to allow particular dreamers to use the technology in different ways. In my case I quickly applied it to the Medical Realm. Using augmented reality to help bystanders give efficient CPR we created “CPRGLASS” A few months later, we demonstrated that potential it had in TeleMentoring medical procedures. “Visionary–Google Glass Moves From Concept To Clinical Care”.
Since then I have been trying to incorporate Augmented Reality in different settings. Experimenting with different smartglasses (Atheer Labs, META glasses, ODG) has been important and each one has its strengths and weaknesses. Without a doubt there is tremendous potential in Augmented Reality but technology is moving in a very fast pace and therefore choosing a technology that has the least resistance for implementation is crucial to make the innovation process more exciting and less of a hurdle jumping scenario.
This is the reason why focusing on Virtual Reality was an intuitive approach. The concept of AR/VR is not new and has been explored since 1968 with the creation of “The Sword of Damocles” by Ivan Sutherland. Since then, it has been following Moore’s law which has allowed a device that was so heavy that needed to be suspended from the ceiling to a normal pair of glasses. Below is “The Sword of Damocles” (and you thought glass looked weird)
There are many different options of VR headset and they will continue to grow. (This is a good List) In addition eye tracking options are now being created and FOVE is one of them (still in Kickstarter and met funding goals in less than 3 days)
How can we use Virtual Reality to improve Patient Care?
Many options exist regarding applications of Virtual Reality in Healthcare; Psychology/Psychiatry, Meditation, Medical Education. As someone who enjoys videogames (by the way your average gamer is 35 years old now) I have always dreamed of incorporating a fun experience to learning and Virtual Reality, in my opinion is just the way to do it. Initially it would be easier to record it and in the future to create a Computer Generated Interphase that we could interact with. This is why I explored recording a 360 immersive experience with the help of Jaunt VR to film 2 cases. More on this can be found here “Using Virtual Reality in Interventional Cardiology”
Below is a Screen Recording to try and simulate what a VR experience would be using the right gear. It is really not the same thing but I believe it gives an idea.
As I presented the findings people have asked how it is that they could create such videos. Well, Moore’s law is allowing everyone to do this now and a lot of news has been unraveled with the last Google I/O conference. Relevant to the subject is the Google Jump. Jump is an entire ecosystem for creating virtual reality video. As Clay Bavor, Google’s vice president of product, detailed on stage Jump consists of three parts: 1) the camera rig itself 2) software that automatically assembles and processes the footage, and 3) the player.
Virtual Reality now has the attention of a lot of people since this has happened and many have been asking me ways to capture 360 immersive experiences since not many have the opportunity have work with a company like Jaunt VR (which I am very grateful).
With the recent acquisition of Kolor by GoPro it seems that the consumer approach to capture Virtual Reality relies on using several GoPro cameras and it is not as expensive as you may think.
Once the setup is assembled, platforms like jump will allow consumers to stich and assemble the footage to later share with the world. The beauty of this is that experiencing the footage will be accessible to millions of people. Basically anyone with a smartphone will be able to do so if they have something like GoogleCardboard.
The impact that this can have on different fields is only limited by the individual’s creativity. At the moment I am exploring this for medical education, but as we start exploring and the technology continues to evolve we will see the convergence of this technology with ubiquitous networking, robotics, sensors and haptic feedback which will initially enrich our learning experience and in the future could potentially have a keyrole in medical interventions/procedures.
Talking about Convergence, I am beginning to explore the capacity of EchoPixel. It is an amazing empowering technology that goes hand in hand with Virtual Reality and Augmented Reality. More about it can be found in this Wired article “Holograms Will Let Doctors See 3-D Views of Our Insides” as well as this one published in Newsweek “Bringing Virtual Anatomy to the Operating Room”
My recommendation to everyone would be get GoogleCardboard now. It is available for both Android and iOS. Once you experience it then you will understand the potential and depending on your profession you might find exciting uses for it which are now becoming widely available! Good Luck!
Relevant to this is an interview with Michael Gibson on AR/VR
Since the release of the Google Glass explorer program in 2013 different approaches for their implementation in medicine have been explored by medical pioneers. Over the last 2 years we have witnessed a boom in different augmented reality technologies. In addition we also have witnessed the rise of virtual reality.
While augmented reality (AR), in a nutshell, is the overlay of digital content into one’s field of view virtual reality (VR) completely replaces the individual’s field of view. VR is a powerful brainhack that can lead the individual’s brain to believe he is in a completely different location. It is also interesting to mention that we are activating our brains in ways we have never done before and scientists are currently exploring this realm.
A common denominator that I have found regarding the implementation of new technologies into whichever field, is that an individual needs to experience it to understand what it can truly do, what it can become and then how to use it in his profession or environment. When I initially tried to describe the augmented reality experience to the medical community, words never did justice to it. Once they experienced it, the most common reaction was “Now I get it”. The same concept applies to Virtual Reality, but in my opinion in a far more drastic manner. One thing is to add content into one’s field of view and the other is to completely shock the brain’s conception of environment and teleport it into another. This Virtual environment has now become the “true” environment for a period of time.
Shortly after I had the opportunity to explore Virtual Reality I realized the enormous potential it had in clinical practice. From potential therapeutic implications in the psychiatric realm, heart failure and cardiac rehabilitation to medical education. For this reason I decided to find an accessible way to start exploring and working with such. The most convenient option at the time was using the Gear VR Innovator edition for Note 4.
Depending on one’s environment, location and network, a different approach to how they should use it will vary. In my case, as an interventional cardiologist, I have seen training courses in which physicians travel form different states, even countries to a particular location to learn new procedure and techniques. For this reason I saw a window of opportunity in such space. If we learn and become experts by repetition, Virtual Reality is just the perfect setting for this. The tangents that can emanate from such are vast, to the point we could potentially add haptic feedback for additional tactile sensation incorporation into the learning experience. In a future, even telementoring-telerobotic applications.
In order to test the hypothesis that Virtual Reality will have an important role in the future of medical education I researched ways and techniques to capture a 360 immersive video during particular procedures. After exploring ways and techniques to do so I was lucky to be introduced to Jaunt VR. Needless to say they automatically understood the potential of how this could help medical education and ultimately improve patient care. We explored two Scenarios 1) The ultrasound guided initiative, priceless in deciding approach and exit strategy in particular interventions 2) Preparing and deploying the Medtronic CoreValve.
Once the filming was done, I had an enthusiastic expectation of what the outcome would be but I was speechless after witnessing it in VR. I had the opportunity to give a keynote at the Society for Coronary Angiography and Interventions (SCAI) in San Diego in May 6 regarding potential in implementing exponential technologies into medicine, and this recent Virtual Reality was one of them. Trying to explain and transmit these findings as well as other emerging technologies with words is close to impossible. It is like trying explaining sound to someone with hearing impairment. The brain perceives Virtual Reality in a different way than our true reality, and when an individual perceives it “awakens” a new realm and with it the potential. In order for me to relay my message in the most efficient manner, I decided to take the Gear VR to the meeting in order to get interested individuals to test the experience and understand what the words meant. From a total of 20 individuals (most MDs) who have experienced the cases with the VR Headset, their response was always a “WOW” like reaction followed by “This is amazing” and the “Now I get it”.
By understanding the implication of being able to re-live a procedure as a secondary operator, attendees quickly understood the potential in medical education. For those that did interventional procedures, they seamlessly understood how this could help in acquiring and learning new techniques.
To our knowledge, this was the first time that true 360 immersive technology has been used and incorporated to Oculus Rift type technology. The proof of concept and acceptance by the Interventional Community at SCAI was a unanimous positive one which will lead to future projects later to be announced.
There are several reasons why I believe virtual reality will be an extremely disruptive technology in medicine, and it will start with education. In order for people to understand why this is the case they first need to understand what exactly virtual reality is. I would describe virtual reality as a technology that allows the individual to ‘teleport’ to a different environment as an invisible spectator. Effective VR is a ‘brain hack’ so strong that it can actually trick the user into believing they are someplace else, which potential leads to interesting therapeutic applications1. A virtual reality scene can be either computer-generated or captured with 360º cameras. Both approaches have tremendous potential for medical applications.
When I experienced Virtual Reality for the first time, the first thought that came to mind was the tremendous potential of such technology in medical education. For this reason I decided to explore possibilities to capture an immersive 360 environment. Shortly after my search began I quickly realized the task was more daunting than I initially expected. After evaluating the options one quickly stood out from the rest. Jaunt VR, a startup in Palo Alto, CA developing 360º video technology and considered by many as the leader in cinematic VR production. According to their senior engineer, Adam Somers, “Jaunt VR develops end-to-end solutions to enable the capture, production, and distribution of cinematic VR content. There is tremendous potential to apply live-action VR recordings to a number of industries, including medicine. Our collaboration with Dr. Assad represents an important first-step in the exploration of immersive media for educational and therapeutic applications.”
On March 25, 2015 at El Camino Hospital, Mountain View, California we explored the potential of virtual reality by capturing important teaching pearls in interventional procedures that could potentially lead to better patient care and outcomes.
- The Ultrasound Guided Access initiative. This I learned in my advanced peripheral endovascular fellowship with Dr. James Joye and has proven to be a very valuable technique that without a doubt can avoid disastrous complications if used properly.
- With the assistance of the Medtronic team we were able to setup up a simulator to evaluate feasibility. We explored capturing the steps involved in the preparation and deployment of a Medtronic CoreValve. The value of capturing and perfecting this content in VR could completely change the dynamics of training courses, not only for this device but for all others as well.
Once the footage was obtained and processed by the Jaunt VR team I had the opportunity to experience it through Oculus technology. The results surpassed my expectations and therefore I stand by my initial vision. I firmly believe that with the right leadership and initiative this will be a game-changer in medicine. Why you may ask?
- Physicians learn procedures by repetition. The concept of ‘see one, do one, teach one’ is, in my opinion, is outdated. Maybe this worked in 1850 since it was the only option, but it is unacceptable in this day of age. VR allows the in-person demonstration to be repeated as many times as desired, and you can change your point-of-view with each viewing to take in even more information than if you were actually present for the procedure.
- The fact that one can be next to the individual performing the procedure over and over and focus on different aspects is priceless. Unlike traditional video displays, VR provides a human-scale, interactive facsimile that is potentially indistinguishable from reality.
- Exponential growth in technology drives down the price of many devices, including smartphones. By 2020 80% of the adult population will have a smartphone2. This is important because along with a $5 to $20 the Google Cardboard, your smartphone turns into a VR Headset3. Most recent android phones support Cardboard and you can try it for yourself with Jaunt VR’s publically available apps4. This does not mean that everyone should be producing content but it does mean that everyone can learn from it, from simple stitching to familiarizing oneself with more advanced technologies like CoreValve implantation.
Generating and sharing content has the potential to empower and educate thousands, maybe even millions, by experiencing procedures as the second operator or from any perspective that the operator deems to be best. As Dr. Oliver Aalami clearly put it “The reality is that it would take me one hour to review every step of a complex procedure that a video could show me in 10 minutes. In my opinion that is the true benefit…more immersive education…better comprehension in less time”
Technology is changing at an exponential rate. Just think of what merging VR technology with haptic feedback, ubiquitous networking, and tele-robotics can enable. Initially, education is the obvious application, but there is no reason why performing surgeries this way could not be done. Consider, for example, the successful telementoring project done back in 2003 between Brazil and the United States5.
I would like to emphasize that this would not have been possible without the support from Jaunt VR who allowed me to dream a little and get a taste of what I believe will be the beginning of a great thing for medical education.
In May 6, 2015 I will give a talk at SCAI regarding the incorporation of implementation of exponential thinking and technologies. Here I will share the immersive 360 experience with my Samsung Gear VR and other means as well.
Virtual Reality is picking up tremendous speed and it is easy to see why. It’s potential is tremendous. In the following slides you will see once again the exponential trend in technology and its implementation impact on different fields. From a device created by Ivan Sutherland in 1965, which was so heavy that it needed to be attached to the ceiling to the Oculus Rift prototype created in a garage by Palmer Luckey (18) who then sold it for 2 billion dollars to Facebook.
Observe the number of devices created, in addition, they are now being incorporated to brain computer interfaces. This opens a whole new field for research potential, medical education, and therapeutic treatments. Imagine what this could do in a nursing home, cardiac rehabilitation? Virtually Teleporting patients to a beach, museum, or the rehab center?
The speed in which this and other technologies get incorporated into our profession is directly proportional to the number of individuals trying to do so. One of the beauty’s of VR is that anyone can experiment with it if they so please for 5-30dlls. Just get google cardboard and download the app to your android phone and see what the big fuzz is about. Got an iphone? No worries, this technology is soon to arrive to iOS also. Apple plans here, seems it will be available in the iPhone 6s.
I am currently working on interesting projects that I will mention later, but I wanted to create a small introduction since I have been getting a lot of questions!
Imagine piloting an airplane with your mind. No hands on the controls, no feet on the pedals, just thoughts and then movement through the sky. For all of you who aren’t Jan Scheuermann, this is just fantasy.
Editor’s note: Sasa Marinkovic is the head of software marketing for the computing and graphics group at AMD. I’m standing on the roof of a tall building, looking down at the street hundreds of feet below as I catch my breath.