After Action Review for Mixed Reality Conscious Sedation Simulator


Inter-patient variability is a major contributing factor to adverse outcomes in many areas of medical practice and especially in conscious sedation procedures. For example, two patients of similar height, weight, and age may react very differently to the same sedative, resulting in over sedation or under sedation. In many of these cases, patients who are assessed to be at minimal risk are actually at high risk. Then, problems, such as unexpected respiratory depression, cause an adverse outcome, such as neurologic damage or death. Current training programs mainly train students through lectures and real world experience, which logistically cannot bring students to competency in an acceptable amount of time. To address this problem, this proposal aims to develop an immersive mixed reality (MR) based training simulator for educating students about variability in conscious sedation procedures.

This project is an essential part of the conscious sedation simulator. After Action Review module allows research team to review past training experiences of the users by providing playback of recorded training experiences.

System Components


Upper GI Wizard of Oz Interface

The Wizard of Oz interface will be operated by a member of the research team as a control mechanism accepting communication from the PKPD Variable Modeler, Human Patient Simulator and any other components built or included in the future.  At any time a researcher will be able to send operations to the virtual patient. These actions can also be delayed based upon the Upper GI protocols and sedation reactions.  In addition the WoZ application will control sound, character animations, drug administration, and provide output of the data from the PKPD modeler representing vital statistics.

Interaction with the PKPD Modeler is accomplished using TCP/IP socket connectivity. Data received from the PKPD modeler will be parsed forming the appropriate RenMessage to send to the REN Application.

PKPD Variable Modeler

The PKPD model (PKPD) is an executable program for Windows OS that controls drug and patient variables.  Pharmacokinetics (PK) describes how the drug distributes through and is eliminated by the body.  It is often represented by a multi-compartment numerical model.  Pharmacodynamics (PD) is what the drug does to the body.  The PK model will track drug concentrations in several interconnected compartments that are roughly analogous to different body tissues.  The PD model will read the drug concentration in one of the compartments and scale a physiologic change such as heart rate, breathing, airway obstruction, oxygen saturation, exhaled carbon dioxide or blood pressure.

Upper GI Ren Application

The Upper GI REN Application accepts RenMessage from the WoZ application. The type of RenMessage will dictate the type of animations, sound, and emotional state of our virtual patient.

Event Centric Logging

After Action Review(AAR) module should be event centric. A global data structure is needed to hold all of the events during an exam in chronological order. AAR should have a GUI interface showing the event and its index number so that AAR users can see the events in a time centric view. PKPD data is needed for all the events. For example, event name for moving the leg at maximum distance is “move left leg max”. It will play the animation ‘leg_max’ and will respond with the text message ‘left leg max’. There is no sound or emotion tied to this event yet. We also needs event so that we can re-play the proper animation. Students input should be retrieved too so that we can replay that as well. Here is the general format for the data structure:

Event Index     |    Event Name         |   Event information

1                        Leg Movement Max     PKPD string, animations, speech etc…

Given the log file information, the system can parse it and get certain data. In order to replay the animation, the system has to get the timestamp of the vent and response animation name. There is no way to record all the animation so the only way to replay is to trigger animations based on the log. Also, corresponding PKPD info should be updating in the AAR interface during replay.

AAR can replay the animation like a media player. User should be able to go back and forth in all the events easily like a video media player.


VirtualPatient AAR