Author + information
- Received January 21, 2014
- Revision received March 7, 2014
- Accepted March 7, 2014
- Published online August 1, 2014.
- Partho P. Sengupta, MD, DM∗∗ (, )
- Nupoor Narula, MD∗,
- Karen Modesto, MD∗,
- Rami Doukky, MD, MSc†,
- Sarah Doherty‡,
- Jeffery Soble, MD†,‡ and
- Jagat Narula, MD, PhD∗
- ∗Zena and Michael A. Wiener Cardiovascular Institute, Icahn School of Medicine at Mount Sinai, New York, New York
- †Division of Cardiology, Rush University Medical Center, Chicago, Illinois
- ‡TeleHealthRobotics, Chicago, Illinois
- ↵∗Reprint requests and correspondence:
Dr. Partho P. Sengupta, Icahn School of Medicine at Mount Sinai, One Gustave L. Levy Place, PO Box 1030, New York, New York 10029.
We discuss the concept of ultrasound imaging at a distance by presenting the evaluation of a customized, lightweight, human-safe robotic arm for low-force, long-distance, telerobotic ultrasonography. We undertook intercity and trans-Atlantic telerobotic ultrasound simulation from master stations located in New York, New York and Munich, Germany, and imaged a phantom and a human volunteer located at a slave station in Burlington, Massachusetts, using standard Internet bandwidth <100 Mbps and <50 Mbps, respectively. The data from the robotic arm were tracked for understanding the time efficiency of the human interactions at the master stations. Comparison of a beginner in ultrasound operation with a professional sonographer revealed that although proficiency in using ultrasound was not a prerequisite for operating the robotic arm, previous experience in using clinical ultrasound was associated with progressively lower probe maneuvering time and speed due to an enhanced ability of the veteran operator in adjusting the finer angular motions of the probe. These results suggest that long-distance telerobotic echocardiography over a local nondedicated Internet bandwidth is feasible and can be rapidly learned by sonographers for cost-effective resource utilization.
Clinical applications of ultrasound have grown and become an integral part of contemporary clinical medicine. In addition to diagnostic applications, ultrasound is widely used for procedural planning, guidance, and patient monitoring. However, the obvious advantages in safety, cost, and portability of ultrasound systems are offset by the need for education and training for providing high-quality examinations and proper interpretations. This is particularly relevant for clinical assessments in underserved areas (where resources and skills are limited), evaluation of patients with complex disease (such as guidance for acquired structural and congenital heart disease interventions), and workflow-related scenarios (such as performance of ultrasound imaging after-hours).
Improvements in computer and digital technology have revolutionized the storage, transmission, and interpretation of medical images, including ultrasound, allowing images to be reviewed at remote locations (1). The combination of digital imaging and telerobotics may expand the use of ultrasound even further, allowing an expert to perform an examination from a distance, virtualizing both ultrasound image acquisition and interpretation. The robotic arm of the telerobotic ultrasound system must be intrinsically safe for patient interaction (i.e., limited in weight and force-generation capability), able to hold a stable contact position on the body, and able to be manipulated with very fine adjustments through the necessary degrees of freedom for optimizing image quality. It will require safety systems that prevent accidental movement of the probe or robotic arm, and it must be able to provide the operator with precise spatial location (e.g., infra-red localization) and force-feedback (e.g., pressure sensors, haptic). Although robotic ultrasound systems have been under investigation for a long time, most were developed as guidance systems for surgical procedures. Recent technological advances and cost reductions in robotic hardware and control systems have enabled development of affordable and effective telerobotic ultrasound systems to cater to broader clinical practice. These updates will help bring much-needed resources and expertise to remote (e.g., rural, frontier) and dangerous (e.g., battle zones) locations and expand the applicability of community screening procedures (e.g., carotid intima-media thickness and plaque imaging), which are currently largely constrained by the availability and throughput of trained operators. In addition to remote locations, tele-echocardiography can aid practice within institutions, providing timely studies for patients who are hospitalized or waiting in emergency triage locations. Moreover, it could also allow the cardiologists to collaborate virtually and on-demand during advanced procedures (e.g., elective cardiac structural interventions). During the structural interventions performed under fluoroscopic guidance, the ability to hold an ultrasound probe in a stable position over prolonged periods may be particularly helpful in reducing undue radiation exposure.
There are several factors that may affect the performance of remote telerobotic ultrasound. First, exercising remote control of electromechanical systems through software over long distances requires optimal bandwidth. The majority of systems in the past have advocated the use of a high-bandwidth, dedicated telecommunication line or a dedicated high-speed terrestrial fiberoptic network for remote operations, which may limit applicability (2). Although the Internet is low-cost and widely available, there are few data on using it with nondedicated bandwidth for long-distance telerobotic ultrasound. Second, remote telerobotic ultrasound is dependent on the intrinsic capabilities of the robotics, as well as the human factors that influence the interaction with the control systems and the software interface. The robotic arm does not perform the examination but simply acts as an extension for mechanizing the remote operator’s intentions. The amount of training required for successful navigation of the remote robotic arms over the Internet by clinical personnel may vary. In particular, it is unclear whether experience with clinical ultrasound will affect the performance of steering the ultrasound probe with a robotic arm.
In May 2013, we used a new prototype medical telerobotic ultrasound system and tested the feasibility for long-distance imaging over local Internet service. A vascular ultrasound phantom at a slave station in Burlington, Massachusetts (near Boston), was imaged successively from 2 long-distance master stations, 1 created in New York, New York and 1 at a trans-Atlantic site in Munich, Germany. We also examined the blinded interactions of a beginner sonographer and an expert sonographer in their operation of the robotic arm’s software control system to determine the learning curve for the control system and understand the nuances of robotic arm motion that correlated with the overall efficiency of remote ultrasound probe navigation.
The Slave Station
The telerobotic platform (TeleHealthRobotics, Chicago, Illinois) had several core components including a lightweight (2.0 kg), 7-degree-of-freedom, servo-actuated robotic arm (Cyton Gamma configuration, Energid Technologies Corporation, Cambridge, Massachusetts) (Fig. 1). The arm was capable of (low) 1.5 kg payload, low force, and composed of minimum power servos operating at its joints. The robotic arm was controlled over the Internet, supported by video and sound feed that would allow the operator to interact with the subject in real time while acquiring real-time ultrasound images.
The arm was attached to a customized end effector holder for housing a linear L38/10-5 MHz transducer from a SonoSite ultrasound processing unit (Titan model no. PO4101-26, Philips Healthcare, Bothell, Washington) with video output capability (Fig. 1).
The camera system included a combination of Hitachi and Dragonfly cameras (Point Grey, Richmond, British Columbia, Canada) positioned in front of the phantom, overhead, and to the side of the phantom (facing the phantom). A personal computer running Windows was also set up in the slave station for collecting information and transmitting this information to the control system in the master station.
The Master Station
The telerobotic system in the slave station was controlled by remote operators stationed at a generic personal computer (Fig. 2). A robotic control, safety, and sensing software system used for local or remote probe navigation (remote control user interface) was installed on the personal computer. This interface was adjustable in size and configurations for enabling the operator to understand the location and motion of the robotic arm and patient through synthetic and real imagery and to control the robotic arm while continuously observing the target (Fig. 3, Online Videos 1 and 2). The interface enabled the operator to make any single image or group of 4 real or synthetic images dominant so that they could be viewed in a larger frame (instead of a thumbnail). In this case, bandwidth from the source of the dominant image feed(s) was automatically throttled to adjust to the overall video bandwidth.
In addition to the viewing controls, the user interface control panel included the ability to drive the motion of the robotic arm. The interface permitted tight control of the linear and angular motion of the robotic arm, allowing operators to mimic the finite motion capabilities of a human wrist. The remote operator controlled linear and angular motion by using a conventional mouse, the dials of the remote control interface, and the keyboard.
The Intercity and Trans-Atlantic Simulations
The investigators performed intercity and trans-Atlantic vascular ultrasound examinations on a 2-vessel, vascular access simulation-training phantom (Blue Phantom, Kirkland, Washington). One experienced principal operator (P.P.S.) first tested the feasibility and time efficacy of the remote intercity and trans-Atlantic imaging separately. Thirty runs were attempted initially between New York (master station) and Burlington (slave station). The Internet bandwidth used was <100 Mbps. The first intercity simulation required 231 s for initial visualization and localization of the ultrasound vascular phantom on the remote controller interface. The time to visualization was progressively shortened to <10 s by the end of the first 30 simulations. Subsequently, 15 runs were performed on a separate day between Munich (master station) and Boston (slave station). All trans-Atlantic simulations were performed at <50 Mbps with all runs performed in <40 s. No Internet latency was experienced in controlling the robotic arm motion during the simulations from either location.
To explore the learning curve in trainees with varying degrees of proficiency in ultrasound imaging, we performed 2 additional sets of simulations. The imaging times and robotic arm movements were compared between personnel with advanced training in clinical ultrasound (advanced trainee) and a beginner with minimal training in cardiovascular ultrasound (early trainee). Both personnel were blinded to each other and to the previous experimental runs. For exploring the motion kinematics, a series of data points were collected and exported from the user interface tracking software to an Excel file. From these data, we analyzed the maximum and minimum linear and angular velocities of the ultrasound probe in x, y, and z directions. We also calculated the maximum and minimum average linear and angular velocities for each simulation.
The data files from the first 3 simulations performed by the advanced trainee failed to record on the Excel file and were therefore excluded. When comparing the remaining 27 simulations performed by the advanced trainee and the 30 simulations performed by the early trainee, the linear and angular velocities were found to be significantly smaller for the advanced trainee (Table 1). When comparing the overall temporal trends of the simulations, a progressive exponential reduction of the time to appearance of the first ultrasound image (visualization of the phantom) was seen for the advanced trainee but not for the early trainee (Figs. 4A and 4B). This finding suggests that the advanced trainee reduced the time to visualization by moving the robotic arm more efficiently in space. Furthermore, although the movements of the advanced trainee had lower average velocities, the SDs were higher, suggesting that the operator likely continuously adapted to the speed of the flight path.
We further explored the component velocity that correlated with the overall time for remote visualization. Angular velocities in x (r = –0.76, p < 0.001), y (r = –0.63, p < 0.001), and z (r = 0.53, p < 0.001) directions and linear velocities in x (r = –0.53, p < 0.001) and z (r = 0.32, p = 0.01) directions were significantly correlated with the visualization time. Figure 4C shows the correlation between average angular velocity and imaging time. The correlation between visualization time and angular velocities suggests that efficient imaging is achieved by making use of the very fine angular rotation of the robotic arm enabled by the interface.
We subsequently tested the settings required for remote robotic carotid imaging in a healthy volunteer (Fig. 3C, Online Video 3). The 2-dimensional ultrasound probe was successfully moved by the robotic arm and brought close to the right side of the volunteer’s neck. A synthetic image view of the patient’s body was configured in the software control system at the master station, and a B-mode image of the right internal carotid artery was developed along its short axis. The common carotid artery (Fig. 3C, Online Video 3) was localized along its short axis within 60 s. The artery was also visualized along its long axes; the entire evaluation was successfully completed in 4 min. No system latency was experienced, and the volunteer remained comfortable throughout the duration of the procedure.
We confirmed the feasibility of long-distance intercity and trans-Atlantic telerobotic ultrasound by using routine undedicated household bandwidth. The phantom studies and first-in-man trans-Atlantic volunteer study served as a vital test for operability over the Internet because networking speeds may differ between countries and cities due to throughput issues (limited bandwidth due to network congestion), network delay (jitter based on network traffic), and packet loss (routers may drop packets in cases of heavy network traffic). This successful experience helped operators to outline the settings required for human long-distance remote imaging, and a full-scale, 2-phase, 2-institution human feasibility study for telerobotic ultrasound has been planned for 2014. Cohort A of the study will be composed of 26 healthy volunteers who will undergo 2 manual and 2 telerobotic ultrasound acquisitions with the robotic arm controlled over the Internet between institutions in Chicago and New York, respectively. Cohort B will enroll 100 patients who will undergo 1 manual and 1 telerobotic ultrasound acquisition for each of their left and right carotid arteries for detection of carotid plaque.
The simulations data provided key insights into human behavior regarding operation of—and interaction with—the telerobotic ultrasound system. Both the advanced and early trainees were able to learn to operate the telerobotic platform, and both started out with a comparable overall efficiency. Although these findings suggest potential clinical applicability over a wide range of clinical experience, the progressive shortening of imaging time by the advanced trainee suggests a clear advantage in having previous proficiency in the use of ultrasound. In particular, slow (deliberate) average angular speeds of the probe were associated with lower overall imaging time. This finding may be related to less overshoot while obtaining proper horizontal or vertical alignment with the phantom vessel, suggesting that the motion of the robotic arm in three-dimensional space is optimized through slow, methodical fine adjustments once the probe is in the correct approximate location. These findings reinforce the intuitive impression that an experienced ultrasound imager is more likely to translate their prior experience in accurately developing a mental model of the remote ultrasound examination, and adapting the movements necessary to execute it. Although sensor and/or image feedback may help to semi-automate an ultrasound examination in the foreseeable future, our data reinforce that overall clinical experience will be useful in image acquisition using robotic arms. Furthermore, the use of robotic arms may be advantageous in producing more precision, steadiness, dexterity, and maneuverability, while also reducing musculoskeletal and posture-related injuries associated with prolonged imaging for sonographers.
The evolution of long-distance telerobotic ultrasound is expected to complement the emerging field of telemedicine and expand the role of sonographers and physicians in serving as a virtual online “helping hand” to accomplish a remote ultrasound evaluation or guiding paramedic workers at a distance. The emergence of such an efficient system would be valuable in numerous situations; for example, within an institution for guiding emergency triage or interventional procedures, in trauma during civilian accidents or war zones, for screening strategies within local or remote communities, and in natural disasters or remote space expeditions, allowing wider delivery of services more efficiently and cost-effectively. It is likely that evolution of the robotic system would be matched with equal advances in remote controlling interfaces that could allow sonographers to be mobile and operate the remote arms by using touch screens, touch pads, tablets, or wearable devices. Advances in technology could also potentially enable sonographers to use gestures, voice- driven commands, or other immersive interfaces. As technology and medicine continue to merge, there will be evolving issues of data security, patient confidentiality, licensure, consent, authentication, and remuneration that will need pragmatic solutions. Such regulatory and ethical issues are not unique to telerobotic ultrasound and are likely to be addressed through the adoption of electronic health (eHealth) and mobile health (mHealth) systems.
Our evaluation suggests that long-distance telerobotic vascular ultrasound over a local nondedicated Internet connection is feasible and can be rapidly learned by experienced sonographers for cost-effective resource utilization. These advances are particularly relevant for the emerging field of telemedicine and will help address the existing deficiencies in global healthcare access.
The authors thank Markus Kasel, MD, The German Heart Centre, Munich, for allowing the trans-Atlantic feasibility studies.
For supplemental videos and their legends, please see the online version of this article.
Dr. Sengupta holds a licensed patent on “Method for Imaging Intracavitary Blood Flow Patterns”; is an advisor to Saffron Technology, Inc., Medical Intelligence LLC, TeleHealthRobotics, LLC; and is a consultant to Edward Lifesciences Corp. Dr. Doukky has received research grants and served on the advisory board for Astellas Pharma, U.S. Ms. Doherty is chief technology officer of TeleHealthRobotics and Dr. Soble is a shareholder in TeleHealthRobotics. Dr. J. Narula has received funding from GE Healthcarehttp://dx.doi.org/10.13039/100006775 and Philips Healthcare. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose. Sherif Nagueh, MD, served as the Guest Editor for this article.
- Abbreviations and Acronyms
- common carotid artery
- electronic health
- mobile health
- Received January 21, 2014.
- Revision received March 7, 2014.
- Accepted March 7, 2014.
- American College of Cardiology Foundation