Programming sensitivity: Robot arms in the operating room
How can humans and robots cooperate to perform tricky surgeries – such as removing a gallbladder? Mechanical engineer Paul Maria Scheikl is conducting research at HIDSS4Health to determine how robot arms need to be programmed for such uses. To train robot AI, he has to integrate practical experience from the operating room.
When Paul Maria Scheikl, 29, observes a gallbladder operation, he is always astounded at how the surgeons’ nimble hands do everything at once: grasping tissue with small metal grippers and keeping it taut so that it can be cut cleanly, little by little, repeatedly reorienting the camera at the end of the endoscope’s arm so that the gripper and cutting instrument can find their way through the patient’s abdomen to all the bifurcations in the gallbladder, and slowly cutting the tissue. “I thought it was cut with little scissors,” Scheikl says. What actually happens is that the surgeon carefully pushes a small hook under the tissue, pulls it towards himself, checks that the right little bit is in the hook, and cuts the fine fibers by heating them so that even the smallest blood vessels are immediately cauterized. “I would never have thought of that,” says Scheikl.
Scheikl is a mechanical engineer and data scientist. He has been doing research for his doctoral thesis at the Helmholtz Association’s HIDSS4Health graduate school since the summer of 2020. He is one of the 38 doctoral candidates participating in a special program for young scientists there: At HIDSS4Health, medical specialists, life scientists, and data scientists collaborate to gain new insights and develop new methods in fields ranging from health research to genetic analysis to robotics for surgery. The program is a collaboration involving the German Cancer Research Center in Heidelberg DKFZ, Heidelberg University, and the Karlsruhe Institute of Technology KIT.
Bringing together the knowledge of engineers and physicians
Before starting his doctoral studies, Scheikl, a mechanical engineer, had nothing to do with gall bladders or surgical equipment. Now he analyzes surgeons’ movements during such operations whenever he gets a chance. This is because his doctoral thesis is geared towards discovering how robot arms have to be programmed in order to effectively assist surgeons during these procedures. “I had no idea of the incredible number of little steps that have to be choreographed.”
In close cooperation with life science experts at HIDSS4Health, the 29-year-old is making incremental progress towards his goal. Things get especially tricky when robots and humans work closely together: Designing devices for humans involves ergonomic criteria, for example, whereas a robot arm has trouble gripping anything round. “Feedback from medical experts is extremely important. Engineers can easily get lost in the technical details and overlook what is relevant to surgeons,” Scheikl says. “Medical experts, on the other hand, naturally have less of a grasp of what is technically feasible.”
Paul Maria Scheikl, who is a native of Heidelberg, noticed early in his mechanical engineering studies that it was not heavy-duty industry systems that interested him, but intricate robotics. And since, at the Karlsruhe Institute of Technology (KIT) where he studied, robotics is part of informatics and not the mechanical engineering department itself, "I slipped into the field of robotics," says Scheikl. The mix fascinates him. “It is challenging mathematically and scientifically, yet more practical than theoretical informatics.” For his master’s project, he ultimately developed a program for teaching human-robot interaction to robots that provide nursing assistance. It covered such applications as picking up a key that has fallen to the floor and handing it to a patient. Such an application is much more complex than it sounds. When should the machine arm grip, when should it release, and how is the entire operation to be coordinated? “I always wanted to solve problems, not just work through tasks,” says Scheikl. “It’s great fun to find out if and how something new can be done.”
“I really enjoy not having to have a specific plan at HIDSS4Health; it’s great to have so much freedom in research to incrementally pursue the ideal path to my research problem’s solution.”
Paul Maria Scheikl, doctoral researcher at HIDSS4Health
After completing his master’s, Scheikl found just the right environment at HIDSS4Health. During his doctoral studies, he will be mentored by outstanding specialists from both medicine and medical engineering and by data science experts. “Some medical colleagues have also studied medical informatics, which makes discussion that much easier,” says Scheikl.
His daily research routine is jam-packed. Every day he thumbs through specialist magazines, watches videos suggested by his specialist colleagues, and looks for inspiration in research groups in related fields. What methods have proven effective? What could work for me? The tightly packed lecture series by life science professionals and data scientists from various institutions is quickly giving him a broad overview of his field of research. This overview is necessary if he is to find suitable methods of achieving the goals of his doctoral thesis. And where could he network more easily in person with other researchers who combine media research and data sciences? “I really enjoy not having to have a specific plan at HIDSS4Health; it’s great to have so much freedom in research to incrementally pursue the ideal path to my research problem’s solution.”
Prior to the theory: training in the operating room
Scheikl has slowly acclimated himself to organic work; he has removed gallbladders from pig livers himself. What does it feel like? How is the tissue structured? How can fat, organs, and veins be distinguished precisely? He examined the images provided by the camera on the endoscopic arm and analyzed exactly how surgeons see tissue during an operation. After “many training stations”, Scheikl was allowed to be present during operations. Today, he refers to that experience as a “culture shock”. “It was incredible to see how focused medical personnel are on helping people.” Planned operations kept getting postponed because of intervening emergencies. During each operation, Scheikl found “the concern for saving someone’s life palpable – the team was not optimizing a technical system that may be of use sometime down the road, which is the approach engineers often take when they do scientific work.”
Now Scheikl is training a robot to do what a surgeon does. He has identified the method: reinforcement learning, as the technical jargon has it. The AI is not told exactly how to approach the problem. Instead, the system learns based on rewards and negative feedback: That was good, and that was bad. “That’s how it eventually finds its own strategy.”
But to do so, Scheikl also has to find out what is right, of course, so he knows what to reward (and what not to reward). From hundreds of endoscope camera demo videos of surgeons removing gallbladders, he creates simulations in sub-steps, which he uses to develop the necessary reward information with which the neural network is trained: Pull tissue aside, grip something, maneuver the gallbladder to a specific position without damaging anything. “It takes forever to get the AI to actually be able to do it. After all, it has to learn tons of details and basic competencies a person has 18 years to learn before he even starts his medical training,” says Scheikl. These competencies range from hand-eye coordination to quick image recognition. “The robot starts at zero.”
A long way to use on humans
That makes it all the more important to include hefty safety buffers before the AI starts working on a living organism. It is also why, before it begins the next action, Scheikl asks the AI what it will be doing next – so he can stop it if he needs to. To do this, he uses “classical robotics collision-avoidance methods as training wheels” in case the neural network takes the wrong path – even in simulations. Is it too shaky? Did it overlook something? Are the images of a specific patient different from the standard ones the robot has learned, and is that confusing it? “Living organisms aren’t machines, and things are often different from what is expected,” says Scheikl. “If the AI suggests cutting through the abdominal wall, we see from the sensor data that it is messing up and can stop it.”
AI surgeries on people are still a long way off. After successful operations in the simulation, the AI will have to tackle a silicon gallbladder, then an organ from a slaughterhouse, then a live pig, and only sometime after that, working shoulder to shoulder with surgeons, will it be allowed to work on a human being. There are still lots of problems to solve. And that’s exactly what interests Scheikl. For instance, how can a robot, an object made of metal, be taught such skills as touch sensitivity, which is so important for working with fine instruments inside the body? Scheikl describes this as “the ability to detect and communicate clearly to a physician what is happening with the interaction forces between tissue and instrument”. And how can communication between humans and machines be made comprehensible for both? “If the instrument is obscuring the tissue and the human says ‘I can’t see anything’, the neural network has no way of understanding that.” Scheikl grins. “Looking for a solution is fascinating, isn’t it?”
The doctoral candidate still has two years to find it. His dream is to get to live animal experiments. Here, at HIDSS4Health, he has excellent starting conditions for success. “It is just great to be able to work at the interface between medicine, mechanical engineering, and data science and do something that really matters.”
Author: Anja Dilk
