A robot is a system that has three main components: a set of sensors that detect the robot’s environment, actuators (or end effectors) that interact with and within the environment and a control architecture that processes sensory data and generates actions ^[3](Figure 1). ML is mainly involved in the control architecture, enabling the robot to “understand” the sensory input and generate a proper action. In order to do so, the agent must learn to generate a certain action in the context of a set of sensory inputs and desired goals. So, how can a robot learn a surgical skill? First, it could learn from human demonstration by observing experiments conducted by trained experts. The robot could also learn from its own interaction with the environment, by evaluating the appropriateness of its own actions to reach certain target states or goals ^[3,6]. In order to understand current autonomous robotic surgery technologies and future perspectives, a general knowledge of the main subsets of machine learning is needed. The three main categories of machine learning are supervised learning (SL), unsupervised learning (UL) and reinforcement learning (RL) ^[7]. Many agents use variations and combinations of these three categories.

In SL, training data are considered “labeled “, ie, the data consist of a set of known input vectors along with a set of known matching target vectors. The software creates a function that links the input object with the corresponding output ^[8]. For example, consider we have a set of photos of thousands of lung pathologies (pneumothorax, pneumonia etcetera). The data is considered labeled if each photo identifies certain features of the radiograph such as opacification values (input vectors) and each type of pathologies (target vector). Eventually, SL seeks to build a predictor model that predicts target vectors for new input vectors. Learning consists of finding optimal parameter values for the predictor model ^[3].
In UL, the training data is considered unlabeled and consists of a set of input vectors without their corresponding target vectors. UL aims to discover correlations and structure in the data. Whether using UL or SL, Gaussian Mixture Model (GMM) and Gaussian Mixture Regression (GMR) based learning could be added to fine tune the learning process and get more reliable demonstration ^[3,9].
Reinforcement learning (RL) is concerned with how intelligent agents ought to take actions in an environment in order to maximize cumulative reward. The difference between RL and SL is that RL does not need labelled input/ output data pairs, the training data is mostly generated through direct interaction with the environment, and RL does not need explicit correction of sub-optimal actions. RL focuses on exploring the environment (states) in a trial and error approach in order to create and adjust a policy that permits the agent to perform a certain task (action). RL’s end goal is to learn a policy that represents a mapping from states to actions. For example, suppose that we want to train a robot to tie a knot. The agent will explore the environment, gradually develop a certain policy (creating and adjusting a suturing technique) in order to be eventually capable of performing the task of tying the knot. A value of a state-action pair is created and represents how good it is for the robot to perform an action in a given condition or state ^[3,7].
RL can be accelerated using implicit imitation learning (IML), a technique that allows the agent to learn a skill through the observation of an expert mentor ^[10]. For example, IML can be used to teach an agent a surgical skill by observing and imitating the performance of an actual surgeon. The agent observes and analyzes the state transitions of the surgeon’s actions (for example, how the surgeon moves from one position to another, and the composition of every maneuver) and uses the information to update its own states and actions. In a series of works, trajectories recorded from human subjects are used to generate an initial policy (the action generated based on certain state). Additionally, some algorithms for imitation learning can learn from several mentors and are being used to transfer knowledge between agents with different reward structures ^[3,10].
Inverse RL (IRL) is a technique for imitation learning that also consists of an observer agent and a mentor. In IRL the agent learns the reward function of the environment from the observer. Then the agent builds the policy that maximizes the reward function using classical RL’s trial and error approach ^[3].
Another layer of ML is deep learning (DL). DL allows for processing of huge chunks of data to find relationships in sets that are often impossible to explicitly label, such as the pixel in a given set of pictures. The basic architecture of DL consists of what are called neural networks, that are analogous to the neurons and synapses of the human brain, providing much of the ability to learn. Deep learning can be used in combination with the previously mentioned techniques ^[11].
Other aspects of ML deals with fine tuning the agent to account for the unpredictability of, or to accurately represent physical systems. These techniques include system identification, high-quality rendering ^[12–14], domain adaptation ^[15] and iterative learning control ^[16]. Although these methods have some advantages, they generally require large amounts of data and are still labile to unexpected changes and unaccountable environmental elements ^[17]. It is important to note that ML is generally used with other methods in order to optimize robotic capabilities. Even though the control architecture (where ML is mostly needed) governs the action performed based on the current state of the environments, two important components cannot be overlooked; the sensors needed for environment observation and the effectors to perform a certain task. Indeed, surgical interventions include interactions with delicate and deformable structures. So, in order to operate, sensory and motor apparatus are being used to detect, for example, the depth needed for a given maneuver, tissue consistency and to generate the required force and direction of a specific maneuver ^[3,4]. For example, Sozzi et al. used real-time adaptive motion planner (RAMP) to generate collision-free robot motions to avoid obstacles within the workspace ^[18]. For the Smart Tissue Anastomosis Robot (STAR), in order to achieve proper suturing, the team used force sensors and infrared bio-glue to prevent tissue deformation and to guide the robot for needle insertion ^[19].
Next, we will be dealing with current examples of autonomous surgical robots that have incorporated one or more of the methods presented.

Van den Berg et al. employed imitation learning to develop an agent that learns tasks from multiple human demonstrations, optimizing speed and smoothness of task execution. The technique was employed on the Berkeley Surgical Robot and used for knot-tying and drawing figures ^[16]. Schulman et al. also developed an agent that learns by human demonstration using a trajectory transfer method. The agent was able to learn five different types of knots ^[20]. Recently, Calinon et al. used inverse RL to transfer skill from a surgeon teleoperator to a flexible robot. In this method the agent and the mentor may have different morphological structures and still handle the case of skill transfer ^[21].
Moreover, Mayer et al. apply ML for suturing and knot-tying, using recurrent neural networks (a subset of deep learning), publishing a series of work ^[22,23]. Also, they used imitation learning to create a suturing robot using principles known from fluid dynamics ^[24].
Weede et al. developed an intelligent autonomous endoscopic guidance system that anticipates the surgeon’s next action during a procedure and adjusts the position of the endoscopic camera accordingly. The system uses information on the movements of the instruments from previous procedures ^[25].
Krieger el al. used The STAR robot to suture bowels in pigs. They used the concept of supervised autonomous suturing, where the surgeon outlines the incision area then the robot uses sensors and combined 3D imaging to assist in suturing of intestinal anastomosis. The STAR robot was able to place evenly spaced and leak-proof sutures in a trial with live pigs. However, sometimes the surgeons had to make small adjustments to the thread’s position for accurate suturing ^[19]. Moreover, Krieger has taught the robot to remove tumors with infrared markers that were used to mark cancerous areas, the robot then excises these parts with, though preliminary, human level accuracy ^[5].
Mylonas et al. used GMM in designing an algorithm that learns from human demonstration. They created a basic autonomous eFast ultrasound scanning by a robotic manipulator ^[26]. Kassahun et al. used GMM modeled joint probability densities to make their agent capable of understanding the model of interaction between the aorta and the catheter in interventional procedures ^[27].
Another agent used in interventional procedures developed by Fagogenis et al. used what they called “Haptic vision” to assist in paravalvular leak closure of prosthetic valves. They designed a robotic catheter that can navigate autonomously (using leak locations localized from pre-operative imaging) to the aortic valve and deploy an occluder into the site of a leak. An operator then deploys the occluder. Haptic vision combines machine learning with intracardiac endoscopy and image processing algorithms to form a hybrid imaging and touch sensor. Machine learning was primarily used to enable the catheter to distinguish the blood and tissue from the prosthetic aortic valve ^[28].
Last but not least, AI agents are being introduced in Urological procedures. The AquaBeam™ robotic system was approved as a water ablative therapy for the resection of the prostate. Although the technology requires human impute and image planning, the robotic system assists in resecting, surgeon defined, prostatic tissue using high velocity saline jet while autonomously adjusting various flow rates based on the depth, length and width of the area being resected ^[29].

All current ML technologies used in ARS, variably share some common drawbacks, mainly, the highly unpredictable nature of the physical world, restraints regarding the training environment, and the amount of data and time needed to train an optimal machine. Training and testing an intelligent agent using available models tends to be time-consuming which generally involves manually collecting and labelling huge amounts of data, for example when using supervised learning. This is problematic when the job requires data that are difficult to obtain in large quantities with necessary variability, labels that are difficult to specify, and/or expert knowledge ^[30]. The end goal of overcoming these limitations is, broadly, to create a system that can implement the desired actions (suturing for example) with great reliability, flexibility and safety ^[4]. The question is whether it is possible to create the optimal environment for ML to overcome the physical constraints of the real world (including the data availability issue), and increase the speed of skill acquisition while being adaptable to random changes and the complex nature of the physical reality. We will be focusing on modeling challenges, data limitation and adaptation to physical changes. The solutions suggested here are based on developments outside the medical field, as we suggest the need to incorporate these techniques in future autonomous surgical robotics designs.

Modeling challenges and physical data limitations

One of the major challenges in modeling the surgical environment is the deforming and dynamic nature of the living body due to physiological, pathological and even external phenomena. For that purpose, mechanical, geometric, and physiological behavior of the environment should be considered. The current methods that rely on intraoperative inputs are not optimal as they involve theoretical and technical challenges related to the interpretation of sensory information, such as sensor co-registration, synchronization and information fusion, which are highly fragile as well as the need for annotated real world data ^[3]. Moreover, applying ML sometimes employs random exploration, which can be hazardous in any real physical training set. ML might also often require thousands or millions of samples, which could take a tremendous amount of time to collect in a real physical word setting, making it impractical for many applications ^[31]. One way of overcoming this limitation is through learning in simulation.

Data driven physics simulation

Recent results in learning in simulation are promising for building robots with human-level performance on a number of complex tasks ^[32,33]. Ideally, an agent should learn policies that encode complex behaviors utterly in simulation and apply those policies successfully via physical robots.
Moreover, one can speculate that an optimal simulation should be as faithful as possible to the real world. Here comes the role of physics simulators. High quality physics simulations are being used in computer graphics to replicate the physical world, from dynamic fracture animation to fluids and particles simulations ^[34,35]. Although they do not provide an interactive environment for real time simulation, they could be used as a cornerstone to replicate physical reality. On the other hand, there are other methods that allow for real time interactions within the virtual world. For example, Holden et al. developed a data-driven physics simulation method that supports real time interactions with external objects.
Their method combines ML with subspace simulation techniques which enables a very efficient physics simulation that supports accurate interactions with external objects, surpassing existing models ^[36]. Seunghwan et al. created a physics-based simulation of a human musculoskeletal model composed of a skeletal system and 300 muscles with a control system, creating a reliable simulation of anatomical features with robust control of dynamical systems that generates highly realistic human movements. Also, their model demonstrates how movements are affected in specific pathological conditions such as bone deformities and when applying various prostheses ^[37]. The same authors also formulated a technique called VIPER that creates realistic muscle models that simulate controllable muscle movement and even muscle growth ^[38]. Moreover, one can use MuJoCo physics engine which is commonly used to create advanced virtual environments for ML. MuJoCo was used by OpenAI research to virtually train a robotic hand that can solve a RubiK’s cube ^[39]. It is only a matter of time before we reach ultra-realistic real time physics simulators that include complex anatomical and physiological elements. So how do these approaches attempt to replicate reality? The aforementioned models and other approaches sometimes make the simulator to closely match physical reality by performing a variety of techniques including system identification, high-quality rendering ^[12–14], domain adaptation ^[15] and iterative learning control ^[16]. Although they are the best methods to account for known physical entities and generally do not directly rely on physical data, the problem in these techniques is that they are still suboptimal in accounting for the randomness of the real world, requires large sets of synthetic data impute (both data demanding and time consuming) and many times still requires additional training on real-world data ^[17].
So, we have techniques to simulate the real world that can be trained fast and does not require real physical data input, now the question is how can we ensure that our model can deal with uncounted randomness, can be trained fast, and does not require large synthetic data input?

Adaptation to unknown situations and overcoming synthetic data limitation

Any system with decision-making power in the operating room should guarantee the safety of the patient while being able to cope with unpredictable events and the uncertainty of the living body. A critical challenge is to develop intelligent agents that are able to adapt the learned skills to unexpected and novel situations ^[3]. For solving the modelling problem, we suggested the use of virtual physics simulation. Unfortunately, incongruities between reality and simulators make transferring skills from simulation problematic. For example, system identification, a process used for adjusting the parameters of the simulation to match the characteristics of the real world and the behavior of the physical system (i.e robot), is error-prone and time-consuming. Even with other techniques such as high-quality rendering ^[12–14], domain adaptation ^[15] and iterative learning control ^[16], the real world has physical effects that are hard to model and are not captured by current real-time physics simulators like gear backlash, nonrigidity, fluid dynamics and wear-and-tear. Furthermore, simulators are often unable to produce the noise and richness embedded in the real physical world. These differences, known as the reality gap, are considered bottleneck to the usage of simulated data on real physical robots ^[17,39].
Bridging the ‘reality gap’ that separates experiments on hardware from simulated robotics might accelerate autonomous robotic developments through improved synthetic data availability. This brings us to domain randomization (DR), a new method for training agents on simulated environments that transfer to the real environment by randomizing rendering in the simulator. In other words, the parameters of the simulation are randomized in a way that with enough variability in the simulator, the real world appears to the agent as just another variation. So, the underlying hypothesis is this: if the variability in simulation is significant enough, agents trained in simulation will generalize to the real world. It is important to note that researchers can also try to use DR in combination with other techniques that optimize the physics simulation, which might improve the results ^[17,39].

Domain randomization

In DR, the parameters of the simulator—like lighting, pose, object textures, and other physical aspects—are randomized to oblige the agent to learn the essential features of the object and task of interest. DR requires us to specify what aspects we want to randomize, and specify the variable testing states. Although in its early form, the importance of DR is that it allows for the possibility to produce an agent with strong performance using low-fidelity synthetic data. This introduces the possibility of using inexpensive artificial (synthetic) data for training agents while avoiding the need to collect and label incredible amounts of real-world data or to generate highly realistic artificial worlds ^[30]. One of the earliest works on domain randomization was presented by Tobin el al. The team used DR in the setting of RL and managed to train a robot virtually to localize presented objects. They were able to train an accurate real-world object detector that is resistant to partial occlusions and other distractors using synthetic data from a simulator with non-realistic random textures. The detectors were also used to perform grasping in a messed up real environment ^[17]. Also, OpenAI used RL with DR to enable a robot to learn dexterous in-hand manipulations ^[40].
Automatic domain randomization (ADR), is a variation of DR that randomizes the parameters of the simulator automatically, without the need to specify what elements of the simulation we want to change. ADR automatically generates a distribution over randomized environments of ever-increasing difficulty, thus creating millions for scenarios for the learning algorithm. The latest implementation of ADR was conducted by the OpenAI team, who trained a robotic arm to solve a rubik’s cube. The robot was able to solve the cube in a real world setting, even with intentional disturbance of the environment by the researchers ^[39].
Compared to iterative learning control and domain adaptation which are important tools for addressing the reality gap, DR does not require additional training on real-world data. Although DR requires no additional real world training, it can also be combined easily with most parallel techniques, and we should consider using it in combination with realistic physics simulations and possibly even, when possible, other training methods such as imitation learning ^[17]. DR (or ADR), through randomization, reduces the need of synthetic data while creating an agent robust to changes in the real world. It is important to note that DR is still a new technique and will require further optimization to improve its usefulness and applicability.
In brief, what we are suggesting in this review is to try to combine optimal realistic physical simulation techniques with DR (or ADR) and other parallel techniques in a sense that we can create an environment as faithful to the real world, and as random as the real world in order to produce optimal training environment for surgical agents thus creating the most reliable autonomous surgical robots.

Autonomous robots will be needed to address the decreased work force, the increased demand for surgery and the high risk of medical errors. Current technologies used in medical robotics require huge amounts of data input, are difficult to train, and prone to minor changes in the environment. Using physics simulation techniques combined with DR might be what is needed to overcome these limitations in order to create the advances desirable in autonomous robotic surgery. Our approach focuses on virtual training with domain randomization. Of course, to apply the learned skills in the real world certain specific hardware will be needed, however these technologies are currently available in the form of sensors and other hardware already in use in many current medical and non-medical models. Here we are adding this layer of training that if linked properly to a physical hardware, the aforementioned limitations should be eventually overcome. In essence, more trial and less error.