RL microfluidics

Learning microfluidic control from expert demonstrations

We are developing a graphical user interface (GUI) that allows microfluidics users to control an experiment while automatically recording what they see and what they do. These synchronized expert demonstrations will provide training data for behavioural cloning: learning a control policy that imitates the actions an experienced operator would choose from the current experimental state.

Our goal is to make specialist operating knowledge reusable. Instead of asking a reinforcement-learning agent to discover every useful control strategy through trial and error on physical hardware, we first capture successful human workflows and use them to initialize automated controllers.

Why microfluidics is challenging to automate

Microfluidic experiments are controlled through a manageable number of inputs, such as applied pressures or flow rates, but their behaviour can be difficult to predict. A useful operating regime depends on interactions between channel geometry, fluid properties, tubing and reservoirs, surface condition, temperature, device history and the state of connected modules. Moreover, hysteresis-like responses and competing effects occuring at timescales ranging from milliseconds to minutes limit the efficacy of control linear PID controllers (although in many special cases these can be very effective!).

The same control actioncs may produce different outcomes between devices or over the lifetime of one experiment. Responses may be delayed, disturbances can propagate through connected channels, and important events are often visible only through microscopy. Expert users handle this by continuously interpreting the image, remembering recent changes and making small, context-dependent adjustments. All these aspects contribute to the need for several months of training for non-expert users to adopt sophisticated microfluidic devices into their experimental workflows. We want to democratise access to challenging microfluidic methods, e.g double emulsion droplet and giant lipid vesicle production, through automation.

For automation, this means that a controller must do more than hold a single set point. It must recognize the current experimental state, respond to drift and recover when production moves away from the desired regime.

Naive RL is too costly to implement

Reinforcement learning (RL) can learn control policies by exploring actions and using a reward signal to distinguish good and bad outcomes. On a physical microfluidic system, however, naive exploration is expensive. Each training step consumes experimental time and may consume reagents, generate unusable samples, block or damage a device, or require an operator to reset the system.

The example below illustrates this challenge for one of our microfluidic methods: Octanol-assisted Liposome Assembly (see the modular microfluidics pages). Training trajectories can contain both improving periods and severe failures: similar-looking changes in evaluation score may correspond to very different physical states at the chip. Obtaining enough real-world interactions for robust learning can therefore be impractical if the agent begins without prior knowledge.

Behavioural cloning may provide a more data-efficient starting point. A policy is first trained to reproduce expert actions in states already encountered by human operators. RL or other adaptive-control methods can then refine that policy with substantially less unconstrained exploration on the physical device.

Open-source control software for collecting training data

Behavioural cloning depends on high-quality examples of the relationship between observations and expert actions. Conventional experiment logs rarely contain this relationship at sufficient resolution: a lab notebook might record a target pressure or a final result, but not the sequence of image observations, adjustments and responses that led there.

Our GUI is designed to collect these data during normal operation. A demonstration can record:

• timestamped microscope images or video frames;
• image-derived measurements describing the experiment, such as vesicle and lipid-droplet area;
• the pressures or flow commands applied to each inlet;
• the expert’s control changes and their timing; and
• episode information needed to identify stable operation, recovery and failure.

The visualization below shows the structure of a recorded demonstration. The microscope view is synchronized with measured system responses and the pressure commands applied by the user. This turns an expert session into a time-aligned trajectory of states and actions suitable for model training and evaluation.

We intend the control and recording tools to be open source so that demonstrations can be collected across users, devices and operating conditions. A shared data format will make it easier to compare learning methods, combine compatible datasets and build controllers that generalize beyond a single experiment. We are finalising the development of the control software and will release it soon!

Project direction

The next stages are to standardize the demonstration format, provide practical tools for reviewing and annotating recorded episodes, and benchmark behavioural-cloning policies against both expert operation and controllers trained through direct exploration. Longer term, expert demonstrations could support hybrid approaches that combine imitation learning, reinforcement learning and conventional feedback control.