This is a great list! I had not seen the presentation from Jensen and Hendricks - thanks for sharing!
As @evwolfson mentioned, I created Summit for solving scientific problems with Bayesian optimization. We really focus on making it easy to get started, but if you try it and have any issues let me know!
Also, I’d classify the applications of Bayesian optimization into three categories:
BO is something I want to work into my own LC validation efforts - but haven’t quite gotten to point where I have spare time to plan and work on it. Do any of these solutions keep track of or measure a value relevant to time spent on optimization? This is typically coined “regret” from what I’ve seen, and is a metric that can be used to summarize loss of materials/time/resources from not using optimal variables. A combination of time spent optimizing and the difference between the highest optimal returns from a black box function and those used during optimization; something like sum[F(optimal)-F(tests)] + scaled_time_metric… the higher the value, the higher the regret.
From my understanding, it would be a great incentivizing variable to keep track of for relaying the importance of optimization, and the impact of BO on minimizing that “regret”.
Yes, that is the correct definition of regret. There are some nuances here. BO algorithms have an acquisition function which scores the quality of potential new experiments. While most acquisition functions do not explicitly keep track of cost, they will implicitly have a trade-off between exploring (and possibly finding a better solution) and exploiting (optimizing around the best known parameters).
For example, expected improvement is known for being quite exploitative. Upper confidence bound (with the correct hyperparameters) tends to explore more. There are more acquisition functions that have different trade-offs.
One important thing that people often are confused by is that BO algorithms usually do not converge to an optimal point. Common acquisition functions (expected improvement, upper confidence bound) will, to some extent, continue exploring as long as they have the budget and there is uncertainty in the model. So, you typically will set a budget in advance and stop once you hit that or are satisfied with the results.
Finally, if the cost of experiments varies as a function of parameters (e.g., certain liquid class parameters take longer to evaluate or need more material), you could look into cost-aware BO.
I am currently applying Bayesian Optimisation to yeast optimisation. I will have access to a Beckman Biomek i7 and we will use our Matterhorn Studio platform to host Bayesian Optimisation algorithms that can suggest new candidates experiments 24/7, i.e. whenever an experiment has finished. If nothing goes wrong, this should close the loop! (famous last words)