Trustworthy Benchmarking for Black-Box Single-Objective Optimization
Tome Eftimov, Jožef Stefan Institute, Slovenia
Search Trajectories Illuminated
Gabriela Ochoa, University of Stirling, United Kingdom
Predictive Maintenance for Industry 4.0 & 5.0
Rita P. Ribeiro, University of Porto, Portugal
João Gama, University of Porto, Portugal
(Cancelled)
Trustworthy Benchmarking for Black-Box Single-Objective Optimization
Tome Eftimov
Jožef Stefan Institute
Slovenia
![]()
Brief Bio
Tome Eftimov is a senior researcher at the Computer Systems Department at the Jožef Stefan Institute. He is a visiting assistant professor at the Faculty of Computer Science and Engineering, Ss. Cyril and Methodius University, Skopje. He was a postdoctoral research fellow at Stanford University, USA, where he investigated biomedical relations outcomes by using AI methods. In addition, he was a research associate at the University of California, San Francisco, investigating AI methods for information extraction from electronic health records. He obtained his PhD in Information and Communication Technologies (2018). His research interests include statistical data analysis, metaheuristics, natural language processing, representation learning, meta-learning, and machine learning. He has presented his work as 81 conference articles, 50 journal articles, and one Springer book published 2022. He was selected in Stanford University's top 2% of influential scientists worldwide in all disciplines for AI contributions for 2022. The work related to Deep Statistical Comparison was presented as a tutorial (i.e. IJCCI 2018, IEEE SSCI 2019, GECCO 2020, 2021, 2022, 2024, PPSN 2020, 2022, IEEE CEC 2021, 2022, 2023) or as an invited lecture to several international conferences and universities. He is an organizer of several workshops related to AI at high-ranked international conferences. He is an Editor in Evolutionary Computation Journal and Associate Editor in Expert Systems with Applications He is involved in both national and European projects. Currently, he is coordinating bilateral projects with Sorbonne University, France (algorithm selection and configuration), Leibniz University Hannover, Germany (fair benchmarking for dynamic algorithm configuration), and the University of Banja Luka, Bosnia and Herzegovina (theoretical and machine learning approaches for graph data). He has previously coordinated national projects on representation learning for stochastic optimization algorithms (2022-2024) and robust statistical analysis for single-objective optimization (2019-2021), as well as an EFSA-funded project on natural language processing for food science (2021-2022).
Abstract
At the start of 2022, the evolutionary computation scientific community published a call for action pointing to the elephant in the room in metaphor-based metaheuristics used for black-box optimization (BBO). This highlighted three core issues: useless metaphors, lack of novelty, and biased experimental validation and comparison. This talk will provide an overview of recent advances in benchmarking approaches that lead to more robust and reliable results and meta-learning approaches used to select the best algorithm for a particular optimization problem. We will zoom specifically into two approaches: i) a selection of more representative data instances that can generalize the study results and ii) algorithm footprints that provide sets of easily or challenging solvable problem instances for a particular problem instance together with an explanation about which problem landscape characteristics are related to that outcome. The end vision is a paradigm shift in black-box optimization algorithms to decrease ineffective use of resources and duplications of efforts, leading to more focused advances in the field. Indirectly, such approaches will facilitate new directions in automated algorithm configuration and selection by providing explanations rooted in advances in the field to transfer academic knowledge to new optimization problems better.
Search Trajectories Illuminated
Gabriela Ochoa
University of Stirling
United Kingdom
![]()
Brief Bio
Gabriela Ochoa is a Professor of Computing Science at the University of Stirling in Scotland, UK. Her research lies in the foundations and applications of evolutionary algorithms and metaheuristics, with a recent emphasis on fitness landscape analysis and visualisation. She holds a PhD from the University of Sussex, UK, and has worked at the University Simon Bolivar, Venezuela, and the University of Nottingham, UK. Her Google Scholar h-index is 44. She has published over 180 refereed articles and obtained 6 best-paper awards and 10 other nominations (GECCO, EvoStar and PPSN), many of them related to her work on Local Optima Networks (LONs) and Search Trajectory Networks (STNs). She collaborates cross-disciplines to apply evolutionary computation in healthcare and conservation. She has been active in organisation and editorial roles in venues such as the Genetic and Evolutionary Computation Conference (GECCO), Parallel Problem Solving from Nature (PPSN), the Evolutionary Computation Journal (ECJ) and the ACM Transactions on Evolutionary Learning and Optimisation (TELO). She is a member of the executive board for The ACM interest group in evolutionary computation, SIGEVO, and the editor of the SIGEVOlution newsletter. In 2020, she was recognised by the leading European event on bio-inspired algorithms, EvoStar, for her outstanding contributions to the field.
Abstract
Many nature-inspired optimisation algorithms have been proposed over the years. It is unclear, however, to what extent recent algorithms are really “new”, or how and why to select one of them to solve a given task. Search trajectory networks (STNs) are a data-driven, graph-based modelling tool to analyse, visualise and contrast the behaviour of different types of optimisation algorithms. STNs offer a visual and intuitive fresh perspective to explain and interpret search and optimisation. This talk overviews our methodology including recent developments: applications to neuroevolution, multi-objective optimisation, STNWeb, and the use of generative AI to automate the analysis.
Predictive Maintenance for Industry 4.0 & 5.0
Rita P. Ribeiro
University of Porto
Portugal
![]()
Brief Bio
Rita P. Ribeiro is an Assistant Professor in the Department of Computer Science at the Faculty of Sciences of the University of Porto (FCUP) and co-director of the Bachelor in Artificial Intelligence and Data Science program at the University of Porto. She is also a Researcher at the Laboratory of Artificial Intelligence and Decision Support (LIAAD) at the Institute of Systems Engineering and Computing, Technology and Science (INESCTEC). Her main research interests revolve around learning problems in imbalanced domains, anomaly detection, explainability, evaluation issues in learning tasks, and application problems related to social good and environmental impact. She has participated in several research projects on environmental issues, fraud detection, and predictive maintenance applications. She is also a member of the program committee of several international conferences. She serves as an editor and reviewer for various international journals, including the Machine Learning Journal (MLJ), Journal of Data Science and Applications (JDSA), Information Fusion (INFFUS), and Intelligent Data Analysis (IDA). She has also co-organized numerous scientific events and is currently involved in the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases - ECML PKDD 2025 as one of the Research Track Chairs.
Abstract
Data-driven Predictive Maintenance (PdM) is becoming increasingly important in many industries, particularly in the context of Industry 4.0 and the emerging Industry 5.0. Industry 4.0 focuses on the automation and digital transformation of manufacturing processes by integrating advanced technologies like the Internet of Things (IoT) and Artificial Intelligence (AI). Meanwhile, Industry 5.0 extends this paradigm by emphasizing human-machine collaboration and sustainability. Predictive Maintenance (PdM) plays a vital role in both frameworks by leveraging machine learning algorithms on historical and real-time data from various system parts to detect anomalies and possible defects in equipment before they lead to failure. However, implementing PdM efficiently can be challenging due to system complexity, data integrity, non-stationarity of environments, and limited data availability on rare or dangerous degradation behaviours. Black-box models based on deep-learning techniques are popular due to their high predictive accuracy. However, as these systems become more complex with many interacting components, it's crucial to ensure the trustworthiness of these models through explainability. This talk will explore some of these challenges, current trends, and promising research directions and conclude with case studies in the public transport sector.
Recent Advances in Learning from Data Streams
João Gama
University of Porto
Portugal
* CANCELLED *
Brief Bio
João Gama is a Full Professor at the Faculty of Economy, University of Porto. He is a researcher and vice-director of LIAAD, a group belonging to INESC TEC. He got the PhD degree from the University of Porto, in 2000. He is a Senior member of IEEE.He has worked on several National and European projects on Incremental and Adaptive learning systems, Ubiquitous Knowledge Discovery, Learning from Massive, and Structured Data, etc. He served as Co-Program chair of ECML 2005, DS 2009, ADMA 2009, IDA 2011, and ECML/PKDD 2015. He served as track chair on Data Streams with ACM SAC from 2007 till 2016. He organized a series of Workshops on Knowledge Discovery from Data Streams with ECML/PKDD, and Knowledge Discovery from Sensor Data with ACM SIGKDD. He is the author of several books on Data Mining (in Portuguese) and authored a monograph on Knowledge Discovery from Data Streams. He authored more than 250 peer-reviewed papers in areas related to machine learning, data mining, and data streams. He is a member of the editorial board of international journals ML, DMKD, TKDE, IDA, NGC, and KAIS. He (co-)supervised more than 12 PhD students and 50 MSc students.
Abstract
Learning from data streams is a hot topic in machine learning and data mining. In this talk, we present two different problems and discuss streaming techniques to solve them. The first problem is the application of data stream techniques to predictive maintenance. We propose a two layer neuro-symbolic approach to explain black-box models. The explanations are oriented toward equipment failures. For the second problem, we present a streaming algorithm for online hyper-parameter tuning. The Self hyper-Parameter Tunning (SPT) algorithm is an optimization algorithm for online hyper-parameter tuning from non-stationary data streams. SPT works as a wrapper over any streaming algorithm and can be used for classification, regression, and recommendation.