prof. Ing. RNDr. Martin Holeňa, CSc.

Optimization tasks encountered in real-world applications more and more frequently optimize objectives that are not mathematically calculated functions, but results of computer simulations or experimental measurements. That kind of optimization, called black-box optimization, presents two great challenges: 1. it is possible to get only the values of such a black-box objective, not its gradient or higher derivatives, 2. the evaluation of the objective typically costs much time and/or money. As far as the first challenge is concerned, evolutionary algorithms have proven very successful for optimization using only the values of the objective during the last decades. However, they typically require a large number of evaluations, which conflicts with the second challenge. That conflict incited an intense reseaech into black-box evolutionary optimization during the last decade, bringing a broad spectrum of topics suitable for PhD theses.

As black-box optimization, we refer to optimization in which the mathematical expression of the optimized function is not used (typically because no such expression is known), but the optimization algorithm only has its values at specific points available. These values are usually obtained empirically by measurement or through experiments, either physical or in the form of simulations. Black-box optimization uses algorithms that make almost no assumptions about the mathematical properties of the optimized function, most often evolutionary algorithms and other nature-inspired algorithms such as particle swarms. Since these algorithms work only with the functional values of the optimized function, they approach its optimum much more slowly than the optimization methods for smooth functions, which also use information about the gradient or second derivatives of the optimized function. This property is particularly disadvantageous in conjunction with the fact that empirically obtaining the value of the optimized function is usually quite expensive and time-consuming. However, evolutionary algorithms can be significantly accelerated by using the empirical black-box function only occasionally for evaluating the value of the optimized function, whereas mostly only its sufficiently accurate regression model is evaluated. Among the regression models used for this purpose, artificial neural networks have been for about 20 years, first multilayer perceptrons and later networks with radial basis functions. Due to the current popularity of modern types of neural networks, often referred to as deep neural networks, new approaches usable speed up black-box optimization based on modern neural networks have nevertheless been proposed in recent years, such as deep Gaussian processses, using Bayesian neural networks, optimization in a latent space of a lower dimension, mapped by a generative neural network into the spacein which the inputs of the optimized black-box function lie, or making use of GAN-type networks (generative adversarial network), the two components of which are used for the exploration and exploitation components of the optimization.

Graph data are used to keep knowledge about many important areas of the present world, such as computer networks, social networks, chemical molecules, communication systems, industrial processes or text documents. However, data-based methods for data analysis and modelling, such as classification, regression and clustering, have been developed primarily for vector data, and graph data need to be for using those methods first represented in some vector space. Most successful at such representation are artificial neural networks. Due to the need to learn graph representation, a specific kind of neural networks appeared, called graph neural networks. However, also graph neural networks have the propertzy of an overwhelming majority of neural networks that the rtransformation of network inputs to their outputs is a black-box mapping, which for a given network input does not enable to explain its output. In connetion with traditional neural networks, primarily multiayer perceptrons and radial basis function networks, attention has been paid, already since the 1990s, to methods enabling to describe the dependence of network output on its input by means of logical implications and equivalences, or to explain in some other way the value of its output for a given input. In the case of graph neural networks, however, research into explainability methods is only starting. Contribute to it should also the proposed thesis.

Evolutionary algorithms are, in the last 20 years, one of the most successful methods for solving non-traditional optimization problems, such as search for the most suitable documents containing required information, discovery of the most interesting knowledge in available data, or other kinds of optimization tasks in which the values of the objective function can be obtained only empirically. Because evolutionary algorithms employ only function values of the objective function, they approach its optimum much more slowly than optimization methods for smooth functions, which make use of information about the objective function gradients as well, possibly also about its second derivatives. This property of evolutionary algorithms is particularly disadvantageous in the context of costly and time-consuming empirical way of obtaining values of the objective function. However, evolutionary algorithms can be substantially speeded up if they employ the empirical objective function only sometimes when evaluating objective function values, whereas they mostly evaluate only a sufficiently accurate regression model of that function. It is the accuracy of the model that determines how successful surrogate of the original empirical function it will be. Therefore, the model is made more accurate after obtaining each new generation of points in which the empirical function has been evaluated, through re-learning including those points. However, it is possible to go even further and to consider, when choosing points for empirical evaluation, besides the value of the empirical function also how they contribute, during model re-learning, to making it more accurate. Such an approach is termed active learning. However, using active learning to accelerate evolutionary algorithms is only at a very beginning, and should be supported also by the proposed thesis.

Transfer learning is a method of algorithmic extraction of knowledge from a solved problem and its incorporation into the solution of another problem. It began to be studied more deeply about 20 years ago, in connection with the development of modern types of neural networks, often referred to as deep networks. It allows using a network trained on a large amount of data to train another network of similar quality on a much smaller amount of data.

In recent years, there have been also attempts to use transfer learning in black-box optimization. This is an optimization in which the mathematical expression of the optimized function is not used (typically because no such expression is known), but only its values at specific points are available to the optimization algorithm. These values are usually obtained empirically by measurement or by means of experiments, whether they take place physically or in the form of simulations. Black-box optimization uses algorithms that make almost no assumptions about the mathematical properties of the optimized function, most often evolutionary algorithms and other nature-inspired algorithms such as particle swarm optimization. As for transfer learning, it turns out that, similar to the case of neural networks, it allows to train a network of the same quality with a smaller amount of training data, it makes it possible to find the optimum based on a smaller number of values of the black-box function during black-box optimization. This is very promising because empirically obtaining the value of the optimized function is usually quite expensive and time-consuming.

However, research on methods for transfer learning in black-box optimization is still in a very early stage. A contribution to it should also be the proposed thesis.

prof. Ing. RNDr. Martin Holeňa, CSc.

Ing. Jan Žegklitz

Department of Software Engineering

This thesis investigates the performance of Gaussian processes (GP) in the context of Covariance Matrix Adaptation Evolution Strategy (CMA-ES), the state-of-the-art evolutionary optimization method for black-box continuous optimization, using niching functions from the CEC 2013 competition, which are characterized by a high number of local optima. It describes the integration of CMA-ES and GP surrogate model and compares its performance to Model Guided Sampling Optimization.

Thesis on DSpace

prof. Ing. RNDr. Martin Holeňa, CSc.

Ing. Petr Pulc

Department of Applied Mathematics

One of the key preprocessing tasks in information retrieval from a video is the segmentation of the scene, primarily its segmentation into foreground objects and the background. This is actually a classification task, but with the specific property that it is very time consuming and costly to obtain human-labelled training data for classifier training. That suggests to use semi-supervised classifiers to this end. The presented work reports the investigation of semisupervised classification methods based on cluster regularization and on fuzzy c-means in connection with the foreground / background segmentation task. To classify as many video frames as possible using only a single human-based frame, the semi-supervised classification is combined with a frequently used keypoint detector based on a combination of a corner detection method with a visual descriptor method. The paper experimentally compares both methods of semi-supervised classification in this context with traditional algorithm GMM.

Thesis on DSpace

prof. Ing. RNDr. Martin Holeňa, CSc.

Department of Theoretical Computer Science

Semi-supervised learning is characterized by using the additional information from the unlabeled data. In this thesis, we compare two semi-supervised algorithms for deep neural networks on a large real-world malware dataset. Specifically, we evaluate the performance of a rather straightforward method called Pseudo-labeling, which uses unlabeled samples, classified with high confidence, as if they were the actual labels. The second approach is based on an idea to increase the consistency of the network's prediction under altered circumstances. We implemented such an algorithm called Pi-model, which compares outputs with different data augmentation and different dropout setting. As a baseline, we also provide results of the same deep network, trained in the fully supervised mode using only the labeled data. We analyze the prediction accuracy of the algorithms in relation to the size of the labeled part of the training dataset.

Thesis on DSpace

prof. Ing. RNDr. Martin Holeňa, CSc.

Ing. Jan Žegklitz

Department of Theoretical Computer Science

This diploma thesis draws on the research by L. Bajer, Z. Pitra, J. Repický and M. Holeňa, which deals with the surrogate modeling in the CMA-ES algorithm. The aim of this work was to verify the quality of the current design and then to propose the variants with testing of multiple models. The main contribution of this thesis is the proposal of the evolution of surrogate models in the CMA-ES algorithm, which has a great improvement potential.

Thesis on DSpace

prof. Ing. RNDr. Martin Holeňa, CSc.

Ing. Tomáš Borovička

Department of Theoretical Computer Science

The aim of the thesis is a research of early detection of cow heat on modern automated farms. The main goal is to compare several knowledge-based engineering techniques with reference solution. Specifically Naive Bayes kernel, Decision tree, ARIMA and Cox's regression techniques are used for the experiments. Results show that it is possible to outperform precision as well as sensitivity of reference solution. The most interesting result suggests that precision can be outperformed by Naive Bayes kernel by 12% while sensitivity stays 2% higher than sensitivity of reference solution. No less important is also the data preprocessing step which aims to remove noise and extract features from the data. The techniques used for removing the noise and smoothing the data are Moving average and Butterworth filter.

Thesis on DSpace

prof. Ing. RNDr. Martin Holeňa, CSc.

Department of Software Engineering

This thesis is focused on using multiple Gaussian processes (GP) simultaneously as surrogate models for the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) method, which is an important algorithm in the black-box optimization. The thesis contains an implementation in the MATLAB enviroment, which uses the Gaussian Processes for Machine Learning (GPML) library and benchmark tests from the COmparing Continuous Optimisers (COCO) platform. It also investigates the process of choosing the right models, which will be trained, as well as algorithms for the selection of the best surrogate model, which will be used for the next generation of CMA-ES. The thesis also contains the results of performed experiments that prove the improvement of overall performance when compared to using only one surrogate model.

Thesis on DSpace

prof. Ing. RNDr. Martin Holeňa, CSc.

Ing. Petr Pulc

Department of Software Engineering

This thesis focuses on searching structures in compilations of popular songs by using well known self-organizing maps (SOM) as well as by using context relevant self-organizing maps (CRSOM). This thesis also describes all steps taken in relation to the general process of Knowledge discovery in databases and the required studies and implementations of aforementioned maps. The outcome of the investigation is a concept that is used to predict popularity of a new song based on the structure that is found in a compilation of songs represented by acoustic properties.

Thesis on DSpace

prof. Ing. RNDr. Martin Holeňa, CSc.

Ing. Petr Pulc

Department of Software Engineering

With the quickly rising popularity of multimedia, especially of the audiovisual data, the need to understand the inner structure of such data is increasing. In this thesis, a method for structure discovery in multimedia data is proposed. It consists in integrating a self-organizing map (SOM) and hierarchical clustering to find a suitable cluster structure in the multimedia data. The output of every SOM is evaluated by various levels of hierarchical clustering with different number of clusters mapped to the SOM. Within those mapped levels, the one with the lowest average within-cluster distance is searched, which then determines the most appropriate clustering for the map. In experiments, the proposed approach was applied, with SOMs of four different sizes, to nearly 16 000 segments of audiovisual recordings of lectures. Several examples of the interpretation of obtained results are provided.

Thesis on DSpace

prof. Ing. RNDr. Martin Holeňa, CSc.

Department of Theoretical Computer Science

The thesis analyses the use of Gaussian processes as a surrogate model for Covariance Matrix Adaptation Evolution Strategy. Specifically, the thesis examines and tests the improvements which use Gaussian processes in a space with reduced dimensionality.

Thesis on DSpace

prof. Ing. RNDr. Martin Holeňa, CSc.

Department of Software Engineering

prof. Ing. RNDr. Martin Holeňa, CSc.

Ing. Tomáš Kouřim

Department of Theoretical Computer Science

This thesis focuses on performance gain investigation for Gaussian process-based surrogate modeling techniques in the field of continuous black-box optimization by means of the Covariance Matrix Adaptation Evolution Strategy (CMA-ES). We outline several state-of-the-art techniques and experimentally evaluate them within the optimization framework using recently proposed CEC'2013 benchmarking testbed.

Thesis on DSpace

prof. Ing. RNDr. Martin Holeňa, CSc.

Ing. Tomáš Borovička

Department of Theoretical Computer Science

The aim of this thesis is modelling of behaviour of dairy cows behavior and clustering of these models. Behavioral modelling can help to better understand dairy cows' behavior and to improve their welfare. With cluster analysis of these models, cows can be categorized and divided into groups based on their behavior patterns. In the thesis three different approaches of modelling animal behavior are proposed. For each approach the possibility of cluster analysis is discussed. Hidden Markov model was implemented and evaluated on real data and a dissimilarity measure used for model clustering was described. It has been shown, that hidden Markov model is able to model cows' behavior and models of particular animals can be clustered. Comparing the clusters we can find the differences between groups of cows that behave similarly.

prof. Ing. RNDr. Martin Holeňa, CSc.

Ing. Martin Flusser

Department of Applied Mathematics

This thesis deals with the problem of sentiment analysis from utterances by using LSTM networks. These are compared with some more widespread classification methods. Several approaches are proposed, implemented and compared to each other. The results are summarized.

Thesis on DSpace

prof. Ing. RNDr. Martin Holeňa, CSc.

Department of Software Engineering

This thesis is dedicated to modifying an existing system for discriminant chronicles mining, resulting in a system for discriminant chronicles mining capable of handling multi-dimensional input data. The modified system will be applied to real-world data concerning crystal growth.

Thesis on DSpace

prof. Ing. RNDr. Martin Holeňa, CSc.

Department of Software Engineering