Ing. Miroslav Skrbek, Ph.D.

Publications

Adaptive Input Normalization for Quantized Neural Networks

Year
2024
Published
Proceedings of the 27th International Symposium on Design and Diagnostics of Electronic Circuits & Systems. Piscataway: IEEE, 2024. p. 130-135. ISBN 979-8-3503-5934-3.
Type
Proceedings paper
Annotation
Neural networks with quantized activation functions cannot adapt the quantization at the input of their first layer. Preprocessing is therefore required to adapt the range of input data to the quantization range. Such preprocessing usually includes an activation-wise linear transformation and is steered by the properties of the training set. We suggest to include the linear transform into the training process. We document that it improves accuracy, requires the same resources as standard preprocessing, plays a role in network pruning, and is reasonably stable with respect to initialization.

Evaluation of the Medium-sized Neural Network using Approximative Computations on Zynq FPGA

Year
2023
Published
Proceedings of 2023 12th Mediterranean Conference on Embedded Computing (MECO). Piscataway: IEEE, 2023. p. 1-4. ISSN 2637-9511. ISBN 979-8-3503-2291-0.
Type
Proceedings paper
Annotation
Integrating artificial intelligence technologies into embedded systems requires efficient implementation of neural networks in hardware. The paper presents a Zynq 7020 FPGA implementation and evaluation of a middle-sized dense neural network based on approximate computation by linearly approximated functions. Three famous benchmarks were used for classification accuracy evaluation and hardware testing. We use our highly pipelined neural hardware architecture that takes weights from block RAMs to save logic resources and enables their update from the processing system. The architecture reaches excellent design scalability, allowing us to estimate the number of neurons implemented in programmable logic based on single-neuron resources. We reached nearly full chip utilization while preserving the high clock frequency for the FPGA used.

Approximate arithmetic for modern neural networks and FPGAs

Year
2022
Published
Proceedings of the 11th Mediterranean Conference on Embedded Computing (MECO 2022). Institute of Electrical and Electronics Engineers, Inc., 2022. p. 351-354. ISSN 2377-5475. ISBN 978-1-6654-6828-2.
Type
Proceedings paper
Annotation
Approximate arithmetic is a very important approach for implementing neural networks in embedded hardware. The requirements of real-time applications with respect to the size of deep neural networks force designers to simplify the neural processing elements. Not only the reduction of precision of model parameters to a few bits, but also the use of approximate arithmetic increases computational power and saves on-chip resources beyond exact computation. Since it was shown that linearly approximated functions are suitable for implementing neural networks in hardware, FPGAs have improved. So we decided to re-implement the processing element on a modern FPGA and to present implementation results regarding speed and resource consumption. A neural processing element based on linearly approximated functions was implemented in Vivado and tested on an xc7 FPGA. The results show that the architecture saves significant resources and a clock frequency above 100 MHz can be achieved in pipelined design.

Feasibility of a Neural Network with Linearly Approximated Functions on Zynq FPGA

Year
2022
Published
2022 29th IEEE International Conference on Electronics, Circuits and Systems (ICECS). New York: Institute of Electrical and Electronics Engineers, 2022. ISBN 978-1-6654-8823-5.
Type
Proceedings paper
Annotation
This paper is focused on the feasibility of a neural network with linearly approximated functions on modern FPGA. An approximate multiplier and linearly approximated activation functions were used for a neural network implemented on Zynq FPGA. We proposed a novel architecture for a fully functional, layered, and configurable neural network.

High-Performance Spiking Neural Network Simulator

Authors
Year
2019
Published
Proceedings of the 8th Mediterranean Conference on Embedded Computing - MECO'2019. Institute of Electrical and Electronics Engineers, Inc., 2019. p. 88-91. ISSN 2377-5475. ISBN 978-1-7281-1739-3.
Type
Proceedings paper
Annotation
Simulation of neural networks is a significant task for contemporary artificial intelligence research. Despite the availability of modern processing hardware, the task is still too demanding to be done in a sequential way. Therefore, a parallel computation approach is almost always necessary. Modern graphical accelerators (GPUs) represent highly parallel machines with a significant computational performance that can be unleashed only under certain conditions including threads scheduling, proper sources occupation, aligned data access, communication management, etc. We have proposed a novel acceleration approach for large neural networks. It is using a GPU and incorporating biologically highly precise spiking neurons that can imitate real biological neurons. The simulator can be, for example, used for research of communication dynamics of large neural networks with tens of thousands of spiking neurons.

Humanoid Robot Control by Offline Actor-Critic Learning

Authors
Danel, M.; Skrbek, M.
Year
2017
Published
ITAT 2017: Information Technologies – Applications and Theory. Aachen: CEUR Workshop Proceedings, 2017. p. 71-77. vol. 1885. ISSN 1613-0073.
Type
Proceedings paper
Annotation
In this paper, we present our results on application of reinforcement learning on full body control of a humanoid robot. The task we try to learn is achieving vertical position of robot’s torso from an initial position of laying flat on the ground. Our experimental setup includes an instance of the NAO robot in theWebots simulation environment. We use an actor-critic neural agent. As this is a work in progress, we only train offline from a sample of random movements. We present a series of experiments on a simplified task and a final evaluation on the humanoid robot control task that shows improvement over random policy.

Exploration Robot with Stereovision

Authors
Skrbek, M.; Richter, V.
Year
2014
Published
Proceedings of the 2nd Prague Embedded Systems Workshop. 2014.
Type
Proceedings paper
Annotation
This article presents a four-wheeled robot equipped with two cameras for stereoscopic vision. The robot is remotely controlled from a mobile device running the Android operating system. On the display, an operator sees 3D anaglyph image from stereoscopic cameras and the depth map that allows determining the distance to nearby objects in front of the robot. The robot has rich set of advanced sensors like inertial sensors, infrared low distance sensors, microphone array for sound localization, speech recognition and voice output. Wi-Fi is used for connection between the robot and the mobile device. Control commands are sent via the TCP/IP protocols to the robot. A robot control application on the mobile device has been developed for the Android operating system. This is a layered application consisting of the communication and robot control API and the graphical user interface. On start, the application retrieves a list of available commands in the XML format describing capabilities of the robot. The application can adapt its user interface according to this list. We also tested the Kinect sensor as a replacement of cameras and have developed a control application receiving all data available on Kinect.

Spatio-Temporal Data Classification using CVNNs

Authors
Zahradník, J.; Skrbek, M.
Year
2013
Published
Simulation Modelling Practice and Theory. 2013, 33(33), 81-88. ISSN 1569-190X.
Type
Article
Annotation
This paper presents two new approaches of spatio-temporal data classification using complex-valued neural networks. First approach uses extended complex-valued backpropagation algorithm to train MLP network, whose output’s amplitudes are encoded in one-of-N coding. It makes a classification decision based on accumulated distance between network output and trained pattern. The second approach is inspired in RBF networks with two layer architecture. Neurons from the first layer have fixed position in space and time encoded into theirs weights. This layer is trained by presented extension of neural gas algorithm into complex numbers. The second layer affects which neurons from the first layer belong to specific class. Paper contains details on experimenting with proposed approaches on artificial data of hand-written character recognition and comparison of both methods.

Supervised two-step feature extraction for structured representation of text data

Authors
Háva, O.; Kordík, P.; Skrbek, M.
Year
2013
Published
Simulation Modelling Practice and Theory. 2013, 33(33), 132-143. ISSN 1569-190X.
Type
Article
Annotation
Training data matrix used for classification of text documents to multiple categories is characterized by large number of dimensions while the number of manually classified training documents is relatively small. Thus the suitable dimensionality reduction techniques are required to be able to develop the classifier. The article describes two-step supervised feature extraction method that takes advantage of projections of terms into document and category spaces. We propose several enhancements that make the method more efficient and faster than it was presented in our former paper. We also introduce the adjustment score that enables to correct defected targets or helps to identify improper training examples that bias extracted features.

Vector representation of context networks of latent topics

Authors
Háva, O.; Skrbek, M.; Kordík, P.
Year
2013
Published
Proceedings of the World Congress on Engineering 2013. Hong Kong: Newswood Limited - International Association of Engineers, 2013. pp. 286-290. ISSN 2078-0958. ISBN 978-988-19251-0-7.
Type
Proceedings paper
Annotation
Transforming of text documents to real vectors is an essential step for text mining tasks such as classification, clustering and information retrieval. The extracted vectors serve as inputs for data mining models. Large vocabularies of natural languages imply a high dimensionality of input vectors; hence a substantial dimensionality reduction has to be made. We propose a new approach to a vector representation of text documents. Our representation takes into account an order of latent topics that generate observed words; an extracted document vector includes information about the adjacency of words in a document. We experimentally proved that the proposed representation enables to build document classifiers of higher accuracy using shorter document vectors. Short but informative document vectors enable to save memory for storing data, to use simpler models that learn faster and to significantly reduce an overfit effect.

Contextual latent semantic networks used for document classification

Authors
Háva, O.; Skrbek, M.; Kordík, P.
Year
2012
Published
Proceedings of the International Conference on Knowledge Discovery and Information Retrieval. Porto: SciTePress - Science and Technology Publications, 2012. pp. 425-430. ISBN 978-989-8565-29-7.
Type
Proceedings paper
Annotation
Widely used document classifiers are developed over a bag-of-words representation of documents. Latent semantic analysis based on singular value decomposition is often employed to reduce the dimensionality of such representation. This approach overlooks word order in a text that can improve the quality of classifier. We propose language independent method that records the context of particular word into a context network utilizing products of latent semantic analysis. Words' contexts networks are combined to one network that represents a document. A new document is classified based on a similarity between its network and training documents networks. The experiments show that proposed classifier achieves better performance than common classifiers especially when a foregoing reduction of dimensionality is significant.

Document Classification with Supervised Latent Feature Selection

Authors
Háva, O.; Skrbek, M.; Kordík, P.
Year
2012
Published
Proceedings of the 2nd International Conference on Web Intelligence, Mining and Semantics. New York: ACM, 2012. pp. 70-74. ISBN 978-1-4503-0915-8.
Type
Proceedings paper
Annotation
The classification of text documents generally deals with large dimensional data. To favor generality of classification researcher has to apply a dimensionality reduction technique before building a classifier. We propose classification and reduction algorithm that makes use of latent uncorrelated topics extracted from training documents and their known categories. We suggest several latent feature selection options and provide their testing.

Classification of Spatio-Temporal Data Using Complex-Valued Neural Networks

Authors
Zahradník, J.; Skrbek, M.
Year
2011
Published
Proceeding of the 7th Doctoral Workshop on Mathematical and Engineering Methods in Computer Science. Brno: Brno University of Technology, 2011. pp. 94-101. ISBN 978-80-214-4305-1.
Type
Proceedings paper
Annotation
This paper presents a new approach in spatio-temporal classification utilizing the fact, that spatio-temporal data can be easily transformed into polar form of complex numbers. The proposed classifier uses two-layer complex-valued neural network inspired by the RBF networks. Weights of the first layer determine spatio-temporal position of neurons and are trained using neural gas algorithm to appropriately cover the training data set. The second layer divides neurons from the first layer into different classes and is trained in single pass. The classification itself is based on accumulating distance error from neural network. Paper contains details on experimenting with proposed approach on artificial data of hand-written character recognition.

Time Series Classification by Reconstruction and Approximation of Phase Space

Authors
Podhorský, P.; Skrbek, M.
Year
2011
Published
Artificial Intelligence and Soft Computing 2011. Calgary: IASTED, 2011. pp. 273-279. ISBN 9780889868946.
Type
Proceedings paper
Annotation
In this paper, a new algorithm for time series classification is proposed. It is based on comparison of approximated phase spaces reconstructed from data for classification. The reconstruction process embeds advanced feature selection controlled by a genetic algorithm providing the most significant subset of components of phase vectors with respect to maximization of separability of different data classes. Phase space is approximated by a set of Gaussian kernels located in a regular multidimensional grid. Phase spaces are compared point by point within the grid. The proposed algorithm has been tested on artificial data to study its behaviour and properties, and on real data of human motion measured using 3-axis accelerometer.

Classification of Spatio-Temporal Data

Authors
Zahradník, J.; Skrbek, M.
Year
2010
Published
Proceedings of the 7th EUROSIM Congress on Modelling and Simulation, Vol. 2: Full Papers. Prague: Department of Computer Science and Engineering, FEE, CTU in Prague, 2010. pp. 1168-1173. ISBN 978-80-01-04589-3.
Type
Proceedings paper
Annotation
This paper presents a new approach in spatio-temporal data classification. Due to easy transformation of time dimension of spatio-temporal data into the phase of complex number, the presented approach uses complex numbers. The classification is based on a complex-valued neural network with multilayer topology. The paper proposes an extension of complexvalued backpropagation algorithm, which uses activation function applying nonlinearity on the amplitude only (preserving the phase). In order to transform the input data into complex numbers, a new coding technique is presented. Output coding is developed allowing the classification from complex numbers. It is solved with introduction of one-of-N coding extension into complex numbers, which is used as network's output coding. This approach is verified in application of hand-written character recognition, using the data collected during the writing process. The simulation results of this application are presented in the paper.

FAST SUPERVISED FEATURE EXTRACTION FROM STRUCTURED REPRESENTATION OF TEXT DATA

Authors
Háva, O.; Skrbek, M.
Year
2010
Published
Proceedings of the 7th EUROSIM Congress on Modelling and Simulation, Vol. 2: Full Papers. Prague: Department of Computer Science and Engineering, FEE, CTU in Prague, 2010. pp. 149-155. ISBN 978-80-01-04589-3.
Type
Proceedings paper
Annotation
Classification of text documents is challenging problem not only when browsing the web. Structure representation of documents is necessary to build up appropriate classifier. Unfortunately the document-term matrices are usually so sparse and of high dimensionality due large number of terms representing usually smaller number of the documents. Thus the suitable dimensionality reduction technique is required to be able to develop the classifier. The article deals with supervised extraction method that results to small number of sensitive features derived from the initial document-term matrix. The extraction process simulated by neural network is remarkably fast and utilizes all available supervised information from training data.

ROAD MESH MODELLING USING THE SPATIOTEMPORAL CLUSTERIZATION

Authors
Marek, R.; Skrbek, M.
Year
2010
Published
Proceedings of the 7th EUROSIM Congress on Modelling and Simulation, Vol. 2: Full Papers. Prague: Department of Computer Science and Engineering, FEE, CTU in Prague, 2010. pp. 139-142. ISBN 978-80-01-04589-3.
Type
Proceedings paper
Annotation
The GPS navigation is widely used aid for travelers. However, good navigation depends on good maps, which are sometimes hard to get. In this paper we explore a method to model a road mesh using self-organizing spatiotemporal data clustering of collected GPS data. The resulting road mesh model is obtained from simulated self organizing neural network, which clusters the individual data vectors and infers the time dependencies between the clusters. This allows to detect one way roads, or slow traffic automatically. To achive this goal we model the road mesh with the Temporal Hebbian Self-organizing map (THSOM). This paper presents a novel training method for the simulated THSOM neural network, which reduces training period and improves model the convergence of original THSOM neural network. The road mesh model is obtained from real GPS data as well as from simulated data set.

Simulation and Genetic Evolution of Spiking Neural Networks

Authors
Podhorský, P.; Skrbek, M.
Year
2010
Published
Proceedings of the 7th EUROSIM Congress on Modelling and Simulation, Vol. 2: Full Papers. Prague: Department of Computer Science and Engineering, FEE, CTU in Prague, 2010. pp. 1161-1167. ISBN 978-80-01-04589-3.
Type
Proceedings paper
Annotation
In this paper, an implementation of a simulator for spiking neural networks and learning algorithm using genetic evolution is described. We have implemented two neural models (simplified Spike Response Model and Integrate and Fire model) and two learning algorithms - SpikeProp and a simple genetic learning algorithm (presented in this paper). This learning algorithm is easy to understand and does not require any special network topology like feed-forward networks and back-propagation algorithms or thorough investigation of network architecture. It is based on an assumption that small changes in network variables have a small impact on the output and big changes have a big impact, thus calculating a difference between a desired output and a real output and mutating individuals according to a size of this difference we can expect a population to converge.

SIMULATION OF ANT COLONIES WITH HINTS GENERATED BY PARALLEL HEURISTICS

Authors
Kovářík, O.; Skrbek, M.
Year
2010
Published
Proceedings of the 7th EUROSIM Congress on Modelling and Simulation, Vol. 2: Full Papers. Prague: Department of Computer Science and Engineering, FEE, CTU in Prague, 2010. p. 126-130. ISBN 978-80-01-04589-3.
Type
Proceedings paper
Annotation
In this paper we present a new approach for combining discrete optimization algorithms, which resulted from a simulation of several scenarios of information exchange between algorithms running in parallel. We show that a group of parallel metaheuristics can create probabilistic "hints" for colony of artificial ants, which solves the same instance of a Traveling Salesman Problem (TSP). These "hints" are realized by creating intersections of solutions from fast independent solvers and they can be used to improve optimization process. A convergence speedup is shown for a group of Simulated Annealing algorithms together with Max-Min Ant System on non-geometric TSP and for a given time window and random instances of sizes in of order of magnitude of 10^3. Presented model of cooperation between algorithms is a first step on the path to a complex metaoptimization system based on combinations of heuristics.