cgeb2001's DokuWiki!

Deep Learning - A hyperparameter tuning game

Imagine one day you could look like alchemist to play magic or player in ball balance game to adventure.

To better understand the main point of this essay, it will be helpful to brush up some basics from another one Machine learning made easy. As for machine learning method, it usually includes six steps whether or not it is regression or classification model:

Loading the data
Create your model
Training and test data
Preprocessing the data
Evaluating your model’s performance
Tune your model

While for the deep learning, it will include more complex steps as follow:

Create your architecture (layers, nodes, activation)
- how many layers do you want? how many nodes in each layer? What activation function do you want to use in each layer?
Compile the model (loss function, and optimizer )
- This specifies the loss function, and some details about how optimization works
Fit the model
- Which is that cycle of back-propagation and optimization of model weights with your data.
Make predictions
- finally you will want to use your model to make predictions

No matter machine learning or more advanced deep learning algorithms, they all include a necessary step to re-tune the hyperparameters of the models being used. Hyperparameters are also known as human-tunable parameters but can be different when referring to various machine learning and artificial intelligence strategies. For examples:

In the Random Forest (RF) classifier, each tree is only allowed to consider a randomly chosen subset of features at each decision split. Users can specify the number of decision trees in the ensemble, the number of features to consider at each split, and the minimum number of instances per region;
In the Support Vector Machine (SVM) classifier, the binary classification is determined by the widest possible boundary between classes. Users can tune the parameter C, which defines the maximum number of misclassified instances allowed when maximizing the margin size;
In the K nearest neighbour (kNN) classifier, it predicts the label of a given instance based upon the majority label of its k nearest neighbors. Users can tune the value of k to acquire the best results.

While in the more advanced deep learning models, more hyperparameters will be involved. For examples, Artificial neural network (ANN) can include input layer, hidden layers and output layer connected by lines, and in each layer they can have certain number of nodes, which looks like a neuron network. Deep learning is one type of machine learning methods based on neural network, automatically extract features without manually prepare the features to be trained. If the output is incorrect, the network can re-adjust the weights of nodes to improve performance (back-propagation). Users can have many hyperparameters to re-tune the model, for examples:

(1) the number of hidden layers;
(2) the learning rate (which controls how the new parameters are set after back-propagation);
(3) how many instances to give the ANN at a time (batch size which is a number of samples processed before the model is updated. It must be more than or equal to one and less than or equal to the number of samples in the training dataset);
(4) how many times the NN should be trained on the entire input datasets (Epoch is defined as the total number of iterations for training the machine learning model with all the training data in one cycle. It is allowing the learning algorithm to run until the error from the model has been sufficiently minimized);
(5) how many epochs it must continue after the loss stopped from decreasing (patience). If you don't set patience value, the training will continue for all the epochs you set, even if your training results are not getting any better.
(6) early stopping is a form of regularization used to avoid overfitting when training a learner with an iterative method, such as gradient descent.

Sometimes, there are deep learning-specific parameters such as the models of convolutional neural network (CNN), recurrent neural network (RNN) and probabilistic neural network (PNN). CNN often used for image classification which utilize a convolution layer that preserves positional relationships between inputs (e.g. pixels in an image), thus capturing any dependencies among inputs. RNN discovers conditional dependencies of inputs by utilizing the output of previous inputs as features in classification, and can help model long range interactions. PNN estimates the probability distribution of each class, and the class with the highest posterior probability is assigned to each input.

Far beyond this, the selections of activation function (which decides whether a neuron should be activated or not by calculating the weighted sum and further adding bias to it, e.g., rectified linear unit (‘ReLU’) and ‘softmax’), optimizer (used to adjust the parameters for a model such as ‘adam’ or ‘sgd’. The purpose of an optimizer is to adjust model weights to maximize a loss function) and loss of function (which is a mathematical function that quantifies the difference between predicted and actual values in a machine learning model such as ‘mean_squared_error’ and ‘categorical_crossentropy’) can also impact the parameter selection of different Machine learning or deep learning models.

Fortunately, the new developed platforms (e.g., DataCamp), and tools like tensor flow and keras making it easier to use deep learning algorithms which were built-in as python library. Although the barrier has been lower to access the door, it is still challenging to optimize / re-tune the model to acquire best predictions. Although deep leaning require larger amount of training data and computational power comparing to other typical machine learning methods (such as, random forests (RF), support vector machine (SVM) k-nearest neighbors (kNN)), their accuracy evaluation methods are similar. For the classification model of deep learning or machine learning, Overall classification performance can be quantified by different metrics: TNR = true negative rate (also specificity), TPR = true positive rate (also sensitivity, recall), precision, accuracy, area under the receiver operating characteristic (AUROC) curve, PRC = precision-recall curve. For all of the above metrics, values closer to 1 indicate increasingly optimal performance. L is the positive class. TP = true positive, FP = false positive, TN = true negative, FN = false negative.

For the input data set may be split into a Training Set (usually 75–90% of the input data) and a Test Set. When a Test Set cannot be made, cross validation (CV) may be sufficient to estimate the algorithm’s error in classification of test instances. Here, five-fold CV is shown. In each fold, 20% of the input training data is randomly chosen to be the Validation Set. The model is trained using the training data, and its performance on new instances is determined through the Validation Set. Once the benchmark dataset has been spited into test data and train data, they will be fitted in different models to evaluate the performance. Since the data points in the test set may not be representative of the model's ability to generalize to unseen data. To combat this dependence on what is essentially an arbitrary split, we use a technique called cross-validation.

After knowing these performance formula and cross validation methods, it is important to understand the concepts of overfitting and underfitting, the former refers to your model will make accurate predictions on training data, but it will make inaccurate predictions on validation data and new datasets. Underfitting is the opposite. That is when your model fails to find important predictive patterns in the training data. Increasing the number of nodes in a hidden layer or add layers which can increase the model capacity.

References:

Li, R., Li, L., Xu, Y. and Yang, J., 2022. Machine learning meets omics: applications and perspectives. Briefings in Bioinformatics, 23(1), p.bbab460.
Mahood, E.H., Kruse, L.H. and Moghe, G.D., 2020. Machine learning: A powerful tool for gene function prediction in plants. Applications in Plant Sciences, 8(7), p.e11376.
Soltis, P.S., Nelson, G., Zare, A. and Meineke, E.K., 2020. Plants meet machines: Prospects in machine learning for plant biology. Applications in Plant Sciences, 8(6).