import sklearnScikit-Learn is a machine learning library based on Numpy, Scipy and Matplotlib. It stands out with its simple and easy to use syntax, setting defaults, which make it very beginner friendly, while also providing a wide spectrum of tools and the scalability required to make it applicable beyond basic custom test cases, by providing wrappers to other libraries.
Available models¶
In order to make an educated choice for the model to be applied, it helps to understand the structure of the data at hand. This is, however, a rather vague statement, which should encourage you to play around with models sometimes.
There are three basic approaches, depending on the type of data, namely supervised, unsupervised and reinforcement learning.
For the actual model, one can pick from a vast zoo, with common ones being
You can setup various different models with just a single line of code. For instance a neural network can be instantiated as
nn = sklearn.neural_network.MLPRegressor(
hidden_layer_sizes=(8, 8),
activation="tanh",
solver="lbfgs",
max_iter=5000,
)Let us get some data with 1,000 samples and a single feature organized in column vectors.
import numpy as np
X = np.linspace(0, 4 * np.pi, 1000).reshape(-1, 1)
y = np.sin(X).ravel()We split the date into training and test sets in proportions of 80% and 20%.
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(
X,
y,
test_size=0.2,
shuffle=True,
)Train the neural network using scaled feature data.
scaler = sklearn.preprocessing.StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)nn.fit(X_train_scaled, y_train)Predict output for the scaled test data.
y_pred = nn.predict(X_test_scaled)Plot the neural network predictions against the unscaled feature data.
Neural network design¶
This example illustrates that the choice of activation function is an important part of neural network design.
ReLU is often an excellent default choice for deep learning applications.
Smooth activation functions such as
tanhmay be better suited for approximating smooth, bounded functions such as here.There is no universally optimal activation function.
The performance of a neural network depends on the interplay between the problem, the network topology, the activation functions, and the training procedure.
Selecting an effective network architecture is therefore both a science, guided by theory and experience, and an art, refined through experimentation.
Example: Recognition of handwritten numbers¶
This tutorial uses scikit-learn (or sklearn), an open source machine learning (ML) framework based on numpy, scipy and matplotlib.
# Import datasets, classifiers and performance metrics
from sklearn import datasets, metrics, svm
from sklearn.model_selection import train_test_splitA widely used example for the use of machine learning is the recognition of handwritten numbers, for which we will build, train and assess the accuracy of a machine learning model. The example is so common in fact that sklearn, like many other ML frameworks, come with a built-in data set of handwritten numbers.
# Loading a data set containing 1797 8 x 8 images of digits
digits = datasets.load_digits()
print(type(digits.images))
print(digits.images.shape)<class 'numpy.ndarray'>
(1797, 8, 8)
fig, axes = plt.subplots(nrows=3, ncols=5, figsize=(10, 7))
for i, row in enumerate(axes):
for j, ax in enumerate(row):
ax.set_axis_off()
ax.imshow(
digits.images[i * 10 + j], cmap=plt.cm.gray_r, interpolation="nearest"
)
ax.set_title(f"Ground truth: {digits.target[i * 10 + j]}")
Splitting the dataset¶
For ML applications the data is always split in at least two datasets, since for proper evaluation of the performance of the trained model unseen data is required. Whether additional separation of the data set is required depends on the ML model applied, as well as the optimization algorithm.
# Flatten the images
n_samples = len(digits.images)
data = digits.images.reshape((n_samples, -1))
# Split data into 80% train and 20% test subsets, which is a standard ratio
X_train, X_test, y_train, y_test = train_test_split(
data, digits.target, test_size=0.2, shuffle=True
)Pick the model¶
With a rather modest amount of annotated data at hand to solve a classification type problem, it makes sense to use an SVM, since they can make use of different kernels, projecting lower dimensional parameter spaces into higher dimensional ones and therefore saving a lot on the parameters to fit. NNs are more flexible in their basic form, which comes with the price of more parameters.
# Create SVM classifier with a radial basis function (RBF) kernel
my_classifier = svm.SVC(gamma=0.001)Train the model¶
# Train the model on the train subset
my_classifier.fit(X_train, y_train)
# Remember to save your model to a file, if you want to use it again!!!
# This can be achieved e.g. by pickling my_classifier
# import pickle
# save
# with open('my_model.pkl','wb') as f:
# pickle.dump(my_classifier,f)
# load
# with open('my_model.pkl', 'rb') as f:
# my_classifier = pickle.load(f)Assess the accuracy¶
# Predict the value of the digit on the test subset
predicted = my_classifier.predict(X_test)
# Plot some predictions
fig, axes = plt.subplots(nrows=1, ncols=6, figsize=(13, 3))
for ax, image, prediction, GroundTruth in zip(axes, X_test, predicted, y_test):
ax.set_axis_off()
image = image.reshape(8, 8)
ax.imshow(image, cmap=plt.cm.gray_r, interpolation="nearest")
ax.set_title(f"Prediction: {prediction} \nGround truth: {GroundTruth}")
# Show how the classifier performs for the individual numbers
print(
f"Classification report for classifier {my_classifier}:\n"
f"{metrics.classification_report(y_test, predicted)}\n"
)Classification report for classifier SVC(gamma=0.001):
precision recall f1-score support
0 1.00 1.00 1.00 43
1 0.97 1.00 0.99 38
2 1.00 1.00 1.00 32
3 1.00 0.98 0.99 41
4 1.00 1.00 1.00 37
5 0.97 1.00 0.98 32
6 1.00 1.00 1.00 34
7 1.00 1.00 1.00 31
8 0.97 0.97 0.97 35
9 1.00 0.97 0.99 37
accuracy 0.99 360
macro avg 0.99 0.99 0.99 360
weighted avg 0.99 0.99 0.99 360
# Show confusion matrix
disp = metrics.ConfusionMatrixDisplay.from_predictions(y_test, predicted)
disp.figure_.suptitle("Confusion Matrix")
# uncomment to show it as 2D array
# print(f"Confusion matrix:\n{disp.confusion_matrix}")
plt.show()
# amount of coefficients to be fitted for the SVM used here
# 45 * 8 * 8 + 45
print(
f"The amount of coefficients used by the SVM model are "
f"{np.prod(my_classifier._get_coef().shape) + len(my_classifier.intercept_)}"
)
# amount of coefficients to be fitted for the NN from the previous notebook
# bias of first layer 28 * 28, hidden layer 512, and output layer 10
# connections 28 * 28 * 512 + 512 * 10
print(
f"The amount of coefficients used by the NN model of the previous "
f"notebook are {28 * 28 + 512 + 10 + 28 * 28 * 512 + 512 * 10}"
)
# adapted to the problem at hand with 8 x 8 images
print(
f"The amount of coefficients used by an NN model for this problem inspired "
f"by the previous notebook are {8 * 8 + 512 + 10 + 8 * 8 * 512 + 512 * 10}"
)The amount of coefficients used by the SVM model are 2925
The amount of coefficients used by the NN model of the previous notebook are 407834
The amount of coefficients used by an NN model for this problem inspired by the previous notebook are 38474
# build a NN with similar amount of parameters than SVM and compare performance
from sklearn.neural_network import MLPClassifier
my_nn_classifier = MLPClassifier(hidden_layer_sizes=(45))
my_nn_classifier.fit(X_train, y_train)
my_nn_prediction = my_nn_classifier.predict(X_test)
print(
f"The SVM has an accuracy of {metrics.accuracy_score(y_test, predicted):.4f}, "
f"while the NN has an accuracy of {metrics.accuracy_score(y_test, my_nn_prediction):.4f}"
)
nn_disp = metrics.ConfusionMatrixDisplay.from_predictions(y_test, my_nn_prediction)
nn_disp.figure_.suptitle("Confusion Matrix")
plt.show()The SVM has an accuracy of 0.9917, while the NN has an accuracy of 0.9444
As can be seen the overall accuracy is very good, with the confusion matrix showing which numbers have been confused with each other how often. It is also interesting to see how the model performance improved over the training iterations, so-called epochs.
max_iterations = np.array([0, 1, 2, 5, 10, 20, 50, 100, 200])
accuracies = []
for iterations in max_iterations:
my_classifier = svm.SVC(gamma=0.001, max_iter=iterations)
my_classifier.fit(X_train, y_train)
predicted = my_classifier.predict(X_test)
acc = metrics.accuracy_score(y_test, predicted)
accuracies.append(acc)Epochs Accuracy
0 0.102778
1 0.647222
2 0.811111
5 0.933333
10 0.983333
20 0.997222
50 0.988889
100 0.991667
200 0.991667
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(10, 4))
axes[0].set_title("first iterations")
axes[1].set_title("until convergence")
axes[0].plot(max_iterations[:4], accuracies[:4])
axes[1].plot(np.log10(max_iterations[1:]), accuracies[1:])
axes[0].set_xlabel("Number of epochs")
axes[1].set_xlabel("log(Number of epochs)")
axes[0].set_ylim(0, 1)
for ax in axes:
ax.set_ylabel("Accuracy with respect to the test set")
One can also investigate how the performance differs when using more or less data.
from sklearn.model_selection import LearningCurveDisplay, ShuffleSplit
# start from untrained model again
my_classifier = svm.SVC(kernel="rbf", gamma=0.001)
common_params = {
"X": np.concatenate((X_train, X_test), axis=0),
"y": np.concatenate((y_train, y_test), axis=0),
"train_sizes": np.linspace(0.1, 1.0, 5),
"cv": ShuffleSplit(n_splits=50, test_size=0.2, random_state=0),
"score_type": "both",
"n_jobs": 4,
"line_kw": {"marker": "o"},
"std_display_style": "fill_between",
"score_name": "Accuracy",
}
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(10, 6), sharey=True)
LearningCurveDisplay.from_estimator(my_classifier, **common_params, ax=ax)
handles, label = ax.get_legend_handles_labels()
ax.legend(handles[:2], ["Training Score", "Test Score"])
ax.set_title(f"Learning Curve for {my_classifier.__class__.__name__}")
plt.show()
Comparison between classifier models¶
Now one can start comparing the performance between different models, as e.g. done here with a naive Bayes classifier:
from sklearn.naive_bayes import GaussianNB
naive_bayes = GaussianNB()
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(10, 6), sharey=True)
for ax_idx, estimator in enumerate([naive_bayes, my_classifier]):
LearningCurveDisplay.from_estimator(estimator, **common_params, ax=ax[ax_idx])
handles, label = ax[ax_idx].get_legend_handles_labels()
ax[ax_idx].legend(handles[:2], ["Training Score", "Test Score"])
ax[ax_idx].set_title(f"Learning Curve for {estimator.__class__.__name__}")
plt.show()
Additional material¶
In order to make an educated decision on which model to pick, you should read through additional material, starting with very short summaries. YouTube also offers comprehensive introductions, like the one from scikit-learn itself.
from IPython.display import YouTubeVideo
YouTubeVideo("playlist?list=PL2okA_2qDJ-m44KooOI7x8tu85wr4ez4f")You can also use ChatGPT to get familiar with which machine learning solution fits your kind of problem considering the data and computational resources you have at hand. You should also watch out for comparisons between models like this, to get a glimpse of how the numerical trends differ between the methods.