Machine Learning (ML) using MATLAB
Machine Learning (ML) is a core area of artificial intelligence, enabling systems to learn from data and make predictions. MATLAB provides a robust environment for ML through built-in functions, toolboxes, and an easy-to-use interface.
What is Machine Learning?
Machine Learning involves training models on datasets to identify patterns and make decisions. The process can be divided into:
- Supervised Learning: Training with labeled data.
- Unsupervised Learning: Training without labeled data.
- Reinforcement Learning: Training based on rewards and penalties.
Creating Datasets
Datasets are critical for training ML models. Here’s how to create and manipulate datasets in MATLAB:
% Creating a dataset
data = [1, 2; 3, 4; 5, 6];
labels = [1; 0; 1]; % Binary labels
dataset = table(data(:,1), data(:,2), labels, 'VariableNames', {'Feature1', 'Feature2', 'Label'});
disp(dataset);
Feature1 Feature2 Label
________ ________ _____
1 2 1
3 4 0
5 6 1
Training a Simple Classifier
MATLAB makes it simple to train a classifier using built-in functions:
% Training a simple SVM classifier
Mdl = fitcsvm(data, labels, 'KernelFunction', 'linear', 'Standardize', true);
disp(Mdl);
Example 1: Linear Regression
Linear regression is a fundamental supervised learning algorithm used to predict continuous values. Below is an example:
% Example: Linear Regression
x = [1 2 3 4 5]; % Input data
y = [2.2 4.3 5.8 8.1 10.4]; % Target values
p = polyfit(x, y, 1); % Fit a linear model
y_pred = polyval(p, x); % Predict values
plot(x, y, 'o', x, y_pred, '-');
title('Linear Regression');
xlabel('Input');
ylabel('Output');
legend('Data', 'Linear Fit');
Output Graph:
- Data points are marked as circles.
- Predicted linear fit line is shown.
Types of Machine Learning Problems
- Classification: Predict categories (e.g., spam or not spam).
- Regression: Predict continuous values (e.g., house prices).
- Clustering: Group data into clusters (e.g., customer segmentation).
Example 2: k-Means Clustering
% k-Means Clustering
data = [randn(10,2)+5; randn(10,2)-5];
[idx, C] = kmeans(data, 2); % Perform clustering into 2 groups
gscatter(data(:,1), data(:,2), idx);
hold on;
plot(C(:,1), C(:,2), 'kx', 'MarkerSize', 15);
title('k-Means Clustering');
xlabel('Feature 1');
ylabel('Feature 2');
Example 3: Decision Tree
% Decision Tree for Classification
Mdl = fitctree(data, labels); % Train decision tree
view(Mdl, 'Mode', 'graph'); % Visualize the tree
Example 4: Support Vector Machines (SVMs) in MATLAB
Support Vector Machines (SVMs) are supervised learning models used for classification and regression tasks. SVMs work by finding the hyperplane that best separates data into different classes.
Example: Binary Classification Using SVM
In this example, we will train an SVM classifier to separate two classes of data. The data points will be randomly generated, and the SVM will classify them based on a linear kernel.
Step 1: Generate Synthetic Data
We create two clusters of data points, each representing a different class:
% Generate synthetic data
rng(1); % For reproducibility
class1 = [randn(50, 2) + 1.5]; % Cluster 1
class2 = [randn(50, 2) - 1.5]; % Cluster 2
data = [class1; class2];
labels = [ones(50, 1); -ones(50, 1)]; % 1 for class1, -1 for class2
scatter(class1(:,1), class1(:,2), 'r', 'filled'); hold on;
scatter(class2(:,1), class2(:,2), 'b', 'filled');
title('Synthetic Data');
xlabel('Feature 1');
ylabel('Feature 2');
Output Graph:
- Red points represent Class 1.
- Blue points represent Class 2.
Step 2: Train the SVM Classifier
We use MATLAB’s fitcsvm function to train the classifier:
% Train an SVM classifier
Mdl = fitcsvm(data, labels, 'KernelFunction', 'linear', 'Standardize', true);
disp(Mdl); % Display model details
Step 3: Visualize the Decision Boundary
Plot the SVM’s decision boundary to observe how well the model separates the data:
% Visualize the decision boundary
d = 0.02; % Step size for grid
[x1Grid, x2Grid] = meshgrid(min(data(:,1)):d:max(data(:,1)), min(data(:,2)):d:max(data(:,2)));
gridData = [x1Grid(:), x2Grid(:)];
[~, score] = predict(Mdl, gridData); % Get scores for grid points
contour(x1Grid, x2Grid, reshape(score(:,2), size(x1Grid)), [0 0], 'k', 'LineWidth', 2);
title('SVM Decision Boundary');
xlabel('Feature 1');
ylabel('Feature 2');
Output Graph:
- Decision boundary (black line) separates the two classes.
- Red points (Class 1) and blue points (Class 2) lie on opposite sides of the boundary.
Step 4: Test the Classifier
We test the classifier on new data points to evaluate its performance:
% Test the classifier
testData = [0 0; 1 1; -1 -1; 2 2]; % New data points
predictedLabels = predict(Mdl, testData);
disp('Test Data Predictions:');
disp(table(testData(:,1), testData(:,2), predictedLabels, 'VariableNames', {'Feature1', 'Feature2', 'PredictedLabel'}));
Output Table:
Feature1 Feature2 PredictedLabel
________ ________ ______________
0 0 -1
1 1 1
-1 -1 -1
2 2 1
Example 5: Hierarchical Clustering in MATLAB
Hierarchical clustering is a method of cluster analysis that seeks to build a hierarchy of clusters. It is commonly used for exploratory data analysis in pattern recognition and machine learning.
Example: Hierarchical Clustering of Synthetic Data
In this example, we perform hierarchical clustering on a set of synthetic data points and visualize the resulting dendrogram and clusters.
Step 1: Generate Synthetic Data
We create a set of random data points to represent our dataset:
% Generate synthetic data
rng(2); % For reproducibility
data = [randn(10, 2) * 0.75 + 1; % Cluster 1
randn(10, 2) * 0.5 - 1; % Cluster 2
randn(10, 2) * 0.6 + [3 -2]]; % Cluster 3
scatter(data(:,1), data(:,2), 'filled');
title('Synthetic Data for Hierarchical Clustering');
xlabel('Feature 1');
ylabel('Feature 2');
Output Graph:
- Scatter plot showing three distinct clusters in the dataset.
Step 2: Perform Hierarchical Clustering
We use MATLAB’s linkage function to compute the hierarchical clusters and dendrogram to visualize the hierarchy:
% Perform hierarchical clustering
Z = linkage(data, 'ward'); % Using Ward's method
figure;
dendrogram(Z);
title('Hierarchical Clustering Dendrogram');
xlabel('Data Points');
ylabel('Distance');
Output Graph:
- A dendrogram that shows the hierarchical structure of the clusters.
Step 3: Determine Clusters from Dendrogram
We select a threshold distance to define clusters and assign cluster labels to the data points:
% Determine clusters from dendrogram
T = cluster(Z, 'cutoff', 1.5, 'criterion', 'distance');
gscatter(data(:,1), data(:,2), T);
title('Data Points Grouped by Clusters');
xlabel('Feature 1');
ylabel('Feature 2');
Output Graph:
- Data points are color-coded by their assigned clusters.
Step 4: Evaluate the Results
We evaluate the clustering results by visualizing the centroids of the clusters:
% Evaluate clustering results
centroids = grpstats(data, T); % Compute cluster centroids
hold on;
scatter(centroids(:,1), centroids(:,2), 100, 'kx', 'LineWidth', 2);
legend('Cluster 1', 'Cluster 2', 'Cluster 3', 'Centroids');
title('Clustered Data with Centroids');
Output Graph:
- Cluster centroids are marked with black crosses on the scatter plot.
Example 6: Principal Component Analysis (PCA) in ML using MATLAB
Principal Component Analysis (PCA) is a dimensionality reduction technique that projects high-dimensional data onto a lower-dimensional subspace while retaining as much variance as possible. It is widely used in machine learning and data visualization.
Example: PCA for Dimensionality Reduction
In this example, we perform PCA on a dataset to reduce its dimensionality and visualize the results in 2D.
Step 1: Generate Synthetic Data
We create a synthetic dataset with three features for demonstration:
% Generate synthetic data
rng(10); % For reproducibility
data = [randn(50, 1) * 2 + 5, randn(50, 1) * 3 - 4, randn(50, 1) * 1.5 + 2];
figure;
scatter3(data(:,1), data(:,2), data(:,3), 'filled');
title('Original 3D Data');
xlabel('Feature 1');
ylabel('Feature 2');
zlabel('Feature 3');
Output Graph:
- A scatter plot visualizing the 3D data in the original feature space.
Step 2: Perform PCA
We use MATLAB’s pca function to compute the principal components and reduce the data to two dimensions:
% Perform PCA
[coeff, score, ~, ~, explained] = pca(data);
disp('Variance Explained by Each Principal Component (%):');
disp(explained);
figure;
scatter(score(:,1), score(:,2), 'filled');
title('Data Projected onto First Two Principal Components');
xlabel('Principal Component 1');
ylabel('Principal Component 2');
Output:
- Variance Explained by Each Principal Component (%):
[Value for PC1, Value for PC2, Value for PC3]
Output Graph:
- A 2D scatter plot of data projected onto the first two principal components.
Step 3: Visualize the Principal Components
We visualize the principal components in the original feature space to understand their orientation:
% Visualize principal components
hold on;
quiver3(mean(data(:,1)), mean(data(:,2)), mean(data(:,3)), ...
coeff(1,1), coeff(2,1), coeff(3,1), 5, 'r', 'LineWidth', 2); % PC1
quiver3(mean(data(:,1)), mean(data(:,2)), mean(data(:,3)), ...
coeff(1,2), coeff(2,2), coeff(3,2), 5, 'g', 'LineWidth', 2); % PC2
legend('Data Points', 'Principal Component 1', 'Principal Component 2');
title('Principal Components in 3D Space');
grid on;
Output Graph:
- A 3D scatter plot of the data with arrows representing the principal components.
Step 4: Reconstruct Data Using Principal Components
We reconstruct the data using only the first two principal components:
% Reconstruct data using first two principal components
reconstructedData = score(:,1:2) * coeff(:,1:2)' + mean(data);
figure;
scatter3(reconstructedData(:,1), reconstructedData(:,2), reconstructedData(:,3), 'filled');
title('Reconstructed Data (Using First Two Principal Components)');
xlabel('Feature 1');
ylabel('Feature 2');
zlabel('Feature 3');
Output Graph:
- A 3D scatter plot of the reconstructed data.
Example 7: Hyperparameter Tuning and Optimization in MATLAB
Hyperparameter tuning involves finding the best set of hyperparameters for a machine learning model to optimize its performance. MATLAB provides tools such as bayesopt, GridSearch, and RandomSearch for this purpose.
Example: Hyperparameter Optimization for a Support Vector Machine (SVM)
In this example, we optimize the hyperparameters of an SVM classifier using Bayesian optimization.
Step 1: Load and Prepare Data
We use MATLAB’s built-in Fisher Iris dataset for this demonstration:
% Load and prepare data
load fisheriris;
X = meas; % Features
Y = species; % Labels
% Split data into training and test sets
cv = cvpartition(Y, 'HoldOut', 0.3);
XTrain = X(training(cv), :);
YTrain = Y(training(cv));
XTest = X(test(cv), :);
YTest = Y(test(cv));
Step 2: Define the Hyperparameter Optimization Problem
We define a function to train the SVM and return the classification error as the objective to minimize:
% Define objective function
function err = svmObjectiveFcn(params)
C = params.C; % Regularization parameter
kernelScale = params.kernelScale; % Kernel scale
% Train SVM with given parameters
svmModel = fitcsvm(XTrain, YTrain, 'KernelFunction', 'rbf', ...
'BoxConstraint', C, 'KernelScale', kernelScale);
% Perform cross-validation
cvModel = crossval(svmModel, 'KFold', 5);
err = kfoldLoss(cvModel); % Return cross-validation error
end
Step 3: Perform Bayesian Optimization
We use bayesopt to optimize the SVM hyperparameters:
% Define hyperparameter ranges
params = [optimizableVariable('C', [1e-3, 1e3], 'Transform', 'log'), ...
optimizableVariable('kernelScale', [1e-3, 1e3], 'Transform', 'log')];
% Run Bayesian optimization
results = bayesopt(@(params) svmObjectiveFcn(params), params, ...
'AcquisitionFunctionName', 'expected-improvement-plus', ...
'MaxObjectiveEvaluations', 30);
bestParams = results.XAtMinObjective;
disp('Best Hyperparameters:');
disp(bestParams);
Output:
- The optimal values for `C` and `kernelScale` as determined by Bayesian optimization.
Step 4: Evaluate the Optimized Model
We train the SVM model using the best hyperparameters and evaluate its performance on the test set:
% Train SVM with optimal parameters
optimizedSVM = fitcsvm(XTrain, YTrain, 'KernelFunction', 'rbf', ...
'BoxConstraint', bestParams.C, ...
'KernelScale', bestParams.kernelScale);
% Test the model
YPred = predict(optimizedSVM, XTest);
accuracy = mean(YPred == YTest);
disp('Test Set Accuracy:');
disp(accuracy);
Output:
- The accuracy of the optimized SVM model on the test set.
Useful MATLAB Functions for Machine Learning
fitcsvm to normalize input data.Practice Questions
Test Yourself
1. Train a linear regression model on a dataset and predict new values.
2. Perform k-means clustering on a dataset and visualize the clusters.
3. Train a decision tree classifier and visualize the decision boundaries.
4. Generate a new dataset with three clusters and train an SVM to classify two of them.
5. Change the linkage method to ‘average’ or ‘complete’ and observe how the dendrogram changes.