Scikit-Learn - Support Vector Machine¶
Table of Contents¶
Introduction ¶
Support Vector Machine constructs a hyperplane or list of hyperplanes in high dimensional space, which are then used for classification/regression tasks or other tasks like outlier detection.
Below is a list of SVM versions provided by sklearn.
- Classification Tasks
- LinearSVC
- SVC
- NuSVC
- Regression Tasks
- LinearSVR
- SVR
- NuSVR
import numpy as np
import pandas as pd
import sklearn
import warnings
warnings.filterwarnings('ignore')
np.set_printoptions(precision=2)
%matplotlib inline
Load Dataset¶
We'll be loading below mentioned two for our purpose.
- Digits Dataset: We'll be using digits dataset which has images of size
8x8for digits0-9. We'll use digits data for classification tasks below. - Boston Housing Dataset: We'll be using the Boston housing dataset which has information about various house properties like average no of rooms, per capita crime rate in town, etc. We'll be using it for regression tasks.
Sklearn provides both of this dataset as a part of the datasets module. We can load them by calling load_digits() and load_boston() methods. It returns dictionary-like object BUNCH which can be used to retrieve features and target.
from sklearn.datasets import load_boston, load_digits
digits = load_digits()
X_digits, Y_digits = digits.data, digits.target
print('Dataset Size : ',X_digits.shape, Y_digits.shape)
boston = load_boston()
X_boston, Y_boston = boston.data, boston.target
print('Dataset Size : ',X_boston.shape, Y_boston.shape)
LinearSVR ¶
The support vector machine model that we'll be introducing is LinearSVR. It is available as a part of svm module of sklearn. We'll divide the regression dataset into train/test sets, train LinearSVR with default parameter on it, evaluate performance on the test set and then tune model by trying various hyperparameters to improve performance further. We'll also introduce various important attributes of the trained model which can give useful insights once the model is trained.
Splitting Dataset into Train & Test sets¶
We'll split the dataset into two parts:
Training datawhich will be used for the training model.Test dataagainst which accuracy of the trained model will be checked.
train_test_split function of the model_selection module of sklearn will help us split data into two sets with 80% for training and 20% for test purposes. We are also using seed(random_state=123) with train_test_split so that we always get the same split and can reproduce results in the future as well.
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X_boston, Y_boston, train_size=0.80, test_size=0.20, random_state=123)
print('Train/Test Sizes : ', X_train.shape, X_test.shape, Y_train.shape, Y_test.shape)
Fitting Default LinearSVR to Train Data¶
from sklearn.svm import LinearSVR, NuSVR, OneClassSVM
linear_svr = LinearSVR(max_iter=1000000)
linear_svr.fit(X_train, Y_train)
Evaluate Model Accuracy on Test Set¶
Y_preds = linear_svr.predict(X_test)
print(Y_preds[:10])
print(Y_test[:10])
print('Test R^2 Score : %.3f'%linear_svr.score(X_test, Y_test)) ## Score method also evaluates accuracy for classification models.
print('Training R^2 Score : %.3f'%linear_svr.score(X_train, Y_train))
Important Attributes of Estimator¶
LinearSVR provides a list of important attributes that can provide important insights one model is trained. Below is a list of attributes available through LinearSVR.
coef_- It returns an array representing weights assigned to each feature by model. It represents the importance of each feature as per model trained.intercept_- It represents intercept of linear kernel function.
print("Feature Importances :", linear_svr.coef_)
print("Model Intercept :", linear_svr.intercept_)
Finetuning Model By Doing Grid Search On Various Hyperparameters¶
Below is a list of common hyperparameters that need tuning for getting the best fit for our data. We'll try various hyperparameters settings to various splits of train/test data to find out best fit which will have almost the same accuracy for both train & test dataset or have quite less difference between accuracy.
- C - It represents regularization applied to the linear kernel function. The strength of normalization is inversely proportional to C which means that low C will result in high regularization and vice-versa. The default value of
1.0is set. - max_iter - It specifies the number of iteration to try before stopping algorithm. The value of
-1represents no limit and algorithm runs until convergence.
We'll below try various values for the above-mentioned hyperparameters to find the best estimator for our dataset by doing 5-fold cross-validation on data.
%%time
from sklearn.model_selection import GridSearchCV
params = {
'C': [0.1, 0.5, 1.0, 10.0],
}
linear_svr_regressor_grid = GridSearchCV(LinearSVR(random_state=1, max_iter=1000000), param_grid=params, n_jobs=-1, cv=5, verbose=5)
linear_svr_regressor_grid.fit(X_train,Y_train)
print('Train R^2 Score : %.3f'%linear_svr_regressor_grid.best_estimator_.score(X_train, Y_train))
print('Test R^2 Score : %.3f'%linear_svr_regressor_grid.best_estimator_.score(X_test, Y_test))
print('Best R^2 Score Through Grid Search : %.3f'%linear_svr_regressor_grid.best_score_)
print('Best Parameters : ',linear_svr_regressor_grid.best_params_)
Printing First Few Cross Validation Results¶
cross_val_results = pd.DataFrame(linear_svr_regressor_grid.cv_results_)
print('Number of Various Combinations of Parameters Tried : %d'%len(cross_val_results))
cross_val_results.head() ## Printing first few results.
LinearSVC ¶
The support vector machine model that we'll be introducing is LinearSVC. It is available as a part of svm module of sklearn. We'll divide classification dataset into train/test sets, train LinearSVC with default parameter on it, evaluate performance on the test set, and then tune model by trying various hyperparameters to improve performance further. We'll also introduce various important attributes of the trained model which can give useful insights once the model is trained.
Splitting Dataset into Train & Test sets¶
Please make a note that we are also using stratify parameter which will prevent unequal distribution of all classes in train and test sets.For each classes, we'll have 80% samples in train set and 20% samples in test set. This will make sure that we don't have any dominating class in either train or test set.
X_train, X_test, Y_train, Y_test = train_test_split(X_digits, Y_digits, train_size=0.80, test_size=0.20, random_state=123)
print('Train/Test Sizes : ', X_train.shape, X_test.shape, Y_train.shape, Y_test.shape)
Fitting Default LinearSVC to Train Data¶
from sklearn.svm import LinearSVC
linear_svc = LinearSVC(max_iter=1000000)
linear_svc.fit(X_train, Y_train)
Evaluate Model Accuracy on Test Set¶
Y_preds = linear_svc.predict(X_test)
print(Y_preds[:15])
print(Y_test[:15])
print('Test Accuracy : %.3f'%(Y_preds == Y_test).mean())
print('Test Accuracy : %.3f'%linear_svc.score(X_test, Y_test)) ## Score method also evaluates accuracy for classification models.
print('Training Accuracy : %.3f'%linear_svc.score(X_train, Y_train))
Important Attributes of Estimator¶
The LinearSVC has the same attributes as that of LinearSVR.
print("Feature Importances Shape :", linear_svc.coef_.shape)
print("Model Intercept :", linear_svc.intercept_)
Finetuning Model By Doing Grid Search On Various Hyperparameters¶
Below is a list of common hyperparameters that needs tuning for getting the best fit for our data. We'll try various hyperparameters settings to various splits of train/test data to find out best fit which will have almost the same accuracy for both train & test dataset or have quite less difference between accuracy.
- C - It represents regularization applied to the linear kernel function. The strength of normalization is inversely proportional to C which means that low C will result in high regularization and vice-versa. The default value of
1.0is set. - penalty - It accepts one of the two string values. It applies a penalty to linear kernel function and prevents it from overfitting data.
l1Penaltyl2Penalty(default)
- max_iter - It specifies the number of iteration to try before stopping algorithm. The value of
-1represents no limit and algorithm runs until convergence.
We'll below try various values for the above-mentioned hyperparameters to find the best estimator for our dataset by doing 5-fold cross-validation on data.
%%time
params = {
'C': [0.1, 0.5, 1.0, 10.0],
}
linear_svc_classifier_grid = GridSearchCV(LinearSVC(random_state=1, max_iter=1000000), param_grid=params, n_jobs=-1, cv=5, verbose=5)
linear_svc_classifier_grid.fit(X_train,Y_train)
print('Train Accuracy : %.3f'%linear_svc_classifier_grid.best_estimator_.score(X_train, Y_train))
print('Test Accuracy : %.3f'%linear_svc_classifier_grid.best_estimator_.score(X_test, Y_test))
print('Best Accuracy Through Grid Search : %.3f'%linear_svc_classifier_grid.best_score_)
print('Best Parameters : ',linear_svc_classifier_grid.best_params_)
Printing First Few Cross Validation Results¶
cross_val_results = pd.DataFrame(linear_svc_classifier_grid.cv_results_)
print('Number of Various Combinations of Parameters Tried : %d'%len(cross_val_results))
cross_val_results.head() ## Printing first few results.
SVR ¶
The support vector machine model that we'll be introducing is SVR. It is available as a part of svm module of sklearn. We'll divide the regression dataset into train/test sets, train SVR with default parameter on it, evaluate performance on the test set, and then tune model by trying various hyperparameters to improve performance further. We'll also introduce various important attributes of the trained model which can give useful insights once the model is trained.
Splitting Dataset into Train & Test sets¶
X_train, X_test, Y_train, Y_test = train_test_split(X_boston, Y_boston, train_size=0.80, test_size=0.20, random_state=123)
print('Train/Test Sizes : ', X_train.shape, X_test.shape, Y_train.shape, Y_test.shape)
Fitting Default SVR to Train Data¶
from sklearn.svm import SVR
svr = SVR(cache_size=1000)
svr.fit(X_train, Y_train)
Evaluate Model Accuracy on Test Set¶
Y_preds = svr.predict(X_test)
print(Y_preds[:15])
print(Y_test[:15])
print('Test R^2 Score : %.3f'%svr.score(X_test, Y_test)) ## Score method also evaluates accuracy for classification models.
print('Training R^2 Score : %.3f'%svr.score(X_train, Y_train))
Important Attributes of Estimator¶
SVR provides a list of important attributes that can provide important insights one model is trained. Below is a list of attributes available through SVR.
support_vectors_- It represents support vectors of the trained model.intercept_- It represents intercept of linear kernel function.
print("Support Vectors Shape:", svr.support_vectors_.shape)
print("Model Intercept :", svr.intercept_)
Finetuning Model By Doing Grid Search On Various Hyperparameters¶
Below is a list of common hyperparameters that need tuning for getting the best fit for our data. We'll try various hyperparameters settings to various splits of train/test data to find out best fit which will have almost the same accuracy for both train & test dataset or have quite less difference between accuracy.
C - It represents regularization applied to the linear kernel function. The strength of normalization is inversely proportional to C which means that low C will result in high regularization and vice-versa. The default value of
1.0is set.kernel - It specifies kernel type to be used in SVM. It accepts either one of the below
strings or callable.- linear
- poly
- rbf (default)
- sigmoid
- precomputed
- degree - It accepts integer values and represents a degree for
polykernel. It's ignored when other kernels are used. - gamma - It represents kernel coefficient for
rbf,polyandsigmoidkernels. It accepts one of the below strings orfloatas value.scale(default) - 1 / (n_features * X.var())auto- (1/ n_features)
- cache_size - It accepts float values representing kernel cache size in
MB. The default value is200 MB. It's suggested to increase cachec_size based on RAM of computer to increase performance of SVM. - max_iter - It specifies number of iteration to try before stopping algorithm. The value of
-1represents no limit and algorithm runs until convergence.
We'll below try various values for the above-mentioned hyperparameters to find the best estimator for our dataset by doing 5-fold cross-validation on data.
%%time
from sklearn.model_selection import GridSearchCV
params = {
'C': [0.1, 1.0,],
'kernel': ['linear','rbf', 'sigmoid', ],
'gamma': ['auto', 'scale']
}
svr_regressor_grid = GridSearchCV(SVR(cache_size=1000), param_grid=params, n_jobs=-1, cv=5, verbose=5)
svr_regressor_grid.fit(X_train,Y_train)
print('Train R^2 Score : %.3f'%svr_regressor_grid.best_estimator_.score(X_train, Y_train))
print('Test R^2 Score : %.3f'%svr_regressor_grid.best_estimator_.score(X_test, Y_test))
print('Best R^2 Score Through Grid Search : %.3f'%svr_regressor_grid.best_score_)
print('Best Parameters : ',svr_regressor_grid.best_params_)
Printing First Few Cross Validation Results¶
cross_val_results = pd.DataFrame(svr_regressor_grid.cv_results_)
print('Number of Various Combinations of Parameters Tried : %d'%len(cross_val_results))
cross_val_results.head() ## Printing first few results.
SVC ¶
The support vector machine model that we'll be introducing is SVC. It is available as a part of svm module of sklearn. We'll divide classification dataset into train/test sets, train SVC with default parameter on it, evaluate performance on the test set, and then tune model by trying various hyperparameters to improve performance further. We'll also introduce various important attributes of the trained model which can give useful insights once the model is trained.
Splitting Dataset into Train & Test sets¶
X_train, X_test, Y_train, Y_test = train_test_split(X_digits, Y_digits, train_size=0.80, test_size=0.20, random_state=123)
print('Train/Test Sizes : ', X_train.shape, X_test.shape, Y_train.shape, Y_test.shape)
Fitting Default SVC to Train Data¶
from sklearn.svm import SVC
svc = SVC(cache_size=1000)
svc.fit(X_train, Y_train)
Evaluate Model Accuracy on Test Set¶
Y_preds = svc.predict(X_test)
print(Y_preds[:15])
print(Y_test[:15])
print('Test Accuracy : %.3f'%(Y_preds == Y_test).mean())
print('Test Accuracy : %.3f'%svc.score(X_test, Y_test)) ## Score method also evaluates accuracy for classification models.
print('Training Accuracy : %.3f'%svc.score(X_train, Y_train))
Important Attributes of Estimator¶
SVC has the same set of attributes as that of SVR.
print("Support Vectors :", svc.support_vectors_.shape)
#print("Feature Importances :", svc.coef_) ## Only for Linear Kernel
print("Model Intercept :", svc.intercept_)
Finetuning Model By Doing Grid Search On Various Hyperparameters¶
SVC has the same parameters as that of SVR
%%time
params = {
'C': [0.1, 1.0, ],
'kernel': ['linear', 'rbf', 'sigmoid',],
'gamma': ['auto', 'scale']
}
svc_classifier_grid = GridSearchCV(SVC(cache_size=1000), param_grid=params, n_jobs=-1, cv=5, verbose=5)
svc_classifier_grid.fit(X_train,Y_train)
print('Train Accuracy : %.3f'%svc_classifier_grid.best_estimator_.score(X_train, Y_train))
print('Test Accuracy : %.3f'%svc_classifier_grid.best_estimator_.score(X_test, Y_test))
print('Best Accuracy Through Grid Search : %.3f'%svc_classifier_grid.best_score_)
print('Best Parameters : ',svc_classifier_grid.best_params_)
Printing First Few Cross Validation Results¶
cross_val_results = pd.DataFrame(svc_classifier_grid.cv_results_)
print('Number of Various Combinations of Parameters Tried : %d'%len(cross_val_results))
cross_val_results.head() ## Printing first few results.
NuSVR ¶
The support vector machine model that we'll be introducing is NuSVR. It is available as a part of svm module of sklearn. We'll divide the regression dataset into train/test sets, train NuSVR with default parameter on it, evaluate performance on the test set, and then tune model by trying various hyperparameters to improve performance further. We'll also introduce various important attributes of the trained model which can give useful insights once the model is trained.
Splitting Dataset into Train & Test sets¶
X_train, X_test, Y_train, Y_test = train_test_split(X_boston, Y_boston, train_size=0.80, test_size=0.20, random_state=123)
print('Train/Test Sizes : ', X_train.shape, X_test.shape, Y_train.shape, Y_test.shape)
Fitting Default NuSVR to Train Data¶
from sklearn.svm import NuSVR
nu_svr = NuSVR(cache_size=1000)
nu_svr.fit(X_train, Y_train)
Evaluate Model Accuracy on Test Set¶
Y_preds = nu_svr.predict(X_test)
print(Y_preds[:15])
print(Y_test[:15])
print('Test R^2 Score : %.3f'%nu_svr.score(X_test, Y_test)) ## Score method also evaluates accuracy for classification models.
print('Training R^2 Score : %.3f'%nu_svr.score(X_train, Y_train))
Important Attributes of Estimator¶
NuSVR provides a list of important attributes that can provide important insights one model is trained. Below is a list of attributes available through NuSVR.
support_vectors_- It represents support vectors of the trained model.coef_- It returns an array representing weights assigned to each feature by model. It represents the importance of each feature as per model trained.intercept_- It represents intercept of linear kernel function.
print("Support Vectors :", nu_svr.support_vectors_.shape)
#print("Feature Importances :", nu_svr.coef_) ## Only for Linear Kernel
print("Model Intercept :", nu_svr.intercept_)
Finetuning Model By Doing Grid Search On Various Hyperparameters¶
Below is a list of common hyperparameters that needs tuning for getting the best fit for our data. We'll try various hyperparameters settings to various splits of train/test data to find out best fit which will have almost the same accuracy for both train & test dataset or have quite less difference between accuracy.
- nu - It accepts float value between
0-1. It represents an upper bound of the fraction of margin errors and lowers bound on the fraction of support vectors. - kernel - It specifies kernel type to be used in SVM. It accepts either one of the below
strings or callable.- linear
- poly
- rbf (default)
- sigmoid
- precomputed
- degree - It accepts integer values and represents a degree for
polykernel. It's ignored when other kernels are used. - gamma - It represents kernel coefficient for
rbf,polyandsigmoidkernels. It accepts one of the below strings orfloatas value.scale(default) - 1 / (n_features * X.var())auto- (1/ n_features)
- cache_size - It accepts float values representing kernel cache size in
MB. The default value is200 MB. It's suggested to increase cachec_size based on RAM of the computer to increase the performance of SVM. - max_iter - It specifies the number of iteration to try before stopping algorithm. The value of
-1represents no limit and algorithm runs until convergence.
We'll below try various values for the above-mentioned hyperparameters to find the best estimator for our dataset by doing 5-fold cross-validation on data.
%%time
from sklearn.model_selection import GridSearchCV
params = {
'nu': [0.1, 1.0,],
'kernel': ['linear', 'rbf', 'sigmoid',],
'gamma': ['auto', 'scale']
}
svr_regressor_grid = GridSearchCV(NuSVR(cache_size=1000), param_grid=params, n_jobs=-1, cv=5, verbose=5)
svr_regressor_grid.fit(X_train,Y_train)
print('Train R^2 Score : %.3f'%svr_regressor_grid.best_estimator_.score(X_train, Y_train))
print('Test R^2 Score : %.3f'%svr_regressor_grid.best_estimator_.score(X_test, Y_test))
print('Best R^2 Score Through Grid Search : %.3f'%svr_regressor_grid.best_score_)
print('Best Parameters : ',svr_regressor_grid.best_params_)
Printing First Few Cross Validation Results¶
cross_val_results = pd.DataFrame(svr_regressor_grid.cv_results_)
print('Number of Various Combinations of Parameters Tried : %d'%len(cross_val_results))
cross_val_results.head() ## Printing first few results.
NuSVC ¶
The support vector machine model that we'll be introducing is NuSVC. It is available as a part of svm module of sklearn. We'll divide the regression dataset into train/test sets, train NuSVC with default parameter on it, evaluate performance on the test set, and then tune model by trying various hyperparameters to improve performance further. We'll also introduce various important attributes of the trained model which can give useful insights once the model is trained.
Splitting Dataset into Train & Test sets¶
X_train, X_test, Y_train, Y_test = train_test_split(X_digits, Y_digits, train_size=0.80, test_size=0.20, random_state=123)
print('Train/Test Sizes : ', X_train.shape, X_test.shape, Y_train.shape, Y_test.shape)
Fitting Default NuSVC to Train Data¶
from sklearn.svm import NuSVC
nu_svc = NuSVC(cache_size=1000)
nu_svc.fit(X_train, Y_train)
Evaluate Model Accuracy on Test Set¶
Y_preds = nu_svc.predict(X_test)
print(Y_preds[:15])
print(Y_test[:15])
print('Test Accuracy : %.3f'%(Y_preds == Y_test).mean())
print('Test Accuracy : %.3f'%nu_svc.score(X_test, Y_test)) ## Score method also evaluates accuracy for classification models.
print('Training Accuracy : %.3f'%nu_svc.score(X_train, Y_train))
Important Attributes of Estimator¶
NuSVC has the same set of attributes as that of NuSVR.
print("Support Vectors :", nu_svc.support_vectors_.shape)
#print("Feature Importances :", nu_svc.coef_) ## Only for Linear Kernel
print("Model Intercept :", nu_svc.intercept_)
Finetuning Model By Doing Grid Search On Various Hyperparameters¶
NuSVC has same parameters as that of NuSVR
%%time
params = {
'nu': [0.1, 1.0, ],
'kernel': ['linear', 'rbf', 'sigmoid',],
'gamma': ['auto', 'scale']
}
svc_classifier_grid = GridSearchCV(NuSVC(cache_size=1000), param_grid=params, n_jobs=-1, cv=5, verbose=5)
svc_classifier_grid.fit(X_train,Y_train)
print('Train Accuracy : %.3f'%svc_classifier_grid.best_estimator_.score(X_train, Y_train))
print('Test Accuracy : %.3f'%svc_classifier_grid.best_estimator_.score(X_test, Y_test))
print('Best Accuracy Through Grid Search : %.3f'%svc_classifier_grid.best_score_)
print('Best Parameters : ',svc_classifier_grid.best_params_)
Printing First Few Cross Validation Results¶
cross_val_results = pd.DataFrame(svc_classifier_grid.cv_results_)
print('Number of Various Combinations of Parameters Tried : %d'%len(cross_val_results))
cross_val_results.head() ## Printing first few results.
This ends our small tutorial introducing various SVM estimators available as a part of sklearn. Please feel free to let us know your views in the comments section.
References ¶
Sunny Solanki
Sunny Solanki
Intro: Software Developer | Youtuber | Bonsai Enthusiast
About: Sunny Solanki holds a bachelor's degree in Information Technology (2006-2010) from L.D. College of Engineering. Post completion of his graduation, he has 8.5+ years of experience (2011-2019) in the IT Industry (TCS). His IT experience involves working on Python & Java Projects with US/Canada banking clients. Since 2020, he’s primarily concentrating on growing CoderzColumn.
His main areas of interest are AI, Machine Learning, Data Visualization, and Concurrent Programming. He has good hands-on with Python and its ecosystem libraries.
Apart from his tech life, he prefers reading biographies and autobiographies. And yes, he spends his leisure time taking care of his plants and a few pre-Bonsai trees.
Contact: sunny.2309@yahoo.in
![YouTube Subscribe]()
Comfortable Learning through Video Tutorials?
If you are more comfortable learning through video tutorials then we would recommend that you subscribe to our YouTube channel.
![Need Help]()
Stuck Somewhere? Need Help with Coding? Have Doubts About the Topic/Code?
Stuck Somewhere? Need Help with Coding? Have Doubts About the Topic/Code?When going through coding examples, it's quite common to have doubts and errors.
If you have doubts about some code examples or are stuck somewhere when trying our code, send us an email at coderzcolumn07@gmail.com. We'll help you or point you in the direction where you can find a solution to your problem.
You can even send us a mail if you are trying something new and need guidance regarding coding. We'll try to respond as soon as possible.
![Share Views]()
Want to Share Your Views? Have Any Suggestions?
Want to Share Your Views? Have Any Suggestions?If you want to
- provide some suggestions on topic
- share your views
- include some details in tutorial
- suggest some new topics on which we should create tutorials/blogs
svm, sklearn
Subscribe to Our YouTube Channel
