Support Vector Machines

Support Vector Machines (SVM) are one of the supervised learning methods and are used for both classification and regression problems. SVM is a powerful algorithm that performs effectively especially on high-dimensional datasets. Its primary objective is to find an optimal hyperplane that separates or classifies the data.

Definition

Although support vector machines are used in both classification and regression analysis in the literature, they are generally preferred as a machine learning method for classification tasks. This method is based on a supervised learning model. During the algorithm’s operation, different kernel functions can be applied depending on the type of data, enabling both linear and nonlinear classification. If separable data are used in the classification process, all data points can typically be classified by a single hyperplane. However, if the data are not fully separable, classification using a single plane of the same dimension is often not possible. Therefore, different kernel functions are employed.

Working Principle

The logic of SVM is based on the following steps:

1- Data Transformation:

2- Finding the Hyperplane:

3- Support Vectors:

4- Kernel Trick:

• For data that are not linearly separable, SVM uses kernel functions. Common kernel functions include:

• Linear Kernel: K(x,y)=x^Ty
• Polynomial Kernel: K(x,y)=(x^Ty+c)^d
• Radial Basis Function (RBF) Kernel: K(x,y)=exp(−γ∥x−y∥²)
• Sigmoid Kernel: K(x,y)=tanh(αx^Ty+c)

5- Optimization:

Applications

• Text Classification: Used in problems such as spam email detection, sentiment analysis, and document categorization.
• Image Classification: Effective in areas such as object recognition, face recognition, and medical image analysis.
• Bioinformatics: Applied to problems such as protein structure prediction, gene expression analysis, and disease diagnosis.
• Financial Forecasting: Used in areas such as stock price prediction, credit scoring, and risk analysis.
• Handwriting Recognition: Applied in tasks such as postal code recognition and classification of handwritten characters.

Two-dimensional representation of Support Vector Machines

Application Example on the Iris Dataset Using the Support Vector Machine Algorithm

To observe correlations between species, the distribution of features—sepal length and sepal width—is analyzed.

Visualization of feature distribution:

Similarly, the distribution of petal length and petal width from the dataset is examined.

Feature distribution on the graph:

A SVM model is constructed using the first two features (sepal length and sepal width) to predict the species of the flower.

Kernel operations are performed on the building using both linear and nonlinear modeling.

When the classification models are visualized, the following graphs are obtained:

Hyperparameter tuning is performed by observing various kernels with different gamma (γ) values (0.1, 1, 10, 100).

In the resulting graphs, an increase in the gamma value leads to increased model fit.

Advantages and Disadvantages

• A fast and highly accurate method for small-scale datasets.
• Training time is high on large datasets, resulting in longer output generation times.
• Applicable to both linear and nonlinear datasets.
• Less effective and sensitive in nested datasets, leading to lower accuracy rates.
• An effective solution for high-dimensional datasets with focused results.
• High accuracy can be achieved with minimal tuning of hyperparameters (gamma/C).
• Kernel trick operations involve significant computational time.