SVM

Support Vector Machine (SVM) is a supervised learning model used for classification, regression, and outlier detection, though most commonly for classification. SVMs seek a decision boundary, a hyperplane, that best separates data points of different classes by maximizing the margin—the distance between the hyperplane and the nearest points from each class. In the linear case, the hyperplane is defined by w·x + b = 0, and the sign of w·x + b determines the predicted class. To allow some misclassifications, a soft margin introduces slack variables and a penalty parameter C that controls the trade-off between margin width and misclassification.

The learning problem can be formulated as a convex optimization task. In the primal form, minimize (1/2)||w||^2 + C∑ξ_i subject to y_i(w·x_i + b) ≥ 1 − ξ_i and ξ_i ≥ 0. In the dual form, maximize ∑α_i − 1/2∑∑α_i α_j y_i y_j x_i·x_j subject to ∑α_i y_i = 0 and 0 ≤ α_i ≤ C. Only a subset of training points—support vectors—have α_i > 0 and define the decision boundary. The decision function is f(x) = sign(∑α_i y_i K(x_i, x) + b), where K is a kernel function.

To handle non-linear boundaries, SVMs use the kernel trick, mapping inputs into a high-dimensional feature space without explicit computation. Common kernels include linear, polynomial, radial basis function (RBF), and sigmoid. Support Vector Regression (SVR) adapts the framework for regression tasks with an insensitive loss. SVMs are sensitive to feature scaling and can be computationally intensive on large datasets, but they perform well in high-dimensional spaces and with clear margin separation.