Logistic Regression

Sigmoid, log loss, regularisation and gradient descent for binary classification, worked through with code and plots.

Logistic regression is a machine learning algorithm used for binary classification problems, where the goal is to predict the probability of an event occurring (such as a customer buying a product or a patient being diagnosed with a disease) based on one or more input features.

Sigmoid function as $f_{w, b} (x)$

Logistic regression doesn’t output a single continuous variable like Linear Regression. Instead, the output is categorical, or binary. It outputs a 0 or a 1. We use sigmoid function to achieve that, which is defined as:

$f_{w, b} (x^{(i)}) z^{(i)} g (z^{(i)}) = g (z^{(i)}) = w \cdot x^{(i)} + b = \frac{1}{1 + e ^{- z^{(i)}}}$

You can recall that the function $z^{(i)}$ is the function we normally use for linear regression. It’s a fundamental equation in machine learning - simply a function returning a straight line.

Loss

Unlike Linear Regression, using a Mean Squared Error won’t be a good choice for the loss function when we’re aiming at the non-linear function $f_{w, b} (x)$ . Logistic Regression uses a loss function more suited to the task of categorization where the target is 0 or 1 rather than any number.

Loss is a measure of the difference of a single example to its target value while the
Cost is a measure of the losses over the training set

This is defined:

$l oss (f_{w, b} (x^{(i)}), y^{(i)})$ is the cost for a single data point, which is:

$l oss (f_{w, b} (x^{(i)}), y^{(i)}) = {- lo g (f_{w, b} (x^{(i)})) - lo g (1 - f_{w, b} (x^{(i)})) if y^{(i)} = 1 if y^{(i)} = 0$

$f_{w, b} (x^{(i)})$ is the model’s prediction, while $y^{(i)}$ is the target value.
$f_{w, b} (x^{(i)}) = g (w \cdot x^{(i)} + b)$ where function $g$ is the sigmoid function.

The defining feature of this loss function is the fact that it uses two separate curves. One for the case when the target is zero or ( $y = 0$ ) and another for when the target is one ( $y = 1$ ). Combined, these curves provide the behavior useful for a loss function, namely, being zero when the prediction matches the target and rapidly increasing in value as the prediction differs from the target. Consider the curves below:

The loss function above can be rewritten to be easier to implement.

$l oss (f_{w, b} (x^{(i)}), y^{(i)}) = (- y^{(i)} lo g (f_{w, b} (x^{(i)})) - (1 - y^{(i)}) lo g (1 - f_{w, b} (x^{(i)}))$

This is a rather formidable-looking equation. It is less daunting when you consider $y^{(i)}$ can have only two values, 0 and 1. One can then consider the equation in two pieces:
when $y^{(i)} = 0$ , the left-hand term is eliminated:

$l oss (f_{w, b} (x^{(i)}), 0) = (- (0) lo g (f_{w, b} (x^{(i)})) - (1 - 0) lo g (1 - f_{w, b} (x^{(i)})) = - lo g (1 - f_{w, b} (x^{(i)}))$

and when $y^{(i)} = 1$ , the right-hand term is eliminated:

$l oss (f_{w, b} (x^{(i)}), 1) = (- (1) lo g (f_{w, b} (x^{(i)})) - (1 - 1) lo g (1 - f_{w, b} (x^{(i)})) = - lo g (f_{w, b} (x^{(i)}))$

OK, with this new logistic loss function, a cost function can be produced that incorporates the loss from all the examples.

Cost Function

Recall, loss is defined to apply to one example. Here you combine the losses to form the cost, which includes all the examples.

Recall that for logistic regression, the cost function is of the form

$J (w, b) = \frac{1}{m} i = 0 \sum m - 1 [l oss (f_{w, b} (x^{(i)}), y^{(i)})]$

where

$l oss (f_{w, b} (x^{(i)}), y^{(i)})$ is the cost for a single data point, which is:

$l oss (f_{w, b} (x^{(i)}), y^{(i)}) = - y^{(i)} lo g (f_{w, b} (x^{(i)})) - (1 - y^{(i)}) lo g (1 - f_{w, b} (x^{(i)}))$

where m is the number of training examples in the data set and:

$f_{w, b} (x^{(i)}) z^{(i)} g (z^{(i)}) = g (z^{(i)}) = w \cdot x^{(i)} + b = \frac{1}{1 + e ^{- z^{(i)}}}$

Regularization

To prevent overfitting and improve the generalization performance of the model, we can add regularization terms to the cost function:

$J (θ) = - \frac{1}{m} L (θ) + \frac{λ}{2 m} j = 1 \sum n θ_{j}^{2}$

where $λ$ is the regularization parameter that controls the strength of regularization, and $n$ is the number of features. Ridge (L2) and Lasso (L1) are two common types of regularization used in logistic regression.

Cost function for regularized logistic regression

For regularized logistic regression, the cost function is of the form

$J (w, b) = \frac{1}{m} i = 0 \sum m - 1 [- y^{(i)} lo g (f_{w, b} (x^{(i)})) - (1 - y^{(i)}) lo g (1 - f_{w, b} (x^{(i)}))] + \frac{λ}{2 m} j = 0 \sum n - 1 w_{j}^{2}$

where:

$f_{w, b} (x^{(i)}) = s i g m o i d (w \cdot x^{(i)} + b)$

Compare this to the cost function without regularization (which you implemented in a previous lab):

$J (w, b) = \frac{1}{m} i = 0 \sum m - 1 [(- y^{(i)} lo g (f_{w, b} (x^{(i)})) - (1 - y^{(i)}) lo g (1 - f_{w, b} (x^{(i)}))]$

As was the case in linear regression above, the difference is the regularization term, which is $\frac{λ}{2 m} \sum_{j = 0}^{n - 1} w_{j}^{2}$

Including this term encourages gradient descent to minimize the size of the parameters. Note, in this example, the parameter $b$ is not regularized. This is standard practice.

Gradient descent with regularization

The basic algorithm for running gradient descent does not change with regularization, it is:

$repeat until convergence: {w_{j} = w_{j} - α \frac{\partial J ( w , b )}{\partial w _{j}} b = b - α \frac{\partial J ( w , b )}{\partial b}} for j := 0..n-1$

Where each iteration performs simultaneous updates on $w_{j}$ for all $j$ .

What changes with regularization is computing the gradients.

Computing the Gradient with regularization (both linear/logistic)

The gradient calculation for both linear and logistic regression are nearly identical, differing only in computation of $f_{w b}$ .

$\frac{\partial J ( w , b )}{\partial w _{j}} \frac{\partial J ( w , b )}{\partial b} = \frac{1}{m} i = 0 \sum m - 1 (f_{w, b} (x^{(i)}) - y^{(i)}) x_{j}^{(i)} + \frac{λ}{m} w_{j} = \frac{1}{m} i = 0 \sum m - 1 (f_{w, b} (x^{(i)}) - y^{(i)})$

m is the number of training examples in the data set
$f_{w, b} (x^{(i)})$ is the model’s prediction, while $y^{(i)}$ is the target
For a linear regression model
$f_{w, b} (x) = w \cdot x + b$
For a logistic regression model
$z = w \cdot x + b$
$f_{w, b} (x) = g (z)$
where $g (z)$ is the sigmoid function:
$g (z) = \frac{1}{1 + e ^{- z}}$

The term which adds regularization is the $\frac{λ}{m} w_{j}$ .

Evaluation Metrics

In binary classification problems, we typically use evaluation metrics such as accuracy, precision, recall, and F1 score to assess the performance of the model. These metrics can be calculated from the confusion matrix, which shows the number of true positive, true negative, false positive, and false negative predictions made by the model.

Here are some common evaluation metrics:

Accuracy: the proportion of correct predictions out of the total number of predictions.

Precision: the proportion of true positive predictions out of the total number of positive predictions. Recall: the proportion of true positive predictions out of the total number of actual positive instances.

F1 score: the harmonic mean of precision and recall, which gives equal weight to both metrics.

Conclusion

Logistic regression is a powerful and widely used algorithm for binary classification problems. By modeling the probability of the positive class as a function of the input features, logistic regression can make accurate predictions and provide insights into the underlying relationships between the variables.

Examples

Training a model on hand-written data

import numpy as np
import matplotlib.pyplot as plt
import matplotlib

X = np.array([[0.5, 1.5], [1,1], [1.5, 0.5], [3, 0.5], [2, 2], [1, 2.5]])
y = np.array([0, 0, 0, 1, 1, 1])

from sklearn.linear_model import LogisticRegression

lr_model = LogisticRegression()
lr_model.fit(X, y)

y_pred = lr_model.predict(X)

print("Prediction on training set:", y_pred)
print("Accuracy on training set:", lr_model.score(X, y))

Prediction on training set: [0 0 0 1 1 1]
Accuracy on training set: 1.0

plt.scatter(X[:,0],X[:,1],c=y)
plt.scatter(X[:,0],X[:,1],c=y_pred)

Generating synthetic data

Src: https://stackabuse.com/generating-synthetic-data-with-numpy-and-scikit-learn/

import sklearn.datasets as dt

fig,ax = plt.subplots(nrows=1, ncols=3,figsize=(16,5))
plt_ind_list = np.arange(3)+131
rand_state=1
color_map_discrete = matplotlib.colors.LinearSegmentedColormap.from_list("", ["red","cyan","magenta","blue"])

for class_sep,plt_ind in zip([0.1,1,10],plt_ind_list):
    x,y = dt.make_classification(n_samples=1000,
                                 n_features=2,
                                 n_repeated=0,
                                 class_sep=class_sep,
                                 n_redundant=0,
                                 random_state=rand_state)

    plt.subplot(plt_ind)
    my_scatter_plot = plt.scatter(x[:,0],
                                  x[:,1],
                                  c=y,
                                  vmin=min(y),
                                  vmax=max(y),
                                  s=35,
                                  cmap=color_map_discrete)
    plt.title('class_sep: '+str(class_sep))

fig.subplots_adjust(hspace=0.3,wspace=.3)
plt.suptitle('make_classification() With Different class_sep Values',fontsize=20)
plt.show()

Training a model on synthetic data

from sklearn.model_selection import train_test_split

x,y = dt.make_classification(n_samples=10000,
                                 n_features=2,
                                 n_repeated=0,
                                 class_sep=3,
                                 n_redundant=0,
                                 random_state=rand_state)

plot = plt.scatter(x[:,0],
                    x[:,1],
                    c=y,
                    vmin=min(y),
                    vmax=max(y),
                    s=35,
                    cmap=color_map_discrete)
plt.title('classification data')
print(plot)
print(x.shape, y.shape)
x_train, x_, y_train, y_ = train_test_split(x, y, test_size=0.4)
x_cv, x_test, y_cv, y_test = train_test_split(x_, y_, test_size=0.5)
del x_, y_
print('x_train: ', x_train.shape, y_train.shape)
print('x_cv: ', x_cv.shape, y_cv.shape)
print('x_test: ', x_test.shape, y_test.shape)
model = LogisticRegression()
model.fit(x_train,y_train)
y_pred_train = lr_model.predict(x_train)
y_pred_cv = lr_model.predict(x_cv)
y_pred_test = lr_model.predict(x_test)

# print("Prediction on training set:", y_pred_train)
print("Accuracy on training set:", model.score(x_train, y_train))

# print("Prediction on CV set:", y_pred_cv)
print("Accuracy on CV set:", model.score(x_cv, y_cv))

# print("Prediction on test set:", y_pred_test)
print("Accuracy on test set:", model.score(x_test, y_test))

<matplotlib.collections.PathCollection object at 0x7f53796ef880>
(10000, 2) (10000,)
x_train:  (6000, 2) (6000,)
x_cv:  (2000, 2) (2000,)
x_test:  (2000, 2) (2000,)
Accuracy on training set: 0.9945
Accuracy on CV set: 0.9955
Accuracy on test set: 0.9935

Plotting functions used by Logistic Regression

Sigmoid

# F_w,b(x) function plot

def sigmoid(x):
  return 1 / (1 + np.e ** -(x))

X = np.arange(-10, 10)
y = [sigmoid(x) for x in X]


plt.title('Sigmoid function')
plt.xlabel('X')
plt.ylabel('f(X) [sigmoid]')
plt.plot(X, y)

[<matplotlib.lines.Line2D at 0x7f5379488d30>]

Log loss AKA Binary Cross-Entropy Loss:

from sklearn.metrics import log_loss

# target is 0
y_pred = np.linspace(0,1,1000) #np.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, .8, .9])
m = len(y_pred)
y = np.zeros(m)
losses = []
for i in range(m):
  loss = log_loss([y[i]], [y_pred[i]], labels=[0,1])
  losses.append(loss)


# target is 1
y_pred2 = y_pred.copy()
y2 = np.ones(m)
losses2 = []
for i in range(m):
  loss = log_loss([y2[i]], [y_pred2[i]], labels=[0,1])
  losses2.append(loss)  # loss = log_loss()
plt.title('Log loss')
plt.plot(y_pred, losses, color='red', label='0 target')
plt.plot(y_pred2, losses2, color='green', label='1 target')
plt.xlabel('y^ (predicted y)')
plt.ylabel('Log loss(y^)')
plt.legend(loc="upper left")