为了得到梯度，我们将损失与个分量进行区分。$i$$w$

将铰链损失重写为其中和$w$$f(g(w))$$f(z)=\max(0,1-y\ z)$$g(w)=\mathbf{x}\cdot \mathbf{w}$

使用链式法则我​​们得到

$$\frac{\partial}{\partial w_i} f(g(w))=\frac{\partial f}{\partial z} \frac{\partial g}{\partial w_i}$$

当和 0 当变为时被评估。二阶导数项变为。所以最后你得到 $g(w)=x\cdot w$$-y$$\mathbf{x}\cdot w<1$$\mathbf{x}\cdot w>1$$x_i$

$$\frac{\partial f(g(w))}{\partial w_i} = \begin{cases} -y\ x_i &\text{if } y\ \mathbf{x}\cdot \mathbf{w} < 1 \\ 0&\text{if } y\ \mathbf{x}\cdot \mathbf{w} > 1 \end{cases}$$

由于涵盖的分量，您可以将上述内容视为一个向量，并将写为$i$$x$$\frac{\partial}{\partial w}$$(\frac{\partial}{\partial w_1},\frac{\partial}{\partial w_2},\ldots)$

这已经晚了3年，但仍然可能与某人有关......

让表示点的样本和相应标签的集合。我们寻找一个能够最小化总铰链损失的超平面： 求对总铰链损失求导。每个分量的梯度为： $S$$x_i \in R^d$$y_i \in \{-1,1\}$$w$

$$\begin{equation} w^* = \underset{w}{\text{argmin }} L^{hinge}_S(w) = \underset{w}{\text{argmin }} \sum_i{l_{hinge}(w,x_i,y_i)}= \underset{w}{\text{argmin }} \sum_i{\max{\{0,1-y_iw\cdot x}\}} \end{equation}$$ $w^*$ $$\frac{\partial{l_{hinge}}}{\partial w}= \begin{cases} 0 & y_iw\cdot x \geq 1 \\ -y_ix & y_iw\cdot x < 1 \end{cases}$$

总和的梯度是梯度的总和。 Python 示例，使用 GD 查找铰链损失最佳分离超平面如下（它可能不是最有效的代码，但它有效）

$$\frac{\partial{L_S^{hinge}}}{\partial{w}}=\sum_i{\frac{\partial{l_{hinge}}}{\partial w}}$$

import numpy as np
import matplotlib.pyplot as plt

def hinge_loss(w,x,y):
    """ evaluates hinge loss and its gradient at w

    rows of x are data points
    y is a vector of labels
    """
    loss,grad = 0,0
    for (x_,y_) in zip(x,y):
        v = y_*np.dot(w,x_)
        loss += max(0,1-v)
        grad += 0 if v > 1 else -y_*x_
    return (loss,grad)

def grad_descent(x,y,w,step,thresh=0.001):
    grad = np.inf
    ws = np.zeros((2,0))
    ws = np.hstack((ws,w.reshape(2,1)))
    step_num = 1
    delta = np.inf
    loss0 = np.inf
    while np.abs(delta)>thresh:
        loss,grad = hinge_loss(w,x,y)
        delta = loss0-loss
        loss0 = loss
        grad_dir = grad/np.linalg.norm(grad)
        w = w-step*grad_dir/step_num
        ws = np.hstack((ws,w.reshape((2,1))))
        step_num += 1
    return np.sum(ws,1)/np.size(ws,1)

def test1():
    # sample data points
    x1 = np.array((0,1,3,4,1))
    x2 = np.array((1,2,0,1,1))
    x  = np.vstack((x1,x2)).T
    # sample labels
    y = np.array((1,1,-1,-1,-1))
    w = grad_descent(x,y,np.array((0,0)),0.1)
    loss, grad = hinge_loss(w,x,y)
    plot_test(x,y,w)

def plot_test(x,y,w):
    plt.figure()
    x1, x2 = x[:,0], x[:,1]
    x1_min, x1_max = np.min(x1)*.7, np.max(x1)*1.3
    x2_min, x2_max = np.min(x2)*.7, np.max(x2)*1.3
    gridpoints = 2000
    x1s = np.linspace(x1_min, x1_max, gridpoints)
    x2s = np.linspace(x2_min, x2_max, gridpoints)
    gridx1, gridx2 = np.meshgrid(x1s,x2s)
    grid_pts = np.c_[gridx1.ravel(), gridx2.ravel()]
    predictions = np.array([np.sign(np.dot(w,x_)) for x_ in grid_pts]).reshape((gridpoints,gridpoints))
    plt.contourf(gridx1, gridx2, predictions, cmap=plt.cm.Paired)
    plt.scatter(x[:, 0], x[:, 1], c=y, cmap=plt.cm.Paired)
    plt.title('total hinge loss: %g' % hinge_loss(w,x,y)[0])
    plt.show()

if __name__ == '__main__':
    np.set_printoptions(precision=3)
    test1()

我修复了你的代码。主要问题是您对铰链和 d_hinge 函数的定义。这些应该一次应用一个样品。相反，您的定义会在取最大值之前聚合所有样本。

#Run standard gradient descent
gradient_descent<-function(fw, dfw, n, lr=0.01)
{
    #Date to be used
    x<-t(matrix(c(1,3,6,1,4,2,1,5,4,1,6,1), nrow=3))
    y<-t(t(c(1,1,-1,-1)))
    w<-matrix(0, nrow=ncol(x))


    print(sprintf("loss: %f,x.w: %s",sum(mapply(function(xr,yr) fw(w,xr,yr), split(x,row(x)),split(y,row(y)))),paste(x%*%w, collapse=',')))
    #update the weights 'n' times
    for (i in 1:n)
    {
      w<-w-lr*dfw(w,x,y)
      print(sprintf("loss: %f,x.w: %s",sum(mapply(function(xr,yr) fw(w,xr,yr), split(x,row(x)),split(y,row(y)))),paste(x%*%w,collapse=',')))
    }
}

#Hinge loss
hinge<-function(w,xr,yr) max(1-yr*xr%*%w, 0)
d_hinge<-function(w,x,y){ dw<- apply(mapply(function(xr,yr) -yr * xr * (yr * xr %*% w < 1),split(x,row(x)),split(y,row(y))),1,sum); dw}
gradient_descent(hinge, d_hinge, 100, lr=0.01)

我需要 n=10000 才能收敛。

[1]“损失：0.090000，XW：1.0899999999995,08999999999905,0.1.1900000000000008,1.690000000000001,1”[1]“损失：0.100000，XW：1.3399999999995,1.19999999999,0.900000000000075，-1.42000000000011，-1.42000000000011”[1]“损失：0.230000，XW： 0.939999999999948,0.829999999999905，-1.32000000000007，-1.77000000000011" [1] “损失：0.370000，XW：1.64999999999995,1.2899999999999，-0.630000000000075，-1.25000000000011”[1] “损失：0.000000，XW：1.24999999999995,0.999999999999905，-1.05000000000008，-1.60000000000011” [1] “损失：0.240000，XW：1.49999999999995,1.2099999999999，-0.760000000000075，-1.33000000000011”[1] “损失：0.080000，XW：1.09999999999995,0.919999999999905，-1.18000000000007，-1.68000000000011”[1]“损失：0.110000，XW： 1.34999999999995,1.1299999999999,-0.890000000000075,-1.41000000000011"[1]“损耗：0.210000，XW：0.949999999999905，-1.3100000000000007,1.7600000000000011”[1]“损失：0.380000，XW：1.6599999999995,1.29999999999,0.0.620000000000074，-1.2400000000000011”[1]“损失：0.000000，XW： 1.25999999999995,1.009999999999,1 -1.0400000000000008,1.5900000000000011，-1.59000000000011“[1]”损失：0.000000，XW：1.2599999999995,1.00999999999，-1.0400000000000008，-1.59000000000011“