Skip to main content
 首页 » 编程设计

python之如何实现 ReLU 代替 Sigmoid 函数

2025年01月19日19bonelee

import numpy as np  
 
alpha = 0.0251 # as close to true alpha as possible 
def nonlinear(x, deriv=False): 
  if(deriv==True): 
    return x*(1-x) 
  return 1/(1+np.e**(-x)) 
 
#seed 
np.random.seed(1) 
 
#testing sample 
test_x = np.array([[251,497,-246], 
              [299,249,50], 
              [194,180,14], 
              [140,148,-8], 
              [210,140,70]]) 
#Input Array - This input will be taken directly from a Pong game 
X = np.array([[198,200,-2], 
          [90, 280,-190], 
          [84, 256,-172], 
          [140,240,-100], 
          [114,216,-102], 
          [72, 95,-23], 
          [99, 31, 68], 
          [144, 20, 124], 
          [640, 216,424], 
          [32, 464,-432], 
          [176, 64,112], 
          [754, 506,248], 
          [107, 104,3], 
          [116,101,15]]) 
 
#output array - if ball_pos - paddle > 0 move up else move down 
Y = np.array([[0,0,0,0,0,0,1,1,1,0,1,1,1,1,]]).T  
 
syn0 = 2*np.random.random((3,14))-1 
syn1 = 2*np.random.random((14,14))-1 
 
for j in range(60000): 
 
  #forward propagation  
  l0 = X 
  l1 = nonlinear(np.dot(l0, syn0)) 
  l2 = nonlinear(np.dot(l1, syn1)) 
 
  #how much did we miss  
  l2_error = Y - l2  
 
  #multiply how much missed by the slope of sigmoid at the value in l1  
  l2_delta = l2_error * nonlinear(l2, True) 
 
  #how much did l1 contribute to l2 error  
  #(according to the weights) 
  l1_error = l2_delta.dot(syn1.T) 
 
  #in what direction is the target l1? 
  # Sure? 
  l1_delta = l1_error*nonlinear(l1,True) 
 
  #update weight 
  syn1 += alpha * (l1.T.dot(l2_delta)) 
  syn0 += alpha * (l0.T.dot(l1_delta)) 
 
  # display error  
  if(j % 10000) == 0: 
    print("ERROR: " + str(np.mean(np.abs(l2_error)))) 
 
 
#Testing Forward propagation 
l0_test = test_x 
l1_test = nonlinear(np.dot(l0_test,syn0)) 
l2_test = nonlinear(np.dot(l1_test,syn1)) 
 
#Dress up the array (make it look nice) 
l2_test_output = [] 
for x in range(len(l2_test)): 
  l2_test_output.append(l2_test[x][0]) 
 
print("Test Output") 
print(l2_test_output) 
 
#Put all the l2 data in a way I could see it: Just the first probabilites  
l2_output = [] 
for x in range(len(l2)): 
  l2_output.append(l2[x][0]) 
 
print("Output") 
print(l2_output) 

此代码应该接受一组三个数字 [(value_1),(value_2),(value_1-value_2)] 并在第一个和第二个值之间的差为负数时返回“0”或如果差异为正,则为“1”。到目前为止,它实际上运行良好。

这是输出: 错误:0.497132186092 错误:0.105081486632 错误:0.102115299177 错误:0.100813655802 错误:0.100042420179 错误:0.0995185781466 测试输出 [0.0074706006801269686、0.66687458928464094、0.66687458928463983、0.66686236694464551、0.98341439176739631] 输出 [0.66687459245609326, 0.00083944690766060215, 0.00083946471285455484, 0.0074706634783305243, 0.0074706634765733968, 0.00724986378, 0.00724986378 6513183073093, 0.99647100131874755, 0.99646513180692531, 0.00083944572383107523, 0.99646513180692531, 0.98324165810211861629, 0.98324165810211861629, 7,46618, 0.98324165810211861629 .66687459321626519] 错误:0.497132186092

如您所见,给定 alpha = 0.0251 的误差(对于梯度下降 - 通过反复试验发现)仅为 9.95% 左右。

自从我制作这个程序后,我了解到 leaky RelU 是 Sigmoid 函数的更好替代方案,因为它比 Sigmoid 优化和学习更快。我想在此程序中使用 numpy 实现泄漏的 RelU 函数,但我不确定从哪里开始,更具体地说,它的导数是什么。

如何在这个神经网络中实现 leaky RelU?

请您参考如下方法:

我想在这里补充一点,实际上有很多类似 ReLu 的激活函数可以用来代替标准的 ReLu activation。 :

  • Scaled exponential linear unit (SELU) 最近发布。它是 ELU 的扩展,具有特定的参数选择,具有额外的归一化效果并有助于更快地学习。

Here's the list所有激活及其导数。