Skip to main content
 首页 » 编程设计

python之Tensorflow LSTM 有状态选项不维护批处理之间的状态

2024年09月03日30jyk

我是 Tensorflow 的新手，想了解 keras LSTM layer所以我写了这个 测试程序以辨别 stateful 选项的行为。

#Tensorflow 1.x version 
import tensorflow as tf 
import numpy as np 
 
NUM_UNITS=1 
NUM_TIME_STEPS=5 
NUM_FEATURES=1 
BATCH_SIZE=4 
 
STATEFUL=True 
STATEFUL_BETWEEN_BATCHES=True 
 
lstm = tf.keras.layers.LSTM(units=NUM_UNITS, stateful=STATEFUL, 
            return_state=True, return_sequences=True, 
            batch_input_shape=(BATCH_SIZE, NUM_TIME_STEPS, NUM_FEATURES), 
            kernel_initializer='ones', bias_initializer='ones', 
            recurrent_initializer='ones') 
x = tf.keras.Input((NUM_TIME_STEPS,NUM_FEATURES),batch_size=BATCH_SIZE) 
result = lstm(x) 
 
I = tf.compat.v1.global_variables_initializer() 
sess = tf.compat.v1.Session() 
sess.run(I) 
 
X_input = np.array([[[3.14*(0.01)] for t in range(NUM_TIME_STEPS)] for b in range(BATCH_SIZE)]) 
feed_dict={x: X_input} 
 
def matprint(run, mat): 
    print('Batch = ', run) 
    for b in range(mat.shape[0]): 
        print('Batch Sample:', b, ', per-timestep output') 
        print(mat[b].squeeze()) 
 
print('BATCH_SIZE = ', BATCH_SIZE, ', T = ', NUM_TIME_STEPS, ', stateful =', STATEFUL) 
if STATEFUL: 
    print('STATEFUL_BETWEEN_BATCHES = ', STATEFUL_BETWEEN_BATCHES) 
 
for r in range(2): 
    feed_dict={x: X_input} 
    OUTPUT_NEXTSTATES = sess.run({'result': result}, feed_dict=feed_dict) 
    OUTPUT = OUTPUT_NEXTSTATES['result'][0] 
    NEXT_STATES=OUTPUT_NEXTSTATES['result'][1:] 
    matprint(r,OUTPUT) 
    if STATEFUL: 
        if STATEFUL_BETWEEN_BATCHES: 
            #For TF version 1.x manually re-assigning states from 
            #the last batch IS required for some reason ... 
            #seems like a bug 
            sess.run(lstm.states[0].assign(NEXT_STATES[0])) 
            sess.run(lstm.states[1].assign(NEXT_STATES[1])) 
        else: 
            lstm.reset_states()

请注意，LSTM 的权重设置为全 1，并且输入是恒定的以保持一致性。

正如预期的那样，当 statueful=False 没有样本、时间或批间时的脚本输出 依赖:

BATCH_SIZE =  4 , T =  5 , stateful = False 
Batch =  0 
Batch Sample: 0 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch Sample: 1 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch Sample: 2 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch Sample: 3 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch =  1 
Batch Sample: 0 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch Sample: 1 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch Sample: 2 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch Sample: 3 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ]

在设置 stateful=True 后，我期望每个批处理中的样本产生不同的输出( 大概是因为 TF 图在批处理样本之间保持状态)。然而，情况并非如此:

BATCH_SIZE =  4 , T =  5 , stateful = True 
STATEFUL_BETWEEN_BATCHES =  True 
Batch =  0 
Batch Sample: 0 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch Sample: 1 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch Sample: 2 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch Sample: 3 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch =  1 
Batch Sample: 0 , per-timestep output 
[0.86686385 0.8686781  0.8693927  0.8697042  0.869853  ] 
Batch Sample: 1 , per-timestep output 
[0.86686385 0.8686781  0.8693927  0.8697042  0.869853  ] 
Batch Sample: 2 , per-timestep output 
[0.86686385 0.8686781  0.8693927  0.8697042  0.869853  ] 
Batch Sample: 3 , per-timestep output 
[0.86686385 0.8686781  0.8693927  0.8697042  0.869853  ]

特别注意同一批处理的前两个样本的输出是相同的。

编辑:OverlordGoldDragon 通知了我 这种行为是预料之中的，而我的困惑在于 Batch 之间的区别——一组 (samples, timesteps, features) -- 和 Sample 批处理(或批处理的单个“行”)。 用下图表示:

因此，这提出了给定批处理的各个样本之间的依赖性(如果有的话)的问题。来自 根据我脚本的输出，我相信 每个 样本都被馈送到(逻辑上)单独的 LSTM block - 并且 差异样本的 LSTM 状态是独立的。我在这里画了这个:

我的理解正确吗？

顺便说一句，stateful=True 在 TensorFlow 1.x 中似乎被破坏了，因为如果我删除显式 分配上一批的状态:

         sess.run(lstm.states[0].assign(NEXT_STATES[0])) 
         sess.run(lstm.states[1].assign(NEXT_STATES[1]))

它停止工作，即第二批的输出与第一批相同。

我用 Tensorflow 2.0 语法重写了上面的脚本，行为符合我的预期 (无需在批处理之间手动转移 LSTM 状态):

#Tensorflow 2.0 implementation 
import tensorflow as tf 
import numpy as np 
 
NUM_UNITS=1 
NUM_TIME_STEPS=5 
NUM_FEATURES=1 
BATCH_SIZE=4 
 
STATEFUL=True 
STATEFUL_BETWEEN_BATCHES=True 
 
lstm = tf.keras.layers.LSTM(units=NUM_UNITS, stateful=STATEFUL, 
            return_state=True, return_sequences=True, 
            batch_input_shape=(BATCH_SIZE, NUM_TIME_STEPS, NUM_FEATURES), 
            kernel_initializer='ones', bias_initializer='ones', 
            recurrent_initializer='ones') 
X_input = np.array([[[3.14*(0.01)] 
                     for t in range(NUM_TIME_STEPS)] 
                     for b in range(BATCH_SIZE)]) 
@tf.function 
def forward(x): 
  return lstm(x) 
 
def matprint(run, mat): 
    print('Batch = ', run) 
    for b in range(mat.shape[0]): 
        print('Batch Sample:', b, ', per-timestep output') 
        print(mat[b].squeeze()) 
 
print('BATCH_SIZE = ', BATCH_SIZE, ', T = ', NUM_TIME_STEPS, ', stateful =', STATEFUL) 
if STATEFUL: 
    print('STATEFUL_BETWEEN_BATCHES = ', STATEFUL_BETWEEN_BATCHES) 
 
for r in range(2): 
    OUTPUT_NEXTSTATES = forward(X_input) 
    OUTPUT = OUTPUT_NEXTSTATES[0].numpy() 
    NEXT_STATES=OUTPUT_NEXTSTATES[1:] 
    matprint(r,OUTPUT) 
    if STATEFUL: 
        if STATEFUL_BETWEEN_BATCHES: 
            pass 
            #Explicitly re-assigning states from the last batch isn't 
            # required as the model maintains inter-batch history. 
            #This is NOT the same behavior for TF.version < 2.0 
            #lstm.states[0].assign(NEXT_STATES[0].numpy()) 
            #lstm.states[1].assign(NEXT_STATES[1].numpy()) 
        else: 
            lstm.reset_states()

这是输出:

BATCH_SIZE =  4 , T =  5 , stateful = True 
STATEFUL_BETWEEN_BATCHES =  True 
Batch =  0 
Batch Sample: 0 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch Sample: 1 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch Sample: 2 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch Sample: 3 , per-timestep output 
[0.38041887 0.663519   0.79821336 0.84627265 0.8617684 ] 
Batch =  1 
Batch Sample: 0 , per-timestep output 
[0.86686385 0.8686781  0.8693927  0.8697042  0.869853  ] 
Batch Sample: 1 , per-timestep output 
[0.86686385 0.8686781  0.8693927  0.8697042  0.869853  ] 
Batch Sample: 2 , per-timestep output 
[0.86686385 0.8686781  0.8693927  0.8697042  0.869853  ] 
Batch Sample: 3 , per-timestep output 
[0.86686385 0.8686781  0.8693927  0.8697042  0.869853  ]

请您参考如下方法:

一切似乎都按预期工作 - 但代码需要大量修改:

  • Batch: 0 应该是 Sample: 0;您的 batch_shape=(4, 5, 1)，包含 4 个样本、5 个时间步 和 1 个特征/ channel 。 I 在你的例子中是实际的批标记
  • 每个样本都被视为一个独立序列，所以它就像先喂样本 1，然后喂样本 2 - 除了在学习期间，批量样本损失被平均以计算梯度
  • 您的每个样本都是相同 - 因此为每个批处理获得相同的输出是明智的；运行 print(X_input) 来验证
  • Stateful 按预期工作:给定相同 输入，stateful=False 产生相同 输出(因为没有维护内部状态)- 而stateful=True 为不同的 I 产生不同的输出，即使输入是相同的(由于内存)
  • 照原样，您的 lstm 不是学习，所以权重是相同的 - 所有 stateful=False 输出都将是相同的输入相同
  • 强烈建议不要将所有权重初始化为相同的值 - 相反，请使用 random seed


  文章来源：
  https://blog.windname.com/post/hegai2b8.html
阅读延展