我的代理一直在采取随机行动,因此算法未正确训练。我如何确保它采取存储在“next_action, ArgMax = custom_argmax(Q_value)”行中的最佳操作。函数 custom_argmax 计算为每个状态、 Action 对找到的最大 Q 值。
max_episodes = 10
max_steps_per_episode = 1000
discount_rate = 0.99
exploration_rate = 0.5
max_exploration_rate = 1
min_exploration_rate = 0.1
learning_rate = 0.01
explore_decay_rate = 0.2
errors = []
def play_single_game(max_steps_per_episode, render):
global errors
state = env.reset()
# print('We are resetting: ' )
action = env.action_space.sample()
for step in range(max_steps_per_episode - 1):
# if episode == max_episodes - 1:
if render:
env.render()
# print("This is the Ac:", a)
'''
if step%2 == 0:
a = 1
else:
a = 1
'''
new_state, reward, done, info = env.step(action) # declare all, gets new state from taking certain action
# print(info)
next_state = new_state
# print(reward)
old_weights = weights.theta.copy()
if done == True:
weights.theta += learning_rate * (reward - weights_multiplied_by_features(state, action)) * feature_space(state, action)
# print("we are done")
break
else:
# not finished
Q_value= associated_Q_value(next_state)
exploration_rate_threshold = random.uniform(0, 1)
next_action, ArgMax = custom_argmax(Q_value) # is best action
if exploration_rate_threshold < exploration_rate: # take random
r = random.randint(0, len(LEGAL_MOVES) - 1)
next_action = r
# we will update Q(s,a) AS we experience the episode
weights.theta += learning_rate * (reward + discount_rate * ArgMax - weights_multiplied_by_features(state, action)) * feature_space(state, action)
# next state becomes current state
state = next_state
action = next_action
change_in_weights = np.abs(weights.theta - old_weights).sum()
errors.append(change_in_weights)
请您参考如下方法:
您正在进行 epsilon-greedy 探索。你已经设置了 exploration_rate = 0.5 所以你的代理将总是采取 50% 的随机行动,显然。这可能太高了,但这并不意味着您的代理没有学习。
如果您想正确评估您的代理,您必须在禁用探索的情况下运行一集。您不能只禁用随机操作,因为那样它可能永远不会尝试不同的操作;这称为探索/开发权衡。但是,您可以在代理学习时慢慢减少探索,例如在你的循环中使用 exploration_rate *= 0.999 或类似的。
