q_ret update not used

[mvindiola1]

I have enjoyed your really clean implementation of MPO. Thank you for making it available. I was looking at the critic update and think I may have spotted a bug. You update q_ret on line 163 according to retrace but as far as I can see you do not actually use it anywhere. I think you might want to use it recursively on line 161 in place of q_retraces[step + 1].

MPO/mpo.py

Lines 160 to 163 in c84bf23

	for step in reversed(range(nsteps)):
	q_ret = reward_batch[step] + self.γ * q_retraces[step + 1] * (1 - done_batch[step + 1])
	q_retraces[step] = q_ret
	q_ret = (rho_i[step] * (q_retraces[step] - q_i[step])) + val[step]