In 5c, MinGRU training resets hidden state on terminal[t-1], while rollout inference resets on terminal[t]. Since terminal[t] belongs to observation[t], training carries state from the previous episode into the first observation of the next, then resets one step late. Gradients also cross the episode boundary.
This was introduced in a6066343 on July 21th; the previous scan used the current terminal. It potentially affects every native 5c checkpoint & environment trained since then, especially in short-episode environments. In my SCS environment I notice a -1% evaluation score due to this.
Problematic code: forward and backward.
Expected: use terminal[t], matching rollout inference.
In
5c, MinGRU training resets hidden state onterminal[t-1], while rollout inference resets onterminal[t]. Sinceterminal[t]belongs toobservation[t], training carries state from the previous episode into the first observation of the next, then resets one step late. Gradients also cross the episode boundary.This was introduced in
a6066343on July 21th; the previous scan used the current terminal. It potentially affects every native5ccheckpoint & environment trained since then, especially in short-episode environments. In my SCS environment I notice a -1% evaluation score due to this.Problematic code: forward and backward.
Expected: use
terminal[t], matching rollout inference.