Resource exhaustion error

[joeybose]

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[128,78,600]
[[Node: concat_12 = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](Reshape_37, Add, concat_12/axis)]]
[[Node: Reshape_45/_143 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_326_Reshape_45", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

The memory usuage blows up even with an extremely powerful GPU cluster on aws i.e. 64 Gb of GPU memory. The original MemN2N code that this repo is derived from doesn't suffer from this problem so I assume this is a bug.