When I perform adversarial training, I find that the output of BERT is always NAN.

[Struggle-Forever]

• When I perform adversarial training, I find that the output of BERT is always NAN.
• [ ]
  

[akihironitta]

As far as I know, training with AMP (precision=16) can sometimes be unstable and lead to nan as you report.

[Struggle-Forever]

    trainer = Trainer(devices="auto", accelerator="auto", logger=False, multiple_trainloader_mode='max_size_cycle',
                      terminate_on_nan=True,
                      # logger=tb_logger, log_every_n_steps=1,
                      # flush_logs_every_n_steps=5,
                      max_epochs=args.epochs)

I find that sometimes I can train normally and sometimes it is NAN

[Struggle-Forever]

I don't know what happened, but when I chose a new GPU, the NAN problem was solved. I hope this method can help everyone.