-
Notifications
You must be signed in to change notification settings - Fork 1
/
nohuptrain_11Dec_128
61 lines (49 loc) · 3.57 KB
/
nohuptrain_11Dec_128
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
Python Version: 3.10.6 (main, Nov 2 2022, 18:53:38) [GCC 11.3.0]
PyTorch Version: 1.13.0+cu117
Number of GPUs: 1
Save path: exps/ResNetSE34L_AM
Embedding size is 512, encoder SAP.
Initialised AMSoftmax m=0.300 s=30.000
Loaded the model on GPU 0
Initialised Adam optimizer
Initialised step LR scheduler
[W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-11 15:26:45 Epoch 1, TEER/TAcc 0.01, TLOSS 14.512095, LR 0.001000
2022-12-11 18:12:13 Epoch 2, TEER/TAcc 0.31, TLOSS 10.805117, LR 0.001000
2022-12-11 20:56:22 Epoch 3, TEER/TAcc 1.36, TLOSS 9.154469, LR 0.001000
2022-12-11 23:47:18 Epoch 4, TEER/TAcc 2.82, TLOSS 8.212339, LR 0.001000
2022-12-12 02:35:20 Epoch 5, TEER/TAcc 4.35, TLOSS 7.630208, LR 0.001000
2022-12-12 05:30:21 Epoch 6, TEER/TAcc 5.71, TLOSS 7.240783, LR 0.001000
2022-12-12 08:19:06 Epoch 7, TEER/TAcc 7.06, TLOSS 6.938164, LR 0.001000
2022-12-12 11:12:30 Epoch 8, TEER/TAcc 8.29, TLOSS 6.708576, LR 0.001000
2022-12-12 14:05:55 Epoch 9, TEER/TAcc 9.42, TLOSS 6.512892, LR 0.001000
2022-12-12 16:59:02 Epoch 10, TEER/TAcc 10.46, TLOSS 6.350480, LR 0.001000
Traceback (most recent call last):
File "/home/ubuntu/Documents/vt1code/voxceleb_trainer/voxceleb_trainer/./trainSpeakerNet.py", line 277, in <module>
main()
File "/home/ubuntu/Documents/vt1code/voxceleb_trainer/voxceleb_trainer/./trainSpeakerNet.py", line 271, in main
mp.spawn(main_worker, nprocs=n_gpus, args=(n_gpus, args))
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
while not context.join():
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
fn(i, *args)
File "/home/ubuntu/Documents/vt1code/voxceleb_trainer/voxceleb_trainer/trainSpeakerNet.py", line 224, in main_worker
sc, lab, _ = trainer.evaluateFromList(**vars(args))
File "/home/ubuntu/Documents/vt1code/voxceleb_trainer/voxceleb_trainer/SpeakerNet.py", line 171, in evaluateFromList
for idx, data in enumerate(test_loader):
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
data = self._next_data()
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
return self._process_data(data)
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
data.reraise()
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/_utils.py", line 543, in reraise
raise exception
soundfile.LibsndfileError: <unprintable LibsndfileError object>