Trainer not defined.

[boundlessliving]

The past few days, when I start the training, I've been getting this error. I end up having to create an entirely new instance and then it sometimes works. Nothing different in the process. Anyone else? Any way to salvage this error without creating an entirely new instance?

I've now tried 3 different machines on VAST and am getting the same error. I've used both 3090 (my usual) and tried a 4090, same results every time.

Global seed set to 23
Running on GPUs 0,
Loading model from model.ckpt
Traceback (most recent call last):
File "/workspace/Dreambooth-Stable-Diffusion/main.py", line 694, in
model = load_model_from_config(config, opt.actual_resume)
File "/workspace/Dreambooth-Stable-Diffusion/main.py", line 30, in load_model_from_config
pl_sd = torch.load(ckpt, map_location="cpu")
File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 705, in load
with _open_zipfile_reader(opened_file) as opened_zipfile:
File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 242, in init
super(_open_zipfile_reader, self).init(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/Dreambooth-Stable-Diffusion/main.py", line 907, in
if trainer.global_rank == 0:
NameError: name 'trainer' is not defined. Did you mean: 'Trainer'?

[caniyabanci76]

I've seen 'trainer' is not defined when the model didn't download properly or was corrupted upon download. have you tried restarting the kernel and re-run the cells again?