MisconfigurationException: `configure_optimizers` must include a monitor when a `ReduceLROnPlateau` scheduler is used #79

adewdev · 2023-07-11T18:51:39Z

(spacetimeformer) adewdev@bootedeagle:~/Devtools/PyWS0/spacetimeformer-main-original/spacetimeformer$ python train.py spacetimeformer solar_energy --context_points 168 --target_points 24 --d_model 100 --d_ff 400 --enc_layers 5 --dec_layers 5 --l2_coeff 1e-3 --dropout_ff .2 --dropout_emb .1 --d_qk 20 --d_v 20 --n_heads 6 --run_name spatiotemporal_al_solar --batch_size 32 --class_loss_imp 0 --initial_downsample_convs 1 --decay_factor .8 --warmup_steps 1000
Using default wandb log dir path of ./data/STF_LOG_DIR. This can be adjusted with the environment variable STF_LOG_DIR
Forecaster
L2: 0.001
Linear Window: 0
Linear Shared Weights: False
RevIN: False
Decomposition: False
GlobalSelfAttn: AttentionLayer(
(inner_attention): PerformerAttention(
(kernel_fn): ReLU()
)
(query_projection): Linear(in_features=100, out_features=120, bias=True)
(key_projection): Linear(in_features=100, out_features=120, bias=True)
(value_projection): Linear(in_features=100, out_features=120, bias=True)
(out_projection): Linear(in_features=120, out_features=100, bias=True)
(dropout_qkv): Dropout(p=0.0, inplace=False)
)
GlobalCrossAttn: AttentionLayer(
(inner_attention): PerformerAttention(
(kernel_fn): ReLU()
)
(query_projection): Linear(in_features=100, out_features=120, bias=True)
(key_projection): Linear(in_features=100, out_features=120, bias=True)
(value_projection): Linear(in_features=100, out_features=120, bias=True)
(out_projection): Linear(in_features=120, out_features=100, bias=True)
(dropout_qkv): Dropout(p=0.0, inplace=False)
)
LocalSelfAttn: AttentionLayer(
(inner_attention): PerformerAttention(
(kernel_fn): ReLU()
)
(query_projection): Linear(in_features=100, out_features=120, bias=True)
(key_projection): Linear(in_features=100, out_features=120, bias=True)
(value_projection): Linear(in_features=100, out_features=120, bias=True)
(out_projection): Linear(in_features=120, out_features=100, bias=True)
(dropout_qkv): Dropout(p=0.0, inplace=False)
)
LocalCrossAttn: AttentionLayer(
(inner_attention): PerformerAttention(
(kernel_fn): ReLU()
)
(query_projection): Linear(in_features=100, out_features=120, bias=True)
(key_projection): Linear(in_features=100, out_features=120, bias=True)
(value_projection): Linear(in_features=100, out_features=120, bias=True)
(out_projection): Linear(in_features=120, out_features=100, bias=True)
(dropout_qkv): Dropout(p=0.0, inplace=False)
)
Using Embedding: spatio-temporal
Time Emb Dim: 6
Space Embedding: True
Time Embedding: True
Val Embedding: True
Given Embedding: True
Null Value: None
Pad Value: None
Reconstruction Dropout: Timesteps 0.05, Standard 0.1, Seq (max len = 5) 0.2, Skip All Drop 1.0
*** Spacetimeformer (v1.5) Summary: ***
Model Dim: 100
FF Dim: 400
Enc Layers: 5
Dec Layers: 5
Embed Dropout: 0.1
FF Dropout: 0.2
Attn Out Dropout: 0.0
Attn Matrix Dropout: 0.0
QKV Dropout: 0.0
L2 Coeff: 0.001
Warmup Steps: 1000
Normalization Scheme: batch
Attention Time Windows: 1
Shifted Time Windows: False
Position Emb Type: abs
Recon Loss Imp: 0.0

/home/adewdev/Devtools/anaconda3/envs/spacetimeformer/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:287: LightningDeprecationWarning: Passing `Trainer(accelerator='dp')` has been deprecated in v1.5 and will be removed in v1.7. Use `Trainer(strategy='dp')` instead.
rank_zero_deprecation(
/home/adewdev/Devtools/anaconda3/envs/spacetimeformer/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:597: UserWarning: 'dp' is not supported on CPUs, hence setting `strategy='ddp'`.
rank_zero_warn(f"{strategy_flag!r} is not supported on CPUs, hence setting `strategy='ddp'`.")
/home/adewdev/Devtools/anaconda3/envs/spacetimeformer/lib/python3.8/site-packages/pytorch_lightning/loops/utilities.py:91: PossibleUserWarning: `max_epochs` was not set. Setting it to 1000 epochs. To train without an epoch limit, set `max_epochs=-1`.
rank_zero_warn(
GPU available: True, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/home/adewdev/Devtools/anaconda3/envs/spacetimeformer/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1823: PossibleUserWarning: GPU available but not used. Set `accelerator` and `devices` using `Trainer(accelerator='gpu', devices=1)`.
rank_zero_warn(
`Trainer(limit_val_batches=1.0)` was configured so 100% of the batches will be used..
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1

distributed_backend=gloo
All distributed processes registered. Starting with 1 processes

Traceback (most recent call last):
File "train.py", line 869, in
main(args)
File "train.py", line 849, in main
trainer.fit(forecaster, datamodule=data_module)
File "/home/adewdev/Devtools/anaconda3/envs/spacetimeformer/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 771, in fit
self._call_and_handle_interrupt(
File "/home/adewdev/Devtools/anaconda3/envs/spacetimeformer/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 722, in _call_and_handle_interrupt
return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
File "/home/adewdev/Devtools/anaconda3/envs/spacetimeformer/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
return function(*args, **kwargs)
File "/home/adewdev/Devtools/anaconda3/envs/spacetimeformer/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 812, in _fit_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/home/adewdev/Devtools/anaconda3/envs/spacetimeformer/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1218, in _run
self.strategy.setup(self)
File "/home/adewdev/Devtools/anaconda3/envs/spacetimeformer/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 155, in setup
super().setup(trainer)
File "/home/adewdev/Devtools/anaconda3/envs/spacetimeformer/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 139, in setup
self.setup_optimizers(trainer)
File "/home/adewdev/Devtools/anaconda3/envs/spacetimeformer/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 128, in setup_optimizers
self.optimizers, self.lr_scheduler_configs, self.optimizer_frequencies = _init_optimizers_and_lr_schedulers(
File "/home/adewdev/Devtools/anaconda3/envs/spacetimeformer/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 190, in _init_optimizers_and_lr_schedulers
_configure_schedulers_automatic_opt(lr_schedulers, monitor)
File "/home/adewdev/Devtools/anaconda3/envs/spacetimeformer/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 305, in _configure_schedulers_automatic_opt
raise MisconfigurationException(
pytorch_lightning.utilities.exceptions.MisconfigurationException: configure_optimizers must include a monitor when a ReduceLROnPlateau scheduler is used. For example: {"optimizer": optimizer, "lr_scheduler": scheduler, "monitor": "metric_to_track"}
(spacetimeformer) adewdev@bootedeagle:~/Devtools/PyWS0/spacetimeformer-main-original/spacetimeformer$ conda list

packages in environment at /home/adewdev/Devtools/anaconda3/envs/spacetimeformer:

i followed all the instructions but getting above error. not sure if i missed any.

The text was updated successfully, but these errors were encountered:

SEU-ccq · 2023-07-25T07:53:07Z

I'm having the same problem, have you solved it yet?

HiFei4869 · 2023-07-28T08:49:29Z

I have the same problem.
I tried adding the following code to train.py. But it doesn't work.

def configure_optimizers(self):
    optimizer = torch.optim.Adam(self.parameters(), lr=0.02)
    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer,
                                                       mode='min',
                                                       factor=0.2,
                                                       patience=2,
                                                       min_lr=1e-6,
                                                       verbose=True)
return {
   "optimizer": optimizer,
   "lr_scheduler": scheduler, # Changed scheduler to lr_scheduler
   "monitor": "metric_to_track"
}

Sad to find that many people have the same unsolved problem.

HiFei4869 · 2023-07-29T13:56:04Z

@adewdev @seuccq
I tried to modify the spacetimeformer_model.py. It's under spacetimeformer/spacetimeformer_model.
In the function configure_optimizers, I modified the code by adding a monitor in return value.
It looks like this:

def configure_optimizers(self):
    optimizer = torch.optim.AdamW(
        self.parameters(),
        lr=self.base_lr,
        weight_decay=self.l2_coeff,
        )
    scheduler = stf.lr_scheduler.WarmupReduceLROnPlateau(
        optimizer,
        init_lr=self.init_lr,
        peak_lr=self.base_lr,
        warmup_steps=self.warmup_steps,
        patience=3,
        factor=self.decay_factor,
    )
    monitor = 'val/loss'
   
    return {
        "optimizer": optimizer,
        "lr_scheduler": {
            "scheduler": scheduler,
            "monitor": monitor
        }
    }

It seems to work well, at least it won't give me the same error message.

victorventuri · 2023-12-04T22:07:41Z

@adewdev @seuccq I tried to modify the spacetimeformer_model.py. It's under spacetimeformer/spacetimeformer_model. In the function configure_optimizers, I modified the code by adding a monitor in return value. It looks like this:
def configure_optimizers(self):
    optimizer = torch.optim.AdamW(
        self.parameters(),
        lr=self.base_lr,
        weight_decay=self.l2_coeff,
        )
    scheduler = stf.lr_scheduler.WarmupReduceLROnPlateau(
        optimizer,
        init_lr=self.init_lr,
        peak_lr=self.base_lr,
        warmup_steps=self.warmup_steps,
        patience=3,
        factor=self.decay_factor,
    )
    monitor = 'val/loss'
   
    return {
        "optimizer": optimizer,
        "lr_scheduler": {
            "scheduler": scheduler,
            "monitor": monitor
        }
    }
It seems to work well, at least it won't give me the same error message.

I had this issue too, and your solution solved it for me. Thank you!

pdy265 · 2024-03-25T02:28:20Z

@adewdev @seuccq I tried to modify the spacetimeformer_model.py. It's under spacetimeformer/spacetimeformer_model. In the function configure_optimizers, I modified the code by adding a monitor in return value. It looks like this:
def configure_optimizers(self):
    optimizer = torch.optim.AdamW(
        self.parameters(),
        lr=self.base_lr,
        weight_decay=self.l2_coeff,
        )
    scheduler = stf.lr_scheduler.WarmupReduceLROnPlateau(
        optimizer,
        init_lr=self.init_lr,
        peak_lr=self.base_lr,
        warmup_steps=self.warmup_steps,
        patience=3,
        factor=self.decay_factor,
    )
    monitor = 'val/loss'
   
    return {
        "optimizer": optimizer,
        "lr_scheduler": {
            "scheduler": scheduler,
            "monitor": monitor
        }
    }
It seems to work well, at least it won't give me the same error message.

It seems to also worked for me, thank you very much. And I would like to ask how to obtain the data when using the spacetimeformer model for training。

anibalpedraza mentioned this issue Apr 5, 2024

configure_optimizers must include a monitor when a ReduceLROnPlateau scheduler is used. For example: {"optimizer": optimizer, "lr_scheduler": scheduler, "monitor": "metric_to_track"} #94

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MisconfigurationException: `configure_optimizers` must include a monitor when a `ReduceLROnPlateau` scheduler is used #79

MisconfigurationException: `configure_optimizers` must include a monitor when a `ReduceLROnPlateau` scheduler is used #79

adewdev commented Jul 11, 2023 •

edited

Loading

SEU-ccq commented Jul 25, 2023

HiFei4869 commented Jul 28, 2023 •

edited

Loading

HiFei4869 commented Jul 29, 2023 •

edited

Loading

victorventuri commented Dec 4, 2023

pdy265 commented Mar 25, 2024

MisconfigurationException: configure_optimizers must include a monitor when a ReduceLROnPlateau scheduler is used #79

MisconfigurationException: configure_optimizers must include a monitor when a ReduceLROnPlateau scheduler is used #79

Comments

adewdev commented Jul 11, 2023 • edited Loading

distributed_backend=gloo All distributed processes registered. Starting with 1 processes

packages in environment at /home/adewdev/Devtools/anaconda3/envs/spacetimeformer:

Name Version Build Channel

SEU-ccq commented Jul 25, 2023

HiFei4869 commented Jul 28, 2023 • edited Loading

HiFei4869 commented Jul 29, 2023 • edited Loading

victorventuri commented Dec 4, 2023

pdy265 commented Mar 25, 2024

MisconfigurationException: `configure_optimizers` must include a monitor when a `ReduceLROnPlateau` scheduler is used #79

MisconfigurationException: `configure_optimizers` must include a monitor when a `ReduceLROnPlateau` scheduler is used #79

adewdev commented Jul 11, 2023 •

edited

Loading

distributed_backend=gloo
All distributed processes registered. Starting with 1 processes

HiFei4869 commented Jul 28, 2023 •

edited

Loading

HiFei4869 commented Jul 29, 2023 •

edited

Loading