The hyperparameters from Wandb sweep are not being recognized by jsonargparser

[bwdeng20]

🐛 Bug report

To reproduce

You need a wandb account.

Clone my toy repo and then follow the official wandb sweep instructions like the following.

wandb sweep sweeps/swp_nc.yaml  # to  intialize sweep and get `entity/project/sweep_ID`
wandb agent entity/project/sweep_ID --count 2 # count is the number of hyperparameter tuning trials

Then you can see something like

This image shows the terminal output from the first sweep experiment. Note the hyperparameter values in the red rectangles. We can see that jsonargparser is not affected by the hyperparameter values (i.e., mu=5, sigma=2) chosen by wandb sweep . Instead, it still uses the default values (i.e., mu=0, sigma=1). This means that adjusting hyperparameters with wandb sweep is not compatible with jsonargparser .

Expected behavior

The args parsed by jsonargparser can be changed by wandb sweep.

Environment

• jsonargparse version (e.g., 4.8.0): 4.27.2
• Python version (e.g., 3.10): 3.10
• How jsonargparse was installed (e.g. pip install jsonargparse[all]): pip install jsonargparse[all]
• OS (e.g., Linux): Ubuntu22.04
• Wandb version: 0.16

[mauvilsa]

@bwdeng20 thank you for dropping by and thank you for the toy repo. I am not a user of wandb, nor have an account. So I am not able to reproduce your issue.

From what I see in the code, there is no reason yet for me to believe that there is a bug in jsonargparse, rather than a problem in your code. The instructions in wandb are not specific to jsonargparse. And it doesn't explain how the sweeps work internally.

How is it supposed to work? How are the modified parameters of the sweep given to jsonargparse? Is it via sys.argv, via environment variables, something else? The only thing I can deduce from the screenshot is that the configs/nc.yaml file is loaded. This file is fixed and no init_args are given, thus it makes sense that the values are the defaults of the FakeResultGenerator class.

[bwdeng20]

Thanks for the swift response. Maybe I should change the tags from bug to question.

How is it supposed to work?

Solidify a set of experiment configurations (in this case ) into a configuration file (in this case configs/nc.yaml). At this point, running python script.py --configs/nc.yaml will use the default parameter values in configs/nc.yaml for an experiment. When we want to tune some hyperparameters (in this case, the construction parameters mu and sigma of FakeResultGenerator), first define reading the default experiment configuration nc.yaml in tuning config sweeps/swp_nc.yaml by

# a snippet from sweeps/swp_nc.yaml
command:
  - python
  - ${program}
  - --config
  - configs/nc.yaml

and then define the following fields according to the wandb sweep guidelines.

parameters:
  gen.init_args:
    parameters:
      mu:
        values: [ 4., 5. ]
      sigma:
        values: [ 2., 2. ]

We hope the hyperparameter values sampled by wandb sweep can be used by jsonargparser to replace the corresponding values defined in configs/nc.yaml or FakeResultGenerator defaults to run different experiments, given the wandb sweep commands like

wandb sweep sweeps/swp_nc.yaml  # to  intialize sweep and get `entity/project/sweep_ID`
wandb agent entity/project/sweep_ID --count 2 # count is the number of hyperparameter tuning trials

How are the modified parameters of the sweep given to jsonargparse?

I tried to find relevant clues, but I couldn't find any helpful documentation or source code, so I'm sorry that I'm not clear on that.

[mauvilsa]

I tried to find relevant clues, but I couldn't find any helpful documentation or source code, so I'm sorry that I'm not clear on that.

This is kind of key. If the wandb docs are not clear, then the problem is wandb and you should contact them to understand what needs to be done. It isn't a bug in jsonargparse.

Just supposing, imagine that it works by running the command appending more arguments, i.e. sys.argv. The first run of the sweep would be the command plus the first values in the parameters. jsonargparse would expect the command to be like

python ${program} --config configs/nc.yaml --gen.init_args.mu 4. --gen.init_args.sigma 2.

In your code you could print the value of sys.argv to see this. But it seems wandb does not work this way, or there is something you are not doing correctly. Still, it is the wandb people the ones who should help.

[mauvilsa]

Closing this since it seems to be a topic for wandb. But feel free to comment further if there is need.