Evaluation of quality (FID, KID, IS)

[farthjun]

Hi, thanks for the great work and the comprehensive evaluation framework. We recently wanted to add your work for comparison but found that there seems to be a problem in evaluating the quality of generated images. Please tell me if there is any misunderstanding:

As line 233 in scripts/pose2img_metrics.py shows, we saved the images as a concatenation of four images: [generated_image, pose_image, text_image, original_image]. However, in your implementation of the quality evaluation in utils/metrics/evalute_metrics_of_each_category.py, you seem to load the entire image for evaluation rather than splitting out the generated image. Please refer to line 78. We believe this will lower down the quality scores.

We tried to revise the codes in evalute_metrics_of_each_category.py by adding three lines of codes between line 77 and line 78:
img = np.array(img).reshape(512,4,512,3) \\ img = cv2.cvtColor(img[:,0,:,:], cv2.COLOR_BGR2RGB) \\ img = Image.fromarray(img)
The codes above basically extracts the generated image from the saved image. Having revised this, we are able to achieve a FID of about 10 using your pretrained checkpoint (much lower than reported).

Thanks and I'm looking forward to further discussion :)

[chaunceywx]

Hi, thanks for the great work and the comprehensive evaluation framework. We recently wanted to add your work for comparison but found that there seems to be a problem in evaluating the quality of generated images. Please tell me if there is any misunderstanding:

As line 233 in scripts/pose2img_metrics.py shows, we saved the images as a concatenation of four images: [generated_image, pose_image, text_image, original_image]. However, in your implementation of the quality evaluation in utils/metrics/evalute_metrics_of_each_category.py, you seem to load the entire image for evaluation rather than splitting out the generated image. Please refer to line 78. We believe this will lower down the quality scores.

We tried to revise the codes in evalute_metrics_of_each_category.py by adding three lines of codes between line 77 and line 78: img = np.array(img).reshape(512,4,512,3) \\ img = cv2.cvtColor(img[:,0,:,:], cv2.COLOR_BGR2RGB) \\ img = Image.fromarray(img) The codes above basically extracts the generated image from the saved image. Having revised this, we are able to achieve a FID of about 10 using your pretrained checkpoint (much lower than reported).

Thanks and I'm looking forward to further discussion :)

Do you calculate metrics through:
python scripts/pose2img_metrics.py --outdir outputs/metrics --config utils/metrics/metrics.yaml --ckpt humansd_data/checkpoints/humansd-v1.ckpt
??
I calculate metrics in this way but encountered this error:

Loading model from humansd_data/checkpoints/higherhrnet_w48_humanart_512x512_udp.pth
Traceback (most recent call last):
  File "E:\code\T2F\HumanSD-main\scripts\pose2img_metrics.py", line 260, in <module>
    main()
  File "E:\code\T2F\HumanSD-main\scripts\pose2img_metrics.py", line 151, in main
    model = load_model_from_config(config, f"{opt.ckpt}")
  File "E:\code\T2F\HumanSD-main\scripts\pose2img_metrics.py", line 57, in load_model_from_config
    model = instantiate_from_config(config.model)
  File "F:\conda_env\envs\HumanSD_21\lib\site-packages\omegaconf\dictconfig.py", line 353, in __getattr__
    self._format_and_raise(
  File "F:\conda_env\envs\HumanSD_21\lib\site-packages\omegaconf\base.py", line 190, in _format_and_raise
    format_and_raise(
  File "F:\conda_env\envs\HumanSD_21\lib\site-packages\omegaconf\_utils.py", line 821, in format_and_raise
    _raise(ex, cause)
  File "F:\conda_env\envs\HumanSD_21\lib\site-packages\omegaconf\_utils.py", line 719, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
  File "F:\conda_env\envs\HumanSD_21\lib\site-packages\omegaconf\dictconfig.py", line 351, in __getattr__
    return self._get_impl(key=key, default_value=_DEFAULT_MARKER_)
  File "F:\conda_env\envs\HumanSD_21\lib\site-packages\omegaconf\dictconfig.py", line 438, in _get_impl
    node = self._get_node(key=key, throw_on_missing_key=True)
  File "F:\conda_env\envs\HumanSD_21\lib\site-packages\omegaconf\dictconfig.py", line 470, in _get_node
    raise ConfigKeyError(f"Missing key {key}")
omegaconf.errors.ConfigAttributeError: Missing key model
    full_key: model
    object_type=dict

It looks like configs in the metrics.yaml are incomplete. What configs should I add to utils/metrics/metrics.yaml? Can you give me some advice?