Enable AnomalyScoreMeanVarThreshold based on three-sigma

[WenjingKangIntel]

Description

• Provide a summary of the modification as well as the issue that has been resolved. List any dependencies that this modification necessitates.

1. Use train data as validation data during validation and collect all the predictions
2. Based on the predictions above, perform three sigma rule to determine the final threshold: mean - 3 * standard_deviation or mean + 3 * standard_deviation, +/- is determined by config. Currently, only the + scenario should be used, because currently this repo determines the result via checking whether the predictions is larger than the threshold.

• Fixes [Task]: Enabling Unsupervised Workflow: Propose a new thresholding approach #1027

Changes

• Bug fix (non-breaking change which fixes an issue)
• Refactor (non-breaking change which refactors the code base)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist

• My code follows the pre-commit style and check guidelines of this project.
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing tests pass locally with my changes
• I have added a summary of my changes to the CHANGELOG (not for minor changes, docs and tests).

[WenjingKangIntel]

test on:
model: padim
config:

dataset:
  name: mvtec
  format: mvtec
  path: /home/kangwenj/hackathon/data/mvtec_anomaly_detection
  category: bottle
  task: segmentation
val_split_mode: same_as_train

threshold:
  method: meanvar
  std_dev_ratio: 3
  meanvar_mode: add

result:

Test metric	DataLoader 0
image_AUROC	0.9460317492485046
image_F1Score	0.9402984976768494
pixel_AUROC	0.9814534187316895
pixel_F1Score	0.5805442333221436

if using adaptive and val_split_mode: same_as_test:
result:

Test metric	DataLoader 0
image_AUROC	0.9936507940292358
image_F1Score	0.9763779044151306
pixel_AUROC	0.983005702495575
pixel_F1Score	0.72203129529953

Actually, when using this meanvar threshold, increasing std_dev_ratio to 10, can get a better result for image_AUROC/image_F1Score:
image_AUROC 0.9936507940292358
image_F1Score 0.9677419662475586
but the pixel_AUROC/pixel_F1Score is not good:
pixel_AUROC 0.7939819097518921
pixel_F1Score 0.3183630406856537

There might be two different sweet spots for image/pixel threshold.

[ashwinvaidya17]

Thanks. It looks good overall. Just minor comments. Before this can be merged we need to discuss the design for adding more thresholding methods internally.

[ashwinvaidya17]

As you mentioned in your description this does not make sense in the anomaly context. Maybe we shouldn't support minus operation.

[WenjingKangIntel]

We may add it as a placeholder. If in the future, if we use negative outputs of the model, then minus mode can be used?