Another dataset but got 0&nan results

[Rudder573]

How amazing your project is! Hey, I try to run your project on other datasets like KolektorSDD2, and I put it in the 'mvtec' catalogue. But I got no results: I-AUROC: 0.0 AP-det: nan seg-I-AUROC: 0.0 seg-AP-det: nan P-AUROC: 0.0 AUPRO: nan AP-loc: nan. The dataset link is https://github.com/2Obe/BSData. I want to know if it is suitable for the SuperSimpleNet. Please let me know the fact at your convenience.

[ninechi143]

There can be many reasons which lead to this phenomenon, some of them are related to the hyperparameter settings and model computation flow, while the others are about the external bugs (which are irrelevant to Deep Learning algorithm, e.g., the format and expression of your annotation files are somehow incorrect).

If you are certain that the dataset and its format are complete,
I would like to say something happened in your model training, like, the model outputs, losses, or gradients ever exploded in one of parameter updates.

You can check the corresponding values in each iteration, i.e., in train.py, examine the values of losses when you feed each batch in train_loader.

[blaz-r]

Hi, as @ninechi143 already said, I'd first look at the data, if everything with a custom dataset is correct.

Keep in mind that some metrics like AUROC are not defined in cases where you have only one class. So if your test set contains only normal labels, AUROC by definition doesn't have a value. It is possible that torchmetrics implementation in this case just returns 0.

Hope this helps.

[Rudder573]

Well, I received UserWarning: No positive samples in targets, true positive value should be meaningless. Returning zero tensor in true positive score warnings.warn(*args, **kwargs). And the history result is like:
wandb: AP-det ▁▃▅▆▇▃▅▆▆▆█▃▄▆▃▇█▃█▆▇▅▃▅▅▅▅▄▅▆▅▄▇▆▆▅▅▅▄▄
wandb: AUPRO ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb: I-AUROC ▁▃▄▅▆▆▃▃▄▄▅▂▇▂▁▄▅▂▅▁▇▁▆▇▃▃▃▄▃▄▃▄▄▄█▄▃▃▂▂
wandb: P-AUROC ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb: avg_loss █▃▃▃▃▃▃▁▂▃▂▃▂▂▂▁▂▃▂▂▂▂▂▂▃▂▂▂▂▁▂▂▂▂▂▂▂▂▁▂
wandb: batch_loss ▃▃▇▂▂▂▇▅▅▇▂▂▂▇▇▂▆▇▇▆▇▇▂▆▆▇▆▆▁▇▇█▁█▇▆▅▃▅▅
wandb: seg-AP-det ▅█▄▁▆▃▃▂▅▁▃▃▂▃▁▄▄▁▁▁▂▃▂▃▂▃▂▂▁▄▂▂▂▃▃▃▂▂▂▂
wandb: seg-I-AUROC ▅▄▄▃█▁▁▆▄▄▂▂▁▆▄▁▂▆▃▂▁▄▃▁▃▃▃▂▃▂▂▃▃▂▂▃▂▃▃▂
Furthermore, the final result is I-AUROC: 0.396806538105011 AP-det: 0.8442203402519226 seg-I-AUROC: 0.13888710737228394 seg-AP-det: 0.6895762085914612 P-AUROC: 0.0 AUPRO: 0 AP-loc: nan
I found the visualizing result as follows:


, which means however anomalous the sample is, the predicted mask is anomalous, and even the anomalous mask is not predicted. So I doubted if the dataset is suitable for this excellent program.

[blaz-r]

I believe this warning confirms what I said. Since there are no anomalous samples in the test set, the metric values are invalid.

Looking at the metric values, the image-level versions seem to have value, but the pixel ones don't. This would lead me to believe that something is wrong with how you load the dataset, especially the ground truth anomaly mask.
Make sure the GT pixel-label in the test set is read as grayscale single-channel mask with anomalous pixel having value 1, and normal having value 0.

[jin-123-456]

Your work is impressive! However, while replicating your experiments, I noticed a major gap between my results and the paper's reported performance on the second dataset—unlike the first dataset, which matched closely. Could you guide me on how to troubleshoot this discrepancy?

[blaz-r]

I will need more details to help, like which datasets and the results. Could you also please open a separate issue, since this looks unrelated to this one. Thank you.

[jin-123-456]

hank you for your reply. I've already created a new conversation.

[Rudder573]

I believe this warning confirms what I said. Since there are no anomalous samples in the test set, the metric values are invalid.

Looking at the metric values, the image-level versions seem to have value, but the pixel ones don't. This would lead me to believe that something is wrong with how you load the dataset, especially the ground truth anomaly mask. Make sure the GT pixel-label in the test set is read as grayscale single-channel mask with anomalous pixel having value 1, and normal having value 0.

Well, I follow your advice, and it works. Thanks for your instructions. But I got bad results as well.

wandb: Run history:
wandb:      AP-det ▁▅▆▅▃▄▅▅▅▆▇▃▂▃▅▅▂▇▇▅▇▇▃▅▆▂▄▅▄▇██▆▄▅▄▄▄▄▄
wandb:      AP-loc ▁▂▁▁▂▃▂█▁▁▁▁▂▁▁▁▁▁▁▁▂▂▁▂▁▁▁▁▁▂▂▁▁▁▂▂▁▁▂▂
wandb:       AUPRO ▁▇█▁▂▂▂▂▃▅▄▂▄▃▃▂▁▃▄▂▂▃▃▄▃▃▃▅▃▄▄▄▄▄▄▄▄▄▄▄
wandb:     I-AUROC ▁▃▄▅▅▂▄▃▄▄▅▇▁▂▄▄▅▂▇▄▆▆▁▃▃▃▄▂▃▅▄▆█▆▄▃▃▃▂▃
wandb:     P-AUROC ▃▃▃▅▄▄█▁▃▂▂▁▁▃▂▂▄▃▅▂▄▂▃▂▃▂▂▄▂▃▂▂▄▂▃▃▃▃▄▄
wandb:    avg_loss █▅▄▃▂▃▃▃▃▃▃▂▂▃▃▂▃▂▂▃▄▃▃▃▂▃▂▃▃▃▃▂▂▂▂▂▂▂▁▂
wandb:  batch_loss ▃▃▆█▆▂▂▆█▆█▆▂▁▄▁▆▅▁▂▆▆▂▇▆▂▂▁▆▁▆▇▁▂▇▅▅▅▅▇
wandb:  seg-AP-det ▅▆▄█▄▂▁▄▃▂▅▁▂▃▃▃▂▁▃▄▁▁▃▂▃▃▂▂▃▃▁▄▃▂▂▃▃▃▂▂
wandb: seg-I-AUROC ▄▃▄██▄▃▂▂▁▆▄▁▁▃▂▁▃▃▃▁▁▂▃▂▂▁▂▃▁▃▃▂▂▂▂▂▂▂▂
wandb: 
wandb: Run summary:
wandb:      AP-det 0.84422
wandb:      AP-loc 0.00231
wandb:       AUPRO 0.05912
wandb:     I-AUROC 0.39681
wandb:     P-AUROC 0.43062
wandb:    avg_loss 1.00132
wandb:  batch_loss 0.26962
wandb:  seg-AP-det 0.68958
wandb: seg-I-AUROC 0.13889

I set config["perlin_thr"] = 0.7, is that influences the performance? I found the visualizing result as follows:

The predicted masks have no difference between the good and anomalous samples.

[blaz-r]

I suggest you use the same setup as for mvtec for other unsupervised datasets. This means that you set perlin_thr to 0.2 initially. You could then try to increase it, but 0.2 is the most general option.

[Rudder573]

I followed your advice. It had a great improvement: I-AUROC: 0.8289532661437988 AP-det: 0.9622606635093689 seg-I-AUROC: 0.9451583623886108 seg-AP-det: 0.9780187606811523 P-AUROC: 0.7399502992630005 AP-loc: 0.005861310288310051 AUPRO: 0.4162159860134125. But visualizing results are still badly-behaved:




And good samples are not predicted as anomalous mask, which is correct.
Maybe my dataset is not suitable for anomaly detection?

[blaz-r]

Nice to see the improvement. The I-AUROC seems to be a bit low, though; related methods get 90+, and the segmentation results are not great (P-AUROC should be around 90 as well). If you are certain your data is correct, it is also possible that the model is not well-suited for this dataset. Looking at the seg I-AUROC (where the anomaly score is obtained as the max value of seg. map), it achieves 94.5, which is much better and in line with related work. I would try different learning epochs and maybe perlin_thr. Another possibility is not to use the anomaly score from the classification head, but rather the max of the anomaly map.

Looking at the visual results, it seems the top two and bottom two images are quite different (the two images below appear more zoomed in, which might confuse the model if the training set doesn't include this zoom level). If I'm not mistaken, this is the BSData dataset? It should be suitable for anomaly detection. However, I'm not familiar with the data splits, and I think the data should probably be divided into two categories: one containing images similar to the top two, and the other similar to the bottom two. Unless what you've shown here is a standard split. The reason is that most anomaly detection methods work best when the composition of seen images is not vastly different.

[Rudder573]

Thanks for your detailed explanation. It is true that the dataset is the BSData dataset. The normal samples only contain the top two images, and abnormal ones consist of both types. Maybe I delete the below types of images? Or set the zoom level to (512, 512) regarding to the orginal picture is (1130, 460) (1540, 645) (2724, 850)?

[blaz-r]

I inspected the dataset page more carefully, and while it's not entirely clear, I think the data is original made for supervised training. If you used the train_test split, the train set also contains defects, which means that you need to use the config for supervised training (either KSDD2 or Sensum) and not the mvtec option I said before. Additionally, looking at the data dict, it seems the train should contain both Type 1 and 2 images. If you did your own splits, then I'd need more details to help you.

In any case, you need to check carefully that the dataset splits are done correctly and that you correctly load all labels. Otherwise, the poor results could be due to data contamination.

[Rudder573]

Thank you for your continuous support and for patiently answering all my questions. Your help means a lot to me! Well, I checked the dataset again. I found the good samples only contain Type1, while anomalous samples contain Type1 & 2.
Type1_normal

Type1_abnormal

Type2_abnormal

When I deleted all type 2 pictures, I got nice results:

wandb: Run summary:
wandb:      AP-det 0.96888
wandb:      AP-loc 0.29414
wandb:      AUPRO 0.84696
wandb:      I-AUROC 0.86497
wandb:      P-AUROC 0.98591
wandb:      avg_loss 0.00037
wandb:      batch_loss 0
wandb:      seg-AP-det 0.98838
wandb:      seg-I-AUROC 0.94894

Visualizing results are as follows:

However, there're some unpredicted masks:


Well, you can see the anomaly map received the abnormal pixels indeed. Maybe I should increase learning epochs? Because the metric curves is unstable:

I know this dataset may not be suitable for unsupervised anomaly detection, but my spirit originated from this article: A Double Self-Supervised Model for Pitting Detection on Ball Screws. It is for my master thesis, and I already done the supervised method for accurate semantic segmentation. Furthermore, I want to solve the problem 'overdependence on data annotations' , so I choose this method. I really appreciate all the time and effort you've put into answering my questions so thoroughly. Your patience and kindness truly make this project even more inspiring. Thank you!

[blaz-r]

About the unpredicted masks. I'd mostly focus on anomaly map, since the predicted mask is just anomal map thresholded with 0.5. Since all the metrics are threshold agnostic, this 0.5 mioght not be best for visualisation, it was just arbitrarly chosen. From these results it looks like the model is struggling a bit, since you can see that seg-I-AUROC (where anomaly detection is based on max value in anomaly map) works better than the I-AUROC (anomaly score is from classification head).
This can happen for some datasets, so you could keep the anomaly score from classification head, or if you want better results use the max value of anomaly map (seg-I-AUROC) which is mroe in line with what they show in the paper you sent.

Metric curves can be unstable since the model is trained with random noise. I think that's fine for your run. What batch size are you using? Larger batch sizes usually lead to more stable trainign and usually quite better results.