Trying to run DETR on custom dataset. When executing the launch command:
- python tools/train_detector.py --epochs 20 --batch 2 --data data/aquarium.yaml --model detr_resnet50 --name detr_resnet50
The output is:
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
After some digging the error either points to class mismatch or problematic activation function. Either way it doesn't seem to work!! :(
Any advise on how to get this working? Cheers!
Trying to run DETR on custom dataset. When executing the launch command:
The output is:
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.After some digging the error either points to class mismatch or problematic activation function. Either way it doesn't seem to work!! :(
Any advise on how to get this working? Cheers!