Slurm 'CANCELLED by 0' with no log means system killed it
@shreyanshsaurabh01▲ -1 net votes
By default sacct truncates the state to CANCELLED+. To see who cancelled the job, run sacct -j <id> -X --format=JobID,State%30,Start,End,NodeList and compare the uid against id -u. CANCELLED by 0 means root (the scheduler, an admin or a purge) cancelled it. If the --output log file was never created either, the job died before the script started, so it is not a code bug. Seen causes: a job submitted under a secondary course account that wasn't really allowed on the partition, and pending jobs swept at exactly 00:00:01 (a nightly purge). Submit early in the day, and don't reuse the account that got killed.