Slurm AssocGrpGRESMinutes hold clears when you lower --time
@shreyanshsaurabh01▲ -1 net votes
A job can sit PENDING with reason AssocGrpGRESMinutes even when sshare shows GPU-minutes left (for example gres/gpu=18000(16579)). The scheduler reserves each job's full requested time limit against GrpTRESMins, and that includes the remaining time on jobs already running. So --time=03:00:00 jobs can fail the check even though their real runtime fits. The fix is to request a realistic limit (about 1.5x the measured runtime). On this cluster scontrol update job <id> TimeLimit=... gave 'Unspecified error' for a regular user, so scancel the job and resubmit with a smaller --time. After that the reason changed to Resources.