CUDA: read ptxas stack frame and local-memory sector use before tuning a slow kernel
A slow kernel used one thread per time step, and each thread rebuilt the whole model's state, then evaluated every operation. Before reaching for a profiler, nvcc -Xptxas -v already showed the problem: a 113,744-byte stack frame per thread with 0 spill stores. The state was a per-thread local array, and local memory lives in DRAM.
Nsight Compute confirmed it: - DRAM throughput was ~88% of peak, so the kernel looked 'bandwidth-bound'. - SM throughput was ~8.6%. - smsp__sass_average_data_bytes_per_sector_mem_local_op_ld.ratio was ~0.98 bytes per 32-byte sector (stores ~0.86). So ~97% of the moved bytes were wasted sector traffic.
High DRAM % with tiny local bytes-per-sector means you are paying bandwidth for state, not data. A busy GPU doesn't make redundant work free: when the kernel is throughput-bound, wall time is total traffic divided by bandwidth.
The fix is structural, not tuning. Each output depended on only a handful of operations (median 1, p99 ~20), so restructure as one warp per output cone with the lanes over time points. State then lives in registers, all lanes run the same instructions, and change detection uses __shfl_up_sync plus __ballot_sync/popc compaction.
Two other cheap finds in the same nsys run: - cudaFree was called ~2,400 times at ~2.8 ms each, costing ~6.7 s of host time. Allocate once and reuse buffers. - Launch count (~3,000) was negligible, so don't optimise launches first.