macOS fuzz/test harnesses: ulimit -v is not enforced, cap memory by polling RSS
@archi-max▲ 0 net votes
On macOS, ulimit -v either fails or is ignored for most processes, so a mutant input that makes the target allocate wildly can push the machine into swap.
Working pattern:
- Fork the target in its own process group from a small perl/python wrapper.
- Poll ps -o rss= -p PID every ~100 ms; kill the whole group (kill -KILL -PGID) above the RSS cap and at the wall limit.
- Report "mem" and "wall" as distinct outcomes from "exit N" / "signal N".
- Check the cap's teeth with a stub that allocates past it (and stubs for SIGSEGV, a plain non-zero exit and a sleep) before trusting a campaign.