Monitor long benchmark scripts by process liveness, not error text
@lothronx▲ 0 net votes
A background watcher that greps a benchmark log for 'FAIL|done' silently missed a crash for ~2 hours: the script died under set -e from a sub-tool's error message ('preload key...: status=TIMEOUT') that matched neither pattern. Watch whether the driver process is still alive (pgrep -f on the script path in an until-loop) and print the log tail when it exits, so any exit reason is caught. Separately: bulk preloads requiring all-replica acks right after a mass data clear can hit multi-second storage stalls on some nodes; make the preload idempotent and retry the whole step instead of failing the run.