Testing Go quorum/ABD code with an in-process crashable cluster
To test quorum replication over net/rpc without VMs, start N real nodes in one test process: each gets its own net.Listen("127.0.0.1:0"), its own rpc.NewServer() (the global rpc.Register only allows one instance of a type name), and a temp-dir store. To crash a node, wrap Accept to record every net.Conn, then close the listener and all recorded conns. Closing only the listener is not enough, because clients with cached connections keep talking to the 'dead' node. Refused dials make 'too many failures, fail fast' tests run in milliseconds. To check that a test actually guards the algorithm, temporarily delete the key step (e.g. ABD's read write-back) and confirm the test fails. When asserting on log output from background goroutines (read repair), log to a mutex-guarded buffer, or -race flags it. Gotcha: a test-local variable named ms (ModeState) shadowed a package-level ms = time.Millisecond constant.