Refusing to fork a process that holds pipes/sockets: snapshot the fds at library load, or the rule refuses every clone
Context: an in-process library forks a running host process so that N children continue from the same state. Pipes and sockets can't be duplicated meaningfully: a child would share the stream with the parent and interleave reads and writes. So the obvious rule is 'refuse to fork if any pipe or socket is open'.
That rule refused every fork. The host program already held a self-pipe (an event-loop wakeup pair) and a TCP connection to its controller before our library was loaded. Every real process tripped it.
What worked: 1. When the library loads, snapshot the open fds: walk /proc/self/fd and record st_dev/st_ino plus the type from S_ISFIFO/S_ISSOCK. Take the snapshot again after any reconnect that legitimately replaces those fds. 2. When a fork is requested, refuse only on a pipe or socket that is not in the snapshot. Name it in the error, e.g. 'fd 18 is a pipe (<path>)'. Those are the ones the user's code opened and may depend on. 3. In the child, replace the snapshot fds before running any user code. A self-pipe gets a fresh pipe2() pair dup2'd onto the same numbers. Other sockets get /dev/null dup2'd over them, so the child can never write into the parent's connection. 4. Report every replaced fd in the reply (a 'detached' list), so nothing is cut silently.
Also check for threads (/proc/self/task count > 1) and refuse. fork() copies only the calling thread, so a multi-threaded host forks into a broken child.
Test it with a fixture that opens a pipe after load (must refuse, naming the fd), plus one that holds only the pre-load fds (must succeed, with both fds listed as detached).
Follow-up: a load-time snapshot alone is still too permissive, because anything opened before your library loads (an earlier library's constructor, for example) gets detached silently. A tighter rule: classify the fds at load. The host's self-pipe is the single pipe with both ends in the process. Its own network link is the single AF_INET/AF_INET6 SOCK_STREAM socket (check with getsockopt SO_DOMAIN and SO_TYPE). Your own channel fds are the last group. If there are two or more candidates for a role, give none of them that role. Anything else refuses the fork, naming the fd, unless the caller opts in with something like allowDetach. Also re-classify after a checkpoint restore, because the restored image carries the saving process's fd list.
Caveat found later: a load-time snapshot still misses sockets the C library opens on its own. With sssd in nsswitch, a passwd/group lookup that misses sssd's mmap cache (e.g. an unknown user, or an expired entry in a long run) connects to /var/lib/sss/pipes/nss and keeps that socket open, so a 'refuse unknown sockets' rule fires intermittently. What worked: getsockopt(SO_PEERCRED) on each AF_UNIX stream socket. If the peer's uid differs from yours it is a system service (sssd, dbus); replace it with /dev/null in the child rather than refuse. Also name the family, type, peer pid/uid and getpeername path in every refusal, so the next hit identifies itself.