Count calls into a specific shared library's functions across forked Node workers with gdb Python (no root, no uprobes)
Goal: prove whether code inside a vendor .so (e.g. its bundled zstd) is ever executed while a Node app that loads it reads real files. The app forks worker processes. There is no root, so no uprobes or perf.
What worked (gdb 16, RHEL 9):
- Run the app as gdb -q -batch -nx -x count.py --args node app.js. In count.py set detach-on-fork off, follow-fork-mode parent, schedule-multiple on, breakpoint pending on, and handle SIGPIPE/SIGCHLD/SIGUSR1/SIGTERM nostop noprint pass.
- Subclass gdb.Breakpoint with stop() returning False. Count by (gdb.solib_name(pc), frame.name()) and keep a short caller backtrace for the first hit. Create one breakpoint per function name; pending names resolve in every inferior, including exec'd forks.
- Run loop: after run, continue while any inferior has a pid. Before each continue, switch to a live inferior if the selected one has exited. A short-lived child such as getconf otherwise ends the script with "The program is not being run", and gdb batch then kills everything.
- Dump the counts to JSON every N hits and on gdb.events.exited, so a kill still leaves data.
- If the app writes its protocol on fd 3, spawn gdb with stdio [pipe, logfile, inherit, pipe]; the inferior inherits fd 3.
Gotchas:
- Add a positive control: a function you know runs, e.g. the library's zlib entry point. Otherwise a zero count proves nothing.
- Hot controls (inflate internals) at about 100k hits make gdb the bottleneck. Keep one entry point per codec.
- timeout around a wrapper that uses setsid does not kill the setsid child. Kill by recorded process group.
- Symbol tables: a vendor .so that is not stripped exposes its static functions to nm/gdb, so internal copies can be covered too.