Mainnet program deploy fails with 'Too many open files' — reclaim stranded SOL with solana program close

Symptom: anchor deploy (or solana program deploy) against a mainnet RPC stalls and then panics:

Blockhash expired. 4 retries remaining
⠂ 47.6% | Timed out resending 332 transactions...
thread 'main' panicked at quic-client/.../quic_client.rs:111:
QuicLazyInitializedEndpoint::create_endpoint bind_in_range:
Os { code: 24, ... message: "Too many open files" }

The same deploy works on devnet and localnet. Several SOL have already left the wallet.

Two separate things to handle.

1. Recovering the SOL. A partially completed deploy leaves the program bytes in a **buffer account** that holds the rent you paid (multiple SOL for a large program). It is not lost. Open your wallet's recent transactions in an explorer, look at the accounts involved in the latest deploy transaction, find the one whose balance is about the missing amount, and close it: solana program close <BUFFER_ADDRESS> --keypair <UPGRADE_AUTHORITY> returns the lamports. This works even when you no longer have the ephemeral buffer keypair, as long as the closing authority is your wallet. solana program show --buffers lists buffers owned by your authority, which is easier than explorer archaeology.

2. Getting the deploy through. Upgrading the Solana CLI to the latest patch release fixed it in practice (1.18.11 → 1.18.15 in the reported case), because deploy-path retry/QUIC handling changed between patch versions. Note the failure can be version-specific to mainnet's transaction volume while devnet still works. Also raise the file-descriptor limit (ulimit -n 10240) since the panic is literally fd exhaustion from many concurrent QUIC connections, and prefer a paid RPC plus --with-compute-unit-price for the deploy transactions.

Source: adapted from a Stack Exchange answer by Akshay Soam (question by Akshay Soam), https://solana.stackexchange.com/questions/14911/anchor-deployment-on-solana-mainnet-is-failing — CC BY-SA 4.0