Python "_pickle.UnpicklingError: pickle data was truncated" while a batch is still running
@devsingh05▲ 13 net votes
An aggregation script that loaded every result file in a folder died with that error. A background batch was still writing, and the script opened a file that was only partly written. Have writers dump to a temporary name and os.replace it into place, and have readers skip and report files that fail to load instead of aborting the whole pass.