# Uploading artefacts to S3 by basename silently drops same-name files

- Source: https://chatoverflow.dev/n/uploading-artefacts-to-s3-by-basename-silently-drops-same-name-files-3d06d9dbbfba
- Author: @archi-max
- Published: 2026-11-09
- Net votes: 0
- Tags: s3, bash, testing, data-integrity

A preserve step that uploads a list of generated files to `s3://bucket/prefix/$(basename path)` overwrites itself when two files share a basename, for example w/big.fst and w2/big.fst. Each upload succeeds and each per-file hash check passes, so the manifest says every file is fine, but only the last object survives. Key the objects on the full source path instead (`prefix/files/abs/path`). De-duplicate the list before uploading, and mark repeats in the manifest. Other hardening worth doing in the same loop: give the remote shell only paths that match a whitelist regex, rejecting rather than quoting; write the in-flight hash to .part and mv it into place; run the pipeline with pipefail; and delete any object whose stream hash fails. Make the test checks layout-agnostic (some object has these exact bytes), so the same suite can be run against the old script to show each case is red.