When the backup is missing and a script deletes: data loss from a faulty deduplication script

Illustration: faulty deduplication script deletes files; recovery via Git repo, Nextcloud mirror and backup

This experience report summarizes an incident that occurred while consolidating a large set of financial and tax documents. It is anonymized but follows the exact technical failure chain. The goal is to help operators of SMB shares, Nextcloud mirrors, and Git repositories avoid making the same mistakes.

Background

A sole proprietor in a research and freelance context wanted to migrate years of financial documents from a source share (Samba/NAS) into a new, year-based target structure (folders FY2015 … FY2026 and FY_undatiert). The source contained heavily redundant copies; a final deduplication ("no file twice") was planned.

What went wrong

Two mistakes combined into massive data loss:

No verified backup before starting. Only afterwards was an external 8-TB backup (from 16 Aug) mounted. Had it been checked beforehand, the following error would have hardly mattered.
A deduplication script with an unquoted path list. The list of directories to scan was passed unquoted to find. Because some paths contained spaces (e.g. share/Banks/Account 1234 5678/2024), the shell split those paths. This caused parent and child directories to appear multiple times as starting points — find listed the same file multiple times.

The dedup script compared consecutive SHA-256 hashes and deleted the second "identical" entry. Because the same file appeared as a "duplicate of itself" due to multiple listings, it was deleted — along with every further listing. Unique files were destroyed, not redundant copies.

flowchart TD A["Directory list SCOPE unquoted
(find $SCOPE)"] --> B["Shell splits paths
with spaces into fragments"] B --> C["Parent and child dirs appear
multiple times as start points"] C --> D["find lists the same file
N times (N > 1)"] D --> E["Dedup compares consecutive
hash values"] E --> F["h == prev on repeated
listing of the same file"] F --> G["rm the 'duplicates' =
deletion of unique files"] G --> H["~6,400 unique files
destroyed"] style G fill:#c0392b,color:#fff style H fill:#c0392b,color:#fff

The recovery

As soon as the error was noticed, the running operation was stopped immediately. There was no local Git repository as a source, but three additional copies existed on the server or externally:

Nextcloud sync mirror of the source share on the same server → returned ~5,300 files.
Surviving ZIP archives in the target structure → ~70 more files by simply unpacking.
External 8-TB backup (from 16 Aug) → ~600 more files.
Git origin (Gitea) for an affected sub-repository → the full history could be reconstructed via clone.

flowchart LR D["Deleted files
(~6,400)"] --> M["Nextcloud sync mirror
of the source"] D --> Z["Surviving ZIP archives"] D --> B["External 8-TB backup
(16 Aug)"] D --> G["Git origin (Gitea)
for sub-repo"] M -->|"~5,300"| R["Recovered"] Z -->|"~70"| R B -->|"~600"| R G -->|"History"| R R --> S["~98 % saved
(~6,300 of ~6,400)"] style S fill:#27ae60,color:#fff

In total, roughly 98 % of the deleted files were recovered. The remaining ~2 % were files that existed in none of the available copies (including a mailbox export and some documents created after the backup).

Why the GitCover concept helps here

The story shows: those who rely on Git-native structures get back to work quickly even after unintended errors — and simultaneously document in a GoBD-compliant way.

Git repos as a safety net. A commit makes unversioned working files part of the history. An unintended delete or overwrite (even by a script) can always be restored from the repo — you keep working instead of starting over.

GoBD, DSGVO & NIS2 defused. Legal obligations in particular — GoBD retention, DSGVO evidence, or NIS2 reports — can be documented cleanly with versioned, traceable artifacts. An incident becomes a logged, reproducible event rather than an unsolvable problem.

AI agents unleashed on data. Even when using agents that access data automatically, GitCover's basic concepts help: clear provenance (SHA-256), immutable history, and independent copies (mirror, backup, origin) limit damage and make every action auditable.

Lessons Learned

Verify the backup before, not after. Ideally 3-2-1: 3 copies, 2 media, 1 offline. A backup mounted only after the damage does not help.
Always process paths null-delimited and quoted. find … -print0 | xargs -0 instead of unquoted find $LIST. Spaces in paths are the classic breaker.
Dedup must never delete on repeated listing. Correct: keep exactly one copy per distinct path + content; if the same file is listed multiple times, nothing may be deleted (decide by inode or hash+path).
Keep several independent recovery sources. Mirrors, archives, and version control complement each other; no single one suffices.
Protect unversioned working-tree files with a commit. If they sit only as untracked in the repo, a single git clean destroys them. A commit makes them part of history.

Conclusion: A single unquoted variable reference is enough to destroy thousands of unique files. Recovery succeeded only because several independent copies existed. Structure, process, and discipline are not luxuries — they are the only protection there is.

Part of the Experiences rubric. More posts to follow.