hyperindex

A hyperfast, resumable file indexer, checksum verifier and duplicate-file finder for Windows, written in Rust. It walks one or more directories (or every currently plugged-in drive), records size / timestamps / a BLAKE3 content checksum for every file into a persistent SQLite database, and can then report duplicate content across everything it has ever indexed — including across separate USB sticks plugged in on different days.

The key design goal is reuse: plug the same USB stick back in a week later and rescan it — files whose size and modification time haven't changed are recognised instantly and their stored checksum is reused without touching the file's contents again. Only new or changed files get (re-)hashed.

Why

Standard drive/file organisers either don't survive being unplugged (no persistent index) or re-hash everything on every run (slow on large collections). hyperindex is built specifically to answer: "across all these drives I've plugged in over time, what's actually duplicated, and where?" — and to hand that answer to an AI agent (or a person) as an actionable report, not just a wall of find | md5sum output.

Portability

target\release\hyperindex.exe is a genuinely standalone, single-file binary: copy just that one .exe anywhere — another folder, a USB stick, a different Windows machine — and run it. No installer, no companion DLLs, no SQLite runtime to provide (it's compiled in via rusqlite's bundled feature) and, thanks to .cargo/config.toml setting target-feature=+crt-static, no dependency on the Visual C++ runtime redistributable either. Verified with dumpbin /dependents: the only imports left are core Windows OS components present on every Windows 10/11 install (kernel32.dll, ntdll.dll, combase.dll, shell32.dll, bcryptprimitives.dll) — nothing that needs installing.

The .pdb file next to it (debug symbols) is optional — useful for attaching a debugger to a crash, never required to run the program; it's safe to leave behind when copying the .exe elsewhere.

Requirements

• Windows 10/11 (uses the Win32 GetVolumeInformationW API for stable volume identification; the rest of the code is otherwise portable Rust).
• Rust (stable toolchain, edition 2024 — tested with rustc 1.98) via rustup.
• The MSVC C++ build tools (Visual Studio "Desktop development with C++" workload, or the standalone Build Tools) — needed for the linker that Rust's default x86_64-pc-windows-msvc target uses.

Check what you have:

rustc --version
cargo --version

If rustc/cargo are missing: winget install --id Rustlang.Rustup -e, then open a new shell.

Building

cd hyperindex
cargo build --release

The optimized binary lands at target\release\hyperindex.exe. The release profile is tuned for throughput (opt-level = 3, LTO, single codegen unit) since hashing large collections is CPU/IO-bound work where this matters.

Run the test suite (unit tests for the trickiest bits — byte formatting, SQL LIKE-pattern escaping, the duplicate "which copy to keep" heuristic, volume-serial formatting):

cargo test

Quick start

# Index a single drive or folder
.\target\release\hyperindex.exe scan E:\

# Index everything currently plugged in (skips network drives by default)
.\target\release\hyperindex.exe scan --all-drives

# Find duplicates across everything indexed so far, write a handoff report
.\target\release\hyperindex.exe dupes --out report.md

# Folder-level view: which folders contain duplicates, what is safe to remove
.\target\release\hyperindex.exe doubles --out doubles.md

# Continue a scan that was interrupted (nothing to retype)
hyperindex scan --resume

# Sort photos: resolve dates, plan where each one goes, then do it
.\target\release\hyperindex.exe enrich
.\target\release\hyperindex.exe plan --out plan.md
.\target\release\hyperindex.exe apply --plan plan.json --library-root E:\library

# What has each planned file's copy actually done so far?
.\target\release\hyperindex.exe actions

# What has been indexed so far?
.\target\release\hyperindex.exe volumes
.\target\release\hyperindex.exe stats

Re-run scan on a drive you've already indexed and unchanged files are served from the database instead of being re-read:

==> Scanning E:\
    found 128,402 files (411.20 GB) — diffing against existing index...
    done: 812 hashed, 127,590 reused from cache, 0 now missing, 0 inaccessible

Getting help

Every command carries worked examples, not just a flag list:

hyperindex --help            overview and the usual sequence
hyperindex scan --help       everything scan accepts, with examples
hyperindex dupes --help      ... and so on for every command

Windows-style switches work as well as POSIX ones: /?, /h and /help are accepted alongside -h and --help, and /v alongside --version. Only exact matches are translated, so a path or glob that happens to start with a slash is passed through untouched.

Commands

hyperindex scan <paths...> [options]

Walks the given path(s), records metadata for every file, and hashes any file that is new or has changed since the last scan (different size or modification time). Everything is written to the index database.

The same read that computes a file's checksum also records what the file is, judged from its bytes rather than its name: a MIME type from its magic number (image/jpeg, application/pdf, … or text/plain for text in any 8-bit encoding), the EXIF capture time for images, and whether it is plain text. What is looked at is the first 128 KiB — plus, for HEIF/HEIC/AVIF, whose metadata phones usually write at the very end, the last 64 KiB. No extra read for anything being hashed. A file that is unchanged since an index built by 0.100.0 or earlier gets that much read once on its next scan so its row catches up — never the whole file, and never a cloud placeholder. See files under Database schema.

Flag	Meaning
--all-drives	Scan every currently mounted local drive instead of (or alongside) explicit paths.
--include-network	Also scan network-mounted drives when using --all-drives (off by default — different perf/latency profile).
--follow-symlinks	Follow symlinks/junctions while walking (off by default, to avoid infinite loops on self-referential junctions).
--rehash-all	Recompute checksums for every file even if metadata is unchanged — use if you suspect silent bit rot rather than legitimate edits.
--exclude <NAME>	Additional directory base name to skip (repeatable, case-insensitive). Windows bookkeeping directories ($RECYCLE.BIN, System Volume Information, Config.Msi, $WinREAgent, $Windows.~BT, Recovery, OneDriveTemp, …) are always excluded, as are permanently locked system files (pagefile.sys, hiberfil.sys, NTUSER.DAT*, …).
--exclude-path <GLOB>	Skip anything whose full path matches a glob (repeatable, case-insensitive, either slash works), e.g. --exclude-path "**/AppData/Local/Temp/**".
--cloud-files <MODE>	skip (default) records online-only files by metadata but never opens them; hydrate downloads and checksums them. See Cloud-synced folders.
--retries <N>	Extra attempts for a transient failure — a momentary lock, a network blip (default 1). Permission and cloud errors are never retried.
--io-timeout <SECS>	Give up on a single file whose read has not finished in this long (0 = wait forever, the default).
--stall-warn <SECS>	Name any file still being read after this long, so a stuck scan says what it is stuck on (default 60, 0 disables).
--commit-batch <N>	File results per SQLite transaction (default 2000). Smaller means an interrupted scan loses less.
--flush-interval <SECS>	Commit whatever has accumulated at least this often, however few files it is, and fold the write-ahead log into the main .db afterwards (default 300, 0 disables).
--resume	Continue the most recent interrupted scan. Paths and options are read back from the database; files the interrupted run finished are not touched again.
--threads <N>	Hashing worker threads (default: number of logical CPUs).
--min-hash-size <BYTES>	Skip content-hashing (but still record metadata for) files at or below this size.
-q, --quiet	Suppress the status panel and per-file chatter, print only the final summary.

Watching a scan

A scan of several drives is a long, mostly silent operation, so it draws a status panel that is redrawn in place while log lines scroll past above it:

╔═ hyperindex 0.95.0 · scan ════════════════════════════ root 2/5 · 00:41:02 ═╗
║ Volume  F:\  "SANDISK ULTRA"  exFAT  SN 1A2B-3C4D                           ║
║ Root    F:\Fotos                                                            ║
║ Stage   hashing · 16 threads                                                ║
║ Now     …\2019\Sommer\IMG_2381.jpg                        4.20 MB  12s      ║
╟─────────────────────────────────────────────────────────────────────────────╢
║ Root  [██████████████░░░░░░░░░░░░░░░░]   46%  12,345/22,800 files           ║
║       41.20 GB of 88.00 GB · 128.00 MB/s · eta 00:10:44                     ║
║ Job   [█████████░░░░░░░░░░░░░░░░░░░░░]   29%  1 of 5 roots done             ║
║       elapsed 00:41:02 · eta ~01:55:00                                      ║
╟─────────────────────────────────────────────────────────────────────────────╢
║ hashed 8,901 · reused 3,100 · small 210 · cloud 12 · err 4 · db 12,000      ║
╚═════════════════════════════════════════════════════════════════════════════╝

Line by line:

• Volume / Root — which drive and which path is being worked on right now. The volume is named the way you would identify it yourself: drive letter, label, file system, and the serial number the index is actually keyed on, so two identically-labelled sticks are still distinguishable.
• Stage — walking the tree, diffing against the index, hashing, or finishing up. Worth its own line because the first two stages have no percentage to give and can each take minutes on a large volume; without it, the opening of a scan is indistinguishable from a hang.
• Now — the directory the walk is in, or, while hashing, the read that has been running longest, with its age once that exceeds a few seconds. This is the line that tells you a single 40 GB file is what the bar is waiting for.
• Root — progress through the current root only. Precise: the walk has already counted every file and byte underneath it, so the percentage and the ETA are real. Progress is measured in bytes, since that is what hashing time is made of; the throughput figure is a smoothed average of bytes actually read, so a reused checksum does not inflate it.
• Job — progress across all roots of this invocation, and only shown when there is more than one. Deliberately coarse: nothing has counted the roots that have not been walked yet, so this is (finished roots + fraction of the current one) / total roots and its ETA carries a ~. A 64 GB stick and a 4 TB archive both count as "one root".
• Counters — outcomes so far for this root (small = below --min-hash-size, cloud = online-only placeholder, err = unreadable), and db = results committed to the database, which is how much of the run would survive a Ctrl-C.

So: there is no single bar spanning every block of every volume. Each root gets its own precise bar, and the job bar counts roots. hyperindex scan C:\ D:\ F:\Photos scans three roots in sequence, one bar at a time, with the job bar underneath them.

dupes and doubles draw a smaller panel of the same kind — a checklist of their few long steps, so a grouping query that runs for minutes says which step it is on rather than nothing at all.

The frames need a UTF-8 console; hyperindex switches the console's output code page over for the duration of the run and switches it back on exit (which also makes umlauts in paths render correctly). Pass --ascii to leave the console alone and draw with +---+ instead, --progress bar for the single-line progress bar of 0.93 and earlier, or --progress off for nothing but log lines. Redirected output never draws a panel regardless — a display that redraws in place is meaningless in a log file — so piping to a file gives you the log lines and nothing else.

Redrawing in place also needs the console to defer its line wrap — to leave the cursor on a line that fills the window exactly, rather than jumping to the next row — because that is what walking the cursor back up over the last frame depends on. hyperindex turns that mode on at startup and off again on exit; on a console too old to offer it, the live display switches itself off with a note instead of scrolling a fresh copy of the panel past you eight times a second. The panel also steps down to the one-line bar when the terminal window is too short to hold it, so --progress bar is only needed if you want the short form in a window that would have fit the panel.

hyperindex dupes [options]

Finds every group of currently-present files that share both size and BLAKE3 checksum (i.e. confirmed identical content — not just "probably the same"), across every volume ever indexed, and writes:

• a Markdown handoff report (--out, default hyperindex-report.md) — written to be read and acted on by an autonomous agent or a human; see The handoff report below.
• a JSON twin (--json, defaults to the same name with a .json extension) with the identical data in a shape trivial to parse programmatically.

Flag	Meaning
--min-size <BYTES>	Ignore files smaller than this (default 4096 — tiny files produce a lot of low-value "duplicate" noise, e.g. empty marker files). Pass 0 for no filter.
-o, --out <FILE>	Markdown report path.
--json <FILE>	JSON report path.

hyperindex doubles [options]

The folder-level view of the same data. Where dupes answers "which files are duplicated", doubles answers "which folders do I need to look at" — the unit cleanup actually happens in. Output is doubles.md plus a JSON twin, intended as input for a step-2 cleanup tool or for working through by hand.

For every folder holding duplicated content it reports the path, volume, number and combined size of its duplicate files, how many of those are redundant copies, the reclaimable space, the modification-date span, and — crucially — how many files in that folder, or anywhere beneath it, exist nowhere else.

That last number is the safety-critical one, and it deliberately covers the whole subtree: the claim the report makes is that removing the folder is safe, and a folder holding nothing unique itself can still have a subfolder that does. It splits the output into two clearly separated sections:

• Folders that can be removed whole — every file in them is a redundant copy and the index confirms nothing unique inside.
• Folders to clean file-by-file (do NOT delete the folder) — all their duplicates are redundant, but they also hold one-of-a-kind files that wholesale deletion would destroy.

A duplicate report on its own can never make that distinction (a file with no duplicate never appears in one), so doubles consults the index database for it. When the count can't be determined, the folder is shown with ? and is never listed as safe for whole-folder removal — an unknown is not treated as a "no".

A "Folder pairings" section lists which other folders hold the counterpart copies, which is what reveals "folder A is essentially a copy of folder B".

Flag	Meaning
-f, --from-report <FILE>	Reuse an existing dupes report instead of recomputing. Accepts the Markdown report or its JSON twin; given the Markdown, the JSON twin next to it is used automatically because it holds exact values. If a report has been hand-annotated, the Markdown parser tolerates the extra lines.
--fresh	Never read a report — always recompute from the index database.
-o, --out <FILE>	Folder-level Markdown report (default doubles.md).
--json <FILE>	JSON twin (defaults to --out with a .json extension).
--min-size <BYTES>	Ignore files below this size when computing fresh (default 4096). No effect when reading a report — that report's own filter already applied.

Source resolution when --from-report is omitted and --fresh is not given: hyperindex looks for a default-named report (hyperindex-report.json, then .md) in the working directory and uses it if found; otherwise it computes from the database. So the common flow just works:

hyperindex dupes            # writes hyperindex-report.md/.json
hyperindex doubles          # picks that report up automatically

hyperindex enrich [options]

First stage of the (still early) photo-sorting pipeline: resolves a best-guess capture date for every indexed photo, RAW file and its companions (XMP sidecars, Apple's .aae, live-photo .mov), by reading each file's own content rather than just its directory entry. Tries, in order: EXIF DateTimeOriginal → a dated file name (IMG_20230615..., Screenshot_20230615..., WhatsApp's IMG-20230615-WA...) → a dated containing folder → the file's own modification time as a low-confidence last resort.

Also reads GPS EXIF when present and resolves it to a place name — offline, via a bundled city database (reverse_geocoder/GeoNames): no network call, so a photo's location is never sent anywhere. Most files have no GPS tag and simply get no place. Results (date and place) are cached in the database, so a re-run after every scan only costs something for files it hasn't seen before.

Flag	Meaning
--force	Re-derive dates/places for every matching file, not just new ones — e.g. after an upgrade changes how they're resolved.

hyperindex plan [options]

Second stage: builds a dry-run manifest of where every indexed photo would move to — library/photos/<year>/<place?>/<date>_<name>, quarantine/ for anything without a confident date, or neither if it's an exact-content duplicate of a file already in the plan. The place folder (only present when GPS was resolved) is a deliberate, explicit choice — made 2026-08-29 — to fold a location level into the tree despite this project's own earlier warning against folder-encoded location; see obsidian/dev/data-consolidation/HANDOFF.md. Nothing is moved — apply is the step that performs the copies, verifies them by hash and journals them for --undo. Run hyperindex enrich first — files it hasn't reached fall back to modification-time-only and are routed to quarantine rather than guessed at.

Flag	Meaning
-o, --out <FILE>	Markdown manifest (default plan.md).
--json <FILE>	JSON twin (defaults to --out with a .json extension).

Each run also records its decisions in the actions table under a fresh plan_run id, which is printed when the run finishes and written into plan.json. The manifest is still recomputed from scratch every time — that has not changed; what is persisted is the intent, so apply has one auditable row per file to advance rather than a lifecycle re-derived from two reports and a journal. Re-planning under the same id updates rows that are still planned and leaves anything already copied and verified alone.

hyperindex actions [options]

Read-only audit view of the actions table. With no arguments it lists every plan run and how its actions currently stand; with --run <PLAN_RUN> it lists that run's individual actions.

Flag	Meaning
--run <PLAN_RUN>	List one run's actions instead of the per-run summary.
--status <STATE>	Only show actions in this state. Requires --run.
--limit <N>	Cap the listing (default 50; 0 means no limit).

There is deliberately no way to change a row from here. The stages that do the work advance the lifecycle; a human typing a status they would like to be true does not.

duplicate and skip-protected rows stay planned forever, because they are decisions not to place a file — there is nothing for apply to do with them. The listing says so rather than leaving them looking like unfinished work.

hyperindex prefilter [options]

The music equivalent of dupes, purpose-built as input to beet import rather than for human review. Not an enrich/plan pair — the 2026-08-29 decision is that beets already solves tag reconciliation and canonical library layout far better than a bespoke Rust rule set could, so Hyperindex's whole job for music is collapsing exact-content duplicates (the same track sitting on several backup drives) to one canonical copy each, before beet import's own per-file MusicBrainz lookup has to run at all. Canonical-copy choice reuses the same "oldest modification time, shortest path" heuristic as dupes' own keep pick — nothing about it is music-specific. Never runs beet import itself, and never moves or deletes anything.

Writes three files: a Markdown handoff report, its JSON twin (same pattern as every other hyperindex report), and a plain keep-list (.txt, one path per line, no quoting or shell syntax) — the keep-list is the one actually meant to be fed to beet import.

Flag	Meaning
--min-size <BYTES>	Ignore files smaller than this when comparing for duplication (default 4096, same as dupes). Unlike dupes, a file below this never gets dropped — it always comes back "keep", just never compared.
-o, --out <FILE>	Markdown report (default beets-import.md).
--json <FILE>	JSON twin (defaults to --out with a .json extension).
--keep-list <FILE>	Plain keep-list (defaults to --out with a .txt extension).

hyperindex apply [options]

Reads a plan.json (not the .md) and actually performs the moves — the only command that writes anywhere but the index database.

Copies, never moves. Source backup drives are never modified, only read; deleting the now-redundant source afterward is a deliberate, separate human decision. Every file is verified by content hash after the copy — the source's freshly-computed hash, the (optional) hash plan recorded, and the written target's own hash all have to agree, or the file is reported failed and left for a human to look at rather than guessed at. A target that already exists and already matches is left alone (already-done) rather than re-copied, which is what makes an interrupted run safe to simply re-run. A target that exists with different content is never overwritten.

Every attempted file is journaled to the database before the copy starts, under a job id printed at the end of the run (and in the JSON report) — pass that id to --undo to reverse the run later: it deletes exactly the files that run itself placed (never the source, and never a file some other run or a human put at the same path).

Immich note: if --library-root is (or will be) an Immich external library, get placement right the first time. Immich loses an asset's favourites/albums/edits on its next rescan if the underlying file is later moved or renamed — apply never does that on its own, but re-running plan/apply with changed rules over files Immich has already indexed would have the same effect, so don't.

Flag	Meaning
--plan <FILE>	The plan.json to apply. Required unless --undo is given.
--library-root <DIR>	Where the sorted library lives — plan's target paths (e.g. library/photos/2023/...) are resolved relative to this. Required unless --undo is given.
--dry-run	Show what would be copied where, without writing anything.
--undo <JOB_ID>	Reverse a previous run instead of applying a plan. Lists what would be deleted unless --force is also given.
--force	Actually delete when using --undo.
-o, --out <FILE>	Markdown apply report (default apply-report.md).
--json <FILE>	JSON twin (defaults to --out with a .json extension).

Not yet built: a broader verify pass (re-index the target root, confirm every source hash is present at least once, as a safety net before anyone considers deleting sources) — apply's own per-file hash check covers correctness of each individual copy, but not that whole-library assertion.

hyperindex volumes

Lists every volume the database has ever seen (serial number, label, file system, when it was last scanned, current file count and total size).

hyperindex stats

Overall database statistics plus a table of the most recent scan runs (files seen/hashed/reused/errored, bytes hashed, per run).

Global options

Flag	Meaning
--db <PATH>	Which database file to use. Default %LOCALAPPDATA%\hyperindex\index.db — a stable per-user location, so running scan today and dupes next week without passing --db still talks to the same database.
--db-cache <MB>	How much page cache to allow the database. Defaults to a quarter of this machine's memory, clamped to 64–1024 MiB; 0 leaves SQLite's own (2 MiB) default alone. Raising it is the cheapest speed-up available on a large index — see Performance notes.
--progress <MODE>	panel (default) draws the framed status window; bar draws the single progress line of 0.93 and earlier; off prints only the log lines.
--ascii	Draw the display with +---+ instead of box-drawing characters, and leave the console code page alone. The line-wrap mode is still set, since any display has to repaint over itself.

Interrupting and resuming a scan

Results are written to the database as the scan runs, not at the end. Each batch of --commit-batch files is committed, and --flush-interval guarantees a commit at least every so often however large the files are — so an interruption costs at most the last few minutes, never the whole run.

Why the .db file looks frozen

The index runs in SQLite's WAL mode: commits land in index.db-wal and the main index.db is only rewritten at a checkpoint. Watching the .db timestamp during a scan is therefore misleading — it can sit unchanged while tens of thousands of files are being committed next to it. --flush-interval checkpoints on its own cadence for exactly this reason, so the main file visibly advances, and the log is folded back in when the scan ends.

Continuing

$ hyperindex stats
   ID STARTED                  ROOT               SEEN   HASHED   REUSED   ERRORS  HASHED SZ       STATUS
    1 2026-08-24 12:20:35.880  C:\Users\mart…        0        0        0        0        0 B  interrupted

Interrupted scans — continue the newest with `hyperindex scan --resume`:
  #1 C:\Users\marti\Nextcloud9  started 2026-08-24 12:20:35.880
        hyperindex scan C:/Users/marti/Nextcloud9 C:/Users/marti/Nextcloud7 ... --min-hash-size 100

$ hyperindex scan --resume
Resuming scan #1 started 2026-08-24 12:20:35.880
  original command: hyperindex scan C:/Users/marti/Nextcloud9 ... --min-hash-size 100
==> Scanning C:\Users\marti\Nextcloud9
    found 60950 files (122.34 GB) — diffing against existing index...
    10000 of them were already done by the interrupted run; 50950 left

Every invocation records what it was asked to do — the command as typed, the full root list, and every resolved option — on each scans row it creates. That is what lets --resume reconstruct a run without the user remembering the flags they used three hours ago.

It also means a resume finishes roots the original run never reached. A --all-drives invocation killed on the third of five drives continues the third and then scans the fourth and fifth, which no per-root record on its own could know about.

The interrupted scans row is re-opened rather than duplicated, so scan history stays one row per root per invocation; stats marks a scan that was continued with complete*.

If you would rather not use --resume

Just run the original command again. Files already hashed have unchanged size and modification times, so they are served from the index and never re-read — only the remainder is hashed. --resume is faster still (it skips even the diff for finished files) and saves you retyping, but nothing is lost either way.

Surviving a live Windows desktop

A scanner pointed at real user directories meets files that cannot simply be opened and read. The failure modes that matter are not the ones that return an error — those are easy — but the ones that block, because a blocked worker looks exactly like a hung program.

Cloud-synced folders (OneDrive / Nextcloud / Dropbox)

With "virtual files" / "online-only" mode enabled, a synced folder is full of files that look completely normal to stat but whose content lives on a server. Opening one hands control to the sync client, which downloads it — or, if that client is offline, stalled or unauthenticated, puts up its own modal dialog and leaves the read blocked until somebody clicks it away. On a multi-threaded scan the effect is a scan that advances only when you dismiss dialogs, and a scan of a folder you thought was local quietly pulling gigabytes over the network.

hyperindex recognises such files from their file attributes (FILE_ATTRIBUTE_OFFLINE, FILE_ATTRIBUTE_RECALL_ON_OPEN, FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS) before touching them, and by default records their metadata without ever opening them. They are reported separately as online-only rather than as errors, since nothing is wrong with them — they simply are not here:

    found 5090 files (18.49 GB) — diffing against existing index...
    5090 of them are online-only (cloud placeholders); their content will not
    be downloaded. Pass --cloud-files hydrate to checksum them anyway.

Files indexed this way carry metadata but no checksum, so they cannot take part in duplicate detection. Three ways to include them:

1. Tell the sync client to keep the folder on this device ("Always keep on this device" / "Make available locally"), then rescan — this is the fast path, because the files are then simply local.
2. Run hyperindex scan --cloud-files hydrate <path>, which downloads each one on demand. Complete, but slow, bandwidth-hungry, and it needs the local disk space — and it will block on the sync client if that client is unhealthy.
3. Leave them out. For a duplicate hunt across local disks and USB sticks, the cloud copies are usually not what you are looking for anyway.

As a second line of defence, files in such a tree are opened with FILE_FLAG_OPEN_NO_RECALL, so even a file that became a placeholder between the walk and the hash fails cleanly instead of triggering a download.

Windows hard-error dialogs

An empty card-reader slot, a disconnected network path or a failing sector makes Windows display a message box on the process's behalf and block the offending call until it is dismissed. hyperindex disables this at start-up (SetErrorMode, re-asserted per hashing thread), so those conditions come back as ordinary errors.

Errors are grouped by cause, not listed by file

One protected folder or one unhealthy sync client produces thousands of identical error lines, which communicate nothing. Every failure is classified and reported per cause, with the fix:

Unreadable files by cause:
       2314 × cloud-placeholder  online-only files whose content lives on a
                                 server. Bring the sync client online and
                                 re-run, mark the folder as
                                 always-keep-on-this-device, or pass
                                 --cloud-files hydrate ...
         17 × permission-denied  no read permission. Re-run the scan elevated,
                                 or exclude the folder with --exclude ...

The same grouping appears in the dupes report, with the per-file detail folded into a collapsible block and the complete list in the JSON twin. Errors are also de-duplicated across scans and drop out entirely once the file has been hashed successfully, so the list describes the present rather than the whole history.

Other hazards handled

• Locked files — opened with a permissive share mode, so a file another process is actively writing can still be read. Genuinely transient failures (sharing violations, network blips) are retried; permission and cloud errors are not, because they would fail identically every time.
• Long paths — anything approaching MAX_PATH is opened through its extended (\\?\) form, so a 300-character path is indexed rather than reported as an error.
• Junctions and mount points — directory reparse points are not descended into unless --follow-symlinks is given, which is what keeps a scan from re-walking a tree, wandering onto another volume, or looping forever on a self-referential junction such as the legacy Application Data.
• Stalls — a watchdog names any file that has been open for longer than --stall-warn seconds. It cannot cancel the read (Windows offers no way to do that from another thread), but it turns "nothing has moved in ten minutes" into "worker 3 has been reading \\nas\archive\big.iso for ten minutes", which is the difference between a mystery and a decision.
• A drive that disappears mid-scan — the missing-file bookkeeping is skipped when the walk looks incomplete. Otherwise a USB stick that disconnected halfway through would flip every file on that volume to "no longer present" and destroy an index that took hours to build.
• Interruption — results are committed in batches (--commit-batch) as they are produced, so a scan stopped after six hours keeps six hours of hashing instead of losing all of it.
• A file that was readable before and is not now — keeps the checksum it already had. A transient lock never erases work that was expensive to do.

How volume identification works (and why it matters)

Drive letters are not a stable identity — the same USB stick can show up as E: today and F: tomorrow depending on what else is plugged in. hyperindex instead asks Windows for the volume's serial number (GetVolumeInformationW), which is stamped onto the file system at format time and stays constant for the medium's lifetime. Every indexed file is tied to that serial number, not a drive letter, so:

• Plugging the same stick into a different port (different drive letter) is recognised as the same volume — no duplicate volume rows, no redundant re-hashing.
• dupes correctly reports "these two files are identical" even when the two copies live on two different physical drives that happen to both be plugged in (or were, on different days — the index is scanned across everything the database has ever recorded, not just what's currently attached).

UNC network paths (\\server\share\...) are also supported: hyperindex asks the same API, which works for remote NTFS shares; if it doesn't (some share configurations refuse the call), a stable pseudo-serial is derived from the share path itself so the index still round-trips correctly on repeat scans.

The handoff report

hyperindex dupes is explicitly designed to be consumed by an agentic AI as the next step in a cleanup workflow, not just read by a human. The Markdown file:

• states its own usage instructions and the exact "which copy to keep" heuristic up front (oldest modified_at wins, ties broken by shortest path), so the reasoning is auditable and overridable rather than a black box;
• marks exactly one file per duplicate group KEEP and the rest REDUNDANT;
• lists files that could not be read during indexing (permissions, locks, I/O errors) in their own section, so nothing silently vanishes from the picture;
• ends with an explicit checklist of suggested next steps.

hyperindex never deletes or moves anything itself. It only reads and reports; acting on the report (deleting/quarantining redundant copies) is left to whoever — human or agent — consumes it, on the explicit warning that paths/sizes should be re-verified first since the index can go stale if files changed after the last scan.

Database schema

SQLite, WAL mode, at the path from --db (or the default above).

• volumes — one row per physical/logical volume ever seen, keyed by its OS volume serial number (serial), plus label/filesystem/last-seen mount point.
• scans — one row per root per scan invocation: root path, start/finish time, counts (seen/hashed/reused/skipped/online-only/errors), bytes hashed, plus the invocation that created it — job_id (shared by every root of one command), command_line (as typed) and job_json (the full root list and resolved options). A row with no finished_at is an interrupted scan and is what --resume picks up; resumed counts how often it has been continued.
• files — one row per file, keyed by (volume_id, path): size, created/modified/accessed timestamps, BLAKE3 hash (nullable — null means "known to exist but not yet successfully hashed"), read-only flag, and a present flag that is cleared (not deleted) when a rescan of that root no longer finds the file. History survives an unplugged/reformatted drive; nothing is silently thrown away. Since 0.101.0 also the file's sniffed content type: magic (MIME type from the magic number, or text/plain), exif_date (EXIF capture time, ISO-8601 local time, images only), has_text (1 for plain text; NULL otherwise — for a PDF or office file that means "not known yet", not "no text") and sniffed_at. All four are NULL until a scan has read the file, and — like the hash — a scan that could not read a file leaves what is on record alone.
• access_errors — files a given scan attempted to read but couldn't, with both a category slug for the cause (cloud-placeholder, permission-denied, locked, disappeared, device-unavailable, path-too-long, timed-out, other) and the raw OS message.
• enrich — one row per file hyperindex enrich has processed, keyed by file_id: the resolved capture date, which signal it came from (date_source) and how much to trust it (date_confidence), plus an offline-reverse-geocoded place (nullable — most files have no GPS EXIF). A cache, not a report — hyperindex plan reads it via a LEFT JOIN, so a file with no row here is simply "not enriched yet" rather than an error.
• apply_runs — one row per hyperindex apply invocation: which plan.json, which --library-root, the exact command, start/finish time. What --undo <job_id> looks up.
• apply_journal — one row per file a run attempted, written before the copy starts and updated once the outcome is known (verified, already-done or failed, plus the expected and actually-written hash). What --undo walks to know exactly which files one run — and only that run — is responsible for.
• actions — the persisted plan: one row per intended action, keyed by (plan_run, file_id), holding the verb (move, duplicate, quarantine, skip-protected), the destination, the content hash at plan time, and a status that walks the lifecycle planned → copied → verified → source_marked → quarantined → purged. plan writes the planned rows; apply advances them to verified once the destination's hash matches. Read it with hyperindex actions.
• file_tags — per-file classification cache, keyed by file_id: a coarse media_class, a free-form tags_json object of per-class attributes, which extractor produced it, a confidence, and the hash of whatever text was extracted. A cache — every row is regenerable from the file plus its sidecars.
• doc_embeddings — per-file vector for semantic search, keyed by file_id: the model_id that produced it, the dimension, and the vector as a blob of little-endian f32s (the layout sqlite-vec expects, so an index can be layered over the same column later without a rewrite). Explicitly a derived, rebuildable artifact: delete every row and the only thing lost is search speed.

Why actions exists alongside apply_journal

They answer different questions, and collapsing them would lose one of the answers. apply_journal records one execution attempt of one run — it is how apply --undo knows which files a given invocation placed, and it keeps a row for an attempt that died mid-copy. actions records the intent and where it has got to, across however many runs it takes: it is the row you look at to ask "has this file been safely copied and verified, and may its source therefore be retired?"

The three retirement states (source_marked, quarantined, purged) are defined but nothing writes them yet. They are in the vocabulary so that the column has one settled meaning before something starts advancing rows into it, not because such a stage exists.

Because files already carries a BLAKE3 hash, verification never re-reads a source: actions.src_hash is what the index already knew, actions.dst_hash is the destination hashed once after the copy, and "verified" is those two being equal.

Databases written by an older build are migrated in place on open — the index is expensive to rebuild, so new tables and columns are added rather than demanded. An index built by 0.91 opens under 0.101.0, keeps every row, and simply gains the tables it never had; there is no rebuild step and no "please delete your database" path.

Indexed on size, hash, and (size, hash) WHERE present = 1 for fast duplicate-group queries even against a database with millions of rows.

Performance notes

• Directory walk is single-threaded (walkdir) — traversal is metadata-only and cheap; parallelising it buys little and complicates exclude-filtering.
• Diffing against the existing index loads every already-known file under the scanned root into an in-memory map with one query, so deciding "hash or reuse" per file is O(1) instead of one round-trip per file.
• Hashing is parallelised across all CPU cores with rayon, using BLAKE3 — a modern, SIMD- accelerated cryptographic hash function that is dramatically faster than SHA-256 at equivalent collision resistance, which matters a lot when hashing terabytes of data.
• Persistence happens in a single SQLite transaction per scanned root after all hashing is done, rather than one transaction per file — this is the difference between a scan finishing in seconds vs. minutes for large file counts.
• Release builds use LTO + a single codegen unit for maximum throughput at the cost of longer compile times — expected for a tool whose whole point is raw scanning speed.

Reporting is set-based, on purpose

Everything above concerns the scan. The reports have their own rule, learned the hard way: no query inside a loop. An index of a few million files has hundreds of thousands of duplicate groups and folders, so any per-item query is a per-item disk seek, repeated until the run is meaningless. dupes and doubles each answer their question in one pass and do the grouping in memory. Measured on a 1.6 GB index of 2,324,265 files across three volumes:

Step	One query per item	One pass
Duplicate groups + members (311,714 groups)	~35 s	~7 s
Unreadable-file list (4,512 paths)	~8 s per path — hours	~6 s
Unique-file counts (179,074 folders)	~6.4 s per folder — days	~32 s

The middle row is the one to remember when adding a query: nothing indexes files.path on its own — UNIQUE(volume_id, path) leads with the volume — so a lookup by path alone silently degrades into a full sweep of an unrelated index. Restrict the paths you are asking about to a set the database can hold, or join, rather than correlating a subquery per row.

Whole-run times on that index, reports written included: dupes 14–19 s, doubles --fresh ~34 s.

Memory

SQLite's default page cache is 2 MiB, which against a multi-gigabyte index means reading the same b-tree pages from disk over and over. hyperindex raises it to a quarter of the machine's memory (64–1024 MiB) and maps up to 1 GiB of the database, which on the index above took the duplicate query from 18.8 s to 6.8 s without a line of SQL changing. Both are ceilings that fill lazily, not reservations. Use --db-cache <MB> to raise it further on a big machine, or lower it when hyperindex is competing with something else for memory.

Known limitations / roadmap

• Hardlink-aware deduplication (recognising that two paths are the same file via NTFS file IDs, not just identical content) is not implemented; hardlinked files are currently reported as duplicates like any other identical-content pair. std's MetadataExt::file_index() would enable this but is still gated behind the unstable windows_by_handle feature as of Rust 1.98, so it isn't available on stable — a windows-sys-based implementation (mirroring how volume.rs already handles the serial number) is the natural next step.
• No bit-rot-specific verify mode yet beyond scan --rehash-all (which re-hashes everything and will naturally reveal a checksum mismatch if you diff two dupes runs, but there's no dedicated "flag files whose content changed but metadata looks unchanged" report).
• Folder-level deduplication (treating two differently-named folders with identical contents as duplicates) is intentionally still just reporting — see doubles above. What consumes that report to actually reorganise a drive is enrich/plan (currently: photos only), built as subcommands of hyperindex itself rather than a separate tool, so they can reuse db.rs directly instead of a second database connection. ../filesorter and ../Intelectual-PC-Clean-UP-Script were earlier, unfinished attempts at the same goal, superseded by this direction as of 2026-08-28.
• enrich/plan/apply currently cover photos only. Confirmed (2026-08-29, researched against Immich's own docs) that plan's library/photos/<year>/<place?>/... tree works directly as an Immich external-library import path — no separate staging/flattening step needed. Not yet built: the broader verify pass (re-index the target, confirm every source hash is present at least once) as a safety net before anyone considers deleting sources — apply's own per-file hash check is not that whole-library assertion. Music deliberately has no enrich/plan/apply — the decision (2026-08-29) is to let beets do tag-based organising/deduping/import instead; Hyperindex's own part is the prefilter command above. See obsidian/dev/data-consolidation/HANDOFF.md for the full pipeline design and remaining open questions (server layout, the broader verify pass).

Project layout

src/
  main.rs     CLI entry point, subcommand orchestration
  cli.rs      clap argument/subcommand definitions
  model.rs    shared data structures (DB rows <-> in-memory shapes)
  db.rs       SQLite schema + all queries
  volume.rs   Windows volume identification + mounted-drive enumeration
  winfs.rs    Windows hardening: cloud placeholders, error dialogs,
              long paths, failure classification
  scan.rs     the walk -> diff -> hash -> persist pipeline
  sniff.rs    content type + EXIF capture time from a file's head (HEIF: + tail)
  ui.rs       the live status panel, its ASCII fallback, the step panel
  report.rs   file-level duplicate report (Markdown + JSON)
  doubles.rs  folder-level duplicate index, incl. report parsing/reuse
  enrich.rs   capture-date + GPS resolution (EXIF/filename/folder/mtime), cached
  geocode.rs  offline GPS-to-place-name lookup (bundled GeoNames cities)
  plan.rs       photo-sorting rule set (pure function) + manifest report
  prefilter.rs  music exact-dedup pre-filter for beets (pure function) + report
  apply.rs      apply a plan.json: copy, verify by hash, journal, --undo
  util.rs       formatting helpers (byte sizes, timestamps, SQL LIKE escaping)

License

PolyForm Noncommercial License 1.0.0 — free to use, modify and share for any noncommercial purpose, including personal projects, research, education, charities and government use. Commercial use requires a separate license from the copyright holder.

Troubleshooting

• "could not read volume information for '...'" — printed as a warning, not an error; hyperindex falls back to a synthetic but stable volume id derived from the path and keeps going. Typically happens on unusual mount configurations (e.g. a folder mounted as a volume without a drive letter that the API doesn't recognise).
• The .db file does not change while a scan runs — expected, and not a sign that nothing is being saved. Commits go into index.db-wal first; see Why the .db file looks frozen. To confirm progress, watch index.db-wal instead, or lower --flush-interval.
• A scan was interrupted — run hyperindex scan --resume. See Interrupting and resuming a scan.
• A scan seems to hang, and a dialog from a sync client appears — this is the failure mode described under Surviving a live Windows desktop. It should not happen with the default --cloud-files skip; if it does, the client is being asked for a file whose attributes did not advertise it as online-only. Note which path --stall-warn names and re-run that folder with --exclude-path to confirm.
• Lots of files reported as online-only — they live in a cloud-synced folder and were indexed without a checksum, so they take no part in duplicate detection. See Cloud-synced folders for the three ways to include them.
• Files listed as inaccessible — check the "could not be indexed" section of the dupes report. Errors are grouped by cause with the fix for each; the raw OS message is kept in the per-file detail and in the JSON twin.
• A rescan re-hashes everything instead of reusing the cache — this means the file's reported modification time changed since the last scan (common causes: the file was actually edited, or it was copied/restored in a way that reset its mtime, or the file system itself only has coarse timestamp resolution). Confirm with hyperindex stats, which shows hashed vs. reused counts per scan.