hyperindex

A hyperfast, resumable file indexer, checksum verifier and duplicate-file finder for Windows, written in Rust. It walks one or more directories (or every currently plugged-in drive), records size / timestamps / a BLAKE3 content checksum for every file into a persistent SQLite database, and can then report duplicate content across everything it has ever indexed — including across separate USB sticks plugged in on different days.

The key design goal is reuse: plug the same USB stick back in a week later and rescan it — files whose size and modification time haven't changed are recognised instantly and their stored checksum is reused without touching the file's contents again. Only new or changed files get (re-)hashed.

Why

Standard drive/file organisers either don't survive being unplugged (no persistent index) or re-hash everything on every run (slow on large collections). hyperindex is built specifically to answer: "across all these drives I've plugged in over time, what's actually duplicated, and where?" — and to hand that answer to an AI agent (or a person) as an actionable report, not just a wall of find | md5sum output.

Portability

target\release\hyperindex.exe is a genuinely standalone, single-file binary: copy just that one .exe anywhere — another folder, a USB stick, a different Windows machine — and run it. No installer, no companion DLLs, no SQLite runtime to provide (it's compiled in via rusqlite's bundled feature) and, thanks to .cargo/config.toml setting target-feature=+crt-static, no dependency on the Visual C++ runtime redistributable either. Verified with dumpbin /dependents: the only imports left are core Windows OS components present on every Windows 10/11 install (kernel32.dll, ntdll.dll, combase.dll, shell32.dll, bcryptprimitives.dll) — nothing that needs installing.

The .pdb file next to it (debug symbols) is optional — useful for attaching a debugger to a crash, never required to run the program; it's safe to leave behind when copying the .exe elsewhere.

Requirements

• Windows 10/11 (uses the Win32 GetVolumeInformationW API for stable volume identification; the rest of the code is otherwise portable Rust).
• Rust (stable toolchain, edition 2024 — tested with rustc 1.98) via rustup.
• The MSVC C++ build tools (Visual Studio "Desktop development with C++" workload, or the standalone Build Tools) — needed for the linker that Rust's default x86_64-pc-windows-msvc target uses.

Check what you have:

rustc --version
cargo --version

If rustc/cargo are missing: winget install --id Rustlang.Rustup -e, then open a new shell.

Building

cd hyperindex
cargo build --release

The optimized binary lands at target\release\hyperindex.exe. The release profile is tuned for throughput (opt-level = 3, LTO, single codegen unit) since hashing large collections is CPU/IO-bound work where this matters.

Run the test suite (unit tests for the trickiest bits — byte formatting, SQL LIKE-pattern escaping, the duplicate "which copy to keep" heuristic, volume-serial formatting):

cargo test

Quick start

# Index a single drive or folder
.\target\release\hyperindex.exe scan E:\

# Index everything currently plugged in (skips network drives by default)
.\target\release\hyperindex.exe scan --all-drives

# Find duplicates across everything indexed so far, write a handoff report
.\target\release\hyperindex.exe dupes --out report.md

# Folder-level view: which folders contain duplicates, what is safe to remove
.\target\release\hyperindex.exe doubles --out doubles.md

# Continue a scan that was interrupted (nothing to retype)
hyperindex scan --resume

# What has been indexed so far?
.\target\release\hyperindex.exe volumes
.\target\release\hyperindex.exe stats

Re-run scan on a drive you've already indexed and unchanged files are served from the database instead of being re-read:

==> Scanning E:\
    found 128,402 files (411.20 GB) — diffing against existing index...
    done: 812 hashed, 127,590 reused from cache, 0 now missing, 0 inaccessible

Getting help

Every command carries worked examples, not just a flag list:

hyperindex --help            overview and the usual sequence
hyperindex scan --help       everything scan accepts, with examples
hyperindex dupes --help      ... and so on for every command

Windows-style switches work as well as POSIX ones: /?, /h and /help are accepted alongside -h and --help, and /v alongside --version. Only exact matches are translated, so a path or glob that happens to start with a slash is passed through untouched.

Commands

hyperindex scan <paths...> [options]

Walks the given path(s), records metadata for every file, and hashes any file that is new or has changed since the last scan (different size or modification time). Everything is written to the index database.

Flag	Meaning
--all-drives	Scan every currently mounted local drive instead of (or alongside) explicit paths.
--include-network	Also scan network-mounted drives when using --all-drives (off by default — different perf/latency profile).
--follow-symlinks	Follow symlinks/junctions while walking (off by default, to avoid infinite loops on self-referential junctions).
--rehash-all	Recompute checksums for every file even if metadata is unchanged — use if you suspect silent bit rot rather than legitimate edits.
--exclude <NAME>	Additional directory base name to skip (repeatable, case-insensitive). Windows bookkeeping directories ($RECYCLE.BIN, System Volume Information, Config.Msi, $WinREAgent, $Windows.~BT, Recovery, OneDriveTemp, …) are always excluded, as are permanently locked system files (pagefile.sys, hiberfil.sys, NTUSER.DAT*, …).
--exclude-path <GLOB>	Skip anything whose full path matches a glob (repeatable, case-insensitive, either slash works), e.g. --exclude-path "**/AppData/Local/Temp/**".
--cloud-files <MODE>	skip (default) records online-only files by metadata but never opens them; hydrate downloads and checksums them. See Cloud-synced folders.
--retries <N>	Extra attempts for a transient failure — a momentary lock, a network blip (default 1). Permission and cloud errors are never retried.
--io-timeout <SECS>	Give up on a single file whose read has not finished in this long (0 = wait forever, the default).
--stall-warn <SECS>	Name any file still being read after this long, so a stuck scan says what it is stuck on (default 60, 0 disables).
--commit-batch <N>	File results per SQLite transaction (default 2000). Smaller means an interrupted scan loses less.
--flush-interval <SECS>	Commit whatever has accumulated at least this often, however few files it is, and fold the write-ahead log into the main .db afterwards (default 300, 0 disables).
--resume	Continue the most recent interrupted scan. Paths and options are read back from the database; files the interrupted run finished are not touched again.
--threads <N>	Hashing worker threads (default: number of logical CPUs).
--min-hash-size <BYTES>	Skip content-hashing (but still record metadata for) files at or below this size.
-q, --quiet	Suppress the status panel and per-file chatter, print only the final summary.

Watching a scan

A scan of several drives is a long, mostly silent operation, so it draws a status panel that is redrawn in place while log lines scroll past above it:

╔═ hyperindex 0.95.0 · scan ════════════════════════════ root 2/5 · 00:41:02 ═╗
║ Volume  F:\  "SANDISK ULTRA"  exFAT  SN 1A2B-3C4D                           ║
║ Root    F:\Fotos                                                            ║
║ Stage   hashing · 16 threads                                                ║
║ Now     …\2019\Sommer\IMG_2381.jpg                        4.20 MB  12s      ║
╟─────────────────────────────────────────────────────────────────────────────╢
║ Root  [██████████████░░░░░░░░░░░░░░░░]   46%  12,345/22,800 files           ║
║       41.20 GB of 88.00 GB · 128.00 MB/s · eta 00:10:44                     ║
║ Job   [█████████░░░░░░░░░░░░░░░░░░░░░]   29%  1 of 5 roots done             ║
║       elapsed 00:41:02 · eta ~01:55:00                                      ║
╟─────────────────────────────────────────────────────────────────────────────╢
║ hashed 8,901 · reused 3,100 · small 210 · cloud 12 · err 4 · db 12,000      ║
╚═════════════════════════════════════════════════════════════════════════════╝

Line by line:

• Volume / Root — which drive and which path is being worked on right now. The volume is named the way you would identify it yourself: drive letter, label, file system, and the serial number the index is actually keyed on, so two identically-labelled sticks are still distinguishable.
• Stage — walking the tree, diffing against the index, hashing, or finishing up. Worth its own line because the first two stages have no percentage to give and can each take minutes on a large volume; without it, the opening of a scan is indistinguishable from a hang.
• Now — the directory the walk is in, or, while hashing, the read that has been running longest, with its age once that exceeds a few seconds. This is the line that tells you a single 40 GB file is what the bar is waiting for.
• Root — progress through the current root only. Precise: the walk has already counted every file and byte underneath it, so the percentage and the ETA are real. Progress is measured in bytes, since that is what hashing time is made of; the throughput figure is a smoothed average of bytes actually read, so a reused checksum does not inflate it.
• Job — progress across all roots of this invocation, and only shown when there is more than one. Deliberately coarse: nothing has counted the roots that have not been walked yet, so this is (finished roots + fraction of the current one) / total roots and its ETA carries a ~. A 64 GB stick and a 4 TB archive both count as "one root".
• Counters — outcomes so far for this root (small = below --min-hash-size, cloud = online-only placeholder, err = unreadable), and db = results committed to the database, which is how much of the run would survive a Ctrl-C.

So: there is no single bar spanning every block of every volume. Each root gets its own precise bar, and the job bar counts roots. hyperindex scan C:\ D:\ F:\Photos scans three roots in sequence, one bar at a time, with the job bar underneath them.

dupes and doubles draw a smaller panel of the same kind — a checklist of their few long steps, so a grouping query that runs for minutes says which step it is on rather than nothing at all.

The frames need a UTF-8 console; hyperindex switches the console's output code page over for the duration of the run and switches it back on exit (which also makes umlauts in paths render correctly). Pass --ascii to leave the console alone and draw with +---+ instead, --progress bar for the single-line progress bar of 0.93 and earlier, or --progress off for nothing but log lines. Redirected output never draws a panel regardless — a display that redraws in place is meaningless in a log file — so piping to a file gives you the log lines and nothing else.

Redrawing in place also needs the console to defer its line wrap — to leave the cursor on a line that fills the window exactly, rather than jumping to the next row — because that is what walking the cursor back up over the last frame depends on. hyperindex turns that mode on at startup and off again on exit; on a console too old to offer it, the live display switches itself off with a note instead of scrolling a fresh copy of the panel past you eight times a second. The panel also steps down to the one-line bar when the terminal window is too short to hold it, so --progress bar is only needed if you want the short form in a window that would have fit the panel.

hyperindex dupes [options]

Finds every group of currently-present files that share both size and BLAKE3 checksum (i.e. confirmed identical content — not just "probably the same"), across every volume ever indexed, and writes:

• a Markdown handoff report (--out, default hyperindex-report.md) — written to be read and acted on by an autonomous agent or a human; see The handoff report below.
• a JSON twin (--json, defaults to the same name with a .json extension) with the identical data in a shape trivial to parse programmatically.

Flag	Meaning
--min-size <BYTES>	Ignore files smaller than this (default 4096 — tiny files produce a lot of low-value "duplicate" noise, e.g. empty marker files). Pass 0 for no filter.
-o, --out <FILE>	Markdown report path.
--json <FILE>	JSON report path.

hyperindex doubles [options]

The folder-level view of the same data. Where dupes answers "which files are duplicated", doubles answers "which folders do I need to look at" — the unit cleanup actually happens in. Output is doubles.md plus a JSON twin, intended as input for a step-2 cleanup tool or for working through by hand.

For every folder holding duplicated content it reports the path, volume, number and combined size of its duplicate files, how many of those are redundant copies, the reclaimable space, the modification-date span, and — crucially — how many files in that folder, or anywhere beneath it, exist nowhere else.

That last number is the safety-critical one, and it deliberately covers the whole subtree: the claim the report makes is that removing the folder is safe, and a folder holding nothing unique itself can still have a subfolder that does. It splits the output into two clearly separated sections:

• Folders that can be removed whole — every file in them is a redundant copy and the index confirms nothing unique inside.
• Folders to clean file-by-file (do NOT delete the folder) — all their duplicates are redundant, but they also hold one-of-a-kind files that wholesale deletion would destroy.

A duplicate report on its own can never make that distinction (a file with no duplicate never appears in one), so doubles consults the index database for it. When the count can't be determined, the folder is shown with ? and is never listed as safe for whole-folder removal — an unknown is not treated as a "no".

A "Folder pairings" section lists which other folders hold the counterpart copies, which is what reveals "folder A is essentially a copy of folder B".

Flag	Meaning
-f, --from-report <FILE>	Reuse an existing dupes report instead of recomputing. Accepts the Markdown report or its JSON twin; given the Markdown, the JSON twin next to it is used automatically because it holds exact values. If a report has been hand-annotated, the Markdown parser tolerates the extra lines.
--fresh	Never read a report — always recompute from the index database.
-o, --out <FILE>	Folder-level Markdown report (default doubles.md).
--json <FILE>	JSON twin (defaults to --out with a .json extension).
--min-size <BYTES>	Ignore files below this size when computing fresh (default 4096). No effect when reading a report — that report's own filter already applied.

Source resolution when --from-report is omitted and --fresh is not given: hyperindex looks for a default-named report (hyperindex-report.json, then .md) in the working directory and uses it if found; otherwise it computes from the database. So the common flow just works:

hyperindex dupes            # writes hyperindex-report.md/.json
hyperindex doubles          # picks that report up automatically

hyperindex volumes

Lists every volume the database has ever seen (serial number, label, file system, when it was last scanned, current file count and total size).

hyperindex stats

Overall database statistics plus a table of the most recent scan runs (files seen/hashed/reused/errored, bytes hashed, per run).

Global options

Flag	Meaning
--db <PATH>	Which database file to use. Default %LOCALAPPDATA%\hyperindex\index.db — a stable per-user location, so running scan today and dupes next week without passing --db still talks to the same database.
--db-cache <MB>	How much page cache to allow the database. Defaults to a quarter of this machine's memory, clamped to 64–1024 MiB; 0 leaves SQLite's own (2 MiB) default alone. Raising it is the cheapest speed-up available on a large index — see Performance notes.
--progress <MODE>	panel (default) draws the framed status window; bar draws the single progress line of 0.93 and earlier; off prints only the log lines.
--ascii	Draw the display with +---+ instead of box-drawing characters, and leave the console code page alone. The line-wrap mode is still set, since any display has to repaint over itself.

Interrupting and resuming a scan

Results are written to the database as the scan runs, not at the end. Each batch of --commit-batch files is committed, and --flush-interval guarantees a commit at least every so often however large the files are — so an interruption costs at most the last few minutes, never the whole run.

Why the .db file looks frozen

The index runs in SQLite's WAL mode: commits land in index.db-wal and the main index.db is only rewritten at a checkpoint. Watching the .db timestamp during a scan is therefore misleading — it can sit unchanged while tens of thousands of files are being committed next to it. --flush-interval checkpoints on its own cadence for exactly this reason, so the main file visibly advances, and the log is folded back in when the scan ends.

Continuing

$ hyperindex stats
   ID STARTED                  ROOT               SEEN   HASHED   REUSED   ERRORS  HASHED SZ       STATUS
    1 2026-08-24 12:20:35.880  C:\Users\mart…        0        0        0        0        0 B  interrupted

Interrupted scans — continue the newest with `hyperindex scan --resume`:
  #1 C:\Users\marti\Nextcloud9  started 2026-08-24 12:20:35.880
        hyperindex scan C:/Users/marti/Nextcloud9 C:/Users/marti/Nextcloud7 ... --min-hash-size 100

$ hyperindex scan --resume
Resuming scan #1 started 2026-08-24 12:20:35.880
  original command: hyperindex scan C:/Users/marti/Nextcloud9 ... --min-hash-size 100
==> Scanning C:\Users\marti\Nextcloud9
    found 60950 files (122.34 GB) — diffing against existing index...
    10000 of them were already done by the interrupted run; 50950 left

Every invocation records what it was asked to do — the command as typed, the full root list, and every resolved option — on each scans row it creates. That is what lets --resume reconstruct a run without the user remembering the flags they used three hours ago.

It also means a resume finishes roots the original run never reached. A --all-drives invocation killed on the third of five drives continues the third and then scans the fourth and fifth, which no per-root record on its own could know about.

The interrupted scans row is re-opened rather than duplicated, so scan history stays one row per root per invocation; stats marks a scan that was continued with complete*.

If you would rather not use --resume

Just run the original command again. Files already hashed have unchanged size and modification times, so they are served from the index and never re-read — only the remainder is hashed. --resume is faster still (it skips even the diff for finished files) and saves you retyping, but nothing is lost either way.

Surviving a live Windows desktop

A scanner pointed at real user directories meets files that cannot simply be opened and read. The failure modes that matter are not the ones that return an error — those are easy — but the ones that block, because a blocked worker looks exactly like a hung program.

Cloud-synced folders (OneDrive / Nextcloud / Dropbox)

With "virtual files" / "online-only" mode enabled, a synced folder is full of files that look completely normal to stat but whose content lives on a server. Opening one hands control to the sync client, which downloads it — or, if that client is offline, stalled or unauthenticated, puts up its own modal dialog and leaves the read blocked until somebody clicks it away. On a multi-threaded scan the effect is a scan that advances only when you dismiss dialogs, and a scan of a folder you thought was local quietly pulling gigabytes over the network.

hyperindex recognises such files from their file attributes (FILE_ATTRIBUTE_OFFLINE, FILE_ATTRIBUTE_RECALL_ON_OPEN, FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS) before touching them, and by default records their metadata without ever opening them. They are reported separately as online-only rather than as errors, since nothing is wrong with them — they simply are not here:

    found 5090 files (18.49 GB) — diffing against existing index...
    5090 of them are online-only (cloud placeholders); their content will not
    be downloaded. Pass --cloud-files hydrate to checksum them anyway.

Files indexed this way carry metadata but no checksum, so they cannot take part in duplicate detection. Three ways to include them:

1. Tell the sync client to keep the folder on this device ("Always keep on this device" / "Make available locally"), then rescan — this is the fast path, because the files are then simply local.
2. Run hyperindex scan --cloud-files hydrate <path>, which downloads each one on demand. Complete, but slow, bandwidth-hungry, and it needs the local disk space — and it will block on the sync client if that client is unhealthy.
3. Leave them out. For a duplicate hunt across local disks and USB sticks, the cloud copies are usually not what you are looking for anyway.

As a second line of defence, files in such a tree are opened with FILE_FLAG_OPEN_NO_RECALL, so even a file that became a placeholder between the walk and the hash fails cleanly instead of triggering a download.

Windows hard-error dialogs

An empty card-reader slot, a disconnected network path or a failing sector makes Windows display a message box on the process's behalf and block the offending call until it is dismissed. hyperindex disables this at start-up (SetErrorMode, re-asserted per hashing thread), so those conditions come back as ordinary errors.

Errors are grouped by cause, not listed by file

One protected folder or one unhealthy sync client produces thousands of identical error lines, which communicate nothing. Every failure is classified and reported per cause, with the fix:

Unreadable files by cause:
       2314 × cloud-placeholder  online-only files whose content lives on a
                                 server. Bring the sync client online and
                                 re-run, mark the folder as
                                 always-keep-on-this-device, or pass
                                 --cloud-files hydrate ...
         17 × permission-denied  no read permission. Re-run the scan elevated,
                                 or exclude the folder with --exclude ...

The same grouping appears in the dupes report, with the per-file detail folded into a collapsible block and the complete list in the JSON twin. Errors are also de-duplicated across scans and drop out entirely once the file has been hashed successfully, so the list describes the present rather than the whole history.

Other hazards handled

• Locked files — opened with a permissive share mode, so a file another process is actively writing can still be read. Genuinely transient failures (sharing violations, network blips) are retried; permission and cloud errors are not, because they would fail identically every time.
• Long paths — anything approaching MAX_PATH is opened through its extended (\\?\) form, so a 300-character path is indexed rather than reported as an error.
• Junctions and mount points — directory reparse points are not descended into unless --follow-symlinks is given, which is what keeps a scan from re-walking a tree, wandering onto another volume, or looping forever on a self-referential junction such as the legacy Application Data.
• Stalls — a watchdog names any file that has been open for longer than --stall-warn seconds. It cannot cancel the read (Windows offers no way to do that from another thread), but it turns "nothing has moved in ten minutes" into "worker 3 has been reading \\nas\archive\big.iso for ten minutes", which is the difference between a mystery and a decision.
• A drive that disappears mid-scan — the missing-file bookkeeping is skipped when the walk looks incomplete. Otherwise a USB stick that disconnected halfway through would flip every file on that volume to "no longer present" and destroy an index that took hours to build.
• Interruption — results are committed in batches (--commit-batch) as they are produced, so a scan stopped after six hours keeps six hours of hashing instead of losing all of it.
• A file that was readable before and is not now — keeps the checksum it already had. A transient lock never erases work that was expensive to do.

How volume identification works (and why it matters)

Drive letters are not a stable identity — the same USB stick can show up as E: today and F: tomorrow depending on what else is plugged in. hyperindex instead asks Windows for the volume's serial number (GetVolumeInformationW), which is stamped onto the file system at format time and stays constant for the medium's lifetime. Every indexed file is tied to that serial number, not a drive letter, so:

• Plugging the same stick into a different port (different drive letter) is recognised as the same volume — no duplicate volume rows, no redundant re-hashing.
• dupes correctly reports "these two files are identical" even when the two copies live on two different physical drives that happen to both be plugged in (or were, on different days — the index is scanned across everything the database has ever recorded, not just what's currently attached).

UNC network paths (\\server\share\...) are also supported: hyperindex asks the same API, which works for remote NTFS shares; if it doesn't (some share configurations refuse the call), a stable pseudo-serial is derived from the share path itself so the index still round-trips correctly on repeat scans.

The handoff report

hyperindex dupes is explicitly designed to be consumed by an agentic AI as the next step in a cleanup workflow, not just read by a human. The Markdown file:

• states its own usage instructions and the exact "which copy to keep" heuristic up front (oldest modified_at wins, ties broken by shortest path), so the reasoning is auditable and overridable rather than a black box;
• marks exactly one file per duplicate group KEEP and the rest REDUNDANT;
• lists files that could not be read during indexing (permissions, locks, I/O errors) in their own section, so nothing silently vanishes from the picture;
• ends with an explicit checklist of suggested next steps.

hyperindex never deletes or moves anything itself. It only reads and reports; acting on the report (deleting/quarantining redundant copies) is left to whoever — human or agent — consumes it, on the explicit warning that paths/sizes should be re-verified first since the index can go stale if files changed after the last scan.

Database schema

SQLite, WAL mode, at the path from --db (or the default above).

• volumes — one row per physical/logical volume ever seen, keyed by its OS volume serial number (serial), plus label/filesystem/last-seen mount point.
• scans — one row per root per scan invocation: root path, start/finish time, counts (seen/hashed/reused/skipped/online-only/errors), bytes hashed, plus the invocation that created it — job_id (shared by every root of one command), command_line (as typed) and job_json (the full root list and resolved options). A row with no finished_at is an interrupted scan and is what --resume picks up; resumed counts how often it has been continued.
• files — one row per file, keyed by (volume_id, path): size, created/modified/accessed timestamps, BLAKE3 hash (nullable — null means "known to exist but not yet successfully hashed"), read-only flag, and a present flag that is cleared (not deleted) when a rescan of that root no longer finds the file. History survives an unplugged/reformatted drive; nothing is silently thrown away.
• access_errors — files a given scan attempted to read but couldn't, with both a category slug for the cause (cloud-placeholder, permission-denied, locked, disappeared, device-unavailable, path-too-long, timed-out, other) and the raw OS message.

Databases written by an older build are migrated in place on open — the index is expensive to rebuild, so new columns are added rather than demanded.

Indexed on size, hash, and (size, hash) WHERE present = 1 for fast duplicate-group queries even against a database with millions of rows.

Performance notes

• Directory walk is single-threaded (walkdir) — traversal is metadata-only and cheap; parallelising it buys little and complicates exclude-filtering.
• Diffing against the existing index loads every already-known file under the scanned root into an in-memory map with one query, so deciding "hash or reuse" per file is O(1) instead of one round-trip per file.
• Hashing is parallelised across all CPU cores with rayon, using BLAKE3 — a modern, SIMD- accelerated cryptographic hash function that is dramatically faster than SHA-256 at equivalent collision resistance, which matters a lot when hashing terabytes of data.
• Persistence happens in a single SQLite transaction per scanned root after all hashing is done, rather than one transaction per file — this is the difference between a scan finishing in seconds vs. minutes for large file counts.
• Release builds use LTO + a single codegen unit for maximum throughput at the cost of longer compile times — expected for a tool whose whole point is raw scanning speed.

Reporting is set-based, on purpose

Everything above concerns the scan. The reports have their own rule, learned the hard way: no query inside a loop. An index of a few million files has hundreds of thousands of duplicate groups and folders, so any per-item query is a per-item disk seek, repeated until the run is meaningless. dupes and doubles each answer their question in one pass and do the grouping in memory. Measured on a 1.6 GB index of 2,324,265 files across three volumes:

Step	One query per item	One pass
Duplicate groups + members (311,714 groups)	~35 s	~7 s
Unreadable-file list (4,512 paths)	~8 s per path — hours	~6 s
Unique-file counts (179,074 folders)	~6.4 s per folder — days	~32 s

The middle row is the one to remember when adding a query: nothing indexes files.path on its own — UNIQUE(volume_id, path) leads with the volume — so a lookup by path alone silently degrades into a full sweep of an unrelated index. Restrict the paths you are asking about to a set the database can hold, or join, rather than correlating a subquery per row.

Whole-run times on that index, reports written included: dupes 14–19 s, doubles --fresh ~34 s.

Memory

SQLite's default page cache is 2 MiB, which against a multi-gigabyte index means reading the same b-tree pages from disk over and over. hyperindex raises it to a quarter of the machine's memory (64–1024 MiB) and maps up to 1 GiB of the database, which on the index above took the duplicate query from 18.8 s to 6.8 s without a line of SQL changing. Both are ceilings that fill lazily, not reservations. Use --db-cache <MB> to raise it further on a big machine, or lower it when hyperindex is competing with something else for memory.

Known limitations / roadmap

• Hardlink-aware deduplication (recognising that two paths are the same file via NTFS file IDs, not just identical content) is not implemented; hardlinked files are currently reported as duplicates like any other identical-content pair. std's MetadataExt::file_index() would enable this but is still gated behind the unstable windows_by_handle feature as of Rust 1.98, so it isn't available on stable — a windows-sys-based implementation (mirroring how volume.rs already handles the serial number) is the natural next step.
• No bit-rot-specific verify mode yet beyond scan --rehash-all (which re-hashes everything and will naturally reveal a checksum mismatch if you diff two dupes runs, but there's no dedicated "flag files whose content changed but metadata looks unchanged" report).
• Folder-level deduplication (treating two differently-named folders with identical contents as duplicates) is out of scope for this tool by design — hyperindex answers "which files are duplicated", not "how should my folder tree be reorganised". A separate, complementary drive-reorganiser project exists at ../filesorter for that broader workflow.

Project layout

src/
  main.rs     CLI entry point, subcommand orchestration
  cli.rs      clap argument/subcommand definitions
  model.rs    shared data structures (DB rows <-> in-memory shapes)
  db.rs       SQLite schema + all queries
  volume.rs   Windows volume identification + mounted-drive enumeration
  winfs.rs    Windows hardening: cloud placeholders, error dialogs,
              long paths, failure classification
  scan.rs     the walk -> diff -> hash -> persist pipeline
  ui.rs       the live status panel, its ASCII fallback, the step panel
  report.rs   file-level duplicate report (Markdown + JSON)
  doubles.rs  folder-level duplicate index, incl. report parsing/reuse
  util.rs     formatting helpers (byte sizes, timestamps, SQL LIKE escaping)

License

PolyForm Noncommercial License 1.0.0 — free to use, modify and share for any noncommercial purpose, including personal projects, research, education, charities and government use. Commercial use requires a separate license from the copyright holder.

Troubleshooting

• "could not read volume information for '...'" — printed as a warning, not an error; hyperindex falls back to a synthetic but stable volume id derived from the path and keeps going. Typically happens on unusual mount configurations (e.g. a folder mounted as a volume without a drive letter that the API doesn't recognise).
• The .db file does not change while a scan runs — expected, and not a sign that nothing is being saved. Commits go into index.db-wal first; see Why the .db file looks frozen. To confirm progress, watch index.db-wal instead, or lower --flush-interval.
• A scan was interrupted — run hyperindex scan --resume. See Interrupting and resuming a scan.
• A scan seems to hang, and a dialog from a sync client appears — this is the failure mode described under Surviving a live Windows desktop. It should not happen with the default --cloud-files skip; if it does, the client is being asked for a file whose attributes did not advertise it as online-only. Note which path --stall-warn names and re-run that folder with --exclude-path to confirm.
• Lots of files reported as online-only — they live in a cloud-synced folder and were indexed without a checksum, so they take no part in duplicate detection. See Cloud-synced folders for the three ways to include them.
• Files listed as inaccessible — check the "could not be indexed" section of the dupes report. Errors are grouped by cause with the fix for each; the raw OS message is kept in the per-file detail and in the JSON twin.
• A rescan re-hashes everything instead of reusing the cache — this means the file's reported modification time changed since the last scan (common causes: the file was actually edited, or it was copied/restored in a way that reset its mtime, or the file system itself only has coarse timestamp resolution). Confirm with hyperindex stats, which shows hashed vs. reused counts per scan.