mirror of
https://github.com/Heretek-AI/torrent-dedup.git
synced 2026-11-07 10:06:02 -10:00
John Smith
da9de2b766
Collect live inventories from all qBittorrent hosts, choose one complete keeper per duplicated infohash (most free disk), and apply reviewed deletions with entry-only deletes. Every deletion is backed up as a .torrent file first so qui Orphan Scan can reclaim the files afterwards. Includes 32 tests covering login styles, pagination, keeper selection, vetoes, backups, and failure handling.
69 lines
2.2 KiB
Markdown
69 lines
2.2 KiB
Markdown
# torrent-dedup
|
|
|
|
Collect torrent infohashes from multiple qBittorrent WebUI instances into a
|
|
single CSV, with one column per host.
|
|
|
|
## Setup
|
|
|
|
1. Copy the example host list and fill in real credentials:
|
|
|
|
```bash
|
|
cp qbittorrent-hosts.example.csv qbittorrent-hosts.csv
|
|
```
|
|
|
|
2. Install the dependency:
|
|
|
|
```bash
|
|
pip install requests
|
|
```
|
|
|
|
3. Run:
|
|
|
|
```bash
|
|
python dump_torrent_hashes.py
|
|
```
|
|
|
|
Output is written to `torrent-hashes.csv`: the header row lists the hosts, and
|
|
each column contains that host's sorted, de-duplicated infohashes. Hosts that
|
|
fail still get a column (empty) and are reported on the console.
|
|
|
|
## Options
|
|
|
|
| Option | Default | Description |
|
|
|---|---|---|
|
|
| `--hosts` | `qbittorrent-hosts.csv` | Credentials CSV (`host,username,password`) |
|
|
| `--out` | `torrent-hashes.csv` | Output CSV |
|
|
| `--only` | all hosts | Query only the given URLs or ports, e.g. `--only 8083` |
|
|
| `--workers` | `8` | Hosts queried in parallel |
|
|
| `--timeout` | `60` | Per-request timeout in seconds |
|
|
| `--page-size` | `5000` | Torrents fetched per API call |
|
|
| `--retries` | `2` | Retries for timeouts and 5xx responses |
|
|
| `--insecure` | off | Skip TLS verification for self-signed HTTPS hosts |
|
|
|
|
Exit codes: `0` all hosts succeeded, `1` some failed, `2` none succeeded.
|
|
|
|
## Filter duplicates
|
|
|
|
`filter_duplicates.py` reads `torrent-hashes.csv` and writes a new CSV with the
|
|
same columns but only the infohashes that appear on two or more hosts. Hashes
|
|
unique to a single host are removed. The input file is never modified.
|
|
|
|
```bash
|
|
python filter_duplicates.py
|
|
```
|
|
|
|
Output is written to `torrent-hashes-duplicates.csv`. Because columns are
|
|
independent lists, a duplicated hash can sit on different rows in each host's
|
|
column; the console summary reports how many duplicate hashes exist and how
|
|
many extra copies that represents.
|
|
|
|
| Option | Default | Description |
|
|
|---|---|---|
|
|
| `--input` | `torrent-hashes.csv` | Source CSV |
|
|
| `--out` | `torrent-hashes-duplicates.csv` | Filtered CSV |
|
|
| `--min-hosts` | `2` | Keep hashes present on at least this many hosts |
|
|
|
|
`qbittorrent-hosts.csv`, `torrent-hashes.csv`, and
|
|
`torrent-hashes-duplicates.csv` are gitignored because they contain credentials
|
|
and local data.
|