Command Info

Name: websites:cleanup-duplicates

Description: Find and clean up duplicate websites (including IDN Unicode/Punycode duplicates)

Status: completed

Start Time: 2026-07-10 16:09:00

End Time: 2026-07-10 16:09:01

Total Time: 1 second

Dispatched Jobs Count: 0

Successful Jobs Count: 0

Failed Jobs Count: 0

Output

16:09:00 Lock acquired for command: websites:cleanup-duplicates
16:09:00 🔍 Starting duplicate website cleanup...
16:09:00 Found 1 group(s) of duplicate websites.
16:09:00 (dry-run — no data is changed; shows the re-key + merge + keeper-canonicalize plan)
16:09:00 <fg=yellow>Duplicate group for: https://adlogger.org</>
16:09:00 <fg=green>KEEP: ID 3cfc1c33-d85c-456e-8fcb-31cba35e16b8, www.adlogger.org (type LINK, status LIVE)</>
16:09:00 <fg=red>DELETE: ID 3f145f93-5828-402a-b852-856ed442282b, adlogger.org (type LINK, status LIVE)</>
16:09:00 ⚠️ Keeper and losers already share the canonical domain_name. If websites:canonicalize-hosts already ran, data filed under the old variant may be missed.
16:09:00 ↳ mongo:uptime_keywords: 1 doc(s) would reassign
16:09:01 ↳ keeper canonicalize: domain_name adlogger.org→adlogger.org, url/hostname/tld → canonical
16:09:01 📊 Cleanup complete:
16:09:01 ✅ Groups processed: 0
16:09:01 ⏭ Groups skipped (re-key/keeper failures): 0
16:09:01 ❗ Groups aborted (unexpected error): 0
16:09:01 🔄 Domain-keyed docs re-keyed onto keepers: 0
16:09:01 🧹 Logical duplicates merged: 0
16:09:01 ✅ Losers deleted: 0
16:09:01 ⏭ Losers skipped (reassign failures): 0
16:09:01 🔤 Keepers canonicalized: 0
16:09:01 Dry run — no changes written. Take a mongodump of the at-risk collections + a mysqldump of `websites`, then re-run without --dry-run to apply.
16:09:01 Lock released for websites:cleanup-duplicates