Accuracy

How often is a Found right, and how often is a real account missed? This page gives the measured numbers, how they were measured, and what they do not show.

#The numbers

The shipped model, scored on a holdout of 142 site names it never trained on (700 rows: accounts known to exist and usernames made up for the test):

In plain words:

• About two in three Found results on these sites were real accounts.
• About half of the real accounts were found; the rest came back Maybe or Not Found.
• About one in eleven made-up usernames was wrongly reported Found: 47 false Found results on 530 made-up names. The model it replaced, before 2.5.0, managed one in four (133).

These describe particular captured responses from particular sites, not how a live scan of 7,700 sites will do today. Sites change their pages, and some block some networks outright.

#What this means for you

Treat every Found and Maybe as a lead. Open the profile. A false hit is cheap when you check it and costly when you do not.

On F1, the detector is not distinguishable from if status == 200 on held-out sites, and neither are Sherlock's hand-written per-site rules. What the detector buys is fewer false leads: its gain is in precision, at the cost of some recall.

#Measuring it yourself

aliens_eye selfcheck --split holdout --negatives 2
aliens_eye selfcheck --split holdout --report json

selfcheck scans accounts known to exist, and made-up usernames on the same sites, and reports precision, recall, F1 and the false-positive rate overall and per site.

• --split holdout scores the sites the model never trained on: generalisation.
• --split train (the default) scores the sites it trained on, and reads optimistically high.
• --negatives N sets how many made-up names to try per site.
• --detector-only scores the detector's verdict without the checks every Found gets.

#Ground truth

The ground truth is split by site, not by account: the train split (selfcheck.json, 286 site-name entries) and the holdout (eval_holdout.json, 142). The detector learns what each site's pages look like, so holding out accounts of a site it already trained on would overstate how well it does on a new one.

Entries are site names, not independent platforms: aliases such as chess and chess.com are the same site, and older splits let such pairs cross the boundary. New collection keeps host groups together, so a site and its aliases land on the same side. Accounts taken from other projects' lists are their claims, not verified.

#Reproducible evaluation: the frozen corpus

Measuring against the live web is not repeatable: sites change and rate-limit differently from day to day, so a change in the numbers cannot be pinned on the detector. A corpus removes the network from the measurement:

aliens_eye corpus record --out corpus/v1 --split all --negatives 4 --seed 1234
aliens_eye corpus stats corpus/v1
aliens_eye selfcheck --split holdout --corpus corpus/v1 --report json

Replay substitutes only the network fetch, so feature extraction and scoring run the same code on live and replayed responses. Three things keep replays identical: the rate limiter is skipped, the recorded response time is used, and the fingerprint store is read-only.

Made-up usernames come in two kinds: random (14 to 20 random letters and digits) and plausible (quietfalcon84). Random strings alone make an artificially easy test, separable on length and character mix; --plausible-ratio sets the share of plausible ones (default half). A plausible name could, rarely, belong to a real account on a large site.

Corpora are not published: they hold third-party page content, and a full capture is tens of megabytes.

#Ablations

aliens_eye eval ablate --corpus corpus/v1 --split holdout

Scores many detector configurations over the same corpus: status-code baselines, each judge alone, the shipped blend, and the blend with each feature group switched off. Each row has a bootstrap 95% confidence interval on F1, and a paired-bootstrap interval on its difference from the shipped blend. A difference whose interval spans zero is marked ns: on corpora this size, most differences cannot be told apart, and the table says so.

Each row also shows the Maybe rate: a configuration can raise its precision by answering Maybe to everything hard, so precision is never quoted without it.

#Against other tools

aliens_eye eval external --corpus corpus/v1 --sherlock data.json --maigret maigret.json --whatsmyname wmn-data.json

Runs Sherlock's, Maigret's and WhatsMyName's own per-site rules over the same stored responses, so the comparison is between detection logic, not HTTP clients. Their rule files are not bundled: download them yourself and respect their licences (Sherlock and Maigret MIT, WhatsMyName CC BY-SA 4.0). Each run records the file's SHA-256.

A tool is scored only on the rows it has a rule for and decides; the number covered is reported beside its score. WhatsMyName decides only a small share of rows.

#Lessons kept on record

• Everything measured before 2.5.0 was on truncated pages. The HTTP reader parsed only the first network chunk of each page, so older corpora, results and models reflect a random-length prefix of each page.
• A result that vanished. On a 43-site corpus, the ML model alone once beat the shipped blend by a significant +0.126 F1. That corpus included sites the model had trained on. On a site-disjoint holdout, the difference fell to +0.013, not significant.
• The catalog grew; the model did not change. Candidate models trained on pages captured during the 2026-11 catalog check either raised false positives or lost recall on the holdout, so none replaced the shipped model.