Training the model

The shipped model is fine for scanning. Retrain it to experiment with the detector, to add labelled data of your own, or to fit it to the sites you care about.

pip install "aliens-eye[train]"

#The basic loop

# 1. Scan accounts known to exist, and made-up usernames, into a labelled dataset.
#    Reads the train split only; the holdout is never touched.
aliens_eye train collect --out dataset.csv --negatives 4

# 2. Fit the model and export it.
aliens_eye train fit --data dataset.csv --out model.json

# 3. Score it on sites it never trained on.
aliens_eye selfcheck --split holdout --model model.json --report json

# 4. Use it.
aliens_eye someone --model model.json

Compare the candidate's precision and recall with the shipped model's on the holdout before using it. A model that finds more accounts by calling everything Found is worse, not better.

#What fit does

• Fits a logistic regression with a standard scaler, the scaler fitted inside each fold so no information leaks across folds.
• Groups the folds by site when the dataset records which site each row came from, so a site and its aliases are never split between training and testing.
• Picks the blend weight and the two thresholds from nested out-of-fold predictions, and writes them into model.json with the coefficients.
• --max-fpr (default 0.02) caps the false-positive rate on the calibration rows. It is a target on that data, not a promise for unseen sites.

Selection-time F1 is not an independent test score; only the holdout is.

#Without the network

Build the dataset from a frozen corpus instead of live requests:

aliens_eye train collect --corpus corpus --out grouped-dataset.csv
aliens_eye train fit --data grouped-dataset.csv --out candidate-model.json --max-fpr 0.02

Rows built this way record site, site_group, label_source and split with the features. Older CSVs with features only still load, labelled row-stratified-legacy: fine for fitting, not evidence about unseen sites.

#Keeping the holdout clean

train collect reads the train split. --split holdout or --split all would let the model see the sites it is scored on, so they are refused unless you pass --allow-leakage, for research only. Custom ground truth also has holdout host groups removed.

#Adding your own labels

aliens_eye label results/someone_basic_20261107_120000.json --out labeled.csv

Answer whether each Maybe is a real profile, and the answers are appended to labeled.csv as training rows. See Investigations.

#Training on the extended catalog

scripts/train_expanded.py, in the repository, trains a candidate on pages from the extended catalog. Positives are usernames the upstream databases and the curated train split say exist; negatives are generated. Only sites whose profile page is the request take part: API probes, disabled sites, NSFW sites (unless --include-nsfw) and any site whose host group overlaps the holdout are skipped. Every artefact goes in one --work directory, and no stage overwrites a file.

python scripts/train_expanded.py prepare --work runs/exp1 --upstream-data path/to/data.json --max-sites 600
python scripts/train_expanded.py record  --work runs/exp1        # live network
python scripts/train_expanded.py collect --work runs/exp1        # offline from here
python scripts/train_expanded.py fit     --work runs/exp1
python scripts/train_expanded.py compare --work runs/exp1 --holdout-corpus path/to/holdout-corpus

--upstream-data must be the same data.json the catalog was built from; its SHA-256 is checked. Positives that now answer 404 are dropped. compare scores the shipped and candidate models on the holdout corpus (record one with aliens_eye corpus record --split holdout) and writes comparison.json. The shipped model is never replaced automatically.

This trains the HTML model only. A model for JSON API answers would need its own labelled corpus, and is not faked by feeding JSON to the HTML model.

#Rebuilding the ground truth

aliens_eye eval groundtruth --sherlock data.json --whatsmyname wmn-data.json --dry-run

Imports account lists from Sherlock and WhatsMyName into the train and holdout splits, putting a share of newly imported sites (--holdout-fraction, default 0.3) in the holdout. --dry-run reports the changes without writing them.