Blind evaluation of a domain-similarity model
The output: a decision between two neighbour models grounded in a blind-labelled pair sample, confidence intervals and a reading of where the models disagree, not in which model is fancier.
- Based on
- Updated
- ·
Inputs
-
Two neighbour models that give a domain its nearest neighbours from its ranking footprint: cosine similarity and, as the comparison model, Jaccard overlap, both with the same k.
-
The market snapshots the footprints were computed from, with their snapshot dates.
-
A label guide with three classes: direct competitor, meaningful broad competitor and irrelevant overlap.
-
A reviewer who does not know which model surfaced a pair and uses only public web research.
Procedure
Build the pair pool from the union of the models
The pool is the union of both models' top neighbours, so every pair is labelled once whichever model surfaced it. The pair-to-model mapping is stored in a separate, hidden file.
Stratify the sample before labelling
The sample is stratified by snapshot, footprint size and whether one or both models surfaced the pair. It is fixed and committed blank before the first label.
Blind the review
The reviewer gets only the query domain, the candidate domain and the snapshot date. The model's name, score and rank stay out.
Verify integrity before scoring
The labelled file's ids match the blank one 1:1 in the same order, no row is missing or repeated, the preserved fields are unchanged and the blank file is byte-identical to the committed one. Label counts are recomputed from the file, not trusted from a report.
Score as proportions with confidence intervals
Per model, the precision of meaningful neighbours, the precision of direct competitors and the rate of irrelevant overlap are computed, each with a bootstrap confidence interval. Strata are reported apart, because leakage can be a property of the corpus rather than of the model.
Read the disagreements
The pairs only one model surfaced tell the models' character: the broader one finds broad reference sites, the stricter one specialised players. The most informative cases are written into the decision.
Decide with the intervals in view
Overlapping intervals are a tie, and a tie is a result. The decision is recorded as an architecture decision with the evaluation's id, and the labelled file is kept.
Outputs and how to read them
-
A decision record: the chosen model, the metrics with intervals, the strata and the most informative disagreements.
-
A scoring file from which every figure can be recomputed.
-
The labelled review file and its blank counterpart, both in version control.
Failure modes
-
The reviewer sees the model or the score, and the labels follow it.
-
The sample is chosen after labelling, and the result is chosen with it.
-
Proportions are reported without intervals, and a small gap looks like a win.
-
Broad publishers are labelled competitors, and the corpus's leakage is credited or blamed on the model.
-
The more elaborate model is accepted because it is more elaborate.
Where it is used
In the research layer for search data: the blind review decided between cosine similarity and the Jaccard baseline for two Finnish market snapshots, and the decision is an architecture decision in the project.