Evaluation protocols¶

PaMIR provides two evaluation protocols on the same 19 datasets. Both use the same model signature: predict_fn(X_train, y_train, X_test) → scores.

i.i.d. protocol¶

The conventional train/test split used by OpenML-CC18, TabArena, and most tabular benchmarks. Included for direct comparison with published results from those benchmarks.

from pamir import evaluate_iid
results = evaluate_iid(my_model, n_seeds=5)

The procedure:

Shuffle the dataset with a random seed.
Split into train_frac (default 70%) train and the rest test.
Call predict_fn(X_train, y_train, X_test) once.
Compute ROC AUC on the test set.
Repeat for n_seeds (default 5) shuffles; report mean ± std.

i.i.d. parameters¶

Parameter	Default	Description
`train_frac`	0.7	Fraction of rows used for training
`n_seeds`	5	Number of random shuffles
`max_n`	None	Truncate dataset before splitting

Streaming protocol¶

The production-realistic protocol. Applications arrive in a stream, outcomes are revealed only after a maturation delay, and the model is refitted as labels accumulate.

from pamir import evaluate
results = evaluate(my_model, lag=1000)

Why a streaming protocol?¶

Credit-risk models in production face three constraints that the i.i.d. protocol does not capture:

Zero cold-start labels. A new portfolio has no resolved outcomes on day one. The model must score from the first application.
Label-maturation delay. A loan originated today reveals its outcome (default or repayment) months to years later. The model trains only on stale labels.
Pre-commitment. A score assigned to an application is final — there is no post-hoc adjustment once the outcome is known.

Streaming specification¶

for t in 0, 1, ..., N-1:
    resolved_end = max(0, t + 1 - lag)

    if new_default_count_since_last_refit >= k_refit:
        X_ctx = features[0 : resolved_end]
        y_ctx = outcomes[0 : resolved_end]
        X_rem = features[resolved_end : N]

        predictions[resolved_end : N] = model(X_ctx, y_ctx, X_rem)

    metric = roc_auc(outcomes[:resolved_end], predictions[:resolved_end])

Streaming parameters¶

Parameter	Default	Description
`lag`	1000	Label-maturation delay in stream positions
`k_refit`	10	Refit after every k newly resolved defaults
`max_n`	20000	Truncate stream to at most N rows
`min_defaults`	3	Minimum resolved defaults before first refit
`on_error`	`"warn"`	What to do when `predict_fn` raises

These defaults are the reference setting¶

lag, k_refit and max_n change the score, not just the runtime. A smaller k_refit means more refits and therefore fresher predictions, so it reads higher: the same model on bondora scores 0.9090 at k_refit=10 and 0.8625 at k_refit=80.

Two runs are comparable only when all three parameters match. The defaults above are the reference setting; report them alongside any number, and state any deviation explicitly.

k_refit also drives cost — every refit is a full fit plus a scoring pass over the entire remaining stream. At k_refit=10, bondora truncated to 5,000 rows takes 126 refits. A full fleet run at the defaults with a tree ensemble is hours, not minutes.

What the model receives¶

At each refit point the model’s predict_fn is called with:

X_train — features of rows [0, resolved_end), all with known outcomes.
y_train — binary outcomes for those rows.
X_test — features of rows [resolved_end, N), outcomes unknown.

The model must return a score array of length len(X_test). Higher scores must indicate higher probability of default.

What the model must NOT do¶

No access to future labels. The evaluator enforces this structurally: y_train contains only resolved rows.
No access to test-set statistics. The model receives X_test for scoring but must not compute target statistics on it.
No row-level updates. Predictions are fixed from one refit to the next. A model that internally updates per-row is violating pre-commitment.

Metric¶

Both protocols report ROC AUC (higher = better ranking of defaults above non-defaults).

i.i.d.: AUC on the held-out test set, averaged over seeds.
Streaming: cumulative AUC on all resolved rows, using pre-committed predictions.

The fleet metric in both cases is the unweighted mean across all 19 datasets — and it is defined only when the model scored all 19.

Coverage is part of the result¶

If predict_fn raises, the affected rows go unscored and that dataset yields no AUC. Averaging over what is left would report a number the model selected by failing, and a model that crashes on the hard datasets can read higher than one that survives all of them.

pamir.fleet_summary therefore reports auc_mean only for a complete run, and puts the partial average in auc_mean_scored_only, which is diagnostic and not comparable across models. Every result frame carries n_calls and n_failures; evaluate_one additionally returns the distinct error messages under errors.

A submission must report full coverage, or it is not a fleet result.