Getting started¶
Installation¶
PaMIR is not on PyPI yet, so install from a clone:
git clone https://github.com/zypl-ai/pamir-credit
cd pamir-credit
pip install -e ".[data]" # add [dev] for pytest, [synthetic] for pamir.synthetic
The [data] extra installs the fetch dependencies (kagglehub,
huggingface-hub, platformdirs). PaMIR does not distribute the datasets
itself — it ships the catalog, the harmonization recipe, and the code that
fetches each dataset from its original source and harmonizes it locally.
On macOS/arm64, import xgboost before anything that pulls in torch (sdv
does) — the reverse order segfaults. import pamir.synthetic handles this
for you.
Load a dataset¶
from pamir import load_catalog, load_dataset
# Browse all 19 datasets (metadata only — no download)
catalog = load_catalog()
print(catalog[["name", "rows", "DR", "geography", "license"]])
# Load one — fetched from its original source and harmonized on first use,
# then cached locally (default: platformdirs user cache; override with $PAMIR_CACHE)
X, y, meta = load_dataset("taiwan")
print(f"{meta['name']}: {meta['n_rows']:,} rows, DR={meta['DR']:.1%}")
Most datasets are hosted on Kaggle, so fetching them needs Kaggle credentials
(KAGGLE_USERNAME + KAGGLE_KEY, or ~/.kaggle/kaggle.json). UCI, GitHub and
HuggingFace sources need no credentials. You can pre-fetch explicitly:
import pamir
pamir.download("taiwan") # fetch + harmonize + cache one dataset
No Kaggle account? Start with the credential-free set¶
7 datasets come from UCI / GitHub / HuggingFace and need no credentials — they work on a fresh install with zero setup:
pamir.open_datasets() # ['gastonstat', 'lc_clean', 'pakdd', 'poland_1yr', ...]
pamir.download_open() # fetch all of them
Command line¶
pamir list --open # datasets that need no credentials
pamir info taiwan # metadata + recipe for one dataset
pamir download poland_1yr # fetch + harmonize into the cache
pamir cache # where the cache lives
Define your model¶
A model comes in one of two forms, and the reference baselines ship in both:
fit_fn(X_train, y_train) → scorer, withscorer(X_rows) → scores— the form the streaming protocol calls: a refit trains on the matured labels, and the scorer scores rows as they arrive;predict_fn(X_train, y_train, X_test) → scores— the form the i.i.d. protocol calls.
Scores have one entry per row and higher means more likely to default. Most of
the datasets carry object or bool columns, so encode them rather than
dropping them, using the training rows’ levels only:
from pamir.baselines import apply_encoder, fit_encoder
def my_fit(X_train, y_train):
from sklearn.ensemble import HistGradientBoostingClassifier
levels = fit_encoder(X_train) # levels from the training rows
clf = HistGradientBoostingClassifier(max_iter=150)
clf.fit(apply_encoder(levels, X_train), y_train)
return lambda X: clf.predict_proba(apply_encoder(levels, X))[:, 1]
def my_model(X_train, y_train, X_test): # the same model as a predict_fn
return my_fit(X_train, y_train)(X_test)
A scorer must score each row on its own: the streaming protocol checks this on every batch it scores (see Evaluation protocols).
Two baselines ship ready-made if you only need a reference point:
from pamir import logistic_fit, gbdt_fit # fit_fn form
from pamir import logistic_baseline, gbdt_baseline # predict_fn form
i.i.d. evaluation (conventional)¶
Standard random train/test split, repeated across multiple seeds:
from pamir import evaluate_iid, evaluate_iid_one
# One dataset
result = evaluate_iid_one(my_model, "taiwan", n_seeds=5)
print(f"AUC: {result['auc_mean']:.4f} ± {result['auc_std']:.4f}")
# Full fleet
results = evaluate_iid(my_model, n_seeds=5)
Streaming evaluation (with label delay)¶
Applications arrive one at a time, each is scored once on arrival, outcomes are revealed only after a maturation delay, and the model is refitted as labels arrive:
from pamir import evaluate, evaluate_one
# One dataset
result = evaluate_one(my_fit, "taiwan")
print(f"AUC: {result['auc_final']:.4f} ({result['n_refits']} refits, "
f"lag {result['lag_effective']})")
print({k: v for k, v in result.items() if k.startswith("auc_budget_")})
# Full fleet
results = evaluate(my_fit)
Compare both protocols¶
from pamir import evaluate, evaluate_iid
results_iid = evaluate_iid(my_model, n_seeds=5, verbose=False)
results_stream = evaluate(my_fit, verbose=False)
import pandas as pd
comparison = results_iid[["dataset", "auc_mean"]].rename(columns={"auc_mean": "iid"})
comparison = comparison.merge(
results_stream[["dataset", "auc_final"]].rename(columns={"auc_final": "streaming"}),
on="dataset")
print(comparison.to_string(index=False))
By default the i.i.d. protocol uses the full table and the streaming protocol
the first 20,000 rows; pass the same max_n to both for a like-for-like
comparison.
Check the summary before quoting a number¶
If the model raises, the affected rows go unscored. Both protocols record
every failure, and fleet_summary withholds the headline mean unless the run
is valid:
from pamir import evaluate, fleet_summary, gbdt_fit
results = evaluate(gbdt_fit)
summary = fleet_summary(results)
summary["complete"] # False if anything failed
summary["contract_ok"] # False if a table is not the benchmark table
summary["row_independent"] # False if a scorer depended on its batch
summary["auc_mean"] # None unless all three hold
summary["failed_datasets"] # which ones; results["errors"] says why
An average over the datasets a model survived is selected by the model’s own
failures, and can read higher than a working model’s average over all of
them — which is why it is never reported as auc_mean.
While developing, pass on_error="raise" to get the traceback instead of a
warning.
Report your protocol parameters¶
mode, lag, max_lag_frac, k_refit and max_n change the score, not
just the runtime. The package defaults — mode="arrival", lag=1000,
max_lag_frac=0.2, k_refit=10, max_n=20000 — are the reference setting.
Report them with any number, and state any deviation.
Cost scales with the refit count: every refit is a fit plus the scoring of the
rows that arrived since the last one. Raise k_refit (and say so) to trade
resolution for time.