Getting started

Installation

PaMIR is not on PyPI yet, so install from a clone:

git clone https://github.com/zypl-ai/pamir-credit
cd pamir-credit
pip install -e ".[data]"    # add [dev] for pytest, [synthetic] for pamir.synthetic

The [data] extra installs the fetch dependencies (kagglehub, huggingface-hub, platformdirs). PaMIR does not distribute the datasets itself — it ships the catalog, the harmonization recipe, and the code that fetches each dataset from its original source and harmonizes it locally.

On macOS/arm64, import xgboost before anything that pulls in torch (sdv does) — the reverse order segfaults. import pamir.synthetic handles this for you.

Load a dataset

from pamir import load_catalog, load_dataset

# Browse all 19 datasets (metadata only — no download)
catalog = load_catalog()
print(catalog[["name", "rows", "DR", "geography", "license"]])

# Load one — fetched from its original source and harmonized on first use,
# then cached locally (default: platformdirs user cache; override with $PAMIR_CACHE)
X, y, meta = load_dataset("taiwan")
print(f"{meta['name']}: {meta['n_rows']:,} rows, DR={meta['DR']:.1%}")

Most datasets are hosted on Kaggle, so fetching them needs Kaggle credentials (KAGGLE_USERNAME + KAGGLE_KEY, or ~/.kaggle/kaggle.json). UCI, GitHub and HuggingFace sources need no credentials. You can pre-fetch explicitly:

import pamir
pamir.download("taiwan")          # fetch + harmonize + cache one dataset

No Kaggle account? Start with the credential-free set

7 datasets come from UCI / GitHub / HuggingFace and need no credentials — they work on a fresh install with zero setup:

pamir.open_datasets()   # ['gastonstat', 'lc_clean', 'pakdd', 'poland_1yr', ...]
pamir.download_open()   # fetch all of them

Command line

pamir list --open           # datasets that need no credentials
pamir info taiwan           # metadata + recipe for one dataset
pamir download poland_1yr   # fetch + harmonize into the cache
pamir cache                 # where the cache lives

Define your model

Both protocols use the same model signature: predict_fn(X_train, y_train, X_test) → scores, where scores has one entry per row of X_test and higher means more likely to default.

Most of the datasets carry object or bool columns, so encode them rather than dropping them — pamir.encode_features is the glue:

from pamir import encode_features

def my_model(X_train, y_train, X_test):
    from sklearn.ensemble import HistGradientBoostingClassifier
    train, test = encode_features(X_train, X_test)   # shared ordinal codes
    clf = HistGradientBoostingClassifier(max_iter=150)
    clf.fit(train, y_train)
    return clf.predict_proba(test)[:, 1]

Two baselines ship ready-made if you only need a reference point:

from pamir import logistic_baseline, gbdt_baseline

i.i.d. evaluation (conventional)

Standard random train/test split, repeated across multiple seeds:

from pamir import evaluate_iid, evaluate_iid_one

# One dataset
result = evaluate_iid_one(my_model, "taiwan", n_seeds=5)
print(f"AUC: {result['auc_mean']:.4f} ± {result['auc_std']:.4f}")

# Full fleet (all 19 datasets)
results = evaluate_iid(my_model, n_seeds=5)

Streaming evaluation (with label delay)

Simulates production: applications arrive in order, outcomes are revealed only after a maturation delay, and the model is refitted as labels arrive.

from pamir import evaluate, evaluate_one

# One dataset
result = evaluate_one(my_model, "taiwan", lag=1000)
print(f"AUC: {result['auc_final']:.4f} ({result['n_refits']} refits)")

# Full fleet
results = evaluate(my_model, lag=1000)

Compare both protocols

from pamir import evaluate, evaluate_iid

results_iid = evaluate_iid(my_model, n_seeds=5, verbose=False)
results_stream = evaluate(my_model, lag=1000, verbose=False)

# Merge and compare
import pandas as pd
comparison = results_iid[["dataset", "auc_mean"]].rename(columns={"auc_mean": "iid"})
comparison = comparison.merge(
    results_stream[["dataset", "auc_final"]].rename(columns={"auc_final": "streaming"}),
    on="dataset")
print(comparison.to_string(index=False))

Check coverage before quoting a number

If predict_fn raises, the affected rows go unscored. Both protocols record every failure, and fleet_summary withholds the headline mean unless the model scored the whole fleet:

from pamir import evaluate, fleet_summary, gbdt_baseline

results = evaluate(gbdt_baseline, lag=1000)
summary = fleet_summary(results)

summary["complete"]        # False if anything failed
summary["auc_mean"]        # None unless complete
summary["failed_datasets"] # which ones; results["errors"] says why

An average over the datasets a model survived is selected by the model’s own failures, and can read higher than a working model’s average over all 19 — which is why it is never reported as auc_mean.

While developing, pass on_error="raise" to get the traceback instead of a warning.

Report your protocol parameters

lag, k_refit and max_n change the score, not just the runtime: the same model on bondora scores 0.9090 at k_refit=10 and 0.8625 at k_refit=80. The package defaults — lag=1000, k_refit=10, max_n=20000 — are the reference setting. Report them with any number, and state any deviation.

Cost scales with the refit count: every refit is a full fit plus a scoring pass over the entire remaining stream. A full fleet run with a tree ensemble is hours, not minutes; raise k_refit (and say so) to trade resolution for time.