PaMIR — Public Arrival-ordered Measurement for Inference in Risk¶

PaMIR¶

Public Arrival-ordered Measurement for Inference in Risk

The name doubles as the Pamir mountains — the “Roof of the World”; the datasets themselves span Poland, Taiwan, Brazil, Estonia, the US and beyond.

A benchmark of 19 public credit-default datasets (1.2 million rows, 9 named countries, default rates 3%–41%) with two evaluation protocols: a streaming protocol that respects label-maturation delay (the standard condition under which credit-risk models operate in production) and a conventional i.i.d. train/test split for comparison with other tabular benchmarks.

Why another benchmark?¶

Tabular ML benchmarks (OpenML-CC18, TabArena, MultiTab) evaluate models on i.i.d. train/test splits. Credit-risk models in production operate under a different contract:

Applications arrive in a stream, not a batch.
Outcomes are revealed only after a maturation delay (months to years).
The model must score every application before its outcome is known.
A model deployed on day one has zero labels.

No existing tabular benchmark evaluates under streaming conditions with label delay. PaMIR provides both protocols — the streaming evaluation that reflects production reality, and the conventional i.i.d. split for comparison with other tabular benchmarks — on the same 19 credit-risk datasets.

Comparison with existing benchmarks¶

Benchmark	Datasets	Domain	Protocol	Credit-specific
OpenML-CC18	72	General	i.i.d. split	2 datasets
TabArena	51	General	i.i.d. split	2–3 datasets
MultiTab	196	General	i.i.d. split	~5 datasets
Lessmann et al. (2015)	8	Credit	i.i.d. split	Yes
PaMIR	19	Credit	Streaming + lag, i.i.d.	Yes

Synthetic augmentation¶

pamir.synthetic mixes real and synthetic training rows in stated proportions across several generators at once, fits every generator inside the cross-validation fold, and reports both the AUC delta and a full fidelity battery:

from sdv.single_table import GaussianCopulaSynthesizer

from pamir import load_dataset
from pamir.synthetic import CrossValidatedAugmentation, SDVAdapter, SyntheticMixer

mixer = SyntheticMixer(
    generators={"copula": SDVAdapter(GaussianCopulaSynthesizer, name="copula")},
    synthetic_share=0.5,                   # 50% synthetic / 50% real
)

X, y, _ = load_dataset("south_german")
result = CrossValidatedAugmentation(mixer, n_splits=5, seed=42).run(X, y)

result.summary()             # baseline vs augmented OOF AUC, and the delta
result.leakage_frame()       # the probes that say whether to believe that delta
result.fidelity_headline()   # the few fidelity numbers worth reading first

SDVAdapter takes any SDV synthesizer and needs only the synthetic extra, so it is the path to start on. Mixing several generators, including ones that live in their own repos, works the same way:

from pamir.synthetic import ZganAdapter, ZedgeAdapter

mixer = SyntheticMixer(
    generators={"zgan": ZganAdapter(repo_path="~/repos/zgan"),
                "zedge": ZedgeAdapter(repo_path="~/repos/zgn_latdiff", device="cuda:0")},
    weights={"zgan": 0.5, "zedge": 0.5},   # the synthetic part splits evenly
    synthetic_share=0.7,                   # 70% synthetic / 30% real
)

Generators never see a held-out row: the split happens first, each fold gets a freshly reset generator, the test split is never augmented, and pre-generated pools must be registered per fold. The exact-overlap probes that verify this run even with fidelity=False, and hash rows under the real table’s schema, so an engine that returns a column as floats — or as text — cannot slip a reproduced row past them. See the synthetic-augmentation guide in docs/synthetic.md.

Installation¶

PaMIR is not on PyPI yet, so install from a clone:

git clone https://github.com/zypl-ai/pamir-credit
cd pamir-credit
pip install -e ".[data]"

PaMIR ships the catalog, the harmonization recipe and the evaluation code — not the datasets. Each dataset is fetched from its original source and harmonized locally on first use (then cached). The [data] extra installs the fetch dependencies (kagglehub, huggingface-hub, platformdirs). Kaggle sources need Kaggle credentials; UCI/GitHub/HuggingFace sources do not. Start with pamir.download_open() — 7 datasets that need no credentials at all.

Core dependencies: numpy, pandas, scikit-learn, scipy, pyarrow. No GPU required.

Optional extras:

pip install -e ".[dev]"        # pytest
pip install -e ".[synthetic]"  # sdv, sdmetrics, xgboost — for pamir.synthetic

Once published, pip install pamir-credit will be the one-line path.

A note on import order (macOS/arm64)¶

sdv imports torch, and torch and xgboost each ship their own OpenMP runtime. On macOS/arm64, loading torch first makes the next xgboost.train() die with a bare Segmentation fault: 11 and no traceback:

import sdv, xgboost   # xgboost.train() later segfaults
import xgboost, sdv   # fine

import pamir.synthetic loads xgboost for you before anything touches sdv, so importing PaMIR first is enough. If your own script imports sdv directly, put import xgboost above it.

Quick start¶

from pamir import load_catalog, load_dataset, evaluate, fleet_summary
from pamir import gbdt_baseline          # runs on all 19 datasets as-is

# Browse all 19 datasets
catalog = load_catalog()
print(catalog[["name", "rows", "DR", "geography", "license"]])

# Load a single dataset
X, y, meta = load_dataset("gmsc")
print(f"{meta['name']}: {meta['n_rows']} rows, {meta['n_defaults']} defaults")

# Evaluate on the full fleet (streaming protocol)
results = evaluate(gbdt_baseline, lag=1000)
fleet_summary(results)      # headline AUC — and how much of the fleet it covers

# Or conventional i.i.d. evaluation
from pamir import evaluate_iid
results_iid = evaluate_iid(gbdt_baseline, n_seeds=5)

Writing your own model¶

A model is one function, predict_fn(X_train, y_train, X_test) -> scores, used identically by both protocols. X arrives with raw dtypes — most of the datasets carry object or bool columns — so encode them rather than dropping them:

from pamir import encode_features

def my_model(X_train, y_train, X_test):
    from sklearn.linear_model import LogisticRegression
    train, test = encode_features(X_train, X_test)   # shared ordinal codes
    clf = LogisticRegression(max_iter=1000)
    clf.fit(train.fillna(0), y_train)
    return clf.predict_proba(test.fillna(0))[:, 1]

Encoding across train and test together is legal: both protocols hand the model the features of the rows it must score and withhold only their labels.

Always check coverage¶

If predict_fn raises, the affected rows go unscored. evaluate records every failure rather than discarding it, and fleet_summary withholds the headline mean unless the model scored the whole fleet — because an average over the datasets a model survived is not comparable with one over all 19:

results = evaluate(my_model, lag=1000)
summary = fleet_summary(results)

summary["complete"]              # False if anything failed
summary["auc_mean"]              # None unless complete
summary["failed_datasets"]       # which ones, and results["errors"] says why

While developing, pass on_error="raise" to get the traceback instead.

Report your protocol parameters¶

lag, k_refit and max_n change the score, not just the runtime: the same model on bondora scores 0.9090 at k_refit=10 and 0.8625 at k_refit=80. Two runs are comparable only when all three match. The package defaults — lag=1000, k_refit=10, max_n=20000 — are the reference setting; report them alongside any number, and state any deviation.

The streaming protocol¶

The protocol simulates a lender deploying a model on an arriving stream of loan applications.

        flowchart LR
    S["Applications arrive in file order<br/>a[1] a[2] … a[N]"]
    S --> R["<b>Resolved</b><br/>a[1 .. t−lag]<br/>features + matured outcomes"]
    S --> P["<b>Maturing</b><br/>a[t−lag+1 .. N]<br/>features only"]
    R -->|"train on stale labels<br/>refit every k defaults"| M(["Model"])
    M -->|"score — pre-committed"| P
    P -.->|"outcome revealed at t + lag"| R

The rules:

File order = arrival order. Applications arrive in the order they appear in the parquet file (a shuffled database export — no calendar look-ahead is possible).
Label-maturation delay. The outcome of application at position t is revealed only at position t + lag (default: lag=1000). During the first lag positions, the model has zero labels.
Refit cadence. The model is refitted every k newly resolved defaults (default: k_refit=10). This means the model is updated on a schedule driven by information arrival, not wall time.
Pre-committed predictions. At each refit, the model must score the entire remaining stream at once. Predictions are frozen until the next refit — no row-by-row updates.
Cumulative AUC. The metric is ROC AUC computed on all rows whose outcome has been resolved, using their pre-committed predictions.

Why this protocol?¶

Lag is real. A loan originated today defaults (or not) 6–24 months later. Any evaluation that hands all labels to the model at once is measuring a condition that never exists in production.
Pre-commitment is real. In production, a model scores an application the moment it arrives. It cannot wait to see how the rest of the stream turns out.
Cadence is real. Retraining happens when enough new information has arrived, not on a fixed schedule.

Datasets¶

All 19 datasets (1,237,550 rows, 281,766 defaults, DR 3.2–40.9%):

id	name	rows	DR	geography	product	license
bankruptcy	Taiwanese Bankruptcy Prediction	6,819	3.2%	Taiwan	Corporate	© authors
bondora	Bondora P2P Lending	266,482	40.9%	Estonia, Finland, Spain	P2P consumer	CC0
conorsully	Conor Sully Credit Score	1,000	28.4%	Synthetic / educational	Consumer	CC0
dish	Automobile Loan Default	121,856	8.1%	Unspecified	Vehicle finance	CC0
gastonstat	Gaston Sanchez Credit Scoring	4,454	28.1%	Unknown	Consumer	None
gmsc	Give Me Some Credit	150,000	6.7%	USA	Consumer revolving + installment	Unknown
laotse	Laotse Credit Risk	32,581	21.8%	Unspecified	Consumer	CC0
lc_clean	Lending Club (cleaned, 2007–2014)	150,000	20.2%	USA	P2P consumer	None
lc_my	Lending Club (Malaysian variant)	100,000	22.6%	Unspecified	Consumer	Unknown
lc_small	Lending Club (small, Thera-Bank)	9,578	16.0%	USA	Consumer	ODbL
lt_vehicle	L&T Vehicle Loan Default	233,154	21.7%	India	Vehicle finance	Other
pakdd	PAKDD 2010 Credit Data	50,000	26.1%	Brazil	Consumer	None
poland_1yr	Polish Companies Bankruptcy (1-year horizon)	7,027	3.9%	Poland	Corporate	CC-BY-4.0
poland_3yr	Polish Companies Bankruptcy (3-year horizon)	10,503	4.7%	Poland	Corporate	CC-BY-4.0
poland_5yr	Polish Companies Bankruptcy (5-year horizon)	5,910	6.9%	Poland	Corporate	CC-BY-4.0
prosper	Prosper Marketplace Loans	55,084	30.9%	USA	P2P consumer	CC0
sba	U.S. SBA Loan Defaults	2,102	32.6%	USA	Small business	CC0
south_german	South German Credit (corrected)	1,000	30.0%	Germany	Consumer	CC-BY-4.0
taiwan	Taiwan Credit Card Default	30,000	22.1%	Taiwan	Credit card	CC0

Inclusion criteria: a flat table (or reducible to one by the recipe), binary default target, publicly downloadable, ≥1,000 rows, ≥3% default rate.

Data provenance and harmonization¶

PaMIR distributes no data. pamir.download(id) fetches the raw file from the dataset’s original source and harmonizes it locally with a reproducible recipe (pamir.harmonize), so every user reconstructs the identical table. The recipe per dataset (in the catalog’s download / harmonize fields):

Target binarization — a numeric parse or a dataset-specific rule; stored as __target__ (0 = non-default, 1 = default).
Rescaling (lc_my): Credit Score inflated 10x on defaulter rows is divided back down.
Post-outcome leakage removal (semantic cut) — every column the elicitation marked day_zero_available = false (not known to the lender at decision time) is dropped, plus dataset-specific extra drops (e.g. sba: Term, RealEstate, daysterm). This replaces the earlier AUC > 0.95 rule, which let sets of individually-weak columns leak the outcome together.
Column hygiene — drop id / date / constant / surrogate-key columns.
Row shuffle — fixed seed; file order is not an origination sequence.

The harmonized result is validated against pamir/data/expected.json (columns, row count, default rate). pamir.dataset_info(id) returns the full recipe.

API reference¶

`pamir.load_catalog() → DataFrame`¶

Returns a DataFrame with one row per dataset. Index is id. Columns include name, rows, features, defaults, DR, source, source_url, license, geography, product, target_definition.

`pamir.list_datasets() → list[str]`¶

Returns the sorted list of all 19 dataset ids.

`pamir.dataset_info(dataset_id) → dict`¶

Returns the full metadata dictionary for one dataset. Raises KeyError if the id is not in the catalog.

`pamir.download(dataset_id, force=False, quiet=False) → Path`¶

Fetch one dataset from its original source, harmonize it, cache it, and validate against expected.json. Called automatically by load_dataset on first use. Cache location: platformdirs user cache, or $PAMIR_CACHE.

`pamir.load_dataset(dataset_id, max_rows=None, auto_download=True) → (X, y, meta)`¶

Load a single dataset from the local cache (downloading it first if needed; set auto_download=False to require an explicit download).

X: DataFrame of features exactly as stored — int64, float64, bool and object all occur, and most of the datasets carry at least one non-numeric column. Nothing is encoded or imputed for you.
y: numpy array of int (0 = non-default, 1 = default).
meta: dict with dataset metadata plus n_rows, n_defaults, n_features. notes is present only on the datasets that required harmonization, so read it with .get().

The __target__ column is separated into y and not present in X.

`pamir.encode_features(X_train, X_test) → (train, test)`¶

Ordinal-encode the non-numeric columns of both frames against one shared level set, so a category maps to the same code in each. Returns new float frames; the inputs are not modified and numeric NaN is preserved.

`pamir.logistic_baseline(X_train, y_train, X_test) → scores`¶

`pamir.gbdt_baseline(X_train, y_train, X_test, max_iter=150, ...) → scores`¶

Reference baselines that run on all 19 datasets without edits. Both use encode_features; neither needs an optional dependency.

`pamir.evaluate(predict_fn, datasets=None, lag=1000, k_refit=10, max_n=20000, verbose=True, on_error="warn") → DataFrame`¶

Run the streaming protocol on all (or a subset of) datasets.

predict_fn signature: predict_fn(X_train, y_train, X_test) → scores. Called once per refit point. X_train contains rows [0, t−lag) with resolved outcomes; X_test contains the remaining unscored stream. Must return one score per row of X_test (higher = more likely to default); a wrong-length return is counted as a failure.

on_error — "warn" (default) records the failure, warns once per dataset and carries on; "raise" propagates the traceback; "ignore" records it silently. Failures are counted under every policy.

Returns a DataFrame with columns: dataset, n_rows, n_defaults, DR, n_refits, n_calls, n_failures, auc_final. Pass it to pamir.fleet_summary — a mean taken by hand over this frame silently excludes the datasets the model failed on.

`pamir.evaluate_one(predict_fn, dataset_id, ...) → dict`¶

Same as evaluate but for a single dataset. Returns a dict with two additional keys: refit_points (per-refit cumulative AUC) and errors (the distinct failure messages, up to five).

`pamir.evaluate_iid(predict_fn, datasets=None, train_frac=0.7, max_n=None, n_seeds=5, verbose=True, on_error="warn") → DataFrame`¶

Run a conventional i.i.d. train/test split evaluation on the PaMIR fleet. Included for comparison with other tabular benchmarks. Each dataset is shuffled n_seeds times; the reported AUC is the mean across seeds.

Returns a DataFrame with columns: dataset, n_rows, n_defaults, DR, train_frac, n_seeds, n_calls, n_failures, auc_mean, auc_std.

`pamir.evaluate_iid_one(predict_fn, dataset_id, ...) → dict`¶

Same as evaluate_iid but for a single dataset. Returns a dict with additional aucs (list of per-seed AUCs) and errors keys.

`pamir.fleet_summary(results) → dict`¶

Summarize a frame from either protocol. Keys:

key	meaning
`n_datasets`, `n_scored`, `coverage`	how much of the fleet was scored
`complete`	`True` only when every dataset produced an AUC
`auc_mean`, `gini_mean`	the headline numbers — `None` unless `complete`
`auc_mean_scored_only`	the partial average; diagnostic only, not comparable
`n_failures`, `failed_datasets`	what went wrong and where

The headline mean is withheld on a partial run by design: an average over the datasets a model survived is chosen by the model’s own failures, and can read higher than a working model’s average over all 19.

Reference baselines¶

Two baselines ship with the package. Both run on all 19 datasets without edits, and neither needs an optional dependency:

from pamir import evaluate, fleet_summary, logistic_baseline, gbdt_baseline

fleet_summary(evaluate(logistic_baseline, lag=1000))
fleet_summary(evaluate(gbdt_baseline, lag=1000))

To use XGBoost instead, convert the object columns to pandas category dtype first — enable_categorical=True does not accept object:

def xgboost_model(X_train, y_train, X_test):
    import xgboost as xgb
    from pamir import encode_features

    train, test = encode_features(X_train, X_test)   # all numeric
    dtrain = xgb.DMatrix(train, label=y_train)
    dtest = xgb.DMatrix(test)
    bst = xgb.train({"objective": "binary:logistic", "max_depth": 6,
                     "eta": 0.1, "eval_metric": "auc"}, dtrain, 200)
    return bst.predict(dtest)

Cost¶

Every refit is a full fit plus a scoring pass over the entire remaining stream. At the default k_refit=10, bondora truncated to 5,000 rows takes 126 refits — about 73s with a 60-tree HistGradientBoosting — and the default max_n is 20,000 across all 19 datasets. A full fleet run with a tree ensemble is hours, not minutes. logistic_baseline is the cheap reference point; raise k_refit (and report it) to trade resolution for time.

Both streaming and i.i.d. protocols are available — see pamir.evaluate_iid for the conventional train/test split evaluation.

Documentation¶

Sphinx docs (local)¶

Build and view the full documentation:

pip install -e ".[dev]"
pip install sphinx furo sphinx-copybutton sphinx-autodoc-typehints sphinx-llms-txt myst-parser
sphinx-build -b html docs docs/_build/html
open docs/_build/html/index.html

Features: Furo theme with dark/light toggle, copy button on all code blocks, autodoc-generated API reference from docstrings.

LLM-readable docs¶

The Sphinx build automatically generates two files for LLM consumption:

llms.txt — structured index with links to each documentation page.
llms-full.txt — the entire documentation concatenated into a single plain-text file (~800 lines).

Any LLM agent can download yoursite.com/llms.txt to understand the full PaMIR API and protocol in one request.

HuggingFace dataset card¶

The huggingface_card/README.md is the dataset card for the HuggingFace Hub repo. It contains YAML metadata (tags, configs, license) that makes the dataset discoverable through HF search filters, plus a standalone description of the benchmark and its 19 datasets.

Running tests¶

pip install -e ".[dev]"
pytest tests/ -v

The test suite covers: catalog integrity (19 datasets, all metadata fields, DR/feature consistency), harmonized loading, leakage verification (named columns + AUC scan), streaming protocol invariants (no future leakage, lag enforcement, monotonic train growth, pre-commitment), i.i.d. protocol, failure accounting (a partial run yields no fleet mean), the reference baselines on all 19 datasets, and data quality (no duplicates, no constant features). The pamir.synthetic augmentation layer adds its own suite — fold-safe cross-validation, the exact-overlap leakage probes, and the fidelity battery. Data-dependent tests skip cleanly when a dataset is not cached (e.g. CI without Kaggle credentials); catalog and protocol tests always run.

License¶

The PaMIR package code is released under the Apache 2.0 license — see the LICENSE file at the repository root.

PaMIR does not redistribute the datasets — it fetches each one from its original source at the user’s request. Each dataset remains under its own license and terms, as listed in the catalog (pamir.dataset_info(id)["license"]); when you use a dataset you are bound by those terms and are responsible for citing its original authors.

Citation¶

If you use PaMIR in your research, please cite:

@software{pamir2026,
  title  = {PaMIR: Public Arrival-ordered Measurement for Inference in Risk},
  author = {zypl.ai},
  year   = {2026},
  url    = {https://github.com/zypl-ai/pamir-credit},
}

Contents