Contributing¶
Adding a dataset¶
PaMIR ships no data — a dataset is added as a recipe that fetches it from its original source and harmonizes it locally. You never commit a data file.
Check it meets the inclusion criteria: binary default target, publicly downloadable, ≥1,000 rows, ≥3% default rate, and either a flat table or one the recipe can reduce to a flat table (a documented join/aggregation is fine — see
bondora).Add an entry to
pamir/data/catalog.jsonwith the metadata (name,rows,defaults,DR,features,geography,product,target_definition,license,source,source_url, and — where the license or academic origin requires it —attribution), plus:download:{kind, locator, file, sep, encoding, …}— where the raw file comes from.kindis one of:kaggle(slug),uci_zip/github_zip(a zip URL; a member is extracted, one level of nesting supported),github_raw(a direct file URL), orhf(a HuggingFace dataset repo). For a headerless raw file add"header": "none"+"columns": [...]; for ARFF add"format": "arff". Pin the snapshot: a"version"for Kaggle, a"revision"(commit hash) for Hugging Face, a commit hash instead of a branch name in a GitHub URL.sourcenames where the data originate; what the downloader fetches is derived fromdownload(fetched_from).harmonize:{target, target_rule, day_zero_drop, extra_drop, drop_prefix, shuffle_seed, …}.day_zero_dropis the list of post-outcome columns (the leakage-free semantic cut);extra_drop/drop_prefixremove further leaks or identifiers;shuffle_seedfixes the row order.
If it needs a bespoke read, target rule, or rescale, add the branch in
pamir/harmonize.pyand reference it viatarget_rule(seebondora,prosper,south_german).Run
pamir.download("<id>", strict=False), confirm it harmonizes cleanly, then freeze the result intopamir/data/expected.json:columns(name → type),n_cat,n_features,n_rows,n_defaults,DR,raw_sha256(digest of the raw file) andtarget_sha256(digest of the target vector in stored order;pamir.contract.target_sha256). This is the data contract: every later download and every load is checked against it exactly, and a mismatch is an error.Cite the source: put the paper / repository that introduced or used the dataset in the catalog
attributionfield and in Licenses & attribution.pamir/data/*.parquetis git-ignored — never commit the dataset itself.Run the full test suite with the new dataset cached and paste the output into the merge request. Continuous integration downloads only the credential-free datasets, so for a Kaggle-hosted dataset the leakage guards run on your machine and nowhere else.
Add an entry to
CHANGELOG.mdunder the next version (dataset added, removed, or recipe changed); maintainers bump the version andCITATION.cffat release.Open a merge request.
Running tests¶
pip install -e ".[dev,data]"
pytest tests/ -v
Tests that need data skip cleanly when a dataset is not cached (e.g. CI without Kaggle credentials); catalog, recipe and protocol tests always run. The suite checks:
Catalog & recipe integrity — the dataset list matches
catalog.json, all metadata fields, every dataset has a validdownload/harmonizespec,expected.jsonagrees with the catalog, counts innotesmatch the drop lists, and the version agrees acrosspyproject.toml,CITATION.cff,docs/conf.pyandCHANGELOG.md.Data contract — exact validation, strict refusal on download and on load, provenance record in the cached parquet.
Harmonizer/downloader internals — synthetic end-to-end harmonize, cache location, pinned Kaggle handles.
Leakage guards (from the audit) — no single-column AUC outside (0.05, 0.95), rows shuffled (|ρ(position, target)| < 0.1), no
day_zero_dropcolumn survives, no surrogate-key column, no divisibility leak, no missingness leak, no pure level, raw-file scan; each guard has a positive control that reconstructs its leak and asserts that the guard fires.Streaming protocol (synthetic streams, no data needed) — in arrival mode a scorer sees only rows that arrived after its refit and labels that had matured, every score is final, the delay cap, the row-independence probe, failure accounting; the refresh mode reproduces 0.3.0.
Data quality — no duplicate rows, no constant features, licenses and attribution present where required.
All tests must pass before a PR is merged.