OpenNLI
Leaderboard v0.1

An open leaderboard for natural language inference

44 public test sets, grouped into 4 categories

Ranking

Rank by
Parameters

Models within 0.25 points share a rank (shown with =). Green (above median), Red (below median). CAL = calibration error (ECE), lower is better.

Strengths and weaknesses

Individual benchmark scores

Show
Color

Efficiency

Score vs size and speed (dashed line = Pareto front)

Score

Parameters vs Skill Index

parameters · log scale — up and left is better

Throughput vs Skill Index

pairs / second on one GPU (batches of 128, tokenization excluded) · log scale — up and right is better

Calibration

Does a model's confidence match how often it is right? Expected calibration error (ECE), lower is better.

Calibration error vs Skill Index

ECE % (15 bins, averaged)

Reliability diagram

confidence vs accuracy, every dataset weighted equally; on the diagonal = perfectly calibrated; point size = share of rows
Compare (up to 4)

Scoring

Accuracy per dataset (3-way entailment / neutral / contradiction; 2-way sets count neutral and contradiction as “not entailed”).

Contaminated scores. If a model trained on one of the test sets, that score is shown in parentheses and replaced by an estimate of what the model would have scored, based on its results on the other datasets. The estimate is the mean of two predictions: the scores of the 3 most similar models (kNN), and the model's average z-score mapped onto that dataset. It is never higher than the original score.

How to submit

Open a pull request on the Community tab (“New pull request”) that adds your model to requests/. We run the evaluation ourselves and add the result.

Rules

  1. The model must be public on the Hugging Face Hub. For cross-encoders that load in sentence-transformers (three NLI outputs: entailment / neutral / contradiction, with a labelled id2label), you can check your scores with the evaluate.py script. LLMs are welcome too, we'll cross that bridge when we get there.
  2. Declare all training data. Training on train splits and selecting checkpoints on dev splits is allowed. Training on any OpenNLI test set is not allowed.
  3. Every model is also scored on private held-out sets. A model with a large gap between public and private scores will be flagged for review and potentially rejected.
  4. We run every evaluation on our own hardware. The maintainers' run is the official result. Scores within 0.25 points share a rank.
  5. One entry per model version. Newer versions are separate entries.

Pull request: requests/<org>__<model>.json

{
  "model": "org/model-name",
  "revision": "main",
  "license": "apache-2.0",
  "parameters": "150M",
  "trust_remote_code": false,
  "training_data": ["nyu-mll/multi_nli (train)", "..."],
  "contact": "https://huggingface.co/org",
  "notes": ""
}

Check your numbers first with evaluate.py: python evaluate.py --model org/model-name (scores the public sets; the results are not official until we run them).