An open leaderboard for natural language inference
44 public test sets, grouped into 4 categories
Ranking
Rank by
Parameters
Models within 0.25 points share a rank (shown with =). Green (above median), Red (below median). CAL = calibration error (ECE), lower is better.
Strengths and weaknesses
Individual benchmark scores
Show
Color
Efficiency
Score vs size and speed (dashed line = Pareto front)
Score
Parameters vs Skill Index
parameters · log scale — up and left is better
Throughput vs Skill Index
pairs / second on one GPU (batches of 128, tokenization excluded) · log scale — up and right is better
Calibration
Does a model's confidence match how often it is right? Expected calibration error (ECE), lower is better.
Calibration error vs Skill Index
ECE % (15 bins, averaged)
Reliability diagram
confidence vs accuracy, every dataset weighted equally; on the diagonal = perfectly calibrated; point size = share of rows
Compare (up to 4)
Scoring
Accuracy per dataset (3-way entailment / neutral / contradiction; 2-way sets count neutral and contradiction as “not entailed”).
Contaminated scores. If a model trained on one of the test sets, that score is shown in parentheses
and replaced by an estimate of what the model would have scored, based on its results on the other datasets.
The estimate is the mean of two predictions: the scores of the 3 most similar models (kNN), and the model's
average z-score mapped onto that dataset. It is never higher than the original score.
How to submit
Open a pull request on the Community tab
(“New pull request”) that adds your model to requests/. We run the evaluation ourselves and add the result.
Rules
The model must be public on the Hugging Face Hub. For cross-encoders that load in sentence-transformers
(three NLI outputs: entailment / neutral / contradiction, with a labelled id2label), you can check your
scores with the evaluate.py
script. LLMs are welcome too, we'll cross that bridge when we get there.
Declare all training data. Training on train splits and selecting checkpoints on dev splits is allowed.
Training on any OpenNLI test set is not allowed.
Every model is also scored on private held-out sets. A model with a large gap between public and
private scores will be flagged for review and potentially rejected.
We run every evaluation on our own hardware. The maintainers' run is the
official result. Scores within 0.25 points share a rank.
One entry per model version. Newer versions are separate entries.
Check your numbers first with evaluate.py: python evaluate.py --model org/model-name
(scores the public sets; the results are not official until we run them).