TLA+-Bench: An Execution-Grounded Benchmark and Dataset for Natural-Language to TLA+ Specification Generation

Preprint / Under review

Status:

Preprint / Under review

Date:

July 2026

Authors:

Arslan Bisharat, Eric Spencer, Brian Ortiz, Khushboo Bhadauria, Mujtaba Nazari, Beatriz Santos, Anisa Ramos, TaiNing Wang, George K. Thiruvathukal, Konstantin Läufer, Mohammed Abuhamad

Institution:

Loyola University Chicago

Paper:

arXiv:2607.23425 | PDF

DOI:

10.48550/arXiv.2607.23425

Artifact:

Benchmark, dataset, and reproduction code

TLA+-Bench compositionEach of 1,300 dots is one specification: 403 gold specifications checked with TLC and 897 silver specifications checked only for parsing.1,300TLA+ specifications from 13 public repositories403 gold specifications, each model-checked with TLC897 silver specifications, checked for valid form only403 goldRun through TLC.Settings included.897 silverChecked for valid form (parsing) only.
Figure 1. What is in TLA+-Bench. Each dot is one specification. 403 are gold (model-checked with TLC) and 897 are silver (parse-only), from 13 public repositories. Counts are from the paper.

Abstract

TLA+-Bench is an execution-grounded benchmark for measuring whether models can produce correct TLA+ specifications from natural-language descriptions. It contains 1,300 specifications drawn from 13 public repositories: 403 TLC-model-checked gold specifications and 897 parse-only silver specifications. Each gold specification ships with a model-checker configuration, allowing evaluation to measure execution rather than similarity to a reference answer.

Why This Matters

Parsing a generated specification is not the same as establishing that it is correct. TLA+-Bench makes that distinction measurable and exposes how much results change when evaluation choices are made explicit. It supplies a reproducible dataset, grader, and model outputs for research on natural-language-to-TLA+ generation.

The benchmark follows the earlier GCASR 2026 structured-benchmarking poster.

In Plain Words

Imagine asking a computer to write down the rules of a system, such as how several computers agree on one answer. The rules have to be exact, because a small mistake can hide a big problem. TLA+ is a language for writing rules like these. A tool called TLC can then try the ways the system could run and tell you if a rule is ever broken.

Today, people ask AI models to write TLA+ rules from a plain-English description. The hard question is how to tell whether the AI did a good job. One easy check is whether the file is written in the right form. That is like checking spelling. A sentence can be spelled perfectly and still say the wrong thing. A better check is to actually run the rules through TLC and see what happens.

TLA+-Bench is a collection built for that purpose. It holds 1,300 TLA+ specifications taken from 13 public projects. Of these, 403 are “gold”: each was run through TLC, and each comes with the settings needed to run it again. The other 897 are “silver”: we only checked that they are written in a valid form.

When a model is tested, its answer is graded by running it, not by comparing its words with a reference answer. This matters because the same system can be described correctly in many different ways, and a word-by-word match would punish good answers.

The paper also points out that results can change a lot depending on small choices in how a test is run. By making those choices visible, the benchmark lets people see why two studies might disagree. The team released the data, the grader and the models’ answers, so other researchers can repeat the work and compare their own models fairly.

Citation

Arslan Bisharat, Eric Spencer, Brian Ortiz, Khushboo Bhadauria, Mujtaba Nazari, Beatriz Santos, Anisa Ramos, TaiNing Wang, George K. Thiruvathukal, Konstantin Läufer, and Mohammed Abuhamad. TLA+-Bench: An Execution-Grounded Benchmark and Dataset for Natural-Language to TLA+ Specification Generation. arXiv:2607.23425, 2026. https://doi.org/10.48550/arXiv.2607.23425

BibTeX

@misc{bisharat2026tlabench,
  title         = {{TLA+-Bench}: An Execution-Grounded Benchmark and Dataset for Natural-Language to {TLA+} Specification Generation},
  author        = {Bisharat, Arslan and Spencer, Eric and Ortiz, Brian and Bhadauria, Khushboo and Nazari, Mujtaba and Santos, Beatriz and Ramos, Anisa and Wang, TaiNing and Thiruvathukal, George K. and Läufer, Konstantin and Abuhamad, Mohammed},
  year          = {2026},
  eprint        = {2607.23425},
  archivePrefix = {arXiv},
  doi           = {10.48550/arXiv.2607.23425},
  url           = {https://arxiv.org/abs/2607.23425}
}