AI4FM Updates

TLA-Prover Paper Accepted to ICSOFT 2026

2026-06-05T00:00:00+00:00

We are pleased to announce that our latest research paper, “TLA-Prover: Verifiable TLA+ Specification Synthesis via Preference-Optimized Low-Rank Adaptation” (Paper #131), has been accepted for publication at ICSOFT 2026 (International Conference on Software Technologies) in Porto, Portugal.

This marks our second paper accepted to ICSOFT 2026, alongside our systematic evaluation study on LLM-based TLA+ generation. TLA-Prover introduces a 20-billion-parameter model that achieves a 3.5× improvement in semantic correctness over public baselines by combining supervised fine-tuning with repair-based reinforcement learning (GRPO).

Read the full paper details

New Paper: TLA-Prover — Fine-Tuning LLMs for Verifiable TLA+ Specification Synthesis

2026-05-16T00:00:00+00:00

Building on our evaluation study that showed the best public LLMs achieve only 8.6% semantic correctness on TLA+ specification generation, we present TLA-Prover: a 20-billion-parameter model trained specifically for TLA+ specification synthesis.

TLA-Prover combines supervised fine-tuning (SFT) on verified examples with repair-based group-relative policy optimization (GRPO), where the model learns to fix its own rejected specifications. TLC model checking provides the reward signal directly — no learned reward model is needed. A four-tier grading scheme (Bronze, Silver, Gold, Diamond) measures output quality, with Diamond requiring that the model’s correctness property is meaningful enough for TLC to detect deliberate violations.

TLA-Prover achieves 30% pass@1 at Gold and Diamond on a held-out 30-problem benchmark — roughly 3.5× the 8.6% untuned baseline. A DPO ablation trained from the same SFT checkpoint reaches 20% at Diamond.

Read the full paper details

See other posts

Presented Two Posters at GCASR 2026

2026-05-11T00:00:00+00:00

On May 11, 2026 we presented two posters at the 13th Greater Chicago Area Systems Research Workshop (GCASR 2026).

A Structured Benchmarking Dataset for TLA+ Specification Reasoning

Authors: Arslan Bisharat, Eric Spencer, Khushboo Bhadauria, Anisa Ramos, Brian Ortiz, Mohammed Abuhamad, Konstantin Laüfer, TaiNing Wang, George K. Thiruvathukal

This is a currently working paper.

Large Language Models (LLM) and Temporal Logic of Actions (TLA): How Effective are LLMs for Verification Systems

Authors: Brian Ortiz, Arslan Bisharat, Eric Spencer, Khushboo Bhadauria, Anisa Ramos, Mohammed Abuhamad, Konstantin Laufer, TaiNing Wang, George K. Thiruvathukal

See other posts

Our Paper on LLM-Based TLA+ Specification Generation Has Been Accepted

2026-04-30T00:00:00+00:00

We are excited to announce that our paper Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation has been accepted for publication at ICSOFT 2026 (International Conference on Software Technologies) in Porto, Portugal.

The paper presents the first systematic evaluation of LLM-based TLA+ specification synthesis from natural language, evaluating 30 LLMs across eight families on a curated dataset of 205 TLA+ specifications. Key findings include that LLMs achieve up to 26.6% syntactic correctness but only 8.6% semantic correctness, and that model size does not predict quality on formal languages.

Read the full paper details

Interactive Microwave: TLA+ in the Browser

2026-04-17T00:00:00+00:00

We published a browser-native version of the interactive-microwave-tla simulator. It mimics a Java microwave runtime and a small TLA+ model checker that is a literal transcription of the project’s Microwave.tla spec, so students can experiment with the state machine — including the stuttering liveness failure from Laufer, Mertin, and Thiruvathukal, arXiv:2407.21152 — without installing Java, Maven, or TLC.

The stuttering failure, made visible

Section IV.C of the paper introduces a liveness property and its stuttering counterexample:

HeatLiveness == (radiation = ON) ~> (radiation = OFF)

Spec == Init /\ [][Next]_<<door, time, radiation, power>>

Under this original Spec, TLA+ permits behaviors where every step after the microwave begins radiating is a stutter (vars' = vars). The tick never fires, the time never decrements, and radiation = ON holds forever — violating HeatLiveness. The fix (Exercise 3b) is weak fairness on Tick:

Spec == Init /\ [][Next]_<<door, time, radiation, power>> /\ WF_vars(Tick)

Our previous web demo did not model any of this — neither the liveness property, nor the stuttering failure, nor the fairness condition. It merely happened to tick unconditionally in Java and therefore looked correct without explaining why.

What the new build does

The browser port re-implements the spec literally in TypeScript and exposes both the failure and its fix:

A Weak fairness on Tick toggle in the sidebar switches the tick loop between the paper’s original (unfair) spec and the fixed spec with WF_vars(Tick).
With fairness off, the engine lets stutters run even when Tick is enabled. A runtime liveness detector surfaces the trap: after a short window of stutter-while-radiating, the UI flags HeatLiveness as violated and shows a 🔥 overlay — a runtime witness of Figure 7.
With fairness on, the engine forces Tick to fire whenever it is continuously enabled. HeatLiveness holds and the overlay clears.

An exhaustive safety check enumerates every state reachable from Init via the Next relation and confirms zero DoorSafety violations (matching Exercise 2b).

Try it

Reproduce the paper’s Figure 7 by pressing Power, +3s, Start, and then turning off Weak fairness on Tick. The Tick action stops firing, the radiation indicator stays lit, and the liveness signal flips to violated.

Source and details

Repository
Design spec
Browser port (web/)
System Message: INFO/1 (/home/runner/work/ai4fm/ai4fm/src/posts/interactive-microwave-tla.rst, line 9); backlink

Duplicate explicit target name: “arxiv:2407.21152”.

Laufer, Mertin, Thiruvathukal. WIP: An Engaging Undergraduate Intro to Model Checking in Software Engineering Using TLA+. arXiv:2407.21152, 2024.

Eric Spencer Presents ChatTLA+ at Loyola University Chicago

2026-04-17T00:00:00+00:00

Eric Spencer presented ChatTLA+ at the Undergraduate Research and Engagement Symposium at Loyola University Chicago on April 17, 2026 - a talk exploring the use of large language models for generating and verifying TLA+ formal specifications.

Eric Spencer presenting ChatTLA+ at the Undergraduate Research and Engagement Symposium, Loyola University Chicago, April 17, 2026.

The presentation covers how LLMs can be applied to TLA+ formal specification generation and verification, building on the group’s ongoing research into evaluating LLMs on formal language tasks.

View the presentation (PDF)

Read the full details

Brian Ortiz Presents Poster at Graduate School Interdisciplinary Research Symposium 2026

2026-04-11T00:00:00+00:00

Brian Ortiz presented a poster at the Graduate School Interdisciplinary Research Symposium (GSIRS 2026) on April 11, 2026, at Loyola University Chicago.

The poster, co-authored with Arslan Bisharat, Mohammed Abuhamad, Konstantin Läufer, Eric Spencer, Khushboo Bhadauria, George K. Thiruvathukal, and TaiNing Wang, presents the first systematic evaluation of LLM-based TLA+ specification synthesis from natural language. The study evaluates 30 LLMs across eight families on a curated dataset of 205 TLA+ specifications, finding that LLMs achieve up to 26.6% syntactic correctness but only 8.6% semantic correctness. Model size does not reliably predict performance on formal languages.

Brian Ortiz standing next to his poster at the Graduate School Interdisciplinary Research Symposium 2026, Loyola University Chicago.

This poster is based on the published ICSOFT 2026 paper.

View the poster on figshare

Read the full paper details

Can LLMs Write Correct TLA+ Specifications? Our New Evaluation Study

2026-03-29T00:00:00+00:00

We have submitted a new paper evaluating whether large language models can generate semantically correct TLA+ specifications from natural language. We evaluated 30 LLMs across eight families on 205 TLA+ specifications. Best semantic correctness achieved was only 8.6%, and model size did not predict quality.

Read the full paper details

Note

This work is now published in ICSOFT 2026. See the published paper.

Eric Spencer Awarded Mulcahy Scholar Stipend

2025-08-15T00:00:00+00:00

Eric Spencer, an undergraduate researcher in the AI4FM group, has been awarded a $1,000 Mulcahy Scholar stipend from the Loyola University Chicago College of Arts and Sciences to support his research on LLM-based TLA+ specification synthesis.

Eric is conducting this research under the supervision of Professor Konstantin Läufer. The work contributes to the group’s ongoing evaluation of large language models for formal verification tasks, examining whether LLMs can generate semantically correct TLA+ specifications from natural language.

The Mulcahy Scholars Program supports undergraduate research experiences of scholarly significance across the College of Arts and Sciences at Loyola University Chicago.

Brian Ortiz Presents Poster at GCASR 2025

2025-05-08T00:00:00+00:00

Brian Ortiz presented a work-in-progress poster at the 12th Greater Chicago Area Systems Research Workshop (GCASR 2025), held on May 8, 2025 at Loyola University Chicago’s Water Tower Campus.

The poster, co-authored with Mohammed Abuhamad, TaiNing Wang, George K. Thiruvathukal, and Konstantin Läufer, presents an automated pipeline for synthesizing TLA+ specifications from natural language using large language models, validated with the SANY parser and TLC model checker.

Brian Ortiz standing with Konstantin Läufer next to their poster at GCASR 2025, Loyola University Chicago.

Read the full paper details

TLA+ for All: Running Model Checking in a Python Notebook

2025-02-05T00:00:00+00:00

We published a project embedding TLA+ model checking directly into a Python notebook environment, making formal verification accessible without any installation. Users can write specifications, run the TLC model checker, and visualize behaviors in-browser.

Read the full paper details