[NeurIPS 2025] Structure-of-Thought: Why Text-to-Structure is the Next Frontier for LLM Reasoning

Executive Summary

TL;DR: Reasoning over long, complex documents is fundamentally broken in end-to-end LLMs because they lack an explicit "mental map" of the information. Duke University and Meta researchers introduce Structure-of-Thought (SoT)—a prompting technique that forces models to build a graph-based representation (nodes and links) before answering—and T2S-Bench, a rigorous 1.8K-sample benchmark to measure this capability.

Strategic Positioning: This work moves beyond the linear reasoning of Chain-of-Thought (CoT) into the realm of structural grounding. It identifies that a model's ability to "Find, Fuse, and Form" information is bound by its ability to map prose to a formal topology.

The Core Motivation: The "End-to-End" Trap

While Large Language Models (LLMs) excel at simple retrieval, they struggle with "Find-Fuse-Form" pipelines. In these workflows, a user expects a model to identify evidence across multiple sources (Find), integrate conflicting or complementary viewpoints (Fuse), and generate actionable reports (Form).

The authors argue that current models fail here because they lack a Stable Intermediate Representation (IR). Without a structured map, multi-hop reasoning becomes a game of "telephone," where errors accumulate at each step. By looking at how humans handle complex reading—often by marking key points and drawing relationships—the authors realized that structuring is a prerequisite for understanding.

Methodology: Structure-of-Thought (SoT)

The authors introduce Structure-of-Thought (SoT). Unlike CoT, which focuses on the process of reaching an answer, SoT focuses on the organization of the input data.

The SoT Prompt Architecture

Models are instructed to follow a strict format:

1. [Structure]: Extract a JSON-formatted graph of nodes and links from the source text.
2. [Answer]: Synthesize the final response based strictly on that extracted graph.

Figure 1: SoT consistently outperforms Direct Answering and CoT across multiple text-processing benchmarks.

T2S-Bench: The Scientific Stress Test

To evaluate this, they built T2S-Bench, the first comprehensive benchmark using academic papers as a gold standard. Why papers? Diagrams in scientific literature are peer-reviewed and logically dense, providing a "ground truth" that is missing from generic web text.

The benchmark spans 6 domains (CS, Life Sciences, Economics, etc.) and evaluates models on:

• Multi-hop Reasoning (MR): Fault localization, functional mapping, and counterfactual thinking.
• End-to-End Extraction (E2E): Reconstructing the graph from raw text.

Key Experimental Results

The researchers evaluated 45 mainstream models, and the results reveal a massive "reasoning gap" in the industry.

1. The Performance Ceiling: Average Exact Match (EM) on multi-hop reasoning is only 52.1%. Even the most advanced models (Gemini-2.5-Pro, Claude 3.5 Sonnet) struggle when node counts increase beyond 10.
2. Structural Bottleneck: Models are surprisingly good at linking nodes once identified (Link F1 ~80%+), but they are poor at identifying the nodes themselves (Node Accuracy ~50%). This suggests that Entity Recognition is the primary weakness in structural reasoning.
3. Fine-tuning Gains: Training a model like Qwen2.5-7B on the T2S-Train dataset doesn't just help it with structuring; it improves its performance on General Long-Context Tasks (like HotpotQA) by +8.6%, proving that structural thinking is a generalizable skill.

Figure 2: Model performance (Link F1) sharply declines as the number of nodes in the graph increases, highlighting a "complexity wall."

Critical Analysis & Takeaways

Why does SoT work?

SoT acts as an anchor. In standard LLM generation, the model's "attention" can drift. By forcing a structured JSON output first, the model performs a "lossless" compression of the text into its most salient parts. The subsequent reasoning phase then operates on this simplified, verified graph rather than the noisy, high-dimensional raw text.

Limitations

• Structural Ambiguity: The authors admit that a single text can often be represented by multiple valid graphs. T2S-Bench mitigates this by using peer-reviewed diagrams, but real-world application remains "lossy."
• Node Detection: The significant drop in Node Accuracy suggests that LLMs still lack the "precision" required for fine-grained entity segmentation in dense technical text.

The Future of LLM Workflows

The success of SoT suggests that the future of LLM agents isn't just "more parameters," but better Intermediate Representations. We should expect to see more "Structure-Augmented Generation" (SAG) where models maintain dynamic Knowledge Graphs of their conversations to ensure logical consistency over time.

For developers and researchers, the message is clear: If your model is failing at complex reasoning, stop asking it for the answer—start asking it for the map.

Links & Resources:

• Dataset & Code: T2S-Bench GitHub
• Core Methodology: Structure-of-Thought (SoT)