[ER 2024] Tree-of-Thoughts for Domain Modeling: Solving the Complexity of UML Synthesis with LLMs

Executive Summary

TL;DR: Domain modeling is a cognitively demanding task that requires simultaneous management of domain knowledge and abstract modeling notation. This paper proposes adapting the Tree-of-Thoughts (ToT) framework to break down modeling into five manageable sub-tasks. By introducing a Domain-Specific Language (DSL) to orchestrate LLM generators and evaluators, the authors significantly improve the generation of complex UML constructs like association classes, achieving an F1-score of 0.824.

Positioning: This work moves beyond simple "prompt engineering" into "prompt orchestration," providing a structured, algorithmic approach to leveraging LLMs for Model-Driven Software Engineering (MDSE).

The Bottleneck: Why One-Shot Prompting Fails

Current LLM-based modeling relies mostly on Input-Output (IO) or Chain-of-Thought (CoT) prompting. While these can identify simple classes, they consistently fail at:

1. Relationship Classification: Distinguishing between association, aggregation, and composition.
2. Abstract Constructs: LLMs rarely "hallucinate" necessary abstractions like Association Classes unless guided.
3. Consistency: Large models often lose track of long-distance dependencies in complex domain descriptions.

The authors argue that modeling requires deliberate reasoning—the ability to explore multiple solutions and pivot—which standard linear prompting lacks.

Methodology: Branching Out the Modeling Process

The core innovation lies in the task decomposition of the modeling workflow into a tree structure.

1. The Five-Level Tree Structure

Instead of asking for a full diagram at once, the framework follows this path:

• L1: Classes & Attributes: Initial entity identification.
• L2: Associations: General link identification.
• L3: Containments: Refining links into whole-part relationships.
• L4: Inheritance: Identifying "is-a" hierarchies.
• L5: Association Classes: Finalizing complex many-to-many relationship properties.

2. Search & Evaluation (BFS)

The framework employs a Breadth-First Search (BFS). For each level:

• Generate (k=3): Three distinct "thoughts" (partial models) are sampled.
• Evaluate (v=5): The LLM "votes" five times on which thought is the most architecturally sound.
• Select (b=1): The best thought is passed to the next level.

3. Orchestration via DSL

To make this process repeatable, the authors developed a DSL (using TextX). This allows users to define modeling tasks, assessment criteria, and the target notation (e.g., PlantUML) in a clean, declarative configuration file.

Experiments & Results

The team validated the approach using GPT-4 across five diverse domains (Theaters, Robots, e-Commerce, etc.).

Key Findings:

• Classes: Excellent performance (F1: 0.895).
• Association Classes: A major breakthrough (F1: 0.824). Previously, LLMs almost entirely ignored these.
• Relationships: The "Achilles' Heel" remains. Even with ToT, relationship classification precision was low (0.227).

The visual outcome below demonstrates how the ToT Level 5 successfully identifies an "Access" association class that standard prompting would likely have missed.

Critical Analysis & Professional Insight

Why does ToT work for Association Classes?

The authors' insight is brilliant: L5 depends on the quality of L1 and L2. By forcing the model to stabilize its class list and general associations first, the "noise" in the latent space is reduced. By the time the LLM reaches L5, the context contains a structured "winning thought" that makes the abstraction of an association class logically obvious.

Limitations

1. Relationship Ambiguity: The low precision in relationships suggests that LLMs still struggle with the semantic difference between "a part of" and "related to" without specific domain heuristics.
2. Cost: ToT is expensive. With $k=3$ and multiple levels, API costs scale linearly. This is a tool for professional architects, not casual users.

Future Outlook

The next frontier is Hybrid Reasoning. Combining ToT with Few-shot examples at the relationship level or integrating a human-in-the-loop as the "State Evaluator" could solve the current precision bottlenecks.

Takeaway

This research proves that structure precedes intelligence in LLMs. By providing a cognitive framework (ToT) and a control language (DSL), we can transform a general-purpose model into a high-precision domain modeling assistant.

For practitioners, the open-source tool on GitHub serves as a foundation for automating complex system designs.