Better Call the Crowd: Democratizing DSL Notation Design through Crowdsourcing

TL;DR

Coming up with the "right" icons for a specialized modeling language is hard and often subjective. This paper presents a systematic method to let the global crowd decide. By converting a language's abstract structure into automated crowdsourcing tasks, the authors created a notation that outperformed standard expert designs in intuitiveness and clarity.

The "Ivory Tower" Problem in Language Design

Historically, Domain-Specific Modeling Languages (DSMLs) are designed in a closed loop between a few developers and a handful of domain experts. While this ensures technical accuracy, it often results in a "Concrete Syntax" (the visual icons) that is opaque to the broader user base.

The core insight of the authors is that notation is a UX problem, and like many UX problems, it benefits from large-scale testing. Why guess if a "diamond" or a "circle" better represents a gateway when you can have 100 people tell you?

Methodology: From Metamodel to Micro-task

The authors don't just ask random questions; they use a structured pipeline to ensure the resulting language is coherent.

1. Pattern-Based Task Generation

The system identifies specific structures in the language's metamodel (the Abstract Syntax). For example, if it sees a Generalization/Hierarchy pattern (like different types of "Events"), it generates a single task asking about all child elements simultaneously. This ensures the user considers the "look and feel" across related concepts.

Figure 1: The proposed 4-phase lifecycle, integrating the crowd in Step 3.

2. Dynamic Execution

One of the smartest features is contextual consistency. When a participant chooses a specific symbol for a "Task" in Question 1, that chosen symbol is used in the example diagrams for Question 2. This prevents the user from being confused by shifting notations and helps them see how their choices fit into the "big picture."

Experimental Battle-test

The authors tested their approach on a Business Process Modeling notation. They pitted the crowd-chosen notation against "Standard" BPMN icons.

Key Findings:

• Agreement: For 5 out of 7 symbols, the crowd reached an absolute majority, proving that collective intuition is surprisingly consistent.
• Expert vs. Beginner: While experts and beginners agreed on most symbols, some differences appeared (e.g., for "End Events"), suggesting that languages might actually need multiple skins based on user profile.
• Efficiency: Using a "Targeted Agreement" strategy (stopping once 60% agreement is reached) provided the same results as "Static Majority" but with 60% less human effort.

Table 1: Preference distribution across different user expertise levels.

Critical Insight: Scientific "Look and Feel"

The paper proves that the "Physics of Notations" (the theory of how people perceive visual symbols) can be crowdsourced. By moving from a static design to an iterative, crowd-validated process, language designers can reduce the "cognitive load" for their future users.

Limitations: The study focused on graphical icons. It remains to be seen how well this approach translates to textual notations (keywords) or complex dynamic semantics (how the language behaves).

Conclusion

"Better Call the Crowd" marks a shift from "expert-knows-best" to "user-knows-best" in Software Language Engineering. For developers building the next generation of Low-code/No-code platforms, this methodology provides a blueprint for ensuring their visual languages are actually... visual.

Figure 2: A process model built using the "winning" notation from the study.