Towards Microtask Crowdsourcing: Can the Crowd Actually Design Software?

Executive Summary

TL;DR

This research investigates whether the "wisdom of the crowd" can be applied to the creative and often messy process of software design. By breaking down UI design into small, manageable microtasks based on the Morphological Chart method, the authors show that workers on Amazon Mechanical Turk can generate a surprisingly diverse range of design alternatives. While quality varies, the experiment proves that the crowd is a powerhouse for divergent thinking, providing a breadth of solutions that a single designer would struggle to match.

Academic Positioning

Published at ICSE '16, this work represents a pivotal shift from viewing crowdsourcing as a tool for mere "grunt work" (like data labeling) to a potential partner in the high-level Software Design Life Cycle (SDLC).

Problem & Motivation: The Creative Bottleneck

In the world of software engineering, crowdsourcing has long been restricted to "objective" tasks: Is there a bug? Does this test pass? Complex design work was thought to be too monolithic and subjective for the crowd. Prior models like TopCoder rely on competitions, where individuals work in silos. This leads to two major issues:

1. Limited Exploration: You only get as many ideas as you have full-time participants.
2. Lack of Collaboration: There is no mechanism to combine the best "parts" of different designs.

The authors asked: What if we could treat design like a puzzle, where the crowd generates the pieces, and a designer later assembles the best ones?

Methodology: The Morphological Microtask

The core innovation lies in the use of a Morphological Chart. Instead of asking a worker to "Design a Traffic Simulator," the problem is decomposed into specific Decision Points:

• How do we create the map?
• How do we set the timing for lights?
• How do we visualize the state of the simulation?

The CrowdDesign Platform

The researchers built a custom platform to facilitate this. Workers were required to pass a qualification test on UI principles before being allowed to sketch solutions.

Figure 1: The proprietary interface where workers received requirements, used sketching tools, and provided textual justifications for their design choices.

Experiments & Results: Diversity over Perfection

The study analyzed 181 solution alternatives. The findings were polarized but encouraging:

1. Diversity is the Crowd's Superpower: For the "Map Creation" task alone, the crowd produced 11 distinct conceptual categories (e.g., drag-and-drop, node-based, pencil-drawing).
2. The "Wasted Effort" Tax: A large portion of workers (nearly 72%) either failed the qualification or quit, citing that the task was "too hard." This highlights the gap between professional design and general microtasking.
3. High-Quality Gems: Despite the average score being modest, some solutions were rated as "exceptional," meeting all requirements with innovative approaches.

Figure 2: Analysis showing that while individual workers often stayed within a single conceptual lane, the aggregate crowd covered a massive design space.

Critical Analysis & Conclusion

Key Takeaway

The crowd is not a replacement for a Software Architect, but it is an incredible tool for Divergent Design. By offloading the generation of "solution alternatives" to the crowd, professionals can shift their focus from creating to evaluating and synthesizing.

Limitations

• Context Loss: By isolating decision points, there is a risk that the individual pieces won't fit together (the "Frankenstein" effect).
• Platform Friction: Amazon Mechanical Turk workers often expect "monotone" tasks; a creative design task requires a higher cognitive load that wasn't fully supported by the 2016-era tooling.

Future Outlook

This work lays the groundwork for Hybrid Design Workflows. In the modern context, one could imagine an AI agent acting as the "crowd," generating these morphological alternatives, which are then curated by a human lead. The experiment proves that breaking down creativity into microtasks is not just possible—it's productive.