Scaling Web Accessibility Audits: Heterogeneous Crowdsourcing and the Power of Expertise Weighting

TL;DR

Web accessibility evaluation is a bottleneck for an inclusive internet due to the scarcity of experts. This paper presents a crowdsourcing system that uses Golden Set Strategy (GSS) and Time-Based Golden Set Strategy (T-GSS) to turn non-expert efforts into expert-level results. By weighting contributors based on their proven expertise and how long they spend on a task, the system improves accuracy by 7.21% and cuts evaluation time by 50%.

The Scalability Crisis in Accessibility

Ensuring a website is accessible—usable for over a billion people with disabilities—often requires manual conformance testing. While automatic tools exist, they cannot detect semantic or contextual barriers. Unfortunately, manual evaluation is labor-intensive, and true accessibility experts are rare.

Traditional crowdsourcing (like Majority Vote) fails here because accessibility is an "expert-domain" task. If 4 novices say a page is accessible and 1 expert says it isn't, Majority Vote would wrongly favor the novices. The authors recognized that we need a way to filter the noise of the crowd without losing its scale.

Methodology: Beyond Simple Voting

The core innovation lies in how individual opinions are merged into a final decision. The authors move through three levels of complexity:

1. Golden Set Strategy (GSS)

The system "sneaks" hidden tasks with known answers (Golden Tasks) into a worker's queue. A worker's performance on these tasks defines their Golden Accuracy (Weight).

• Logic: If Worker A consistently identifies known barriers, their vote on new, unknown pages counts for more than Worker B, who misses them.

2. Time-Based Golden Set Strategy (T-GSS)

The researchers discovered a critical "Human Factor": Asymmetry in Task Time.

• Finding a barrier is often fast; you see it, you report it.
• Confirming a page is fully accessible takes significantly longer because the worker must examine every element.
• If a worker reports "Accessible" in 2 seconds, they are likely guessing.

T-GSS uses a least-square loss function to optimize weights for accuracy and time cost, treating time spent on "Accessible" vs "Inaccessible" results as separate reliability signals.

Figure 1: The architecture details the pipeline from crawler sampling to the final measuring of the WAEM score.

Experimental Results

The study involved 98 Chinese websites and over 23,000 manual tasks. The findings were stark:

• Accuracy Boost: T-GSS achieved 80.58% accuracy, significantly higher than GSS (74.91%) and Majority Vote (73.37%).
• Efficiency: An expert-only team would have taken 66 hours. The crowd+expert system finished in 31 hours.

Figure 2: Left: Distribution of non-expert golden accuracy; Right: Distribution of accessibility scores across 98 websites.

Critical Insight: The "Accessible" Bias

One of the paper's most salient points is that low-expertise workers have an inherent bias toward marking pages as "Accessible" simply because they lack the knowledge to see the barriers. By penalizing "quick accessible" answers via T-GSS, the system corrects for this "omission bias," which is a common failure mode in regulatory and compliance audits.

Conclusion and Future Outlook

This work demonstrates that we don't need a world of experts to audit the web; we need a world of well-managed contributors. The introduction of temporal logic (time-based weighting) into decision strategies provides a robust framework for any crowdsourcing task that involves expert-level complexity.

In the future, the authors intend to integrate demographic factors and worker motivation into the decision model to further refine result reliability.