[BCU Review] Socially Distanced: How the Pandemic Reconfigured XR User Research

TL;DR

The COVID-19 pandemic forced a fundamental "pivot" in how we test immersive technologies. This paper reviews 951 top-tier publications to track this evolution, revealing a massive surge in remote user studies and a wide variety of hygiene-driven lab adaptations. While the field successfully scaled up participant pools via remote tools, the study warns of a "reporting deficit" regarding ethics and procedural changes.

Problem & Motivation: The XR Research Crisis

In 1990s and 2000s HCI, the "Lab Study" was king. For Extended Reality (XR), this dependency was even stronger due to the specialized, expensive, and non-mobile nature of VR/AR hardware. When COVID-19 hit, the field hit a wall: How do you share an HMD that sits on a user's face during a respiratory pandemic? How do you recruit participants when social distancing is the law?

The authors recognized that while research continued, the way it was being reported—and the methodologies used—might have changed permanently. They sought to quantify this shift to understand the new "Normal" for XR research.

Methodology: Auditing the State of the Field

The researchers analyzed a massive corpus from A* ranked conferences: IEEE ISMAR, ACM CHI, and IEEE VR.

They looked for several key indicators of adaptation:

• Study Type: Was it Co-located (in-person) or Remote (distributed)?
• COVID-19 Considerations: Did the authors mention face masks, 72-hour equipment quarantines, or social distancing?
• Ethical Approval: Was IRB/Ethics board approval explicitly mentioned despite the chaos?
• Hardware Logistics: Were headsets mailed to participants, or did researchers only recruit "HMD owners"?

Figure 1: Breakdown of accepted papers showing the ratio of remote vs. adapted studies across major conferences.

Core Findings: The Rise of the "Remote" Majority

The results show a field in transition. By Q3 2021 (ACM CHI'21), remote studies became a dominant force.

1. The Remote Advantage

Remote studies allowed for significantly larger sample sizes. While lab studies often hovered around 30 participants, remote evaluations reached averages of over 120. Researchers utilized platforms like Prolific, Amazon Mechanical Turk, and specialized tools like the XR Distributed Research Network.

2. Lab Adaptations

For those staying in the lab, "Socially Distanced" research meant:

• Quarantining Hardware: Some labs implemented 72-hour "rest periods" for headsets between users.
• Hygiene PPE: The use of single-use "VR Eye Masks," gloves, and sanitizers became standard.
• Procedural Shifts: Instructors directing users via loudspeakers from separate rooms or using movement "scripts" instead of live confederates for proxemics studies.

3. The Transparency Gap

Despite the radical shifts, the review found a concerning trend: many papers remained silent on when the data was collected or if their new remote/adapted protocols received ethical approval. This lack of context makes it difficult to evaluate the "Freshness" of the results or the safety of the participants.

SOTA Comparison: Beyond 2019 Baseline

Compared to 2019, the 2021 cohort showed:

• CHI: Remote studies increased from nearly zero to roughly 30% of XR evaluations.
• VR: Participant counts increased, driven by remote accessibility, though the variance in data quality (due to unmoderated sessions) became a new challenge.
• ISMAR: Early in the pandemic (Q2 2020), many studies simply stopped, showing high "Premature ending" rates without supplementary studies.

Critical Insight & Guidelines

The value of this paper isn't just in the data—it's in the Future-Proofing. The authors provide concrete guidelines for the community:

• For Remote Studies: Explicitly state if the study was moderated (live video) or unmoderated (automated logs), and how you screened for participant "dishonesty" on crowdsourcing platforms.
• For Lab Studies: Don't just follow hygiene rules; document them. This contextualizes the research for future scholars who may wonder why, for example, a 2021 study had a limited number of experimental conditions.

Conclusion: A More Flexible Future

The pandemic served as a "stress test" for HCI. While it exposed vulnerabilities in our reliance on physical labs, it accelerated the adoption of Distributed XR Research. The legacy of COVID-19 in XR is a move toward more ecologically valid "In the Wild" testing, provided we maintain the academic rigor of reporting our adaptations and ethical oversight.