ADRL: Solving Crowdsourcing Dilemmas with Transformers and Predictive Auxiliary Tasks
Auxiliary-task Based Deep Reinforcement Learning for Participant Selection Problem in Mobile Crowdsourcing
The paper introduces ADRL (Auxiliary-task based Deep Reinforcement Learning), a framework for participant selection in Mobile Crowdsourcing (MCS). By combining Transformers for context extraction, Pointer Networks for combinatorial selection, and an auxiliary task for future task prediction, the method achieves SOTA performance in multi-goal scenarios like ride-sharing.
TL;DR
Mobile Crowdsourcing (MCS) platforms, like Uber or DoorDash, face a daunting challenge: how to pick the right person for the right task while juggling conflicting goals like profit, fairness, and energy efficiency. This paper presents ADRL (Auxiliary-task Based Deep Reinforcement Learning), a sophisticated RL framework that uses Transformers to understand the city's context and an auxiliary prediction task to peek into the future, significantly boosting training efficiency and performance.
Background: The Multi-Goal Headache
In a typical ride-sharing scenario, the "perfect" choice depends on your goal. If you want minimal time, you pick the closest driver. If you want fairness, you pick the driver who hasn't had a job in hours. If you want profit, you might wait for a higher-value task.
Existing methods usually optimize for one of these in isolation. However, real-world platforms need to balance all of them simultaneously. Furthermore, the environment is "structured" (locations, timestamps, traffic) and "uncertain" (tasks appear randomly), making traditional heuristic-based algorithms brittle.
Methodology: The ADRL Architecture
The researchers moved away from simple MLPs and embraced a more "relational" architecture inspired by AlphaStar.
1. The Contextual Encoder (Transformers)
The model uses Multi-Head Self-Attention layers to process participants and tasks. This allows the model to understand the relative importance of participants compared to one another, effectively capturing the spatial "competition" or "density" of the crowd.
2. The Combinatorial Engine (Pointer Networks & LSTM)
Selecting participants for tasks is a combinatorial optimization nightmare. ADRL uses a Pointer Network to map tasks directly to participant indices. An LSTM core ensures that the model remembers which participants were already selected for earlier tasks, preventing collisions and redundant assignments.
Figure 1: The ADRL Policy Network featuring Transformer Encoders and a Pointer Network interaction layer.
3. The Secret Sauce: Predictive Auxiliary Tasks
Standard RL is "blind"—it only learns from rewards received. ADRL introduces an Auxiliary Task: the network is forced to predict what future tasks will look like.
- Why? If the model knows a high-demand surge is coming in a specific neighborhood, it can choose not to assign a local driver to a low-value task now, saving them for the future.
- Result: This supervised signal acts as a regularizer, helping the Transformer learn better embeddings much faster than RL alone could.
Experimental Evidence
The authors didn't just test this in a vacuum; they used over 1.1 billion NYC Taxi and Uber trip records to build a high-fidelity simulator.
Performance Comparisons
ADRL was pitted against several baselines, including:
- NPF/NAPF: Nearest Participant (Available) First.
- PPOWM: Proximal Policy Optimization with MLPs.
- RWM: Reinforce with MLPs.
As shown in the table below, ADRL dominated in nearly every complex environment setting, particularly when the ratio of tasks to participants became challenging.
Figure 2: Performance comparison across Profit-First reward settings.
The Impact of the Auxiliary Task
The ablation study (Figure 6 in the paper) reveals that without the auxiliary task (the "DRL" curve), the model learns significantly slower and reaches a lower final reward peak. This proves that "predicting the future" is a vital part of "deciding the present."
Figure 3: Training efficiency—ADRL (with auxiliary tasks) converges faster and more stably than standard DRL.
Critical Insight & Practical Value
The real breakthrough here isn't just the higher reward; it's the flexibility. Because ADRL is built on RL, a platform manager can change the objective (e.g., "focus on energy efficiency for the next hour due to a heatwave") simply by adjusting the reward weights. The Transformer-Pointer architecture is robust enough to adapt its strategy without needing a complete algorithmic redesign.
Conclusion
ADRL represents a significant step forward in making Reinforcement Learning practical for complex urban systems. By combining the relational power of Transformers with the foresight of Auxiliary tasks, it solves the "Participant Selection Problem" in a way that respects the multi-faceted nature of modern crowdsourcing platforms.
Future Outlook: The next step is integrating real-time traffic dynamics and user behavioral profiles (reliability/preference) into the simulator to see how ADRL handles the "human factor" of crowdsourcing.