quoras: Breaking the English Barrier in Social Science Data Collection
quoras: A Python API for Quora Data Collection to Increase Multi-Language Social Science Research
The paper introduces quoras, an open-source Python API designed for robust data collection from the Q&A platform Quora. It leverages Selenium web automation to enable multi-language data extraction (English, Bengali, French, Hindi, Spanish, and Japanese) to facilitate cross-cultural Computational Social Science research.
TL;DR
The research presents quoras, a Python-based API that automates data collection from Quora. Unlike its predecessors, it supports multiple languages and successfully extracts full-text discussions from JavaScript-heavy pages, providing a vital bridge for computational social scientists studying non-English digital communities.
Background & Positioning
In the landscape of social media research, Twitter and Reddit have dominated the discourse. However, these platforms are often a "Western echo chamber." Quora, with its 17+ language domains, offers a goldmine for understanding how different cultures engage in peer-to-peer knowledge sharing. quoras positions itself as an essential infrastructure tool, moving beyond simple keyword search to offer deep, multi-lingual thread extraction.
The Problem: The "English-Only" Research Trap
Most data collection toolkits are designed with English-language platforms in mind. This creates an inductive bias in social science, where theories of online interaction are built almost exclusively on Western contexts.
Current technical challenges with Quora specifically include:
- JavaScript Rendering: Standard scrapers often miss "hidden" answers that require interaction to load.
- Session Limits: Anonymous browsing is capped, preventing large-scale data mining.
- Fragility: Frequent UI updates render traditional DOM-parsing APIs obsolete.
Methodology: Human-Centric Automation
The authors utilize Selenium to overcome these hurdles. By simulating a real browser session (via ChromeDriver), the API can log into accounts, scroll dynamically to trigger JavaScript events, and navigate language subdomains seamlessly.
The Architecture
The tool is split into three functional modules:
- Quoras Class: The user-facing interface for credentials and language settings.
- Browser Class: Handles the "heavy lifting" of GET requests and DOM scrolling.
- Scraper Class: Parses the final rendered HTML into structured data objects.
Fig 1: The conceptual framework of the quoras API.
Functionality & Usage
The API is designed for accessibility—essential for social scientists who may be more comfortable with Python than Ruby or C++.
| Function | Key Capability |
|---|---|
search() | Fetches URLs for posts, topics, or users based on phrases. |
search_url() | Extracts metadata, including follower counts and related questions. |
get_full_answer() | Captures the complete text of an answer, solving the "read more" expansion issue. |
Multilingual Support
Users can initialize the API with a simple language code:
python quora = Quora(email, password, language='bn') # Target Bengali Quora
Experimental Validation
The paper demonstrates that quoras can retrieve data that previous tools (like quorapy or scrape_quora) simply cannot—specifically related questions suggested by Quora's recommendation engine and full-text content of long answers.
Table 1: Detailed breakdown of the data collection functions provided by the API.
Critical Insight & Future Outlook
While quoras is a significant step forward, it remains a web-automation wrapper rather than a native API, meaning it is still vulnerable to HTML structure changes by Quora’s developers.
Future Work points toward:
- Multimodal Data: Collecting images and videos embedded in answers.
- Reply Threads: Mapping the nested hierarchy of comments within a discussion.
- Adaptive Switching: Automatically moving between language forums to track a single topic globally.
For the research community, this tool represents a move toward a more inclusive and global Social Science, allowing us to ask not just what the world is talking about, but how those conversations differ across cultural and linguistic boundaries.