Can short-form video algorithms be evaluated with reliable real-world evidence?

What counts as reliable real-world evidence for algorithm evaluation?

The most reliable real-world evidence comes from studies that separate problematic use from routine use, because the two have very different relationships with mental health. A 2026 meta-analysis of 58 studies with 96,676 participants found that problematic short-form video use was significantly linked to depression (r=0.24), anxiety (r=0.26), stress (r=0.41), loneliness (r=0.33), and boredom (r=0.42) [1]. In plain terms, these are moderate correlations—meaning that as problematic use increases, these negative states also increase. But routine use (just time spent or frequency) showed no significant link to anxiety and much weaker, inconsistent effects overall [1]. So if you want to evaluate an algorithm's real-world impact, you need to measure the quality of engagement, not just screen time.

How do algorithm features drive the behaviors we measure?

The algorithm's design features—like personalized recommendations, live interaction, and commenting—directly fuel the motivations that lead to excessive use. A 2024 survey of 351 TikTok users found that features like 'recommending' and 'meta-voicing' (commenting, reacting) increase perceived entertainment and social value, which in turn trigger negative affect anticipation (fear of missing out) and ultimately excessive use [5]. This means the algorithm isn't a neutral tool; its affordances are engineered to hook users through avoidance motives. A separate 2026 study of 1,728 users developed a motivation scale showing that avoidance motives (escapism, fear of social disconnection) predict intensive use more strongly than instrumental motives (fun, information) [4]. So reliable evaluation must measure whether the algorithm amplifies avoidance-driven behavior.

What does the content itself tell us about algorithm effects?

The content that goes viral under an algorithm reveals what the system amplifies, and that can be evaluated against real-world baselines. A 2023 analysis of the top 100 IVF-related TikTok videos found that they disproportionately showed same-sex couples (38.1% vs. real-world demographics), gestational carriers (14.0%), and live births (89.3%)—all far above actual population rates [2]. This means the algorithm systematically over-represents certain outcomes, which could skew viewer perceptions. However, the same study found that 93.8% of videos making scientific claims were moderately to highly accurate, suggesting the algorithm doesn't necessarily promote misinformation [2]. This kind of content audit is a valid real-world evaluation method.

Can we measure algorithm effects on real-world behavior?

Yes, and one concrete behavioral measure is risk-taking in decision-making. A 2025 study of 85 college students split into excessive and non-excessive short-form video users found that the excessive group scored significantly lower on the Iowa Gambling Task during the risk phase (last 40 trials), meaning they took more risks [3]. Higher overuse scores correlated with increased risk-taking but not with ambiguous decision-making [3]. This suggests that algorithm-driven excessive use may impair real-world judgment in a way similar to other addictions, providing a behavioral marker for evaluating algorithm impact.

About These Sources

This answer is built on 5 peer-reviewed studies — published from 2023 to 2026, 4 from 2024 or later, 5 in Q1 journals — selected as the most relevant from 5 studies that passed quality screening, drawn from 49 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions