Why would a voice be harder to ignore than a fan?
The leading explanation is not that speech is simply louder or more variable. When researchers matched speech and non-speech sounds for acoustic complexity, then destroyed the speech-like quality of the sounds, speech still disrupted serial recall more than the non-speech sounds did [3]. Serial recall, the task used in these studies, means hearing or seeing a short list and reproducing it in order, a standard way to measure short-term memory.
One interpretation offered is that speech is biologically relevant: a voice may carry information about the surroundings that the brain is inclined to monitor, so it draws processing resources away from the task [3]. That is a proposed mechanism, not a demonstrated purpose. Notably, the same study found that intelligibility itself was not the deciding factor: whispered speech, and even whispered speech with its fine structure reversed so that it was hard to understand, disrupted recall as much as normal whispered speech [3]. What mattered was the formant structure, the vocal-tract resonances that give speech its characteristic sound shape, rather than whether the listener could make out the words [3].
The memory evidence points one way
The clearest result is for short-term memory: speech-like sound is more disruptive than non-speech sound matched for acoustic complexity [3]. The finding also refines what “intelligible” means in this context. Degrading non-words so they were misheard reduced their disruptive effect compared with clear non-words, and sequences with changing vowels were more disruptive than sequences with changing consonants [3]. So the acoustic detail of the speech signal matters, even though full comprehension is not required for the disruption to occur [3].
This matters for interpreting office noise. A conversation you can follow and a conversation you can only half-hear may both interfere with holding a short list of items in mind, because the interference appears to ride on the sound structure of speech rather than on meaning [3]. The practical implication is that masking a conversation with generic broadband sound may not remove the speech-specific part of the distraction.
Mental arithmetic: the comparison is set up, but the answer is not in
Mental arithmetic has been studied alongside office noise, including direct comparisons of office noise containing speech with non-speech office noise [1]. That design is exactly the one needed to answer the arithmetic half of the question. However, the material available here reports the comparison rather than a clear outcome, so it would be overreach to say speech is worse for arithmetic than non-speech noise [1].
Work on background meeting speech likewise frames short-term memory and mental arithmetic as outcomes of interest and examines word recognition and subjective impressions of the acoustic environment [5]. Again, this establishes that the question is actively studied in realistic office settings, but it does not supply a decisive speech-versus-non-speech result for arithmetic [5]. The honest summary is that the memory result is relatively clear while the arithmetic result remains open in this evidence.
Why the task can change the answer
Short-term memory tasks such as serial recall depend on keeping a precise sequence of sound-like items in order, which is plausibly why a changing speech stream interferes [3]. Mental arithmetic draws on different processes, and the same study that proposed an attentional explanation for speech's extra disruptiveness suggested that listeners reallocate processing resources when biologically relevant sound is present [3]. If that reallocation is the mechanism, the size of the effect could depend on how demanding the arithmetic is and how much attention the voice captures, rather than on intelligibility alone.
Individual differences also matter. In one study, musicians outperformed non-musicians on a short-term memory test in two noise conditions, while non-musicians scored higher in quiet, and musicians showed smaller pupil responses, a physiological index of listening effort [4]. That study compared noise levels rather than speech with non-speech office noise, so it does not settle the main comparison, but it is a reminder that “background noise” is not one uniform burden [4]. Separately, memory for non-speech sound features has been linked to speech-in-noise perception and to performance on a cognitive screening test, which suggests that how well a person holds sound features in mind shapes real-world listening [2]. Neither result tells us that intelligible speech beats non-speech office noise for arithmetic; they show that the listener and the task are part of the equation [2][4].
Sources used in this answer
Mental arithmetic and non-speech office noise: An exploration of interference-by-content
The study compares mental arithmetic under office noise containing speech versus non-speech office noise, directly setting up the speech-versus-non-speech comparison for arithmetic.
The contribution of short-term memory for sound features to speech-in-noise perception and cognition
Auditory short-term memory for non-speech sound features predicted speech-in-noise perception and explained part of the variance in a dementia screening test, but the study did not compare intelligible speech with non-speech office noise.
Contrasting effects of irrelevant speech and non-speech sounds on short-term memory
Speech disrupted serial recall more than acoustically matched non-speech sound, and the disruption depended on formant structure rather than intelligibility, since degraded or reversed whispered speech was as disruptive as normal whispered speech.
Auditory Short-Term Memory Evaluation in Noise in Musicians
Musicians had better short-term memory scores than non-musicians in two noise conditions and showed smaller pupil-growth responses, indicating less listening effort, though the study compared noise levels rather than speech with non-speech office noise.
Effects of Background Meeting Speech and Non-Directional Sound Masking on Word Recognition and Subjective Evaluation of the Acoustic Environment
The study examines background meeting speech and non-directional sound masking in relation to word recognition, short-term memory, mental arithmetic, and subjective acoustic evaluation, but does not provide a decisive speech-versus-non-speech result for arithmetic.