Can a mobile app automate multimodal cognitive screening with speech and drawing?

A phone workflow that scores four response types

MemoCare is a React Native app for Android and iOS that runs a cognitive screening session without continuous examiner control. Each item pairs a spoken instruction with one input channel: the microphone for verbal answers, the device GPS for spatial orientation, the touchscreen for commanded actions, and the screen for copying a geometric figure. After completion, the app displays item-level scores and a total score, and stores assessment state locally on the device without requiring user accounts.

The design matters because established screening instruments such as the MMSE mix spoken responses, orientation, memory, calculation, commanded actions, language and visuoconstruction in a single assessment. The paper's premise is that mobile automation must therefore interpret each response type by its own task semantics and then combine them, rather than just display a questionnaire. English and Vietnamese versions share the same interaction flow, while language-specific scripts, normalization rules and spatial vocabularies adapt the workflow to each setting.

Speech scoring begins after transcription

Verbal answers are trimmed, transcribed by Google Speech-to-Text, normalized, and then scored by deterministic task-specific rules. Those rules cover date and number parsing, stateful sequential calculation, multi-target recall matching, ordered phrase matching and sentence-completeness checks. Hesitations and repetitions are tolerated through task-specific rules, and the scoring module combines the normalized transcript with task state so that later answers can depend on earlier ones.

A crucial boundary is built into the test suite: the 138 speech and language cases start from STT transcripts, so they evaluate the downstream scoring logic rather than Google STT accuracy. The paper also notes that for selected tasks, scoring prioritizes recognized task-relevant words and does not infer negation or correction from potentially incomplete transcripts. Readers should therefore treat the speech results as evidence about rule-based scoring, not about end-to-end speech recognition reliability.

Location answers are resolved before matching

Spatial orientation questions depend on where the phone is, and place names can remain valid after administrative renaming or boundary changes. The MemoCare spatial module holds 89,043 administrative boundary records across Vietnam, Switzerland, China and the United States, derived from OpenStreetMap country extracts. It indexes administrative geometries spatially so that candidate regions near the device are retrieved before point-in-polygon testing, instead of scanning the full geometry collection.

For a resolved location, the module returns accepted current and historical place names, normalizes them, and matches them against the STT transcript using exact and approximate name matching. This is how administrative aliases and renamed regions can be accepted. The spatial regression suite passed 48/48 fixed-coordinate cases across the four supported countries, and 8/8 spatial-answer cases passed. These are software verification results on predefined cases, not measures of how well the module handles every real-world location or every user's phrasing.

Touch tasks are verified from recorded events

For mobile-adapted action tasks, MemoCare scores recorded touchscreen events instead of speech. One interaction verifies a multi-step tap-and-drag sequence using action type, target and event order; another checks whether a touch falls within a predefined target region on a displayed image. All touch processing runs locally, without a camera or additional sensor.

The touch module passed 5/5 interaction-scoring cases in the test suite. That is a small number of predefined cases, so it supports the claim that the event-based verification logic works as specified, not that it generalizes across the full range of motor behavior seen in clinical populations.

Drawing accuracy depends on which metric you read

In the visuoconstruction task, a geometric reference appears on screen and the user copies it on the touchscreen. The captured bitmap is resized to 224×224 pixels, and three internal structural criteria, C1–C3, represent component-shape validity and two spatial-relation conditions. The app exposes only a binary item result, which is positive only if all three criteria are satisfied.

Because the manually drawn training set was limited, the authors built a rule-based generator that produced 45,724 synthetic geometric copying examples with known C1/C2/C3 labels, covering valid and invalid structures, relation errors, hard negatives, and visual variations such as blur, perspective, shadow, texture and missing edges. Eight lightweight CNN models from four architecture families were first trained on the synthetic data and then fine-tuned on the manually drawn training set. The development dataset contained 329 training, 66 validation and 71 locked-test drawings.

The two reported drawing metrics answer different questions. Mean balanced accuracy averages the three criterion-specific balanced accuracies, where balanced accuracy is the average of sensitivity and specificity; exact three-criterion accuracy requires all three predictions for the same drawing to be correct. On the locked 71-image test set, the validation-selected ShuffleNetV2 x1.5 reached 91.33% mean balanced accuracy but 78.87% exact three-criterion accuracy. A descriptive consensus of three models reached 95.17% mean balanced accuracy. The gap between the two single-model figures is the practical point: per-criterion performance can look strong while roughly one in five drawings still fails to get all three criteria right.

Clinician ratings cover the workflow, not patients

Four clinician co-authors inspected the end-to-end workflow across eight five-point criteria, yielding a pooled median rating of 4/5, with item-level medians ranging from 3 to 4.5. This is formative inspection by a small group of co-authors, not an independent clinical evaluation. The paper explicitly presents MemoCare as a research prototype rather than a validated diagnostic system, and states that formal patient-level clinical validation remains future work.

The technical test suite passed 151/151 predefined cases: 138 speech/language, 8 spatial-answer and 5 touch/action. Together with the 48/48 spatial regression cases, these results show that the implemented scoring components behave as specified on the authors' own test cases. They do not establish sensitivity or specificity for detecting cognitive impairment in any population.

Modular scoring keeps the workflow replaceable

The processing modules expose a common item-scoring interface, so individual components can be updated without changing the mobile flow. The drawing inference service sits behind a stable AWS endpoint, keeping the mobile client independent of the chosen vision backbone. A separate criterion-aware U-Net provides structured visual interpretation of component shapes and spatial-relation regions, and this explanatory model does not affect the binary drawing score.

This separation is the paper's most transferable engineering choice. It means the speech, spatial, touch and vision implementations can each be swapped or improved while the interaction workflow stays fixed. It also clarifies what the reported numbers measure: each module was verified on its own task-specific cases, and the drawing metrics come from a locked test set of 71 images, not from a clinical outcome study.

The evidence stops short of clinical accuracy

The paper's strongest claim is that a compact phone workflow can capture four response types and score them automatically, with software tests passing 151/151 predefined cases and the selected drawing model reaching 91.33% mean balanced accuracy on a locked 71-image test set. The weakest link for a reader judging clinical usefulness is that no patient-level validation is reported: the clinician input is a median 4/5 inspection by four co-authors, and the speech tests bypass STT accuracy by starting from transcripts.

The priority this paper sets is therefore engineering verification before clinical validation. If you are comparing screening tools, the number to watch is not the 91.33% mean balanced accuracy alone but the 78.87% exact three-criterion accuracy, because the app's binary drawing result requires all three criteria to be correct. MemoCare demonstrates a working multimodal scoring architecture; whether that architecture detects cognitive impairment in real patients remains untested here.

Sources used in this answer