EUVIMEDAtrasti EUVIMED
Crossref

Balancing Human and AI Features for Digital Autism Diagnostics from Home Videos: Human-In-The-Loop Machine Learning Study

Abdur Rasool, Kaitlyn Dunlap, Arman Husic, Aaron Kline, Parnian Azizian, Dennis P Wall, Peter Washington

JMIR AI · 2026-11-20 · 5 · e84699

DOI: 10.2196/84699 ↗

At a glance

Journal article in JMIR AI, published online on 2026-11-20. Written by Abdur Rasool and 6 co-authors. 43 references.

SantraukaBibliografiniai duomenysCiteRelated studies

Abstract (original language)

Background

Autism spectrum disorder (ASD or autism) diagnosis faces significant delays due to specialist shortages and subjective, clinic-based assessments. Although AI offers a potentially scalable solution via automated video analysis, current state-of-the-art deep learning models often fail to capture subtle, clinically nuanced social behaviors essential for reliable detection.

Objective

This study aimed to develop and evaluate a human-in-the-loop (HITL) framework that combines automated AI feature extraction with crowdsourced human annotation for autism classification on naturalistic home videos. We targeted low-confidence AI facial feature extraction outputs to be replaced by human features. We also sought to quantify the relationship between the confidence threshold used for swapping AI features with human features, the volume of human replacement of AI features, and the predictive diagnostic performance.

Methods

We implemented a HITL pipeline integrating deep learning models for facial emotion (7-class) feature extraction. A confidence threshold applied to the emitted probability of the AI feature extractor model was used to identify low-confidence facial features or those that the AI model was unable to make a prediction for (henceforth referred to as “unresolved”), which were then replaced by human “feature extraction” (ie, annotation) conducted by rigorously vetted crowd workers. We evaluated this approach on 100 high-quality videos (50 autism and 50 neurotypical) from the GuessWhat? digital therapeutic app. Performance was assessed across decision thresholds (0.1-0.9 in increments of 0.1) using sensitivity, specificity, F1-score, and area under the receiver operating characteristic curve (AUROC), with 5-fold cross-validation.

Results

Across decision thresholds, performance improved overall but was nonmonotonic at intermediate thresholds. At a threshold of 0.9, 59.7% (597/1000) of frames were replaced by human annotations, and mean sensitivity increased from 0.51 (SD 0.24) at a threshold of 0.1 to 0.66 (SD 0.10) at a threshold of 0.9, an absolute increase of approximately 15 percentage points. The HITL-enhanced multimodal model achieved an aggregate out-of-fold AUROC of 0.76, compared with 0.54 for the AI-only model.

Conclusions

In this proof-of-concept study, replacing AI features with human features tended to be associated with higher observed autism classification performance in this particular dataset. We targeted low-confidence and unresolved frames for human annotation. We note that these findings are exploratory and thus do not isolate the independent causal effect of human replacement or establish clinical diagnostic validity.

Cite

APA

Rasool, A., Dunlap, K., Husic, A., Kline, A., Azizian, P., Wall, D. P., & Washington, P. (2026). Balancing Human and AI Features for Digital Autism Diagnostics from Home Videos: Human-In-The-Loop Machine Learning Study. JMIR AI, 5, e84699. https://doi.org/10.2196/84699

Vancouver

Rasool A, Dunlap K, Husic A, Kline A, Azizian P, Wall DP, et al. Balancing Human and AI Features for Digital Autism Diagnostics from Home Videos: Human-In-The-Loop Machine Learning Study. JMIR AI. 2026;5:e84699. doi:10.2196/84699

BibTeX
@article{Rasool2026,
  author = {Rasool, Abdur and Dunlap, Kaitlyn and Husic, Arman and Kline, Aaron and Azizian, Parnian and Wall, Dennis P and Washington, Peter},
  title = {Balancing Human and AI Features for Digital Autism Diagnostics from Home Videos: Human-In-The-Loop Machine Learning Study},
  journal = {JMIR AI},
  year = {2026},
  volume = {5},
  pages = {e84699},
  doi = {10.2196/84699}
}
Start for free →