Precision telemedicine through crowdsourced ...
Document type :
Article dans une revue scientifique: Article original
DOI :
PMID :
Permalink :
Title :
Precision telemedicine through crowdsourced machine learning: testing variability of crowd workers for video-based autism feature recognition
Author(s) :
Washington, Peter [Auteur]
Stanford Medicine
Leblanc, Emilie [Auteur]
Pediatric Cardiac Surgery Services [Stanford]
Dunlap, Kaitlyn [Auteur]
Penev, Yordan [Auteur]
Kline, Aaron [Auteur]
Paskov, Kelley [Auteur]
Sun, Min Woo [Auteur]
Chrisman, Brianna [Auteur]
Stockham, Nathaniel [Auteur]
Varma, Maya [Auteur]
Voss, Catalin [Auteur]
Haber, Nick [Auteur]
Wall, Dennis P. [Auteur]
Stanford Medicine
Leblanc, Emilie [Auteur]
Pediatric Cardiac Surgery Services [Stanford]
Dunlap, Kaitlyn [Auteur]
Penev, Yordan [Auteur]
Kline, Aaron [Auteur]
Paskov, Kelley [Auteur]
Sun, Min Woo [Auteur]
Chrisman, Brianna [Auteur]
Stockham, Nathaniel [Auteur]
Varma, Maya [Auteur]
Voss, Catalin [Auteur]
Haber, Nick [Auteur]
Wall, Dennis P. [Auteur]
Journal title :
Journal of Personalized Medicine
Abbreviated title :
J Pers Med
Volume number :
10
Pages :
86
Publication date :
2020-08-13
ISSN :
2075-4426
Keyword(s) :
pediatrics
telemedicine
diagnostics
crowdsourcing
machine learning
autism
telemedicine
diagnostics
crowdsourcing
machine learning
autism
HAL domain(s) :
Sciences du Vivant [q-bio]
English abstract : [en]
Mobilized telemedicine is becoming a key, and even necessary, facet of both precision health and precision medicine. In this study, we evaluate the capability and potential of a crowd of virtual workers-defined as vetted ...
Show more >Mobilized telemedicine is becoming a key, and even necessary, facet of both precision health and precision medicine. In this study, we evaluate the capability and potential of a crowd of virtual workers-defined as vetted members of popular crowdsourcing platforms-to aid in the task of diagnosing autism. We evaluate workers when crowdsourcing the task of providing categorical ordinal behavioral ratings to unstructured public YouTube videos of children with autism and neurotypical controls. To evaluate emerging patterns that are consistent across independent crowds, we target workers from distinct geographic loci on two crowdsourcing platforms: an international group of workers on Amazon Mechanical Turk (MTurk) (N = 15) and Microworkers from Bangladesh (N = 56), Kenya (N = 23), and the Philippines (N = 25). We feed worker responses as input to a validated diagnostic machine learning classifier trained on clinician-filled electronic health records. We find that regardless of crowd platform or targeted country, workers vary in the average confidence of the correct diagnosis predicted by the classifier. The best worker responses produce a mean probability of the correct class above 80% and over one standard deviation above 50%, accuracy and variability on par with experts according to prior studies. There is a weak correlation between mean time spent on task and mean performance (r = 0.358, p = 0.005). These results demonstrate that while the crowd can produce accurate diagnoses, there are intrinsic differences in crowdworker ability to rate behavioral features. We propose a novel strategy for recruitment of crowdsourced workers to ensure high quality diagnostic evaluations of autism, and potentially many other pediatric behavioral health conditions. Our approach represents a viable step in the direction of crowd-based approaches for more scalable and affordable precision medicine.Show less >
Show more >Mobilized telemedicine is becoming a key, and even necessary, facet of both precision health and precision medicine. In this study, we evaluate the capability and potential of a crowd of virtual workers-defined as vetted members of popular crowdsourcing platforms-to aid in the task of diagnosing autism. We evaluate workers when crowdsourcing the task of providing categorical ordinal behavioral ratings to unstructured public YouTube videos of children with autism and neurotypical controls. To evaluate emerging patterns that are consistent across independent crowds, we target workers from distinct geographic loci on two crowdsourcing platforms: an international group of workers on Amazon Mechanical Turk (MTurk) (N = 15) and Microworkers from Bangladesh (N = 56), Kenya (N = 23), and the Philippines (N = 25). We feed worker responses as input to a validated diagnostic machine learning classifier trained on clinician-filled electronic health records. We find that regardless of crowd platform or targeted country, workers vary in the average confidence of the correct diagnosis predicted by the classifier. The best worker responses produce a mean probability of the correct class above 80% and over one standard deviation above 50%, accuracy and variability on par with experts according to prior studies. There is a weak correlation between mean time spent on task and mean performance (r = 0.358, p = 0.005). These results demonstrate that while the crowd can produce accurate diagnoses, there are intrinsic differences in crowdworker ability to rate behavioral features. We propose a novel strategy for recruitment of crowdsourced workers to ensure high quality diagnostic evaluations of autism, and potentially many other pediatric behavioral health conditions. Our approach represents a viable step in the direction of crowd-based approaches for more scalable and affordable precision medicine.Show less >
Language :
Anglais
Audience :
Internationale
Popular science :
Non
Administrative institution(s) :
INSERM
Université de Lille
Université de Lille
Submission date :
2022-06-15T13:59:06Z
2023-02-01T10:22:49Z
2023-02-01T10:22:49Z
Files
- jpm-10-00086-v3.pdf
- Version éditeur
- Open access
- Access the document