Evaluation of using the SOCcer automated occupation auto-coding algorithm to assist expert coding of job descriptions
Annals of Work Exposures and Health · 12.05.2026 · 70 (4)
DOI: 10.1093/annweh/wxag031 ↗Kurz gefasst
Evaluationsstudie in der Zeitschrift Annals of Work Exposures and Health, online erschienen am 18.05.2026. Verfasst von Pabitra R Josse und 6 weiteren Autor:innen. Beteiligte Einrichtung: Occupational and Environmental Epidemiology Branch, Division of Cancer Epidemiology and Genetics, National Cancer Institute, 9609 Medical Center Driv…. Hauptthemen laut MEDLINE: Algorithms, Job Description, Soccer, Occupations. Die Arbeit nennt 28 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Intramural Research Program of the National Institutes of Health (NIH), NIH, U.S. Department of Health and Human Services.
Abstract (Originalsprache)
Background/objective Systems to automatically code job descriptions to standardized occupations, such as SOCcer, reduce the burden of expert coding in occupational studies, but expert coding remains needed. We evaluated how frequently experts select a SOCcer-suggested code when using the SOCAssign platform, which shows SOCcer's top 10 scoring codes (Si, with i denoting results ranking, compared to independent coding).
Methods
We evaluated 2 protocols for SOCcer-assisted coding using the SOCAssign platform that we undertook when we had insufficient resources to employ the best practice of independent double-coding, with discordances resolved by consensus. Protocol 1 used a single coder, with a supervisor providing additional review for a subset based on features we identified that were more likely to indicate potential discordance between coders. Protocol 2 allocated jobs to be coded by 1 or 2 coders based on the SOCcer score (the probability an expert would have selected that code) for the highest scoring code (S1 score) and the score difference between the top 2 scoring codes. For comparison purposes, we included a third set of jobs that had previously undergone independent expert coding. For each job, we identified if the expert selected the top-scoring (S1) code, or one of the top 3 (S1–3) or top 10 (S1–10) scoring codes from SOCcer and compared findings across coding protocols, overall and by S1 score.
Results
In Protocol 1, 25% of 6,824 jobs underwent supervisory review. In Protocol 2, jobs with an S1 score <0.2 or S1 − S2 score difference of <0.2 underwent coding by 2 experts (45% of 7,381 jobs); the remainder were coded by one expert. In both protocols and in independent coding, experts selected S1, one of S1–3 or one of S1–10 with similar prevalences (48% to 50%, 62% to 64%, and 73% to 74%, respectively). Expert coders were more likely to deviate from the SOCcer-suggested list for jobs with low score/score differences (39% to 44%) than high score/score differences (11% to 13%). In all evaluations, the frequency of selecting a SOCcer-suggested code increased with score, with some attenuation observed when the S1 − S2 score difference was <0.2.
Conclusions
Our evaluations provide support for SOCcer-assisted expert coding. We provide recommendations for identifying jobs where multiple reviewers may be most needed.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Occupational and Environmental Epidemiology Branch, Division of Cancer Epidemiology and Genetics, National Cancer Institute, 9609 Medical Center Drive, Rockville, MD 20850, United States.
- Data Science and Engineering Research Group, Division of Cancer Epidemiology and Genetics, National Cancer Institute, 9609 Medical Center Drive, Rockville, MD 20850, United States.
- BC Centre for Disease Control, 655 West 12th Avenue, Vancouver, BC V5Z 4R4, Canada.
- BC Cancer, 675 West 10th Avenue, Vancouver, BC V5Z 1L3, Canada.
- Occupational and Environmental Health Division, School of Population and Public Health, 2206 East Mall, UBC, Vancouver, BC V6T 1Z3, Canada.
- Alberta's Tomorrow Project, Cancer Research & Analytics, Cancer Care Alberta, Alberta Health Services, 3395 Hospital Drive NW, Calgary, AB T2N 5G2, Canada.
ORCID-Profile: Pabitra R Josse, Daniel E Russ, Melissa C Friesen
Förderung
- Intramural Research Program of the National Institutes of Health (NIH)
- NIH
- U.S. Department of Health and Human Services
- Intramural NIH HHS
Zitieren
APA
Josse, P. R., Russ, D. E., Orszag, L., Peters, C. E., Shen-Tu, G., Vena, J. E., & Friesen, M. C. (2026). Evaluation of using the SOCcer automated occupation auto-coding algorithm to assist expert coding of job descriptions. Annals of Work Exposures and Health, 70(4). https://doi.org/10.1093/annweh/wxag031
Vancouver
Josse PR, Russ DE, Orszag L, Peters CE, Shen-Tu G, Vena JE, et al. Evaluation of using the SOCcer automated occupation auto-coding algorithm to assist expert coding of job descriptions. Annals of Work Exposures and Health. 2026;70(4). doi:10.1093/annweh/wxag031
BibTeX
@article{Josse2026,
author = {Josse, Pabitra R and Russ, Daniel E and Orszag, Leila and Peters, Cheryl E and Shen-Tu, Grace and Vena, Jennifer E and Friesen, Melissa C},
title = {Evaluation of using the SOCcer automated occupation auto-coding algorithm to assist expert coding of job descriptions},
journal = {Annals of Work Exposures and Health},
year = {2026},
volume = {70},
number = {4},
doi = {10.1093/annweh/wxag031}
}
EUVIMED