EUVIMEDEUVIMED entdecken
EvaluationsstudieOpen Access

Evaluation of using the SOCcer automated occupation auto-coding algorithm to assist expert coding of job descriptions

Pabitra R Josse iD, Daniel E Russ iD, Leila Orszag, Cheryl E Peters, Grace Shen-Tu, Jennifer E Vena, Melissa C Friesen iD

Annals of Work Exposures and Health · 12.05.2026 · 70 (4)

DOI: 10.1093/annweh/wxag031 ↗

Kurz gefasst

Evaluationsstudie in der Zeitschrift Annals of Work Exposures and Health, online erschienen am 18.05.2026. Verfasst von Pabitra R Josse und 6 weiteren Autor:innen. Beteiligte Einrichtung: Occupational and Environmental Epidemiology Branch, Division of Cancer Epidemiology and Genetics, National Cancer Institute, 9609 Medical Center Driv…. Hauptthemen laut MEDLINE: Algorithms, Job Description, Soccer, Occupations. Die Arbeit nennt 28 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Intramural Research Program of the National Institutes of Health (NIH), NIH, U.S. Department of Health and Human Services.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background/objective Systems to automatically code job descriptions to standardized occupations, such as SOCcer, reduce the burden of expert coding in occupational studies, but expert coding remains needed. We evaluated how frequently experts select a SOCcer-suggested code when using the SOCAssign platform, which shows SOCcer's top 10 scoring codes (Si, with i denoting results ranking, compared to independent coding).

Methods

We evaluated 2 protocols for SOCcer-assisted coding using the SOCAssign platform that we undertook when we had insufficient resources to employ the best practice of independent double-coding, with discordances resolved by consensus. Protocol 1 used a single coder, with a supervisor providing additional review for a subset based on features we identified that were more likely to indicate potential discordance between coders. Protocol 2 allocated jobs to be coded by 1 or 2 coders based on the SOCcer score (the probability an expert would have selected that code) for the highest scoring code (S1 score) and the score difference between the top 2 scoring codes. For comparison purposes, we included a third set of jobs that had previously undergone independent expert coding. For each job, we identified if the expert selected the top-scoring (S1) code, or one of the top 3 (S1–3) or top 10 (S1–10) scoring codes from SOCcer and compared findings across coding protocols, overall and by S1 score.

Results

In Protocol 1, 25% of 6,824 jobs underwent supervisory review. In Protocol 2, jobs with an S1 score <0.2 or S1 − S2 score difference of <0.2 underwent coding by 2 experts (45% of 7,381 jobs); the remainder were coded by one expert. In both protocols and in independent coding, experts selected S1, one of S1–3 or one of S1–10 with similar prevalences (48% to 50%, 62% to 64%, and 73% to 74%, respectively). Expert coders were more likely to deviate from the SOCcer-suggested list for jobs with low score/score differences (39% to 44%) than high score/score differences (11% to 13%). In all evaluations, the frequency of selecting a SOCcer-suggested code increased with score, with some attenuation observed when the S1 − S2 score difference was <0.2.

Conclusions

Our evaluations provide support for SOCcer-assisted expert coding. We provide recommendations for identifying jobs where multiple reviewers may be most needed.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

AlgorithmsJob DescriptionSoccerOccupationsHumans

Stichwörter der Autor:innen

epidemiologymethod evaluationoccupation coding

Autor:innen und Institutionen

  • Occupational and Environmental Epidemiology Branch, Division of Cancer Epidemiology and Genetics, National Cancer Institute, 9609 Medical Center Drive, Rockville, MD 20850, United States.
  • Data Science and Engineering Research Group, Division of Cancer Epidemiology and Genetics, National Cancer Institute, 9609 Medical Center Drive, Rockville, MD 20850, United States.
  • BC Centre for Disease Control, 655 West 12th Avenue, Vancouver, BC V5Z 4R4, Canada.
  • BC Cancer, 675 West 10th Avenue, Vancouver, BC V5Z 1L3, Canada.
  • Occupational and Environmental Health Division, School of Population and Public Health, 2206 East Mall, UBC, Vancouver, BC V6T 1Z3, Canada.
  • Alberta's Tomorrow Project, Cancer Research & Analytics, Cancer Care Alberta, Alberta Health Services, 3395 Hospital Drive NW, Calgary, AB T2N 5G2, Canada.

ORCID-Profile: Pabitra R Josse, Daniel E Russ, Melissa C Friesen

Förderung

  • Intramural Research Program of the National Institutes of Health (NIH)
  • NIH
  • U.S. Department of Health and Human Services
  • Intramural NIH HHS

Zitieren

APA

Josse, P. R., Russ, D. E., Orszag, L., Peters, C. E., Shen-Tu, G., Vena, J. E., & Friesen, M. C. (2026). Evaluation of using the SOCcer automated occupation auto-coding algorithm to assist expert coding of job descriptions. Annals of Work Exposures and Health, 70(4). https://doi.org/10.1093/annweh/wxag031

Vancouver

Josse PR, Russ DE, Orszag L, Peters CE, Shen-Tu G, Vena JE, et al. Evaluation of using the SOCcer automated occupation auto-coding algorithm to assist expert coding of job descriptions. Annals of Work Exposures and Health. 2026;70(4). doi:10.1093/annweh/wxag031

BibTeX
@article{Josse2026,
  author = {Josse, Pabitra R and Russ, Daniel E and Orszag, Leila and Peters, Cheryl E and Shen-Tu, Grace and Vena, Jennifer E and Friesen, Melissa C},
  title = {Evaluation of using the SOCcer automated occupation auto-coding algorithm to assist expert coding of job descriptions},
  journal = {Annals of Work Exposures and Health},
  year = {2026},
  volume = {70},
  number = {4},
  doi = {10.1093/annweh/wxag031}
}
Kostenlos starten →