EUVIMEDEUVIMED entdecken
Crossref

Evaluación de desempeño en PIS: Diseño de rúbricas y fiabilidad interevaluador

Margarita Johanna Rocafuerte López, Filomena Agripina Quezada Sanmartin, Nidia Sharon Torres Aguilar, Esperanza Narcisa Mejía Ruiz

Horizonte Científico Educativo International Journal · 2025

DOI: 10.64747/xeasmv38 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Horizonte Científico Educativo International Journal, 2025. Verfasst von Margarita Johanna Rocafuerte López und 3 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To examine inter‑rater reliability of an analytic rubric for Integrated Knowledge Projects (PIS) in 12th grade and to outline design decisions for formative and summative uses.

Methods

Observational study in public urban schools in Guayaquil (−2.170997°, −79.922359°). A total of 400 student products (reports, presentations, prototypes) were scored by 4–6 raters per product using a six‑criterion, four‑level rubric. A pilot (n≈40) informed descriptor refinement and variance estimation. The protocol comprised training and calibration with an anchored bank. Analyses included Gwet’s AC1 per criterion (95% CI), ICC(2,k) for the total score (0–100), mixed‑effects models for product‑type comparisons, and, in a subsample (n≈120), generalizability studies (G‑coefficient and phi), plus sensitivity checks (re‑weighting, adjudication exclusion, and re‑calibration).

Results

AC1 ranged from 0.72 to 0.84, higher for technical criteria (Evidence/Data; Rigor) and lower for interpretive dimensions (Collaboration; Ethical Impact and Feasibility). Overall ICC(2,k) reached 0.88 (mean k≈4.8; ICC(2,1)=0.69); the performance‑by‑k curve showed diminishing returns after k=5, with an operational “sweet spot” at k=4–5. The p×i×r subsample yielded G=0.86 and phi=0.82; most error variance was due to product×rater interactions. Prototypes scored lower than reports and presentations, especially on Communication and Impact. Sensitivity analyses supported metric stability and showed reduced “rater” variance after re‑calibration.

Conclusions

The PIS‑BGU rubric demonstrates good‑to‑excellent reliability for the total score and moderate‑to‑high reliability by criterion, supporting both summative decisions and formative feedback. Recommendations include using k=4–5 raters, protocolized adjudication when discrepancies exceed one level, and strengthened calibration for interpretive criteria. Future work will incorporate models with rater severity and external validation in rural settings and other STEAM areas

Zitieren

APA

López, M. J. R., Sanmartin, F. A. Q., Aguilar, N. S. T., & Ruiz, E. N. M. (2025). Evaluación de desempeño en PIS: Diseño de rúbricas y fiabilidad interevaluador. Horizonte Científico Educativo International Journal. https://doi.org/10.64747/xeasmv38

Vancouver

López MJR, Sanmartin FAQ, Aguilar NST, Ruiz ENM. Evaluación de desempeño en PIS: Diseño de rúbricas y fiabilidad interevaluador. Horizonte Científico Educativo International Journal. 2025. doi:10.64747/xeasmv38

BibTeX
@article{Lpez2025,
  author = {López, Margarita Johanna Rocafuerte and Sanmartin, Filomena Agripina Quezada and Aguilar, Nidia Sharon Torres and Ruiz, Esperanza Narcisa Mejía},
  title = {Evaluación de desempeño en PIS: Diseño de rúbricas y fiabilidad interevaluador},
  journal = {Horizonte Científico Educativo International Journal},
  year = {2025},
  doi = {10.64747/xeasmv38}
}
Kostenlos starten →