Evaluating Program Coherence and Participant Progress in a TOEFL Preparation Course Through a Logic Model
Authors
-
Taufik Suadiyatno
Universitas Pendidikan Ganesha
-
Ni Nyoman Padmadewi
Universitas Pendidikan Ganesha
-
I G.A. Lokita Purnamika Utami
Universitas Pendidikan Ganesha
-
Putu Suarcaya
Universitas Pendidikan Ganesha
DOI:
https://doi.org/10.36312/jolls.v6i3.5600Keywords:
Language program evaluation, Logic model, Participant progress, TOEFL preparationAbstract
Conventional evaluations of TOEFL preparation frequently emphasize end scores or participant satisfaction, which can obscure whether institutional resources, instruction, assessment, and intended longer-term benefits form a coherent program pathway. This study evaluated the coherence and participant-progress evidence of a TOEFL preparation course at Lembaga Pengembangan Pendidikan Nusa Tenggara Barat (LPPNTB) through a logic model. A qualitative case study supplemented by descriptive quantitative data integrated stakeholder interviews, classroom observation, institutional documents, try-out score records, and alumni datasets from 2021, 2022, and 2024. Qualitative evidence was coded deductively by the logic-model categories and inductively for implementation issues; descriptive numerical evidence was embedded to corroborate or complicate the qualitative interpretation. The findings indicate a generally coherent relationship among institutional support, trainer preparation, integrated reading-structure-listening instruction, guided practice, feedback, mock tests, and the intended goal of TOEFL readiness. Among 35 participants with complete Try-Out 1 and Try-Out 3 records, the mean changed descriptively from 426.00 to 474.11, and 32 participants recorded higher later scores. This pattern does not establish causation and must be considered alongside the available pretest-to-final means, which changed from 479.79 to 476.29. Alumni records suggested perceived usefulness for scholarships, English development, and career purposes, but were retrospective and self-reported. The course therefore demonstrated coherent program logic and promising, non-causal evidence of readiness. Priority improvements include diagnostic placement, differentiated digital assignments, intensive listening support, standardized assessment documentation, and longitudinal alumni tracking.
Downloads
References
Abulhul, Z. (2023). A logic model conceptualizes the initiative of applying for an inclusive education program. Journal of Practical Studies in Education, 4(6), 15-19. https://doi.org/10.46809/jpse.v4i6.74
Alderson, J. C., & Hamp-Lyons, L. (1996). TOEFL preparation courses: A study of washback. Language Testing, 13(3), 280-297. https://doi.org/10.1177/026553229601300304
Alderson, J. C., & Wall, D. (1993). Does washback exist? Applied Linguistics, 14(2), 115-129. https://doi.org/10.1093/applin/14.2.115
Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford University Press.
Bachman, L. F., & Palmer, A. S. (2010). Language assessment in practice: Developing language assessments and justifying their use in the real world. Oxford University Press.
Bayram, I., & Canaran, O. (2019). Evaluation of an English preparatory program at a Turkish foundation university. Journal of Language and Linguistic Studies, 15(1), 48-69.
Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 21(1), 5-31. https://doi.org/10.1007/s11092-008-9068-5
Booker, Z.-Z. (2019). A revised logic model for educational program evaluation. Journal of Modern Applied Statistical Methods, 18(2), Article eP3047. https://doi.org/10.22237/jmasm/1604190300
Bowen, G. A. (2009). Document analysis as a qualitative research method. Qualitative Research Journal, 9(2), 27-40. https://doi.org/10.3316/QRJ0902027
Brown, H. D., & Abeywickrama, P. (2019). Language assessment: Principles and classroom practices (3rd ed.). Pearson.
Clapham, K., Manning, C., Williams, K., O’Brien, G., & Sutherland, M. (2017). Using a logic model to evaluate the Kids Together early education inclusion program for children with disabilities and additional needs. Evaluation and Program Planning, 61, 96-105. https://doi.org/10.1016/j.evalprogplan.2016.12.004
Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE.
Creswell, J. W., & Poth, C. N. (2018). Qualitative inquiry and research design: Choosing among five approaches (4th ed.). SAGE.
Czosnek, L., Zopf, E. M., Cormie, P., Rosenbaum, S., Richards, J., & Rankin, N. M. (2022). Developing an implementation research logic model: Using a multiple case study design to establish a worked exemplar. Implementation Science Communications, 3, Article 90. https://doi.org/10.1186/s43058-022-00337-8
Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods (2nd ed.). McGraw-Hill.
Educational Testing Service. (n.d.). The TOEFL ITP Assessment Series. https://www.ets.org/toefl/itp.html
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods designs: Principles and practices. Health Services Research, 48(6 Pt 2), 2134-2156. https://doi.org/10.1111/1475-6773.12117
Guetterman, T. C., & Fetters, M. D. (2018). Two methodological approaches to the integration of mixed methods and case study designs: A systematic review. American Behavioral Scientist, 62(7), 900-918. https://doi.org/10.1177/0002764218772641
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112. https://doi.org/10.3102/003465430298487
Isbell, D. R., Kremmel, B., & Kim, J. (2023). Remote proctoring in language testing: Implications for fairness and justice. Language Assessment Quarterly, 20(4-5), 469-487. https://doi.org/10.1080/15434303.2023.2288251
Kiely, R., & Rea-Dickins, P. (2005). Program evaluation in language education. Palgrave Macmillan.
Koçer, P., & Topkaya, E. Z. (2023). Evaluating an English language writing course through logic model: Links between outcomes and activities. Innovational Research in ELT, 4(1), 14-31. https://doi.org/10.29329/irelt.2023.558.2
Lembaga Pengembangan Pendidikan Nusa Tenggara Barat. (2024). Data alumni peserta Rumah Bahasa LPPNTB tahun 2021, 2022, dan 2024 [Unpublished institutional dataset].
Lembaga Pengembangan Pendidikan Nusa Tenggara Barat. (n.d.-a). Silabus TOEFL [Unpublished institutional document].
Lembaga Pengembangan Pendidikan Nusa Tenggara Barat. (n.d.-b). Tutorial mengakses dan mengerjakan soal TOEFL di Web Rumah Bahasa [Unpublished institutional document].
Mayer, R. E. (2020). Multimedia learning (3rd ed.). Cambridge University Press.
Mazidah, E. N., & Masruroh, L. (2024). CIPP model evaluation on TOEFL course at higher education. Edulitics Journal, 9(2), 112-128.
Miles, M. B., Huberman, A. M., & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook (3rd ed.). SAGE.
Mohamed, F. T., Olamo, T. G., & Yemiru, M. A. (2025). Effect of differentiated instruction on primary EFL students’ writing performance and perception: The case of Grade 7 students in Hawassa City, Ethiopia. Social Sciences & Humanities Open, 11, Article 101230. https://doi.org/10.1016/j.ssaho.2024.101230
Norris, J. M. (2009). Understanding and improving language education through program evaluation: Introduction to the special issue. Language Teaching Research, 13(1), 7-13. https://doi.org/10.1177/1362168808095520
Norris, J. M. (2016). Language program evaluation. The Modern Language Journal, 100(S1), 169-189. https://doi.org/10.1111/modl.12307
Ormsby, S. M., & Morrow, J. A. (2019). Development of a logic model for use in evaluation of learning support programs. Journal of Student Success and Retention, 6(1), 1-26.
Patton, M. Q., & Campbell-Patton, C. E. (2022). Utilization-focused evaluation (5th ed.). SAGE.
Primary, M. A., Emilia, E., & Sukyadi, D. (2023). Program evaluation of English language learning for EYL curriculum development in Indonesia: Teachers’ perception, challenges, and expectation. Journal on English as a Foreign Language, 13(1), 314-338. https://doi.org/10.23971/jefl.v13i1.6068
Sailer, M., Maier, R., Berger, S., Kastorff, T., & Stegmann, K. (2024). Learning activities in technology-enhanced learning: A systematic review of meta-analyses and second-order meta-analysis in higher education. Learning and Individual Differences, 112, Article 102446. https://doi.org/10.1016/j.lindif.2024.102446
Saldaña, J. (2021). The coding manual for qualitative researchers (4th ed.). SAGE.
Shamsi, E., & Bozorgian, H. (2024). Collaborative listening using multimedia through metacognitive instruction: A case study with less-skilled and more-skilled EFL learners. Asian-Pacific Journal of Second and Foreign Language Education, 9, Article 25. https://doi.org/10.1186/s40862-023-00248-8
Smith, J. D., Li, D. H., & Rafferty, M. R. (2020). The Implementation Research Logic Model: A method for planning, executing, reporting, and synthesizing implementation projects. Implementation Science, 15, Article 84. https://doi.org/10.1186/s13012-020-01041-8
W. K. Kellogg Foundation. (2004). Logic model development guide. W. K. Kellogg Foundation.
Wang, P., & Cheng, L. (2025). The impact of TOEFL iBT preparation on Chinese test-takers’ perceptions of integrated speaking and writing design. Critical Inquiry in Language Studies, 22(1), 64-87. https://doi.org/10.1080/15427587.2024.2448815
Yin, R. K. (2018). Case study research and applications: Design and methods (6th ed.). SAGE.
Yu, G., & Green, A. (2021). Preparing for admissions tests in English. Assessment in Education: Principles, Policy & Practice, 28(1), 1-12. https://doi.org/10.1080/0969594X.2021.1880120
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Taufik Suadiyatno, Ni Nyoman Padmadewi, I G.A. Lokita Purnamika Utami, Putu Suarcaya
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with JOLLS agree to the following terms:
- For all articles published in JOLLS, copyright is retained by the authors. Authors permit the publisher to announce the work with conditions. When the manuscript is accepted for publication, the authors agree to implement a non-exclusive transfer of publishing rights to the journals.
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlike 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
