RATERS’ PREJUDICES IN ORAL PERFORMANCE ASSESSMENT

Murat Polat, Emel Akay

Abstract


Research on testing speaking claims that raters’ beliefs, perceptions and even their prejudices may be involved in the process of grading although they are given a set of rubrics to stay on the same track and have stable qualities on the assessment of oral production that is why many researchers have studied the rationale of those beliefs and the amount it affects the scores. This study aimed to find out whether the perceptions and beliefs of raters’ play a significant role in testing speaking and question the role of experience in the involvement of such beliefs. To do that, a group of raters were asked to grade the audio recordings of a group of students twice with one-month-interval in between, being misinformed about the students’ physical appearances each time with the help of different pictures, and were interviewed later to identify whether their pre-conceptions on students’ physical appearances play a role in their grading oral performances. Also, the data obtained were used to draw some conclusions whether the raters intentionally or unintentionally used their beliefs in the grading process. The analysis revealed that student appearance may be significantly effective in teachers’ grading and this is true especially for experienced teachers who believe their judgements are true and unbiased more than the less experienced ones.

 

Article visualizations:

Hit counter

DOI

Keywords


performance assessment, rater prejudice, bias, halo effect, physical appearance

Full Text:

PDF

References


Akay, E. & Toraman, C. (2015). Students' attitudes towards learning English grammar: A study of scale development. Journal of Language and Linguistic Studies, 11(2), 67-82.

Aydin, B., Akay, E., Polat, M. & Geridonmez, S. (2016). Türkiye'deki Hazırlık Okullarının Yeterlik Sınavı Uygulamaları ve Bilgisayarlı Dil Ölçme Fikrine Yaklaşımları. Anadolu University Journal of Social Sciences. 16(2), 1-19.

Bachman, L.F. (2004). Gender Bias in the Classroom. Journal of legal education. 23(4), 137-146.

Banks, L. B. (1998). Teacher Cognition in Grammar Teaching: A Literature Review. Language Awareness, 12(2), 96-108.

Berger, J., Hammit, F., Norman, R. & Zelditch, M. (1977). Status Characteristics and Social Interaction: An Expectation State Approach. NY: Elsevier Scientific Pub. Co. Inc.

Boyce, M.W. (1979). Physical attractiveness- a source of teacher bias? Australian Journal of Teacher Education. 4(1), 34-44.

Centra, J. & Gaubatz, N. (2000). Is there gender bias in student evaluations of teaching? The journal of higher education. 70(1), 17-33.

Crisp, V. (2012). An investigation of rater cognition in the assessment of projects. Educational Measurement: Issues and Practice, 31(3), 10-20.

Crocker, L. & Algina, J. (1986). Introductıon to Classical and Modern Test Theory. CBS Collage Publishers Canpany. USA.

Cronbach, L.J. (1995). Giving method variance its due. In D.T. Gilbert & S.T. Fiske (Eds) Personality research methods and theory (145-157). Hillsdale, NJ. Lawrence Ebaum Ass. Inc.

Dennis, I. & Newstead, S. (1990). Blind marking and sex bias in student assessment. Assessment and evaluation in higher education. 15(2), 132-139.

Eagly, A. & Mladinic, A. (1994). Are people prejudiced against women? Some answers from research on attitudes, gender, stereotypes and judgements of competence. European Review of Social Psychology. 5(1), 1-34.

Eckes, T. (2005). Examining rater effects in TESTDAF writing and speaking performance assessments: A many-facet Rasch analysis. Language Assessment Quarterly. 2(3), 197-221.

Francis, B., Robson, J. & Read, B. (2001). An analysis of undergraduate writing styles in the context of gender and achievement. Studies in Higher Education. 26(3), 313-326.

Harding, S. (1991). Whose science? Whose knowledge? Buckingham, Open University Press.

Hedge, T. (2000). Teaching and learning in the language classroom. Oxford, England: Oxford University Press.

Hoyt, W.T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it? Psychological Methods. 4(2), 403-424.

Johnson, J. & Lim, G. (2009). The influence of rater language background on writing performance assessment. Language Testing. 28(4), 485-505.

Kenyon, D.M. (1992). Introductory remarks at symposium on development and use of rating scales in language testing. Teachers Forum, Columbia University.

Kim, H. J. (2015). A qualitative analysis of rater behavior on an L2 speaking assessment. Language Assessment Quarterly. 12:239-261.

Kondo, K. (2002). A FACETS analysis of rater bias in measuring Japanese second language writing performance. Language Testing. 19(1), 3-31.

Krawczyk, M. (2017). Do gender and physical attractiveness affect college grades? Assessment & Evaluation in Higher Education. DOI: 10.1080/02602938.2017.1307320

Langlois, J.H., Kalakanis, L., Rubenstein, A.J., Larson, A., Hallam, M. (2000). Maxims or myths of beauty? A meta-analytic and theoretical review. Psychol Bull. 26(3):390-423.

Murphy, P. & Elwood, J. (2002). Constructions of achievement and the positioning of students: a gender perspective. Pedagogy, Culture and Society. 10(2), 134-152.

Myword, C. M. & Wolfe, E. W. (2003). Detecting and measuring rater effects using many facet rasch measurements: Part1. Journal of Applied Measurement. 4, 386-422.

Newstead, S. (1996). The psychology of student assessment. The Psychologist: Bulletin of the British Psychological Society. 9, 543-547.

Newstead, S. & Dennis, I. (1990). Blind marking and sex bias in student assessment. Assessment and Evaluation in Higher Education. 15, 132-139.

Polat, M. (2017). Teachers’ attitudes towards teaching English grammar: A scale development study. International Journal of Instruction. 10(4):379-398. DOI: 10.12973/iji.2017.10422a

Read, B., Francis, B. & Robson, J. (2005). Gender, bias assessment and feedback: analysing the written assessment of undergraduate history essays. Assessment & Evaluation in Higher Education. 30(3), 241-260.

Reed, D.J. & Cohen, A.D. (2001). Revisiting rater and ratings in oral language assessment. Studies in Language Testing11: Experimenting with uncertainty (pp82-96). Cambridge, UK.

Schaefer, E. (2008). Rater bias patterns in an EFL writing assessment. Language Testing, 25(4), 465-493.

Talamas, S., Mavor, K. & Perrett, D. (2016). Blinded by beauty: Attractiveness bias and accurate perceptions of academic performance. Plos One. 11(2), 1-18.

Umberson, D. & Hughes, M. (1987). The Impact of Physical Attractiveness on Achievement and Psychological Well Being. Social Psychology Quarterly. 50(3), 227-236.

Wigglesworth, G. (1993). Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction. Language Testing. 10(3), 305-319.




DOI: http://dx.doi.org/10.46827/ejel.v0i0.1249

Refbacks

  • There are currently no refbacks.


Copyright © 2015 - 2023. European Journal of English Language Teaching (ISSN 2501-7136) is a registered trademark of Open Access Publishing Group. All rights reserved.

This journal is a serial publication uniquely identified by an International Standard Serial Number (ISSN) serial number certificate issued by Romanian National Library (Biblioteca Nationala a Romaniei). All the research works are uniquely identified by a CrossRef DOI digital object identifier supplied by indexing and repository platforms.

All the research works published on this journal are meeting the Open Access Publishing requirements and can be freely accessed, shared, modified, distributed and used in educational, commercial and non-commercial purposes under a Creative Commons Attribution 4.0 International License (CC BY 4.0).