Validity in language assessments under scrutiny

The South African Rasch PLC (Professional Learning Community) members met on the Science, Mathematics and Technology Education Department at University of Pretoria’s Groenkloof Campus to further their understanding of measurement and assessment in education early in May 2018.

Dr Graham Dampier (Head of Department of the Integrated Student Success Initiative at the University of Johannesburg) gave an address about validity and language assessment. Dr Dampier provided bold and fascinating insight into the concept of validity and its challenges in language assessments. This sparked a lively debate about the problems of defining validity, the challenges of translating assessments and the difficulties faced in South Africa to follow best practices, such as designing instruments with panels. Some of the myths surrounding validity according to Dr Dampier include: that validity has one definition, that validity is fixed and stable and that all validity is construct validity.

Dr Dampier argued that validity should be a process, one in which a case is made for the applicability of the instrument in a certain context and for a specific population.

Following Dr Dampier’s seminar, Mr Mishack Tshele (Data Manager at the Centre for Evaluation and Assessment) presented a more practical demonstration on how to prepare data for a Rasch analysis. Mr Tshele focused on preparing data for the RUMM programme, but most of his instructions are also useful for other Rasch software. Mr Tshele led the group through the processes of setting up the data in excel, making sure the data are clean, creating code books and importing the data into RUMM. Both the theoretical and practical aspects of measurement using the Rasch models were explored in this successful PLC meeting.