Building A Validity Argument For A Listening Test Of Academic Proficiency PDF Download

Language testing is a ubiquitous practice in immigration contexts used as a data collection procedure to assess immigrants' ability to communicate in the language of the host country to promote social as well as economic integration and productivity in the workplace (McNamara & Shohamy, 2008). Unlike English tests, little attention has been directed to the interpretation and uses of scores from French proficiency tests, which prompts - indeed, requires - validation research to justify test use. Drawing on advances in test validity theory (Kane, 2006, 2013), this study builds a validity argument for the listening component of the Test de connaissance du français (TCF) in the context of Quebec immigration. Test validity theory has evolved considerably since the traditional tripartite model of content, predictive and construct components (Cronbach & Meehl, 1955), have been conceptualized as a unitary construct (Messick, 1989) and more recently have been theorized in terms of argumentation (Kane, 2006, 2013), borrowing concepts from models of inference (Toulmin [1958], 2003), which include scoring, generalization, explanation, extrapolation and decision inferences that play key roles in a validity argument. In an argument-based approach to validity, claims about testing instruments are composed of warrants that must be supported by backings in the form of empirical studies, which are foundational for the claims, but also support the inferences that authorize each of the claims in the argument. More specifically, this study gathered empirical evidence to support the scoring, generalization and explanation inferences, proposing three research questions that addressed construct representation, potential bias and test method usefulness. The questions were concerned with the listening subskills that the TCF assesses, differential item functioning (DIF) across gender, first language, age, and geographical location as well as the option functioning of multiple choice (MC) items in the assessment of second language listening comprehension. Although multiple statistical and measurement models are readily available to analyze test response data, this study privileged confirmatory factor analysis (CFA) to examine the listening subskills operationalized in the TCF, specifying the models following suggestions from a panel of experts. The unidimensional Rasch model was used to generate the difficulty parameters across subgroups of interest to perform the DIF analyses. And the nominal response model (NRM) was used to model the response options of the MC items. The results from these three studies yielded backings for each of the selected inferences in the validity argument for the TCF. Based on the CFA models recommended by the panel of experts, the results suggested that the TCF test forms under study primarily assess examinees' understanding of explicitly stated information in aural discourse, thereby underrepresenting the listening construct. A few items were found to target the ability to infer implicit ideas and understanding of the general topic or main idea, however, this latter subskill was only found in one test form, suggesting that the forms are not equivalent. The DIF analysis flagged multiple items across test forms and between the subgroups of interest, but very few were associated to potential bias, which included speech perception, literary genre and vocabulary familiarity. Thus, given that many items flagged for DIF could not be associated to a potential bias, this question was partially answered and attenuates the validity argument. The results from the NRM suggested that most items functioned well while others were potentially doubled keyed. The argument-based approached to validity proved helpful in putting together empirical evidence into a coherent whole to support and build a case for the interpretation and uses of the TCF in the context of immigration, which in turn can be used to address the identified weaknesses, providing a means to attenuate the potential rebuttals that threaten the validity of the argument. Some caveats in the validation framework were also outlined and relate to the accessibility of data to address the extrapolation and decision inferences in immigration contexts, but as Newton and Shaw (2014, p. 142) advocated "the argument-based approach underlies the fact that validation is not simply a one-off-study but a program: potentially a very intensive program". And this program can include key stakeholders such as government officials that help complete the validity argument for the TCF in Quebec immigration.