Explain what is meant by the terms "validity coefficient" and "reliability coefficient."
Validity and Reliability
1. Validity and Reliability Joanna Ochoa V.
2. Validity To assert that a test has construct validity; empirical evidence is needed. Subordinate forms of validity. A valid test measures accurately what it is intended to measure. Content validity Criterion-related validity
3. Content Validity The content of the test constitutes a representative sample of the skills it is supposed to measure = Content Validity Specification of the skills or structures that the test is meant to cover. The test must include a proper sample of the relevant structures.
4. Criterion-related validity The test and the criterion are administered at the same time. Example: Oral exam. Long vs. The short version of the exam. Random sampling. Levels of agreement = correlation coefficient. Perfect agreement coefficient of 1. Lack of agreement = coefficient of zero. Concurrent validity The degree to which a test can predict candidates’ future performance. Example: Proficiency test to predict a student’s ability to cope with a graduate course at a British university. Criterion measure: student’s English perceived by his supervisor or the outcome of the course. Predictive validity
5. Validity in scoring Items and the way in which they are scored must valid. Example: A reading test. (Should we consider grammar and spelling mistakes in the responses?)
6. A test is said to have face validity….. For example, A test to measure pronunciation ability.
7. How to make tests more valid The scoring must be related to what is testing. Reliability!! Whenever feasible, use direct testing. Write explicit specifications for the test.
8. Reliability We have to: construct, administer and score items in a way that we will obtain similar results in different situations.
9. The reliability coefficient To quantify the reliability of a test. Ideal reliability coefficient 1 Would always give the same results. The reliability coefficient of zero Sets of results unconnected with each other.
10. It is required to have two sets of scores to be compared. A group of students take the same test twice. TEST-RETEST METHOD 1. Too soon (memorization of the answers) 2. Too late (forgetting) solution Alternate forms methods Split half method = only one administration of one test
11. Scorer reliability If the scoring of a test is not reliable, then the test results cannot be reliable either. For example, The scorer reliability coefficient on composition writing test = .92 The reliability coefficient for the test = .84 Variability in the performance of individual candidates accounted for the difference between the two coefficients.
12. How to make tests more reliable Take enough samples of behaviour Exclude items (weaker vs. Stronger students) Do not allow too much freedom Write unambiguous items Provide clear and explicit instructions Ensure that tests are well laid out and perfectly legible Make candidates familiar with the format and testing techniques Provide uniform conditions of administration.
- 1. More items more reliability
- 2. Too easy and too difficult items
- 3. Choice of questions
- 4. Unclear meaning of the items
- 5. The supposition that the students all understand the instructions
- 6. Institutional tests are badly typed
- 7. Unfamiliar aspects of test
- 8. Precautions must be taken
13. Ways of obtaining scorer reliability Use items that permit scoring which is as objective as possible Make comparisons between candidates as direct as possible Provide a detailed scoring key Train scorers Agree acceptable responses and appropriate scorers at outset of scoring Identify candidates by number, not name Employ multiple independent scoring
14. Reliability and validity To be valid a test must provide consistently accurate measurements. A reliable test may not be valid at all. Example: writing test To make tests reliable, we must be wary of reducing their validity.