Choosing Students for the Assessment
Once a program has developed student learning outcomes, identified where in the curriculum students will demonstrate mastery of those outcomes, aligned one or more assignments or (other artifacts) with those outcomes, and selected the right timing for an assessment, it’s important to think about how to choose which students and their work to include in an assessment project.
Most programs will apply a sampling method to obtain student work, especially when undertaking course-embedded assessment. In some cases, it may not always be possible or feasible to collect student work samples from every student in a large program. It also keeps the process manageable where there are multiple or lengthy artifacts to review.
If a program is small, assessing the entire population of students (e.g., all seniors, all students who are part of a leadership program) enables the department to draw more accurate conclusions about student learning. It may take a program longer to gather the relevant evidence for all students in the program (e.g., instead of taking work from a given quarter of the year, it may take a year or more to accumulate the work on the entire population of students).
Sample Size Considerations
Assuming that a program has determined it will sample just part of the population, it then should determine the appropriate sample size and method for the assessment. Four factors should be considered: achieving a reasonable percentage of the available student population; the length and complexity of the artifacts (student work products); the number of faculty or staff who will be involved in evaluating student work; and the expected variation in student work.
If the program is quite large, programs may elect to sample between 10 and 20 percent of the students. If the work samples are large or complex (for example, multiple pieces of student work per student), a program may sample only 10 percent of the students, especially if there aren’t many raters available. If the variation in student work is considerable, a program may choose to expand the sample so that the program is better able to detect the trends in student performance.
When a program elects to sample some students in a program, it is important to select those students randomly. A simple random sample can be achieved by using a random number generator tool (there are many software and online options available for this task.) While faculty or staff may prefer to share work from students that performed the strongest on the selected work products, or select work from students who received varying grades on an assignment (e.g., low, middle-range, high), it’s ultimately more advantageous to stick with randomly selected students. When random sampling is properly applied, a program can learn a lot more about variation in student performance within a program than it can when solely evaluating the work of students who received the highest grades.
The program’s Curriculum Alignment Matrix will identify the relevant courses to include in an assessment. Sometimes only one course is relevant and then it's a good idea to include multiple sections of that course, especially when taught by different faculty. If multiple courses address the learning outcome under study, it's best to sample student work from each. This can be done in any typical quarter or if course offerings are less frequent, sampling can be done throughout the year. These are ways to increase the representativeness of the sample.
Please note that the sample sizes used for assessment are typically smaller than those used for social science research. Additional guidelines for appropriate sample sizes can be found in Mary Allen's, Assessing General Education Programs (available in Educational Assessment). Mary Allen notes that generally if almost all the students in a program are doing well or poorly, small samples (e.g., 25 students) are sufficient, and that many general education or core assessments draw upon approximately 50-75 students.