The goal of Part 1 - Identify Skills is to identify which skills instructors or course designers aim to or claim to address in the course. At the end of Part 1, reviewers should be able to answer these three questions:
CAST Framework
CAST Framework
The Four Concept Areas of the Framework
Identify Skills
Answering this question encourages reviewers to explicitly identify a specific, limited set of skills, beyond topical content, that the course introduces, reinforces and operationalizes for students. This is important/valuable because…
- Being explicit reduces ambiguity for students and instructors; puts emphasis on 21st century skills that are not always ‘called out’ in outcomes
- Being specific encourages quality over quantity; more powerful to have a focal set of skills; prevent overload
- And reflecting can reveal which skills the course is most fit to teach; are these skills in demand in the growing STEM industries; who are the learners?
Example methods:
- Interview course designers/instructors
- Written reflection activity for course designers/instructors
Answering this question encourages reviewers to use a common language to name and describe the nature of the skills aimed to be or claimed to be taught. This is important/valuable because….
- Establishing consistent terminology internally will encourage consistency within the course and course documentation (externally)
- Establishing a common understanding of what those skills mean and how they are operationalized in the course will make it easier to identify where in the course those skills are addressed
- And enable the team to understand what ‘counts’ and what doesn’t count as ‘that skill’ in the context of the course.
Example methods:
- Develop skills rubric
- Adopt/adapt existing skill framework
Answering this question encourages reviewers to better align the goals of the course (what skills students should learn/practice in the course) with learner expectations (what skills do students think they will learn/practice in the course). This is important/valuable because...
- Aligning goals and expectations can improve learner satisfaction
- Strengthening ties between what is taught in the course and what is promised to be taught in external documents— like syllabus, course listing, advertisements— will help learners connect with the upskilling/reskilling opportunities they need.
Example method:
- Conduct a content analysis of externally facing documents or materials (e.g., description in course catalogue, advertising material, publicly published syllabus).
Review Course
The goal of Part 2- Review Course is to understand the ways in which the course structures support students’ learning and practice of priority skills. Part 2, reviewers should be able to answer these questions:
Answering this question encourages reviewers to connect Part 1 with Part 2 — taking inventory of the course and physically pointing to the lesson or activity where the Identified Skills from Part 1 appear.
Example Methods:
- Asynchronous structural review
- Open-ended course observation
- Structured course observation
- Interviews
In addition to noting the occurrence of a skill, reviewers are encouraged to note the characteristics of each occurrence, in search for traces of processes/structures that support deeper learning. For instance, when is the skill being introduced for the first time? And at what point is the skill reinforced with additional instruction or scaffolding? At what point and in what ways are students prompted to practice the skill?
Example Methods:
- Asynchronous structural review - introduced, practiced, reinforced
- Open-ended course observation
- Structured course observation
- Interviews
Assess Outcomes
The goal of Part 3 - Assess Outcomes is to verify if, and the extent to which, students learned the identified skills through the course. In Part 3, reviewers should be able to answer this question:
Answering questions related to student engagement with, growth in, or achievement of target skills is ultimately essential for determining how successfully the course supports student skill development and can provide formative insight into what concepts or skills need to be reinforced or practiced to a greater degree throughout the course.
Results compiled from Review Course and Assess Outcomes can help reviewers understand how well, if at all, the skills most strongly represented in the course align with the skills students report learning and reveal any misalignments.
Example Methods:
- Student pre-post surveys
- Student retrospective survey
- Student interviews
- Embedded assessment
Check for Transfer
The goal of Part 4 - Check for Transfer is to understand which skills students have learned most deeply and have been able to apply in situations beyond the course (aka skill ‘Transfer’ from one setting to another). In Part 4, reviewers should be able to answer these questions:
Answering this question can be challenging, as it may require getting feedback from students after the course concludes. Thus, one suggested strategy is to integrate post-course check-ins (e.g., 3 month, 6 month and/or 12 month) with students as part of the course in the form of surveys, interviews or focus groups. If post-course contact with students is not possible, LinkedIn or university profiles can serve as proxy data.
While it is likely not realistic to make discrete causal links between course learning outcomes and a student’s ability to apply a skill, getting a gauge from students as to how strongly they attribute their ability to use a skill to the course, can further the course instructor or designer’s claim that the course delivers on promised skills (or prompt revisions to make certain skills more visible/explicit if attribution is lower than expected).
Bringing it all Together
Once a course assessment is complete, a meeting amongst the course facilitators (course designer, instructors, etc.) should be conducted to reflect on the outcomes of the assessment and co-interpret results of each Part. In the meeting, course facilitators can decide what, if any, adjustments should be made to the course materials (e.g., syllabus, catalog descriptions, course content) or the structure of the course.
A simple place to start with implementing changes to the course is with the course syllabus and making sure the skills that are being taught in the course are properly described in the syllabus. By using an existing skills terminology a course instructor can “translate” their syllabus into more universally recognized terms for certain skills and define these skills and how they are being introduced and practiced in the course. A course facilitator can do this on their own or use a proprietary service like EMSI’s Skillabi, which helps course instructors translate their syllabus.
The outcomes of the course assessment may result in large structural changes to the course, such as working in new exercises and activities which can help refine 21st-century skills along with technical skills, like adding writing assignments or group assignments.