One major outcome of the Generative AI and Assessment research project has been the development of an assessment design framework based on assessment submissions and follow-up interviews with participants.
The 4C Assessment Design (4CAD) framework has emerged from an identified set of instructor approaches to addressing the increasing availability of generative AI and its use by students in various aspects of their learning.
These approaches are captured by four Cs:
-
Collaboration. The integration of generative AI as a working partner in the completion of the assessment. This approach tends to stress adaptability and interactivity with the technology as a means of demonstrating the learning outcome(s) the assessment is meant to measure.
-
Critique. The analysis and critical evaluation of AI-generated content, often accompanied by a reflection on the human–AI interaction. This approach tends to focus on the relevance, accuracy, utility, and overall value of AI outputs and the use of AI in the process of completing the assessment (or parts thereof).
-
Content Disclosure. The documentation of generative AI use. This approach tends to emphasize transparency around the inclusion of AI-generated content as well as the different ways in which AI is incorporated into the process of completing the assessment (or parts thereof).
-
Confirmation. The verification of student work. This approach tends to seek both compliance with policies at the course, program, and institutional levels and assurance that the student can demonstrate intended learning outcomes without the use of AI.
Putting the Pieces Together
Using these categories to describe and classify assessments offers something closer to a typology of approaches, rather than a scale or hierarchy. Furthermore, with 4CAD, we have also intentionally avoided classifying or evaluating task-based use cases for generative in assessment (e.g., AI is “good” or “bad” for doing X), anticipating that AI development and capability will likely outpace any such claims.
We chose to visualize the framework with an unconventional Venn diagram, showcasing the various interplay and overlap among the four functions.
Figure 1. A Venn diagram designed to allow for 11 possible assessment classifications: (1) Collaboration; (2) Critique; (3) Content Disclosure; (4) Confirmation; (5) Collaboration and Critique; (6) Collaboration and Content Disclosure; (7) Collaboration and Confirmation; (8) Collaboration, Critique, and Content Disclosure; (9) Collaboration, Critique, and Confirmation; (10) Critique and Content Disclosure; (11) Critique and Confirmation. The numbers on the figure in each area correspond to a unique ID associated with each assessment submission. The areas with asterisks (*) and hyphens (-) indicate overlapping categories for which we did not receive any relevant assessment submissions. In the discussion that follows, we consider the possible features of assessments that might hypothetically fall into these categories.
| ID | Assessment Description |
|---|---|
| #10 | Video Game Covers with Firefly |
| #13 | Generative AI and ChatGPT: What Do They Know? |
| #35 | Completing Incomplete Program Evaluation |
| #43 | In-Class Poetry Analysis |
| #68 | AI Use Document |
| #73 | AI Literature Project |
| #78 | Critiquing Multiple Choice Questions |
| #82 | ChatGPT Balanced Scorecard |
| #94 | Online Writing Session |
| #102 | Advanced Editing with AI Prompt Engineering |
| #164 | First-Year Drama Essay |
| —/*** | No assessments submitted (speculative) |
Table 1. A short description and link (if available) to the full assessment text for each assessment ID in Figure 1.
A Few Highlights
In our analysis, we found that most assessments instructors submitted fell into two or more categories. If an assessment suggested using AI as a collaborator on a specific task, it would often pair the collaborative function with some form of critique: at the very least, fact checking and reviewing any AI output before integrating it into the deliverable.
Some assessments did, however, incorporate the use of AI for collaboration or critique exclusively, rather than a mix of both, and a smaller number of assessments that allowed AI use did not require any disclosure of use.
At the time of our research (September 2023–April 2024), confirmation was largely viewed in terms of verifying student completion of an assessment without AI assistance. Hence, there were no assessments that allowed for the use of AI and also implemented some form of confirmation. To clarify, we understand confirmation as an instructor-imposed method of determining how the assessment was completed, rather than a student-centred method. Thus, confirmation differs from content disclosure, which situates agency with the student who is expected to be transparent about how they used AI. Invigilated assessment represents a form of confirmation with which most would be familiar. Some instructors in our study expressed a desire to return to in-class assignments, but we also saw confirmation in the form of oral exams or viva-style assessments following the submission of a written deliverable.
Since conducting this research, tools like Grammarly’s Authorship allow for confirmation to be applied to assessments that permit the use of AI by tracking and identifying human vs AI-generated writing throughout a submission. Instructors using technology like this could design an assessment that would fall into the area of the Venn diagram marked with hyphens (-): an assessment, for example, that tracks student use of AI in the writing process to ensure that it is being used in specific ways (brainstorming, creating an outline, identifying gaps, challenging arguments, etc.) and that it is not compromising the assessment’s overall integrity.
The other area of the Venn diagram to which we were unable to map any of the submitted assessments was the overlap between collaboration and critique without any content disclosure, indicated by the area with asterisks (*). One possible explanation for the lack of assessments in this space is that if an instructor is asking students to use AI in several different ways, they would likely want students to disclose and/or reflect upon these use cases.
What’s to Come
This research has more or less been paused since collecting and analyzing the assessments (winter 2024), conducting the follow-up interviews (winter/spring 2024), and finalizing the interview transcriptions (summer 2024). Next steps include a deeper reading of the interview transcripts within the context of the 4CAD framework and a final, published manuscript. These steps have been on hold due to the Principal Investigator (Ben Lee Taylor) cutting a postdoctoral fellowship short to accept a full-time role at McMaster University.
Despite the delay, however, the 4CAD framework has proven useful to instructors, educational developers, and others trying to grapple with and make sense of generative AI’s impact on teaching and learning in higher education.
Inquiries can be directed to Ben Lee Taylor.