Low Stakes, High Rewards: Cracking the Code to Successful Assessments

Camie Wood 0:00

When was the last time you took a really big test? In this episode, we're going to speak with James Martin, an instructional designer at Global Campus. And he's going to tell us a little bit about low stakes testing how that's beneficial to your students, and how you can incorporate in your class.

Do you ever remember a time when you were a student, or even as a professional person, getting ready to take a high stakes test that either made you really nervous, or you ended up doing worse on? Because it was a lot of information that you were covering? But also, because you were so nervous, you couldn't really concentrate on the material?

James Martin 1:04

Yes. First, the AC t, then the GRE, all of those big bubble dot tests, where you read through some book for you know, the 60 days before you take it and then go in and only hit well, you're stuck there for hours, you know, artificial environment, you can't look anything up. You can't. It's yeah, I hate those things I've never been, I've never been a big fan. And that's what when you say tests, that's what I think of is that. And I think a lot of students when they hear you say test, that's what they think of. And tests don't have to be like that.

Camie Wood 1:37

They don't but and those large tests like that can really induce a lot of testing anxiety, and cause poor performance. And then also, even though you've been reading that big book for 60 days, how much of that? Can you remember now? Absolutely. So James, tell us a little bit about yourself.

James Martin 2:00

a year, I started in December: 2021

Camie Wood 3:23

So James, the other day, you and I were talking about low stakes testing and how much you love it. So tell me a little bit about why you believe low stakes testing is really important in courses.

James Martin 3:35

So one of the things I often recommend to instructors is they include weekly reading quizzes over whatever text they're assigning. But before I do that, I kind of walk them through the value of the things culturally, I think we do a bad job with testing. A lot of when I say test, in fact, you know, you're immediately gonna think about the AC T, or the GRE or some awful thing that you take once and it's an hour's long commitment. And then if you have a bad day, you do a bad job. And it's not really reflective of your ability. It's it's reflective of a lot of things. So I think we do testing wrong in a lot of ways. And we and even in academic classes, sometimes. I know when I was in college, some classes would be based around really two huge exams. And those were most of your grade. That's not what I'm recommending in any class, or what I would recommend in any class. What I'm talking about here, what I do recommend, and there's good science to you reasons to back this up are regular short, low stakes, objective quizzes, let's call them quizzes, that seems less scary if they're quizzes. And I started doing this back when I was teaching online. I taught American Lit and my discussions weren't doing so well. There weren't you know, there wasn't a lot of engagement there. And the reason it wasn't on engagement, there's you know, obviously by the way, we're taught lit if people weren't doing the reading or they weren't doing it in time for that week's discussions, so we weren't have anything going on in the discussion forums. So I started creating these short quizzes, most of them were just five questions, multiple choice so that the, you know, the LMS could grade them for me automatically. And, you know, if I kept those around the next term, I would expand them sometimes to 10 questions, but I kept it in that kind of like, five to 10, maybe as many as 15, for a longer thing. And those were, well, you can cover a lot of ground and 10 questions, so that that I could, I could ask a question from the first page of the story, the last page of the story and going all the way through it. So I could tell as an instructor whether they were getting it or not, or where they dropped off. You know, every if everybody did a bad job on it, then it was usually just a bad quiz, because there's an art and a science to making quizzes that are fair. And I really wanted to be fair, I didn't want them to be tricky. Another bit with reading quizzes, in general, is there's a tendency to want to play gotcha with students. But that doesn't help. There's no pedagogical reason to try to trip people up. So what I wanted was to give them some skin in the game, because students are, are practical, especially non traditional students, they put their time where they have to put their time. And if you don't make it to where they're putting time and your class that they need to, then they won't, and that's reasonable. I think they have to divide their time somewhere. So the whole idea of adding some accountability to the things I'm asking them to read was a way to get that base level understanding of a text so that we could have a discussion about it.

Camie Wood 6:39

Yeah, yeah. I know, we've all experienced those funky test. It's where you feel like you have crammed and crammed and crammed your brain full of everything that you can, and you're probably still going to fail it.

Amalie Holland 6:57

And there have been some studies done to show that huge tests, there is not a lot of retention, with huge tests. And there was there were some studies done where students took an end, of course exam, and then re took that same exam three months later after the summer. And they scored significantly worse. Over that summer, they're not retaining information.

James Martin 7:19

Yeah. And what was cool about what was cool about the science, once I started all this, just intuitively, I was like, it just made sense. I mean, we've all had teachers who quizzed us over stuff to make sure that we read it. So I like I was taking that idea and running with it. But in an electronic version, where it could grade it for me, which was nice. But the what I discovered later, when I dig, dug into the science of it is that it increases retention. So you know, if you do that same sort of experiment where you test somebody right after the test, and you test them again, a week later in a week after that, to see how memory falls off. You, you stay higher on the curve over time, if you've been quizzed over it, just the mere act of having to answer some questions. And I don't know if the effect there is that it sharpens your mind to know that you're going to have to quiz over something or if just the mere act of taking the quiz helps solidify it in your head, regardless of how you do but whatever the case, the the effect is they retain that information longer if he wasn't over it. So in a way, you're doing them a favor on that level, too. You're increasing retention simply by, you know, having some really what seems like a cinci easy, objective quiz. And I want it to be easy. I even told the kids I'm like I said, I would never ask you about a character's eye color. Unless it's relevant to the plot, and it almost never is theirs. I can think of one novel where that's the case. In every other case, it's not, it's not necessary. So again, it's not a matter of gotcha, it's a matter of, of, of taking advantage of this mechanism that exists for retention.

Camie Wood 8:52

So shorter quizzes over long test helps students retain better, but what about it being low stakes? Why is that better for students?

James Martin 9:03

Well, that gets rid of a lot of the problematic side effects of tests. You know, it's there's nobody gets test anxiety over a five question. Objective, easy quiz, right. And so, the nice thing about making them simple, is what you're trying to do here is not so much assess their knowledge that you get to do that too, but to, to just encourage them to do the thing you want them to do, which is to read. And for the students who don't take you seriously, they're not going to bomb too many quizzes before they, you know, say oh, he means it, he really I actually have to read these things. So then they start reading so you kind of there's a gentle and punishment when they when they don't take you seriously and read the stuff but eventually that that goes away really quickly. You get what you get out of that the first week or so. And then what what you find is just a higher level of compliance. Now, I mean, you could abuse that too, I guess, but like, you know, in this case, It was just the base level of reading that you needed in order to have a useful discussion about the things we were talking about.

Amalie Holland: 10:08

I think that's what's interesting is keeping that from being arbitrary questions. I mean, they're objective, but they're sort of pointless. They're just, did you read it? Haha, gotcha, you didn't read it. So, by not doing that, you get the students fail it, it's a consequence, it's not a punishment. It's just, that's the natural consequence of them not doing the reading.

James Martin: 10:32

Yeah, you build trust with them in that way, because you're really rewarding them, you know, they get to take their little five, and you know, the end this whole thing. So low stakes to in terms of their overall grade, like all of these quizzes together, regardless of how many of them you have might be 5% of the grade, maybe 10% of the grade, something like that. So you're still gonna grade them on those higher level tasks, where they're assimilating things and using their creativity. But you can't do those things unless you get the base level covered. Right? If they don't know what happened in the text, then they can't ping off that into interesting directions, right. So this, this lays the foundation. So even in a, even an upper level class, you can you can quiz people on their reading, you're, you're doing them a favor if you do it, because then they they show up.

Amalie Holland: 11:18

If they show up to a task, without the supplies, they can't do the task. And if they're not doing the reading, they're not getting the supplies.

James Martin: 11:28

Quizzes are also a good barometer of class performance. If everyone fails the quiz, then that either means it's a bad quiz, or it means that they didn't get it right. In the same way. If you quiz people and one student does very poorly, then that could be a useful indication that that students struggling with something either with that content or with something outside of that content. And that gives you an opportunity for that regular and substantive interaction that I'm sure we'll be talking about on one of these podcasts. That gives you a indicator that that student might need you to reach out and say, Hey, what what happened with with this quiz, so that gives you some data, you don't have that you can't just stare into people's eyes and know whether they got it or not, it would be great if he could if he feels like you can, but you can't. So giving them these assignments as an instructor also gives you some useful indicators of what's going on with your class, in terms of the knowledge they're picking up or failing to. And doing them

Amalie Holland: 12:21

more frequently means that you're catching things before the very end.

James Martin: 12:27

Yeah, and I started doing this when I was when I was doing it. And I recommend this to for people that want to get started with it is try to do one a week, you know, you might give them several reading assignments in a week, just pick one of them and do a quiz over it. And that gives you an easy way to build into it. And it's still like, you know, rightful semester, that's what 15 quizzes that can take some time building good quizzes is not not easy. But if you make them short and make them objective and not tricky, then those are easier to build, then tricky ones or elaborate ones. And five doesn't seem like enough questions. But it is enough questions, you keep it from being onerous, you you give them your, again, you're trying to just reward a certain sort of behavior and gently punish the lack of it. So that's what they're for, they're not really to try, did they get every detail of this piece, you don't care, you want them to have read it so that they can build on that in other sorts of assignments. So I'd start with one a week, and then eventually you can build it all the way out in my in my own. You know, teaching eventually, I had a quiz over everything, if I gave you something to read, there was going to be a quiz over it. And they knew that and it helped, you just had to read stuff you couldn't like get around it. Now there are kids who like, okay, it's only 5% my grade, you know, I'll do without 5% of my grade, you know, so you can decide where you're going to adjust that value. But those kids, I mean, you know, you can't reach everybody, the rank and file, just say, Oh, this is something I have to do now, instead of something that I might potentially blow off or that I'll do sometimes, but other times. And since you've got a date on it, you know, then you've got, you know, they've crossed that threshold at a certain date. So then you can plan your discussions and other kinds of assignments around their baseline knowledge of the text that you assign. After all, as a instructor, you decided this material was important. You've put together what you think of as essential readings for this topic. Wouldn't it be great if they read them?

Amalie Holland: 14:22

Well, and that comes back to to if you can't come up with the things that they really need to know in that quiz. Maybe you need to rethink whether what you are assigning to them is actually necessary or is it just you really liked it? I was guilty of that all the time.

James Martin: 14:37

Sure, like a lot of instructional design things. It really just it's a matter of being more thoughtful and methodical about the stuff that goes into your class. And so, you know, again, like you said, if you if you can't think of five questions to ask about this, then maybe it doesn't matter. Maybe Maybe you just take it off the list and put something else on there.

Camie Wood: 14:55

So you mentioned that if you assigned it by the end of This experiment that you started in your class, if you assign that reading, they were going to have a quiz on it. So how did you construct that? Did you have a quiz for every single assignment as an individual quiz? Or did you create a quiz that quiz them on other readings at one time?

James Martin: 15:22

Yeah, there's there's a case to be made for cumulative quizzing, but I kept it individual, it was just easier for me that way. Also, when I went back and looked at my class, you know, between semesters, it was easy to look at this quiz for A Rose for Emily and see if it was any good or not. It was like look at it's like, oh, wait, that's kind of a tricky question, or, okay, this is five, it's kind of long story. I'd like to add some questions and make this one a little longer. And over like, really long texts, you can always quiz by chapter of course, or if it's a novel, you can quiz by chapters or sections or whatever is relevant. Or you can you know, and you don't do cover, there's not, you don't cover everything, right. Like a lot of times for really long, you know, because I've teaching novels and things like that. So like, sometimes I'd have initial quiz to get them started, that might cover the first handful of chapters, and then I might quiz some of the middle chapters. And then I might quiz, you know, the final couple of chapters. So I didn't necessarily cover the entire work. But I did like I was, I was thoughtful about where in the work I was, because I didn't want there to be 10 questions on the last page, you know, or on the first I wanted to spread them out through it, because I was looking to try to like drag them along through the text. I mean, that was the goal. So if you keep the goal in mind, then the way you construct it is easier.

Amalie Holland: 16:35

I imagined that allows as an instructor for you to be able to pace the class better. When you have everybody moving at that same, you're checking to see that everybody's at the same spot. Absolutely.

James Martin: 16:45

We just had Richard conversations in the in the discussion boards. And those were like in those classes, that was just participation stuff. So it's like I had a layer of quizzing and a layer of participation. But the participations were better, because, you know, people had done the reading, not that, not that you'll get everybody, and nothing works for everybody.

Camie Wood: 17:03

So James, you talked about how you started doing this in your class. But I know since then, you've also read a book that you said, helped you feel validated in what you did in your class.

James Martin: 17:17

So after I've been doing this for a while, I was reading a book just for professional development as ID here at Big Global Campus, and it's called Make It Stick by Peter Brown, Mark McDaniel, and Henry Rodin. Sure it's from 2014. And it's on Harvard up. And it's a book that gets kicked around in instructional design circles quite a bit. There's a handful of books out there that people say, oh, you should read this. And so it was just on my list of things to read at some point. The subtitle is the Science of Successful Learning, which gives you some idea of what it's about. So it's, it's rather unlike most pedagogy books. Brown, worked as a management consultant. And he's also a novelist. McDaniel and Rancher are both psychologists and researchers at Washington University in St. Louis. McDaniel directs something called circle, which is the Center for Creative research on cognition, learning and education. And the other researcher guy runs something called the memory lab. So they're both involved in they're both curious about learning and retention. They're not educators, per se, but they're, you know, they're psychologists and, and psychological researchers. So a lot of the stuff in their book, the book covers a lot of topics. Some of those are about things you might have heard of, like the Dunning Kruger effect, and the ways that memory you know, doesn't work or your perception of expertise can be off. Metacognition stuff, but like the early chapters of the book are a lot about testing. And I feel like they, they spend some time on it making the case for it. And I've come around to their way of thinking that testing gets a bad rap because we do it so badly. And because we do it in these high stakes, one shot ways, which is not what they encourage. Because as Amelie pointed out, that doesn't help pedagogically speaking, it doesn't help you learn stuff. It just shows that you were able to get over a certain mark on a certain day and prove your worth. But you know, our my goal wasn't that. And I'd say you know, most instructors goal isn't that what they want, is they want their students at the end of the day to learn the stuff that they sat down in front of them because they know it's important because they know their field. So this just gives you a way to help ensure

Amalie Holland: 19:36

that that happens. Teaching is the one profession where we job shadow somebody for 15 years. You have watched teachers your whole life. And the main way that they have done things for most of us is through testing. And so it's natural, I think for teachers to for instructors to fall into that. I'll give them a big test. So that's what we do we do big tests. And this is this is close. This is little tests, it's a little there's a little test, it's a B test. So it's not completely outside the, the wheelhouse. It's not, it's

James Martin: 20:12

not like cheating at first, I thought, you know, and I did it in my high school classes, too. And they're like, really? It's five, it's five question quiz. I'm like, yeah, it's five questions, but there's a method to my madness. I didn't go into it like we have here. I'm gonna six, I just want to make sure you read stuff. You know. So yeah, that's what we would do. But yeah, yeah, it seems, rinky dink five questions, especially for a grad student or something. But it's not the science is there to back it, you know, the intuitive feel of it doesn't really matter here, if the goal is get people reading the stuff so that you can have good conversations or so they can write papers on the stuff or do any of that higher level stuff. Then, this works. And it's easy to implement, every LMS has the ability to create multiple choice questions, and auto grade them for you. So that's, it's teacher friendly, that's something you don't have to deal with. You can deal with, you know, getting the feedback on, you know, essays and stuff like that in other assignments, but it's okay to have this too. Sometimes I think we get in the in a mistaken idea that if it's a higher level class, that everything in it is higher level, but there's, there's base level knowledge type stuff that low on the Bloom's Taxonomy, list, those things you have to understand, too, you know, if you don't understand when the Civil War was, you can't talk about it in a useful way, right? Hey, you have to know these things.

Camie Wood: 21:39

I really think that that's something people misunderstand about Bloom's a lot of times is that it's not that once you get to these higher level class, all you're doing is creating and analyzing and evaluating, it's that you get to that point in the class, because you're starting to analyze some of that foundational knowledge. But you have to go through the other steps to make sure that students are with you and understand that foundational knowledge, because they can't analyze what they don't understand.

James Martin: 22:10

There you go. There's no There's a lot of there's a lot of esoteric knowledge in in higher level classes there, you know, it's knowledge you don't get to in the in the lower level classes, but it's still knowledge, you still have to commit it to memory and habit as a working part of your mental models of the world. So this just covers that this quizzing bit, covers that not drawing inferences, not doing any of that higher level stuff, just knowing what happened. And it works. I've mentioned it in the context of stories, because that's my background. But you can do it with anything, you can do it with any nonfiction piece, too, you can do it with criticism, you can do it with videos, for that matter, I've never taken it quite that far. But that's something that you can totally do. If you assign video content, then you can write quizzes over video content. So anyway, it's a simple idea with broad applicability. And I would encourage anybody building an online class or revising existing one to take advantage of it, because you know, just baby steps, start by adding a quiz over something, you gave them the first week or two classes and build out from there. You don't have to do it all in one day, or already for one term, because you're gonna teach this class over and over. As you as you look at it for revision, take a look at your text and say, Oh, I never, I never gave them a quiz or this, why don't I build a little quiz over this? That'll make it just that much better, that much more effective?

Camie Wood: 23:33

I love it. Absolutely. Love it. How would you define low stakes testing for instructors?

James Martin: 23:42

low stakes in this case, to me is five questions or 10 questions, or five or 10 points, and the whole thing is only five or 10% of your overall grade.

Camie Wood: 23:51

Why is low stakes testing important to include an online courses, online courses?

James Martin: 23:59

Engagement is you know, we're always looking for engagement online. And sometimes we think that means, you know, all kinds of really interesting ways of interacting online. And it does mean that but but this is also a type of engagement, your you're getting them to engage with the text so that then they have knowledge that they can use in other parts of the class, including in online discussions.

Camie Wood: 24:21

What are the best ways for instructors to incorporate low stakes testing in their courses?

James Martin: 24:28

Just take advantage of that quiz tool that's built into Blackboard. And don't be shy about making it just a bunch of multiple choice questions. There's no reason that you need to be using this as a feedback mechanism. It's already going to be a feedback mechanism in there, he'll see how the class performs on that on that quiz. And you'll see how individual people performance is gonna give you some data but this isn't a thing where you want them to draw an inference and then you have to spend time like giving them feedback. Say that feedback for papers and projects and and presentations use this as just a baseline really simple objective thing.

Camie Wood: 25:05

What should instructors do? If students, a lot of students in their class, fail the quiz or fail a particular question?

James Martin: 25:18

Yeah, so, a quiz design is deceptively simple, it's easy to ask a question that you and this is actually one of the topics that gets mentioned later on in, make it stick. You've all had that, that math teacher who was so good at math, obviously gifted, but it could not get it across to you because they couldn't think with that beginner mind. It's really easy. As an as you already know, these texts, probably intimately whatever you're assigning you've read them, lots of times, you're very familiar with what's in them, it can be hard sometimes to do a quiz that is fair to someone brand new to the text. So my first, you know, not that I'm all that humble. But my first approach, if someone if the whole class bombs a test, I think it's probably the test, you know, so I'd want to go back and look at it and make sure these really are questions that that people can get on the first read. And that's tricky. It would be great if you could find somebody, some dear colleague who would be willing to read the thing for the first time and happens to not have read it, whatever it is, and take a quiz over it. That would be helpful. But you can usually suss it out, you can easily look at it and go, Oh, that's kind of that's kind of obscure, that's not really an important takeaway. For someone just reading I think, and you know, other students think what should a person come away from this text, knowing if they read it with 80% of their available attention, you know, because that's more real world. Don't assume that they read it with a fine tooth comb or anything, because then you're going to get gotcha land. And the point here is not to build ill will with your students. The point here is to reward your students for doing what you wanted them to do to reward the behavior that you're asking for, which is to read the stuff.

Camie Wood: 26:57

Yeah. So you would say a key part of low stakes testing is to one evaluate yourself and how you are assessing students. But also, it's a tool of building trust with students in the class.

James Martin: 27:11

Absolutely. And it's weird that it would be but it totally is. They'll be on your side pretty quickly, once they take a couple of really easy quizzes. And realize that what you care about is them reading stuff, not trying to find some way to make sure they don't get full credit, let them get full credit, let everybody in the class get 100% on every quiz, and it's only going to be five or 10% of their overall grade, it doesn't matter. They'll feel like you gave them a big gift.

Amalie Holland: 27:36

It's probably especially helpful when you actually come back to that information that you quiz them on in the discussion or in the rest of the course in that because then it definitely isn't a gotcha. They needed it. You see where it is?

James Martin: 27:47

That's right. There you go. And that would be the ultimate test of was it was it a good question, right? Does it build into the rest of your class in any in any useful way?

Camie Wood: 27:58

Thanks for joining us today on The Pedagogy Toolkit. We'll see you next time.

Transcribed by https://otter.ai