9 - Reassessment Carnivals: An Interview with Kate Owens

Episode 9 • 12th September 2023 • The Grading Podcast • Sharona Krinsky and Robert Bosley

Kate: One of the great things about alternative grading is the fact that the students can go back and see what topics they have successfully done so far and which ones they're still working on, or which ones they still need help with. And I think that even if they come to the carnival and then decide like, wait a second, I didn't know this problem's going to have a sphere and I forgot the formula for the surface area of a sphere, I'm not doing this problem, just their process of looking back through their grade book and identifying what things they feel confident on and which ones they still need to study, and having to make that choice and having that information to help them make that choice, is really important as they're going into their final exam. And before alternative grading, I don't know that my students would've known what they struggled with, if, at best, maybe they knew that, like test two, that was a hard one. That was the one they struggled with. But there was no actual like, learning thing that would've helped them see, oh, you know what it was, it was that geometry stuff, or it was those word problems or those graphs that I was working with. But now my grade book shows them that information, it still available to them later.

Bosley: Welcome to the grading podcast, where we'll take a critical lens to the methods of assessing students' learning. From traditional grading to alternative methods of grading. We'll look at how grades impact our classrooms and our students' success. I'm Robert Bosley, a high school math teacher, instructional coach, intervention specialist and instructional designer in the Los Angeles Unified School District and with Cal State LA.

Sharona: And I'm Sharona Krinsky, a math instructor at Cal State Los Angeles, faculty coach and instructional designer. Whether you work in higher ed or K-12, whatever your discipline is. Whether you are a teacher, a coach, or an administrator, this podcast is for you. Each week you will get the practical, detailed information you need to be able to actually implement effective grading practices in your class and at your institution.

Welcome everyone back to the pod. This is Sharona, one of your co-hosts, and I'm here with Bosley, the other co-host. How you doing today, Boz?

Bosley: I'm doing really well. I'm actually really excited because it's not just the two of us today. We actually have another guest. This is another great math educator and one of the founding members of the grading conference, and I know one of your personal mentors, so I'd like to welcome Dr. Kate Owens to the podcast. How are you doing tonight, Kate?

Kate: Hey guys. It's so good to see you both.

Sharona: I know, it's really fun because we actually get to record and see each other, but as people are listening, they don't get to so. Kind of a fun thing.

Kate: This is true. This is true.

Bosley: So if you'll just, I'd like to turn it over to you just to kind of introduce yourself Kate, and kind of give your origin story of how you came to alternative grading.

Kate: Yeah. My name is Kate Owens and I'm at the College of Charleston, which is a small public liberal arts college in Charleston, South Carolina. I'm the associate chair in the math department and we have majors in math and statistics and data science, and our student population is around 11,000 for our undergraduates.

ices in my own courses around: 2013

So we're coming up on about 10 years that I've been doing this thing. And I started hearing about this from the writings of Robert Talbert and talking to David Clark. And also reading just some different people that were blogging about the way they were approaching their own courses. And Joshua Bowman, who's a mathematician. Now he's at Pepperdine University, he was blogging a lot about his use of standards based grading.

rds-based grading ship around: 2013

Sharona: But, but why? Why were you even open to changing your grading?

Kate: Yes, this is another good question is like, okay, how did that actually come to be?

I mean, I had been thinking about grading practices for years before then. I mean, I had realized that my own traditional grading definitely had some flaws to it. I knew that there were students who I really thought could do better, but they sort of weren't given the opportunity to make up for early missteps or mistakes early on in the semester.

So that was part of it. I definitely knew that there were students who were doing really well in my courses, maybe even making a B+, making an A level grade who had never solved any particular problem fully correctly. They had sort of gotten by on accumulating points and accumulating enough partial credit that somehow it would like cement together and build into something that maybe didn't align with what I thought the goals for the course would be.

So those were two of the big things. It was really hard for me to give meaningful advice to students. I think we've all had the experience of students to come to our office and then say, I really am invested in your course. I really want to do better. Like, what should I do?

Without any more information, you know, I would tell them, well, you need to study more, do more practice problems. But I didn't have any student specific advice for that conversation in that moment. So I was really frustrated with my grade book because there's a lot of different ways a student can earn an 80% on a test. And just knowing that they made an 80%, I don't know what advice to give them.

Are they struggling with one type of problem? Are they getting 80% of every problem? Is there some stuff that they're just never even getting to at all? Like is it just too much at once? All of that information was lost. So I thought that a lot of the alternative grading ideas were ways to sort of patch up some of these holes in the system that I had in my traditional grading.

Sharona: So, I know that it's your fault that I'm sitting here today.

Kate: Oh my goodness.

Sharona: Because you were one of the first two blogs that I read when I went down this path. And I guess my question to you was, why did you start blogging about it? And do you still blog at all?

Kate: I haven't blogged in a long time. I feel that, I feel like I've said everything that I had to say or that a lot of the reason I started blogging was to understand my own thoughts and philosophy and approach a little bit better. Like, what did I really think about partial credit and like, is it worth it to grade using partial credit? Or what is the purpose of an exam? Or what does I want my students to learn in my calculus course? Or any of these kinds of things.

And I didn't have like a two sentence summary of any of those ideas. So for me, blogging was a way to just like garbage words onto pages until I had seven paragraphs. And then I could finally see in all of that garbage, like, oh no, that was just two sentences. It was two sentences were my thoughts about that one thing.

So it was a space for me to be able to just digest stuff mentally and having a blog post that, like, I'm going to click submit on this, forced me to write human readable words and not just like, oh, I'll do it later. So it was a procrastination prevention tool a little bit and it helped me sort of figure out where I was philosophically with a lot of these ideas.

And I'm not really blogging anymore. I don't know, my kids are a little bit older and so maybe I'm just driving around to different sports fields all the time. The time that I used to spend blogging is like, now I have got to drive somebody to a soccer field or something. So maybe that's partially it. And then partially, I have sort of figured out where I am with some of those, like, big questions. And my approach to these, these courses are the way that my grading architecture works. So I have less stuff that I'm trying to digest.

Sharona: So, we wanted to talk specifically with you because, you know, so just so the listeners know, and I'm sure they do know, we've now worked together, the three of us quite a bit, and so...

Kate: many years.

Sharona: ...one of the things, Yes. And you're still talking to us. This is a good thing.

Kate: That's true.

Sharona: So one of the things that, well, we really wanted to talk about two of the four pillars in alternative grading, which is the pillar labeled "marks indicate progress", also known as "assessment of mastery matters", as well as the pillar "reattempts without penalty".

And just sort of explore with you, both your style and, and also compare and contrast with some of ours. How do you actually do your assessments? So, you know, this episode is coming out, we've already had some conversations about clearly defined learning outcomes, the first pillar. So we're sort of assuming that most of the people listening to the pod have already got their learning targets.

So can you share how you do your assessments? And I think Bosley, you wanted to ask her like a compare and contrast.

Bosley: Yeah, I ,was because we, we've talked about the pillar, I wanted to take this opportunity on this episode to, you know, look at the three of us because we do have three different approaches of how we do the reassessments.

So I, I wanted to kind of, all three of us, just share the nuts and bolts of how we address that, that pillar about reat attempts without penalties and what that means to us and our classes.

Kate: Sure. Yeah. Okay. So mark's about showing meaning and then about reassessments. Alright. Sort of two, two different directions to talk about.

For me yeah, I, I do offer my students the opportunity to take reassessments. So they're invited to do this. It's not a, a required part of my course, so if they are unhappy with their performance on any of the assessments or the scores or marks that they have at a particular time they're invited to come take a reassessment.

What that really looks like, or the nuts and bolts, has definitely shifted for me and changed over the last 10 years as I've sort of thought about different ways of doing this. When I started out it was definitely sort of a come to my office hours. You can take a, a quick quiz, one or two questions, sitting outside my office, I've got a big space. So I, you know, ask them if they need any help. We would review previous work over some practice problems. And then when we both agreed that we thought that they were ready they take a, you know, a quick quiz that would take just a few minutes and then we progressed that way and hopefully they would do a little bit better.

And if not, we'd go through that feedback revision cycle and have another conversation. And so that's definitely the way that it worked for a long time for me, and I think that that's ideally, in my ideal world, that functions really well for me. My classes are limited to around 30 students, between 20 students and 30 students in each section.

So I'm working with around 100 to 120 students each semester. So I don't have a tremendous number of students that are coming to do this. It's still humanly manageable for me to meet with them face-to-face for a few minutes at a time and have conversations. So that's really good. As terms of limitations, I do limit my reassessments.

For each student, they have to choose one learning target, I call them course standards, one standard per week. So in my calculus class, this might be something like the quotient rule, or it might be something like a related rate problem or something about understanding the fundamental theorem of calculus, but whatever that topic is, they can only pick one topic per week to retry or reassess with me.

And the reason for this is, I guess there are several reasons. So one reason is that I really want them to focus on that one thing and really think deeply about it until they find some kind of understanding, until they have that light bulb moment, rather than kind of trying to think about too many things at once and getting lost in too many things.

The next thing is that, for me, I can only grade so many pieces of paper. I can only look at so many problems per week. This is like a human being that's looking at this. I'm not using any automated grading system. So I was trying to avoid the situation of a student showing up in my office and saying, I'm going to try 45 quizzes and then see how I do on them.

Because I cannot facilitate that. I just can't grade that many things. So having a smaller limitation on how many things they could try each week was really important just to keep my own sanity and leave my work at work when I come home.

Sharona: How many times can they try that one standard in a week?

Kate: So usually I have office hours on a Monday, so anyone that's like really gung-ho to meet with me right at the start of the week can come on Monday.

And I don't offer any, any kind of limitation on how many times in a week they can retry. So for some students, they'll come by on Monday or Tuesday, they try a reassessment quiz, they're successful, they're happy, I'm happy, and then we're done for that week. So sometimes it's just one or two for some of the more difficult content standards, some of the harder problems, sometimes it takes some students, many, many tries. And so I'll see them a couple times on Monday and they'll pop in my office between classes on Tuesday, and then I'll see them again on Wednesday. So maybe it ends up, you know, it takes them four or five cycles like this throughout the week. And I'm happy with this.

I think that every time they go through this feedback, revision, reattempt cycle, there's some amount of learning that takes place. And I'm there to intervene to make sure there's some amount of learning between tries. But I don't say that there's any particular limit. I don't think it's ever gone above about five or six in a week that I can recall on the last 10 years.

So it's not unlimited, although there is no limit.

Bosley: So, are the students revising their, their own work as the reassessment, or are they revising it and getting that learning and then taking a new question of a similar type?

Kate: Yeah. I think this is one of the ways that the three of us probably approach our systems differently.

When I started I would always generate a new type of problem, like a new problem. So they would be able to go over their previous attempt with me and look over their work. We talk about it, we talk about what was good, what went wrong, what the missing key component was for that try, but then they would have to sort of start over from scratch.

For me, it was important. I really wanted to see a full, complete, correct, explicated, well written, full problem produced just in one shot. And so I didn't want them just going back and, you know, fixing up little pieces here or there and patching it together. I also just don't know if I could track, for me, like the data tracking of this would be very complicated, tracking what was the thing that I thought was missing the first time, and then what if I, when I'm looking at it again, I see a different thing that was missing the first time that I didn't catch the first time I looked at it.

So just starting over with a blank sheet of paper sort of made my life a little bit easier. I do think that this works for me in my courses because I teach a lot of our first year and second year sort of service level courses for our STEM majors. So these are things like Calculus one that are really technical skills where it's easy to generate 45 versions of quotient rule problems.

And maybe if I were teaching, you know, an upper level number theory course or something, this would be a much harder for me to do and I might think about revisions more deeply, but I haven't had to do that yet.

Sharona: And to clarify, I mean, that's not different than one of my classes. I think what I'm hearing is all of your classes have the same basic assessment structure, right?

Kate: All my courses have the same basic assessment structure. I think that that's probably fair. And certainly the reassessments have worked kind of the same way. I will say that, you know, during the big online shift due to COVID-19, you know, a lot of things about all of our courses were changed, including the way reassessments would happen.

Because I certainly didn't want to have 30 students in my little 20 square foot office at once. So a lot of my reassessments and that sort of thing moved to sort of online quizzes where students would submit work as a PDF file and I would offer some kind of feedback, potentially meet with them over Zoom.

So there's some digital measures, but the essence of my reassessment being a particular quiz that takes 10 minutes and has a few problems on it is still the same.

Sharona: And so what I would say is I teach three main different courses and each of the three of them handles reassessments differently. So one of them is the most similar, well, my calculus class, when I taught it, I don't teach in the calculus sequence anymore, but when I taught it, I did the same thing that you do, which is generate new quizzes, new problems, usually in office hours or I think I heard the name Reassessment Carnival from someone someday that we can talk about in a minute.

So I do that, but each of my three courses happens very differently. And, and I know Boz, you have two different structures that you use.

Bosley: Well, one of them is the one we both use. So, and that is that entry level quantitative reasoning with statistics class, which is similar to what Kate was saying with one big difference.

And that is ours has more structure about when they can do those reassessments. So we, we don't have this, you know, open invitation that students can come in and do whichever one they want. We've got structured time where if you need to still assess on standard one, you're doing it week six or week seven for the third time.

And then again week 10. So our, that one has a little bit more structure, but it is similar where we have the feedback loop and give a chance to look at the the student's mistakes and work with them on the revisions. But then they get a new generated question that, because it's statistics, it's a completely different scenario, but with similar concept questions that are on it.

Sharona: Right. And our calendar is set up so that the student gets five attempts through the semester to show success twice but prescheduled.

Kate: I think that I'm also looking at, I want to see that success happening twice. And I think that I've been, I don't know, I haven't paid as much attention to how many times every student gets an attempt at something in my particular class.

I think that it probably varies, just depending on the semester or the particular learning standard. Some of them tend to come up a whole bunch and then some of them maybe not so much. And so that's not something, I couldn't say that everyone gets five chances at this during class time. I don't know that ahead of time. I definitely don't even know that for this upcoming semester.

Bosley: Now but your linear algebra Sharona it is done in a, a very different way, correct?

Sharona: Well, yes and no. It still uses the pre-scheduled reassessment schedule. I just have settled on that being my preferred way of doing things is just pre-schedule it for literally the whole class and then only people who need it take it.

But the biggest distinction is what happens on each assessment. So for each, so if a student takes an assessment on a specific learning target, they're going to get it kicked back from me, either as complete, as a revise, or as a retake. And so I use a revision that where they're not doing it from scratch.

And, and Kate, you and I teach actually the same linear algebra course. But the reason I want to see a revision, or I decided that revisions were good, is because most of the mistakes I see are one of two things. It's either in arithmetic or copying error. I want to teach them to find it and fix it. Or, I see a conceptual issue that, because I'm still requiring them to do it twice, getting them to fix it on the first one actually has them learn it better.

That from, in my perspective, then a new problem. Because they're going to make the exact same mistake on the new problem. And so I would rather sort of require them to try or, or give them the opportunity to try to fix it, because I see the same mistakes and the, and they're very much, they look really simple and yet they're very conceptual.

You probably see it where I get equations that don't equal anything,. And what it equals is actually an important conceptual thing. Yet it looks like all they did is leave off "equals zero". And so getting them to go through the thinking process and I'm finding more learning from the revision.

So,

Kate: Sure. I think that for me, with the students in my office hours, we have that conversation of what happened the last time you tried this problem or this quiz, or whatever the assessment looked like. And then we'll look at it together and then we'll realize like, oh, it looks like you had an equation that didn't have a right side, or you didn't know what it was supposed to equals or something went wrong.

And so that's sort of like, find your mistake. Can you describe it to me so we don't make it again? That conversation is still happening, it's just I'm not awarding credit for it. Right? They're still gonna have to go back and then start over from scratch and, and then try and get full, get it fully correct.

Sharona: Right. And I don't think that either of these is right or wrong. It's just what we're comfortable with as individual instructors accepting as evidence.

Kate: Absolutely.

Bosley: Yeah. And then my other class, my second semester quantitative reasoning class in the high school, is even more different than any of those where, because that's a project based class and it, it is more in the specs, you know, specs realm than the standards realm, I do have, I have no reassessments, new assessment, it's, it's all revision. It's you turn in your project, I go through it and give you direct feedback and then give it back to you and say go and fix this and resubmit it. Now, I don't have unlimited submissions for the same reason, Kate, that you don't let students do all 45 standards in one week.

A, I don't want to drown myself in grading, which is a huge thing to consider, especially for anyone that's listening to this, that is brand new or still fairly new. Looking at this and thinking about your calendar is a real concern because if not done well, you can drown yourself in grading.

So I don't have the unlimited resubmissions and I also have, after the third resubmission, like they've got to come and instead of just looking at the feedback and trying to go from there, they actually have to come in, either outside of class time or schedule some time that we can talk in class.

But you know, that they have to get a little bit more one-on-one with me to really sit down together and look at what are they doing that's getting it kicked back. But also what I really find interesting is there's three of us here and we just described four different ways of doing this.

Sharona: Well, and I'm going add a fifth one.

Bosley: Okay.

Sharona: Because my history of math class is a weird compression or a combination of reattempts and revisions, because I have four projects. The projects are all a revision process. So they give me a project, it has a set of specifications and they have to revise it until it's good, right?

But buried inside the project, I have 10 learning targets that are based on content themes. And so if a student is unable to get that content theme in Project One, but yet they manage to get the project done, they could attempt the theme in project two. So like one of my themes is ancient numerical systems and arithmetic systems, for example.

So, because one of the first things we do in that class is we look at like Egyptian and Babylonian number systems. Then there's some other things that they can also get. So if they get one of the other things in that project, but I'm not happy with their analysis of the arithmetic system, I can tell them in project two because by then we've gotten to like the Roman numeral system or the Hindu Arabic, they can try it in that one.

So it's like a reattempt, even though they still have to like finish project one, but they don't necessarily have to get that content theme in it. So it's like a mishmash of both revisions and new attempts.

Bosley: And why is there, like I said, between three of us, we've got five different ways of doing this, why, why so different? Like kind of interested in both of your opinions of why you think there's so much variety here?

Kate: I think I've probably tried five different things to land on where I am. So I think that also there's this, like, this evolution over time of. If you look at my syllabi over the last 10 years, like there's a lot of structural things that are the same, but some of the minutia have changed just a little bit here and there.

I think that part of that for me is there are some semesters where I start out in August, I have a lot of energy and some really big dreams and hopes and goals, and so I try something that's really, really great and then I realize in October it was not a good idea, don't do that again. And so I think that there's some of that that happens too, is like, I'm not the same syllabus writer in August as I am in January.

Right. I've got different, I've got a different outlook on the world.

Sharona: Well, and my perspective is, is twofold. One is I think that grading is highly relational and very personal. It is the biggest relationship you have with your students. So therefore it has to be an expression of our personal values and beliefs, and an expression to students of what we think of the student. So even if we all have a general agreement on values, the way we express them, because of our personalities, is going to change. So I think that's where I've never seen, I mean even Bosley and I literally teach in the same course with the exact same grading structure.

And I still think that his students would relate to him slightly differently than my students will relate to me. Even though the grading structure is literally identical. And also it differs by the course. So I have in my three courses, different student populations. One is a freshman GE course, one is a sophomore level math and STEM major course, and one is a junior/senior level math and STEM major elective.

And those are three different things. And the linear algebra course is also to some degree in a sequence of content. It is relied upon by future courses whereas the history of math is not.

Bosley: Yeah, and we talked about that when we were talking about developing the learning targets. Some of the things you need to consider are: Who are the students taking your course? Is it in a sequence? Where in the sequence is it? Are there skills that a student really needs to take from this class to be successful in that next course? So all of those things, not just affect the learning targets, but also affect, your grading architecture, how you address these reassessments.

And that was kind of the one of the big points I was hoping to make because even though we described five different ones, we also were describing five very different classes.

Kate: Yeah. And I think that institutional culture is another thing that's really important when thinking about all of this, because I know that at some institutions, having students come to office hours, two or three times in a row throughout a week may not physically be possible for the students for whatever reason.

And so I think that part of my system is set up because it works for my classroom and my campus. And the fact that we're sort of small and everyone's very close by and I can see my students, you know, I see them four days a week, just for being in class. And so to snag them outside of class for five or 10 minutes is not that difficult for me.

But I know in other places it doesn't work the same way. And that's something that we have to be cognizant of, right? Every time we design a system, it's not just about us or our students, but there's some broader context that's going on around us.

Bosley: That's a great point. And I don't know Sharona if we've talked much about that in some of the other episodes. So that is, that's a really good point, Kate. Thank you for bringing that up.

Sharona: Well, I know that we've thought about it, but I don't think it's been on the pod. Our institutional context is very different from yours, Kate. We are a commuter school in a very, very large metropolitan area. We serve a highly underserved population.

We are a Hispanic serving institution, but to be a Hispanic serving institution in this country, you have to have 25% or more of your student population be of Hispanic origin. We are 85%, so we are very, very heavily Hispanic and most of our students commute, often by public transit. So many of our, my students are coming an hour, hour and a half on public transit, and I only see them twice a week, and they're also working 30 to 40 hours a week, many of my students. So yeah, coming to my office hours, even once doesn't work. So I do a lot of Zoom office hours now, which is much better for them. I do them at night. You know, The me of 10 years ago would've said, what crazy person does office hours at eight or nine o'clock at night? I mean those poor students. And now it's like, no, that's when they're available. So context definitely matters.

So I was wondering if you could share, Kate, a little bit about maybe some of your, I wouldn't, I don't want to call them failures, because they're not failures, but times in your journey where it was a little bit harder than other times.

Because I know Boz and I have talked, even on the pod, our first implementation was a god awful disaster and yet still better than traditional grading. But you've, you've definitely had some stories. Can you share some of those?

Kate: Sure. I think that that's probably typical. I mean, I think that part of getting into this is trying to understand what you didn't like about your previous system.

And it's hard to know what those things are until you really figure out what they are. So there's some like feeling around for that, that happens to many of us. I don't know. I Right. I'm not going to describe them as failures. Thank you. I 've tried some things that I'm not doing anymore.

Right. So I'll tell you about the things that I've tried that I'm not doing anymore, but maybe they would work for you or somebody different. One of the things that I really wanted to get away from was the partial credit points in having these calculus problems that would be graded out of 14 points, because I feel like just years of my life were sucked away in trying to figure out if this solution was worth 11 out of 14 points, or maybe 10 out of 14 points, or maybe it's worth 12 and a half out of 14 points.

And I just like that, that mental space, I didn't want to spend time on that anymore. Instead, I really wanted to write better feedback and offer better advice to students and give them the opportunity to like retry these kinds of things. So I tried to get from like the 100 point scale or even the 14 point scale to something that just had a fewer tiers, right, in our proficiency rubric.

So I wanted to go to something that I felt like my students could understand and that I could explain pretty well. And so I went to a GPA scale. And so when I would grade a particular problem or a particular assessment or whatever, I was just using a GPA scale. So if it looks like an A to me, it was a 4.0. And if it looks like, okay, this is, this is passing level work, I'll accept this. You know, it'd be a 2.0 and maybe a 1.0 fell below my threshold for what I would find to be acceptable for a particular learning target. So I had this four point scale that I was using and I don't do that anymore.

So here's what happened with that, is that I found that my students, if you give them numbers, they want to like add them and average them. And even getting my learning management system to stop turning them to percentages. So for me, you know, a 3.0, like this is B level, like this is good work. You did some good stuff here.

And the learning management system would do, well, it's three out of four, so that's a 75%. And my students were like, why did you give me a C? And I'm like, I gave you a 3.0 like that's a B. And they're like, no, you gave me a 75%. So I ran into a lot of problems about implementing numbers where I was really trying to get away from the whole numeric system altogether.

So my first piece of advice is don't put numbers in anywhere. Like this is too hard to not do the kinds of things that numbers make us want to do. And so then I was like, okay, well I have a four tier system, so I switched over to the E M R F rubric. Which I'm sure that several people have talked about.

And so I was using that one, and I used that one for quite a while. It definitely solved my number problem that students were trying to like, add up like an E and an M together in some kind of way. My learning management system had no idea what I was talking about, and so it, it was not trying to give me a percentage for any of these things.

That was great news. I did not like E M R F. The thing about letters is that I think that we are all emotionally traumatized by letter grades. I mean us, the instructors, the students, just everyone, everywhere. So if you put an F on a student's paper, like that is just an emotional gut punch that I really couldn't stand behind.

So I changed it immediately to E M R N. So it wasn't a failure, it was just a not yet. And occasionally it would be like a, no, don't do that again. But N was just a no information available, right? We were just starting ground of N and then I used E M R N for, I don't know, I guess a few years or something.

And my problem with E M R N was that my top two levels were like, meets my expectations and then exceeds my expectations or masters the content and does an exemplary job or something like this. And I found that I couldn't really distinguish, in a lot of cases, between a student who really met my expectations, which I consider my expectations to be really high, and somebody that like exceeded them. Like I'm teaching college algebra or something, and this, I'm trying to assess if they know how to use the quadratic formula and like they've met my expectations.

Like I don't know how you can exceed my expectations about the quadratic formula. Like we either have done the thing or we have not done the thing and there's no like exceptionally beautiful way to apply the quadratic formula.

Sharona: I don't know, maybe interpretive dance, you know.

Kate: I mean, maybe there is some way I just, I couldn't find a succinct way to justify the difference in every case that I could be like, this is what an E is and this is what an M is, but, maybe interpretive dance would be one way. And so I just condensed E and M into a top level tier of like, this seems to be satisfactory to me, or it's a successful attempt. Overall, I'm happy with what I see here. There's no glaring omission or error or conceptual gap or anything like this. So my E M R N scale got condensed from four levels just to three.

So right now my scale, the top level is an S and so I say that that's satisfactory or it's successful. The attempt that was provided to me, I'm overall happy with, it's met my expectations. We are good to go. My middle level, instead of like reassessment or revise or anything like this, I wanted to show that the student had made some progress, right?

Rather than like a deficiency model of like: it would've been successful, but something was missing. I wanted to say, I. It wasn't a blank page. There's something here. That's good. So my middle level, I used a letter G and for me that stands for growth or growing. Like there's something that's here, it's not just topsoil, like I see a little leaf that's sticking through and if we give it some sunlight and some rainwater and a little love, like maybe it could be a plant later. I don't know, it's not a plant yet, but we're like, there's some something's going on there. There's signs of life. So I've got sort of satisfactory at the top, growing in the middle.

And then my bottom category for a student who wasn't there, didn't turn it in, didn't know what the question was asking for, still is an N. So that just means a no information or a not yet, or nothing's written. And that's my lowest level.

Bosley: So there's two transformations in that story I kind of wanted to point out. The first one is, is the use of your language and that really kind of transitioning into that growth mindset, which, growth mindset, you know, is kind of at the heart of these alternative grading systems. But the other one was the transformation that we were talking about, Sharona, on episode seven when we were diving a little bit deeper into the grading architecture and those proficiency skills and how you went from, by far the most traditional proficiency scale, which is a four level, down to a three level, because you weren't finding a good distinction or a use of that fourth level.

And that was one of the things that we talked about back in that episode and that I really tried to harp on is if you don't have a reason for those extra levels, get rid of them. Like it, it doesn't do anything but add complexity that's unnecessary. So if there's not a purpose in your grading architecture, or not a purpose for what you're going to present to the students, like, you know what Joe was saying back when he interviewed, then don't do it like, like you were just like you, you know, discovered through your years of doing it is your top two were basically the same. So why not just make them the same?

Kate: Right. And part of me kept that highest level because I wanted to like give it as a reward to students who did, like, who went above and beyond and, and give them the Gold Star and like the high five on their paper or whatever.

But I found I don't need like a grade to be able to do this. Like, I can just look at Sharona and be like, Sharona, like you are a rock star at this. This was an amazing thing that you did. I was so happy when I read this work. You showed so much growth or whatever. And like, I can have a conversation with a person about this.

I don't need to have this in the grade book to tell them that I was really pleased or happy or whatever the thing was.

Sharona: Well, and I think that's an important point because even, you know, Alfie Kohn's book, Punished by Rewards, I think speaks to some of this. Which is we're so wedded to the idea that these grades actually say something about the quality of the work.

And I agree with you. I can't distinguish between good and great really. And I was basically not sold on a four level until we spoke to Bosley's friend Joe, which that episode is coming out shortly, as of when we record this. He looks at it as you do it if you the instructor, need it for something. So in particular, as a high school teacher, he was responsible for differentiating instruction for every student, no matter how good they were.

So if they hit that grade level, that was that record, that fourth level was a indicator to him that he needed to do more with that student. It wasn't, it didn't make a difference in the actual grading. And that was the first time I really felt that I understood a good reason to have multiple levels.

That being said, we do have four levels in our statistics GE course, because one way to use four levels is when you're managing multiple instructors. So our GE course is a coordinated course, and because we need to be able to calibrate "good enough" among all the instructors, some instructors because of their personality, because of their relational needs with students, they do need that "good" to "great" distinction, or we can't get them to agree that good enough is good enough.

So that's another reason that somebody might use multiple levels of success, is if you're in some sort of a coordinated instructor environment and you have to calibrate among multiple instructors.

Kate: Absolutely. I think that my, the ways that my system changed over time can be summarized as: just simplify as much as possible, right?

So, so for me, like numbers, like this is hard and it was too hard to suppress the urge to add and divide them and all this sort of stuff. So like, I have got to go more simple than that. And then I went with like the published example of the four tier proficiency scale that I could find, because I thought, well, it's published somewhere, so it's a good one for me to try.

And then I realized it's still too complicated for what I'm really trying to target. And so then I bounced down to three. And I've even used two. I have done like a pass and a not yet in some of my course, in some of the circumstances. Not on a global course wide system. But here and there, I've definitely experimented with that a little bit.

So my advice, my advice to Kate of 10 years ago is take whatever you've written down and find five ways to simplify it before you go to bed.

Sharona: Well, and I like what you said about that two level because in that history of math where I'm smooshing together specifications and standards, the specifications for the projects are two level, they're complete or not yet.

And then, or well, it's really complete or revise, like there's no "not yet", that's what I got rid of. It's complete or revise. And then I still use the three level on those content standards, which are complete, revise or redo. So I add that redo back in and if they just, if there's nothing there, it's still a revise for those projects because missing you still have got to fix it or you're not gonna get a complete on the project.

whack myself on the head from: 2017

I just want to hit myself. And by the way, Kate, you said you used the letters and we're also wedded to letters. I've actually gone away from letters. I use emojis because, emojis seem to..

Kate: I know some people have done this. I I don't know. I, I have enough technological issues that, you know, anything I can type and like send on a keyboard. I'm good with that.

Well, the other thing that I've simplified over time is I think that targeting the right number of standards for your course and your semester and your students is like the most important decision to make when setting up a class. And I know working with instructors over the last 10 years who are thinking about doing this, it's so easy for us to believe that we have 100 very important things that we are going to teach this class, right?

And here's my list of 100 things, and that is judy a grade tracking assessment nightmare. So for me, most of my classes end up with about 25 course standards for our 16 week semester. So we're looking at, you know, maybe about one and a half per week. So that's not really one per class period, but about one and a half per week.

So with 25 standards in 16 weeks, since I talked about earlier, students are only allowed to reassess one standard each week. By the pigeonhole principle they have to do something correct during class on the in class quizzes and exams and all of the stuff that happens. And then reassessments are really just there for those topics or ideas or concepts or problems that each student finds individually like really tricky to, to get through, to really understand deeply or whatever the holdup is. So most of the things that they're learning, they're demonstrating that understanding during class time and reassessments are just for those handful of things for each student to kind of work on with me one-on-one.

Sharona: Now, do you still do a reassessment carnival? And what is a reassessment carnival?.

Kate: What is assessment carnival? Yeah, so I, what is a reassessment carnival? So I have about 25 standards for 16 weeks, and students can only pick one thing per week to retry, to reassess during my office hours, hours or wherever those reassessments happen.

So by the end of the semester, I find that my students, some of them have painted themselves into quite a tricky spot where they've sort of put things off a little bit too long. So something that I started doing pretty early on in my alternative grading courses is I offer what I call a reassessment carnival.

That's a nice way to say take as many quizzes as you want to in a single day. But when you tell students, "Hey, do you want to take 12 quizzes in calculus class?" No one says yes. But when you say, do you want to participate in a carnival? Everyone gets very excited. So that, that terminology has worked well for me over the years.

So what the carnival is, is that it usually happens like maybe the last day of class or the day before, the last day or something like this, really right toward the end. And students they fill out some kind of form online, some kind of survey about what topics they want to reassess. And I, I lift the one per week rule.

So they're allowed to reassess as many things as they want to attempt for the class period. So most of my classes meet for 50 minutes. So if they want to try six things in 50 minutes, I'm totally in support of this. They'll fill out this little form ahead of time so that I know like how many photocopies of things and what types of problems I'm supposed to bring in.

It's completely like low stakes. So if they come in and read a problem and say, "Nope, I'm not doing that one", then no one's feelings are hurt. They can just walk out of the room with it. If they turn it in for me to, great. Then that will, of course stick. And they can do them in any order. So they usually come in and students are taking different numbers of quizzes on different things.

They're only allowed to have one on their desk at a time, so they'll work on one problem and turn it in and pick up their next problem. As I'm madly trying to grade all of these as quickly as possible. Usually I end up with about 200 quizzes for my 30 students by the end of the reassessment carnival. So it's quite a lot, but usually I'm able to grade about half of those during the carnival itself, because I'm writing as quickly as they are and assessing them is pretty fast for me by the end. So a couple of reasons for this activity. One is that it allows my students to demonstrate understanding of anything they've learned into the course that far.

So for me, I don't really care when a student learns a particular thing. It doesn't matter to me if it's in February or April. My final deadline, I guess, is like the final exam. So if they want to show me that they've learned something that they were struggling with all semester and today they actually know how to do it on the last day of class, I am so thrilled to see that demonstration of knowledge.

So I'm very excited for them to do this. The second thing is many of my courses have cumulative comprehensive required final examination that covers all of the content from the semester. And this activity really makes the students go back and think about, "what were the things I've struggled with and how can I start studying for them right now?"

So it's sort of a dress rehearsal for the final exam, and it gives them an opportunity to try a couple more problems, to get some more feedback from me, before they're in the high stakes final exam testing situation. So that's another reason that I do it. And then third, it gives me the opportunity to sort of boost everyone's grade a little bit, because by the end of the semester, students are really stressed out.

Some of them know the material well, but just haven't been able to schedule with me because, you know, maybe they're a student athlete or maybe they have an internship or they've got a full-time job and they just haven't been able to come to my office for a couple of weeks because of whatever's going on.

And this gives them that opportunity to have the time to really show me that they've understood some material, that they had a gap in their knowledge at some point along the road.

Bosley: I love that name. And you know, Sharona, our final is kind of similar to that, where, you know, the student can take any of the standards throughout the course.

But they only need to take the ones that they haven't gotten mastery. So maybe we should change the, the name of our final to the Reassessment Carnival

Kate: For many years it's happened in person in the classroom. So, you know, like out of the 25 things, maybe I would have to rewrite quizzes on 20 of them. And so I would come in with like a stack of 20 manila folders, right? And they, they would have on the front of them like what the name of the learning target was, and I would lay them around the room, like around the walls on the floor. And then every student could run up and just grab one thing from a folder and then work on it, and then turn it in and then run over to a different folder and grab it and work on it and turn it in.

And so it for sure has a carnival kind of feel because it is a lot of people running. Literally they're running around the classroom, grabbing things, writing them, turning them in to me, I'm like grading stuff and putting stuff in the grade book. Some students come and try one problem and they leave after five minutes.

Other students are there sweating and finishing their ninth problem in 45 minutes as fast as they can. And so it is, it, it feels like a carnival environment where everybody is there for a different reason and there's some amount of excitement about getting this thing over with.

Bosley: And that's funny. The last time I taught an Algebra two class in high school was one of my first times transitioning into an alternative grading. And that's how I did my final. Like literally I had, you know, 15 manila folders laid out with enough copies for every single student.

So I wasted half a tree because I didn't actually need that many, but I didn't know how many I really needed. So instead, I made like, you know, I'm teaching multiple classes, so we're talking like 90 students and all, and 15 to 17 of these questions. So lots of paper copies made and in these manila folders.

But, slight difference because it was a high school class, like students couldn't leave when they were done, which caused an issue. And then just that kind of chaotic thing might've been great, might've been fun for the students, did not work for me. Like my anxiety and my just, okay, this is just going way too nuts, just did not work. But it goes back to that, that personalization of this kind of grading system. There's two things very, very similar. It works for you. It didn't work as well for me when I tried it in my high school setting. Maybe the college setting would've been different, but...

Kate: It was still a carnival, It sounds like Boz, no matter what it was.

Bosley: Yeah, it was a carnival. It was just the carnival didn't work for me.

Sharona: He didn't want to be the carnival barker.

Bosley: Yeah.

Sharona: Yeah. So I think that mine emotionally was probably in the middle, pre pandemic, pre some of the assessment tools that we now have.

I definitely had the manila folders and the multiple copies of the multiple versions and things like that. It was definitely one of the harder parts of the system. Now that we've transitioned, I've transitioned all of my courses that use the quiz style assessment to work on the assessment tool, I use CheckIt, everything is on Canvas, and so what I've been able to do, which works for me, is I don't want my students to have to try to cram their reassessments into a fixed timeframe. So actually every single one of my assessments is its own quiz on Canvas, they're all set at 30 minutes for a single problem.

So 30 minutes is enough time that they shouldn't have to rush, but marginally short enough time that trying to get someone else to either help them with the work or use something like a Chegg is marginally more difficult. It's not impossible. And if a student wants to take all, you know, 14 assessable standards or 13 assessable standards in statistics and wants to take the maximum amount of time, that means they're going to be taking six and a half hours worth of finals.

But they're all asynchronous. They're open for like three days on Canvas. Half of them are open the first half of the week, half of them are open the second half of the week. And they can just choose. They can just choose what they want to do and when they want to do it. And that has made it a lot more manageable for me.

What I don't do, and in theory I could, but I'm not willing to put the effort in, is actually specifically assign each of those reassessments to the people that still need them. Instead, I open them up for everyone and I tell the students, it's your responsibility. I'll help you, if you come to office hours, we can talk it through together, say, what do you need to still take and what don't you? But I do put that responsibility on the students to figure out what they need to reassess.

Kate: For me, the end of the semester is a, it's sort of a push and pull. It's a balance between my philosophy, my internal philosophy about all of this, and then implementation. Because on the one hand, like I would like to have my students have as much time as they need and not have this time pressure of we have to get this done in 15 minutes.

I'd like to have my students give me better quality solutions because they're not writing as quickly as they possibly can. Maybe having them do them all the same day is probably bad from like a brain perspective. Like it's probably bad to just tax their brains that much. But all of that balanced against like, I have to get grades in, in like 12 and a half minutes and I have a hundred students and I want to give everyone the same opportunity and that has some cost to it for me, that I can't be there for 24 hours in a row. That I'm definitely concerned about academic honesty and this kind of stuff about putting all of this stuff on Chegg or finding it somewhere else that there's kind of like, there's kind of this balance that I have to find between really what I would desire the most and then what I'm physically capable of doing.

And my system I think has found at least some kind of balance for me about what my students can do, what I can do, and how we can make this actually humanly possible given the confines of our system.

Sharona: Well, and you said you have a mandatory final in these classes. Most of them. So I do think that your carnival serves an additional purpose, which is getting them ready for the time pressure of that traditional final.

And I think that's a legitimate thing that we also need to take the context into account for. You know, I luckily I don't have that constraint.

And I think that even if they come to the carnival and then decide like, wait a second, I didn't know this problem was going to have a sphere and I forgot the formula for the surface area of a sphere, I'm not doing this problem. Just their process of looking back through their grade book and identifying what things they feel confident on and which ones they still need to study, and having to make that choice and having that information to help them make that choice is really important as they're going into their final exam.

Before alternative grading, I don't know that my students would've known what they struggled with, if at best maybe they knew that, like, test two, that was the hard one. That was the one they struggled with. But there was no actual like, learning thing that would've helped them see, oh, you know what it was, it was that geometry stuff, or it was those word problems or those graphs that I was working with. But now my grade book shows them that information. It's still available to them later.

Bosley: Alright, so we're starting to kind of get close on time. There was one last question that I've been dying to ask though. Because you've been doing this for so long, were there any changes that was caused by the pandemic that you have continued after the pandemic?

I mean, so not during the pandemic. We know everything changed drastically during, but did any of those changes carry on after the pandemic for you?

Kate: Yeah, I think that I like Sharona mentioned, I'm also using CheckIt problems to help generate some of these, some of the types of problems that we're using.

Because I was able to download the CheckIt problem database and put it into our learning management system, it made doing reassessment quizzes online through our learning management system, a lot more feasible just from a practical standpoint. So one of the reasons I hadn't been doing them online was that I wasn't necessarily using a homework system that would generate these problems.

And so then I'd have to like input problems into the learning management system and at some point it was just easier for me to print them and have them and hand them to a student. But I've moved some of that stuff online and so I have been doing recently allowing them to do reassessment quizzes online where basically there's some kind of password for each of these quizzes.

And then they have to request for me the password for the reassessment quiz on whatever topic. And then that requesting of the password is the time that we could have that conversation of like, okay, great, yeah, let's, before we do that, let's see what happened last time. What were the gaps in your understanding?

Is there a time that we could need to make sure that you understand what you're doing now or that I could offer advice or feedback or help before this problem? And then usually they just get the password and they'll go and work out whatever randomly generated problem is given to them. And I don't think that I have it timed, but I think I tell them I expect to see your written work submitted within 30 minutes or something like this.

And so they work out their work on paper and then upload it as a PDF file. It's a little bit easier for me because I'm not like chained to a Xerox machine somewhere anymore. I'm not running out of copies and having to run anywhere, so that's definitely made it easier. It's nice that the students, once they upload that PDF file, neither of us loses it, right?

So I know that I've definitely lost student work and like, no, no, it's around, I swear I graded it. It's on this desk here somewhere. I've had that conversation a few times or when I've asked a student, well, what did you, what happened on that last quiz that you took? And they say, oh, You know, my roommate, whatever, my dog ate it.

It's not here, it's totally gone. But the fact that now they can scan their stuff to a PDF file and it's there means that we can still look at it later and then look at the growth that's happened. Like, okay, the first time we tried this, this is what happened. And then the second time you grew a whole lot in your knowledge and understanding, except there was these two gaps and now you're a third time.

Now we actually have the full picture for what's going on. So definitely moving some of those kinds of things online has been helpful, both from the logistical standpoint, but also from just like the students understanding their growth over time. Right. And getting that full picture all in one spot.

Sharona: And I have one last question as well. You already gave some of this, but what is like your number one tip for someone considering or starting to look at doing this? They're ready to pull the trigger. What is your one or two, number one things that you want new people to know?

Kate: I think for me, I was very nervous about doing this when I was like thinking about making this leap. So I think that the advice I would give to like former Kate is it's okay to be nervous about this. Like this is a big radical change. And so there's like, you're never going to design a system where you're like, oh, this is going to go very smoothly. I'm not nervous about it at all. So don't wait for that moment because it's not going to happen.

And then secondarily, whatever system you design, I know that you're nervous about it. It's because it's too complicated. So just try and make it way more simple and like you'll thank me for that later. But I think that designing these systems and figuring out what works is like, it's very meta about alternative grading, right?

The first time you try it, it'll go great. It'll be great. And you'll find nine things that you're going to do better next time around after you get the feedback from the students and after you understand your own architecture a little bit more. So the first time doesn't have to be perfect, right?

You just have to submit it the first time, go through a feedback reassessment cycle, and then make it better the second time.

en I started looking at it in: 2016

It's

Kate: been great. It's been a great collaboration that we've had over the years in all of these conversations, and I feel like I learned something new through them every time. And I still have not written my syllabi for this coming semester, so I'm still trying to make all of these decisions and design some system that I'm not going to hate.

So hopefully we'll be able to do it.

Sharona: Yeah. Well, our semester starts in 13 days, so mine are, are well on their way, but they're not done.

Kate: My semester also starts in 13 days. I just haven't No, no, it'll get done.

Sharona: It's, it's always been my coordination.

Bosley: Mine starts in six, so..

Sharona: Yeah. But you're not teaching in that institution this, this year, so that, that doesn't count.

You don't have any syllabi to write. So again, thank you so much for your inspiration, for your energy, for your willingness to share your journey and Is there any last thing you want to share with anyone before we go? You're, you're good.

Kate: No, this is really fun and thank you for having me. This is awesome.

Sharona: Absolutely. Boz, any last words?

Bosley: No, just thank you everyone for joining us and we'll see you on our next episode.

Sharona: Please share your thoughts and comments about this episode by commenting on this episode's page on our website, www.thegradingpod.com. Or you can share with us publicly on Twitter, Facebook, or Instagram.

If you would like to suggest a future topic for the show or would like to be considered as a potential guest for the show, please use the Contact us form on our website. The Grading podcast is created and produced by Robert Bosley and Sharona Krinsky. The full transcript of this episode is available on our website.

Share Episode

Shownotes

Transcripts

Follow

Links

Chapters

Video

More from YouTube