Writing PrairieLearn Questions and Computer-Based Test Practices
University of California Television (UCTV)
0:08 Craig Zilles: I've been using PrairieLearn for a long time,
0:11 so I'm user Number 3e for PrairieLearn.
0:13 Now there are hundreds of thousands of users.
0:17 I want to say I've been using it for about 15 years,
0:20 and then the computer based testing facility,
0:23 we started at Illinois about 12 years ago, and so that started with my class.
0:28 I hired the first proctors I have a bunch of experience,
0:33 and so I'm going to try to share some of that with you,
0:36 but it's hard to remember what it is that I have the expert blind spot,
0:43 so to do ask questions as they come.
0:45 I'm going to focus a bunch on PrairieLearn.
0:51 PrairieLearn is open source software,
0:53 and it was built primarily for this top purpose,
0:58 which is to enable mastery based homework.
1:01 We're strong believers in this notion of mastery oriented teaching of, like,
1:08 you should keep trying something until you get it right that having
1:12 a bunch of homework problems which you try and then submit,
1:15 and a week later, you get feedback about whether you got it right or not,
1:19 that's not going to maximize learning.
1:20 What maximizes learning is being able to give you immediate
1:23 feedback on, did you do the thing correctly or not?
1:26 Then immediately giving you another opportunity to try the thing.
1:29 PrairieLearn has really good support for auto grading,
1:33 and so that enables us giving students immediate feedback,
1:36 and then the ability to write question generators allows students to say,
1:41 give me another version of the question until
1:43 they're satisfied that they can do the thing.
1:46 A few years after the PrairieLearn existed,
1:51 we started running asynchronous exams in our computer based testing facility.
1:57 Asynchronous means that students are taking the exams at different times.
2:00 This turns out to be super important
2:03 for the reducing the administrative overhead of exams.
2:07 There's a non trivial amount of work of students having conflicts and getting
2:11 sick and dealing with all that, that if you're just able to say,
2:14 hey, sign up for a time that works for you, a lot of that overhead goes away.
2:18 But importantly, we can't be giving the same exams to students taking the exam
2:23 at different times because one student will
2:25 tell the other students what's on the exam,
2:28 and so having question generators and pools
2:31 of questions enables us to generate randomized exams.
2:36 Also, auto grading facilitates the use of frequent testing that basically,
2:43 what the CBTF does is it allows you to outsource the proctoring of the exam,
2:49 outsource the scheduling and administrative overhead
2:53 of the exam to Prairie test,
2:55 and then if you can outsource a bunch of the grading,
2:58 it makes running exams very cheap.
3:01 If exams are cheap, then you can
3:03 deploy them in ways that are pedagogically useful.
3:06 We find that if you run many small exams,
3:09 the students end up learning more and are less stressed.
3:13 If you run retake exams, if you give them an opportunity to take an exam, fail,
3:19 figure out what went wrong and take another version,
3:22 they're willing to do the work to learn the material.
3:24 I think this is especially important now
3:27 in the context of GenAI because the students
3:31 are confusing themselves about how much they're
3:34 learning when they're working with AI tools,
3:36 that many of us at the beginning of the semester have gotten up and said,
3:39 here's how you should study for my class,
3:42 and you tell them, here's all the best practices.
3:44 Most of the students go, yeah, whatever,
3:47 I'm going to go do my own thing, and that's fine.
3:51 They're optimizing their lives.
3:54 But many of them misunderstand what they need
3:58 to do in order to succeed in your class.
4:00 Having the opportunity to do whatever they
4:03 think is going to work, have that fail,
4:06 and then have an opportunity to listen to you
4:08 about what actually will work and get a bunch
4:12 of those points back by demonstrating that they actually did
4:14 learn those things on a second exam is really important.
4:17 Being able to run these exams asynchronously with auto grading,
4:22 at least a significant amount of auto grading,
4:24 enables faculty, even in large classes to run many versions,
4:29 many exams and retakes.
4:32 As much as possible, we want to cover all our learning objectives,
4:36 that PrairieLearn was originally built as a reaction to the questions
4:41 that learning management systems at the time and even Canvas now can ask,
4:46 as well as what book publisher homework tools we're providing that we
4:51 weren't satisfied that it could do all the things that we wanted tested,
4:55 and so we didn't want to dumb our classes down
4:59 by being constrained to what a given tool could do,
5:03 and so PrairieLearn has a lot of flexibility and extensibility
5:07 that allows it to not dumb down your course.
5:11 What PrairieLearn is trying to do is be
5:14 in the sweet spot between what LLMs like Canvas can do,
5:21 which Canvas makes it very easy to write a small set of questions.
5:26 On the other side of the spectrum is you could
5:29 build a tool from scratch that does whatever you want.
5:34 That's very flexible, but it's also a lot of work.
5:38 What PrairieLearn is trying to do is be somewhere in between on this Praido
5:43 optimal surface where we get most
5:46 of the flexibility of being unconstrained platform,
5:50 but also we can do a lot of things very easily.
5:53 I'm going to show you some examples
5:57 of that, but there's a couple of key concepts in PrairieLearn.
6:04 One is that maintaining a collection of questions
6:08 is very much like a software engineering exercise,
6:12 and so we should learn from all
6:16 the experience of software engineering exercises.
6:19 In that you're going to have multiple
6:22 people probably simultaneously working on your question bank.
6:25 Those people will be working probably on different parts of it,
6:29 but you want to be able to integrate
6:31 that work without people stepping on each other's toes.
6:34 Behind the scenes, there's for people to know what it is,
6:37 a Git repository which manages the question bank.
6:41 People can develop some code, other people can check it,
6:44 and only when everybody's happy does it actually
6:46 get committed into the main part of the repo.
6:50 Another key thing is that it uses standard technology,
6:53 so it uses HTML5 as the display language.
6:58 It uses Python on the back end.
7:00 These are common systems, and in fact the use of HTML5 allows us to take
7:08 things that were on the web and turn them into PrairieLearn questions.
7:12 If somebody built some neat little widget in your domain,
7:16 you can usually take that if it's
7:18 open source and make PrairieLearn questions from it,
7:22 and I'll show you an example of that.
7:24 The PrairieLearn doesn't have the notion of question types.
7:31 If you go to Canvas, it's like, do you want to do a multiple choice question,
7:34 or do you want to do a fill in the blank or an essay question, whatever?
7:38 There's no notion of question types, but there's things that people do as part
7:43 of questions that they do over and over again.
7:45 For example, you may ask for a numeric
7:48 input or a symbolic input or such and such,
7:52 and so when we identify there's these things
7:54 that people are doing over and over again,
7:57 we can encapsulate them in what we call elements,
8:02 and so that embeds all the codes so that somebody
8:06 can use that code without having to look at it,
8:09 and so we're building up a more and more
8:12 sophisticated collection of elements all the time that let people,
8:16 even without understanding how those elements
8:19 work build more and more sophisticated questions.
8:22 We're hiding a lot of complexity and making the tool still extensible.
8:29 Also, for people in computer science,
8:31 we have very sophisticated auto graders that are very flexible
8:35 because you can configure a docker container anyway you want,
8:40 so again, a lot of flexibility.
8:44 Let me show you a couple examples just to point
8:49 out the extent of the things you can do in PrairieLearn.
8:53 By all means, this is not exhaustive.
8:55 It's just what I cobbled together.
8:58 This is my canonical example of a question generator.
9:02 I teach a course on machine organization in computer science department.
9:06 I'm showing a picture of a circuit, and I'm asking the students to give me
9:10 a mathematical description of what the circuit does,
9:13 and this is a question generator.
9:15 If I say, hey, give me a new variant of this question,
9:18 it gives me a new circuit, and these pictures aren't being stored,
9:22 they're actually being generated dynamically.
9:24 I have a piece of code that knows how to generate Boolean expressions,
9:28 and I have another piece of code that given
9:30 a Boolean expression knows how to draw it as a circuit,
9:33 and so I can make as many of these as I want.
9:37 Basically, again, the idea is that I'm going to give students as much
9:41 practice with this idea as they need in order to master it.
9:46 This is a similar thing in the mechanical engineering domain.
9:50 This is from a dynamics class.
9:52 There's two rods.
9:54 It's giving the position and the velocity of the two points,
10:01 P and Q at the end of the two rods,
10:04 and it's asking what's the velocity of the Point A that's between them.
10:12 This also is a question generator.
10:14 All of these numbers are going to be randomized there, as well as this picture.
10:19 If I say, give me a new variant,
10:21 then it's going to give me a new configuration of the linkage,
10:25 as well as the corresponding locations and velocities of those endpoints.
10:30 Also, in our institution, we're trying to encourage our engineers to use
10:36 programming as a means of solving engineering problems,
10:40 and so we're providing the students all of the constants
10:44 in this problem in a bunch of different languages.
10:47 If they want to write a little code, to solve this problem,
10:52 they can do that without making transcription errors,
10:55 so I can just copy that, write some code,
10:57 come back and solve that, and that's true both on homework
11:00 and exams that we give them access to programming tools on exams.
11:05 This is an example of a thermodynamics question.
11:10 This is a little simpler that in this image, the image is static,
11:15 but we're going to change some of the numbers
11:18 in the image as we randomize the problem, so this one is 18 psi, and if I say,
11:24 give me a new variant of this, now it's 24 psi inside there,
11:29 and actually, it will change units.
11:32 In some cases, it'll use metric units, so it's giving kilopascals instead.,
11:39 This is an example of what's called the order blocks element.
11:44 Basically, I have a bunch of tiles that contain things,
11:47 and the student's job is to order the tiles in a way that solves some problems,
11:52 so this is what are called proof blocks.
11:55 I can build a proof by moving these in some particular order,
12:01 and if I grade it, it will tell me, you got the first one wrong.
12:07 Basically, it's giving feedback until it hits something wrong.
12:11 In this case, I didn't know how to do that problem,
12:13 so I got the first step wrong.
12:16 This is a physics problem or, I guess, arguably just calculus,
12:21 but it's plotting the acceleration as a function of time,
12:26 and it's asking the student, given an initial velocity at times zero,
12:29 such and such, what's the velocity at 12 seconds?
12:33 I as a student, have to manually integrate
12:36 this, so this is asking in numeric terms, and if I say, give me a new variant,
12:44 it gives me a different plot of the accelerations.
12:50 We can ask students to do symbolic math.
12:53 There's a tool that either I can bring up a keyboard that lets them
13:00 enter symbolic math or here's a lookup that, if I use LA Tech notation,
13:08 I can just type it, and it will check,
13:13 is your symbolic expression equivalent to the correct answer using SimPy.
13:20 We have support for graphing questions, so we can,
13:26 if I select that tool, I can drop down asymptotes.
13:31 I'm not looking at the question, but saying mark all the intercepts on the axis
13:38 and then roughly sketch the shape of this plot,
13:45 and we can auto grade that, that will typically have very small
13:50 tolerances for where the asymptotes are
13:52 placed because even after you've placed them,
13:55 you can reposition them, to make sure they're exactly in the right place,
14:01 similarly with the points that maybe that point should have been over here.
14:05 It's harder to free form, draw the line and get it in exactly the right place.
14:10 We'll typically give the students a much wider tolerance,
14:13 which is what we would do on paper,
14:15 is that the calc faculty that we built this for would just have the students,
14:20 like, label the positions of particular points
14:22 and then just eyeball is this sketch, roughly the right shape.
14:26 This is a questions from the robotics faculty.
14:31 It turns out to be a multiple choice question,
14:33 but what's interesting is that the answers are not numeric,
14:38 they're theographical.
14:40 It's saying, it's been a while since I took robotics,
14:45 but what's the manipulability ellipse of the tip?
14:51 When this is randomized, it will regenerate those multiple choice options.
14:58 Those are being generated randomly based on the configuration of this problem.
15:06 Speaker 1: Are you doing that image based on the new L_1 and L_2,
15:13 or is it just saying down below, there's a new L_1 and L_2, the work can be.
15:19 Craig Zilles: I think you could redraw it, I'm not sure this one does.
15:25 Yeah, it looks like it's constant,
15:27 but you could use that implementation to redraw it.
15:37 Again, it's up to what the person writing the question did.
15:42 The question was, is it redrawing this linkage of the L_1 and L_2?
15:46 It certainly could, but I'm not sure it did in this question.
15:50 This is a question asking students to draw free body diagrams,
15:55 and so this tool lets you take and position where the forces would be.
16:08 Again, it can auto grade this, probably shouldn't
16:16 spend time doing this, but so for example,
16:21 I might do something like that and save and grade this, and it says,
16:25 in this case, this is what the answer should have been,
16:28 so I got, I think, three of them, so it gave me 75%.
16:32 But I didn't get the downward force of the thrust piece itself.
16:39 This is an example of taking a piece
16:44 of open source software and building a question around it.
16:47 I'm asking them to design finite state
16:50 machines for people who know what that is.
16:52 That's great.
16:53 If not, it's a certain kind of graph.
16:56 I had found online somebody had built a nice little
17:01 HTML tool to design finite state machines, and I extended it.
17:10 As I'm building this, each time I do some action in this little CAD tool,
17:17 I'm updating a string.
17:20 When the student says, Submit, it's sending that string to the server,
17:25 and so I can auto-grade it.
17:34 If I build questions like this that capture
17:37 the student's work in a digital format,
17:39 I can auto-grade very sophisticated things without using AI.
17:43 What I would call deterministically auto-graded.
17:46 In this case, I haven't actually even finished it,
17:49 and I can say, Hey, go ahead and grade it.
17:52 This question will refuse to grade it because it says,
17:55 Hey, things are incomplete.
17:57 You didn't set the output state in this state,
18:02 and then there's a whole bunch of edges that are missing.
18:06 I, as a student on homework, get this feedback, and I can iteratively try it.
18:13 Or on an exam, I can take that feedback, and as an instructor,
18:18 I can give the students slightly less points each
18:20 time they submit it until they get it correct.
18:24 We have support for sophisticated code-writing problems.
18:30 This is running VS Code inside of PRI Learn.
18:33 If you want to do multi-file programming assignments, we can do that.
18:39 This is support for image capture.
18:43 For our calculus class.
18:45 They wanted to be able to not just grade the final answers,
18:49 but grade the derivations.
18:52 We added support for image capture.
18:54 On homework, we allow them to use their phones.
18:57 In the testing facility, we've added webcams to a bunch of our computers,
19:05 and so we can capture the image,
19:07 and then we can say, Here's the part that I actually want to grade.
19:11 That's not really very interesting math, but.
19:15 Our calc faculty use this for manual grading.
19:18 That PL has manual grading support that's very much like GradeScope.
19:24 You can see it here on the right.
19:28 We've been doing a bunch of work with getting it to do AI grading.
19:32 This is in the experiment that was
19:35 run last fall of auto-grading handwritten work.
19:40 We're randomizing the question.
19:43 We built a rubric.
19:45 In general, if you're building rubrics for AI,
19:48 you need to be much more detailed than you
19:50 would be for humans because the AI is bright, but it doesn't know what you want,
19:54 so you have to be very clear about what you want.
19:56 We'll take the original question, the student's work.
20:01 Usually, we'll compute what the correct answer is
20:03 for the given question along with the rubric, and we'll send that to the AI,
20:08 and it will go through and say, Here's what I think the student wrote,
20:11 and it will set all the rubric items,
20:15 whether they should be granted points or not.
20:19 There's actually two columns here.
20:22 One is the manual grading column.
20:24 One is the AI grading column.
20:28 You can see here that the AI, is actually on this one,
20:35 selected the rubric items similarly to the human.
20:39 Typically, this is being done in bulk
20:42 after the assignment has been submitted by everybody.
20:44 The faculty member or their course staff will
20:46 go through and hand-grade a bunch of elements,
20:50 then run the AI against it to see, is the AI performing similar to the human.
20:56 If not, they refine the rubric until it is.
21:00 My understanding is that AI is actually very
21:03 good at refining the rubric that you can say,
21:07 Here's some questions that you graded differently than I did.
21:13 How would you change the rubric to make sure
21:15 that you grade them the same way that I did?
21:17 Then you can hand-grade a couple more and see if it's on track.
21:23 I'm a long-term.
21:27 It's much more compelling to use AI grading
21:30 for formative things because giving the immediate feedback to students
21:35 is very valuable so they can know exactly
21:37 what they did wrong and try again for exams.
21:41 I think we can probably afford to still
21:45 manually grade stuff for high-stakes assessments because I think
21:49 students are willing to make the trade-off to get
21:55 some uncertainty when the trade-off is getting immediate feedback.
22:03 But they're maybe less excited about
22:08 having AI grade them on high-stakes assessments.
22:12 Other questions?
22:14 There's another question back there.
22:16 Speaker 2: Regarding the variations that it auto-generates.
22:21 Thanks, Eve.
22:23 This is not my field of expertise.
22:26 I'm looking at circuits like that or a logic
22:29 diagram like that versus the variant.
22:32 To me, that might be like one looks significantly simpler than the other,
22:37 but then again, that's me not being in the field.
22:41 My question is, Is there a knob that you
22:45 can tune on how the level of difficulty, it's based on the question?
22:49 I would assume that you want the level
22:50 of difficulties somewhat similar between the variants?
22:55 Craig Zilles: This is all done by me.
22:58 I was the one who decided these are an equivalence class.
23:02 I'm not saying I'm perfect.
23:05 There's a lot of people that are very concerned about perfect fairness.
23:12 I am not one of them, and I think it's misguided that when we run tests.
23:21 On homework, I'm not worried at all because, basically,
23:24 they can repeat the task as many times
23:27 as they want until they're satisfied they do the thing.
23:30 If some of the variants are slightly easier
23:32 than some of the other ones, who cares?
23:35 Obviously the place where the fairness matters is on a high-stakes
23:39 assessment where this is going to nontrivially impact their grade.
23:44 A couple of things.
23:45 One is, I try to avoid where there's any
23:47 one question that's worth a substantial portion of their grade.
23:50 Like, the one time I know of there was
23:54 big fairness concerns was in a class where there was
23:57 a two-hour exam that had exactly big coding problems on them
24:01 that were each worth like 7.5% of their final grade.
24:05 The students were very sensitized to which version of the question they got.
24:11 One thing that is true is that the students are concerned about fairness,
24:16 but they're not very good at actually
24:19 knowing which versions are harder than others.
24:21 Everybody thinks they got the hardest version, which obviously can't be true.
24:28 The other thing is, so I'm the one who's judging, are these roughly equivalent?
24:36 In practice, when I write, so when I write different versions of the problem
24:41 and I put them in a question pool,
24:43 I try to get them so that they're empirically within about 10% of each other.
24:48 If my exam has 10 or 15 questions and they're all within 10%,
24:53 then maybe some are slightly harder and some exams are slightly easier,
24:58 but it's roughly the same thing, and the amount of variants that I'm introducing
25:03 into their grade is much less than if one
25:08 of my colleagues teaches the class in the next
25:11 semester and has a completely different organization of the class,
25:13 that class version, the grades are going to mean different things.
25:18 The other thing that I keep in mind is we teach this much stuff,
25:22 and then we actually only test this much.
25:25 There's a lot of sampling error that's going on that the student that happened
25:28 to study the things that were on the exam versus the one who didn't.
25:32 There's a bunch of measurement error already there.
25:36 You're just trying to get the error that you're introducing through randomness
25:40 to be in the same scale as the existing sort of measurement errors.
25:46 I do my best that there is every once in a while,
25:52 I'll get one version here that seems somewhat simpler.
25:58 We did a study to try to look at this problem that we
26:03 basically looked at all the different versions for a series of exams,
26:09 and there were specific variants that we found that had in general,
26:17 it was rare that there were specific variants that faculty had
26:21 decided these were similarly performing
26:22 and that were wildly different performing.
26:25 Some examples of that are people were doing things
26:28 on the unit circle and things in quadrant three,
26:32 where X is minus and Y is minus,
26:34 turned out to be harder than things in the other three quadrants,
26:36 which, in hindsight, makes sense.
26:39 Sometimes if you're varying something
26:41 from a negative number to a positive number,
26:43 that zero is much easier or much harder than all the other numbers.
26:48 Anywhere there's a discontinuity seems to be a place you should be concerned.
26:54 But we were pleasantly surprised that we were taking random faculty and just
26:59 looking at what questions that they had written that, for the most part,
27:05 the randomness wasn't.
27:07 The faculty's judgment of how to randomize the question was pretty good.
27:13 That their pedagogical content knowledge about what's hard and what's
27:19 easy was a pretty good guide as they designed the questions.
27:23 There's another question here if you want to throw the microphone.
27:26 Speaker 3: Did they have to do this question or not?
27:30 Because I say just Z and not Z.
27:32 I don't know, it's been a while since I looked at those circuits,
27:35 but I think it's Z and not Z,
27:36 which is obviously zero, and then it's going into an OR.
27:39 Basically, this one ends up being not X and not Y or something like that.
27:46 But if they had to simplify and some
27:48 of the other ones didn't simplify so easily, then that would also introduce.
27:52 Craig Zilles: It's funny.
27:53 It's Z and not Z.
27:54 Speaker 3: It has Z and not Z,
27:55 and then it's going into an OR with something else.
27:56 Eventually Y came out of that hole.
27:59 Craig Zilles: You don't need to give it a minimal version.
28:03 Just being dumb.
28:05 But I think you're saying this term actually isn't necessary,
28:10 so it should be sufficient.
28:12 Speaker 3: The next one, too, because it's basically a zero going into an OR.
28:15 In the end, that whole thing comes out to be not X.
28:19 Craig Zilles: What this question is doing behind the scenes,
28:22 the expressions aren't unique.
28:24 There's multiple correct expressions.
28:26 What I'm doing is I'm generating a truth table from the expression
28:30 and then checking the truth table because the truth table is unique.
28:34 Speaker 4: Can you show me?
28:38 Craig Zilles: I can show you other questions too that may be more interesting.
28:41 But there's three main parts to a PrairieLearn question.
28:45 One is the metadata associated with the question.
28:49 This is just like what the name of the question is,
28:52 maybe some tags about what the topic is, and whatever you want to store.
28:57 In this case, it's a circuit and expression question.
29:00 If I want to later search for circuit questions, I can do that.
29:04 There's the presentation layer.
29:07 This is what determines what students see.
29:10 This is in sort of extended HTML.
29:13 Anybody who's familiar with HTML will recognize
29:16 this is wrapped in a paragraph tag.
29:19 There's some special tags.
29:22 This is an example of an element.
29:25 Here I'm saying this is the Prairie string input element.
29:30 I want the users to type something,
29:33 and virtue of a string is that I'm not going to constrain it to anything.
29:38 As compared to an integer input element,
29:42 that would only allow them to type numbers.
29:45 They couldn't type any punctuation or any letters in an integer input element.
29:49 String input element, they can type whatever,
29:51 and I'm just going to deal with whatever they type.
29:54 There's also a figure element that's saying, Hey, I'm going to draw a picture.
30:00 In particular, that picture is being drawn dynamically.
30:04 Then there's also this, so if I go back to the question,
30:14 go back to the view that I had before.
30:17 There's actually three ways that you can view a question.
30:22 This is what we would call the question panel.
30:25 This is what the student gets shown that they
30:27 can interact with initially to answer the question.
30:32 There's what's called the submission panel, which is every time I submit,
30:37 I'm going to record what the student did,
30:39 and I'm going to display them so they can go back and they
30:42 can see their whole history of attempts on this version of the question.
30:48 Then the third is the correct answer panel.
30:51 This is what the student sees when they can no longer attempt the question.
30:56 On an exam, if they can attempt a question four times,
30:59 then after the fourth time,
31:01 we'll show them whatever you decide to render in the correct answer panel,
31:04 or as soon as they get it right, they get shown the correct answer panel.
31:09 On homework, we often give them exactly one attempt.
31:12 You can try this version of the problem once.
31:15 If you didn't get it right, we show you the correct answer,
31:18 and then you can ask for a new variant of the problem.
31:22 PrairieLearn uses a don't repeat yourself concept, which basically,
31:28 we're going to generate all three of those views from one file.
31:32 In this file, you demarcate which things
31:36 should only show up in the question panel.
31:38 This is saying for the answer panel and the submission panel,
31:43 don't show this stuff because it would just be extra garbage on the page.
31:48 That's what you should show on all of the panels is this string input thing.
31:53 This input element gets rendered in three different ways
31:56 in the different panels that in the question panel,
31:59 it lets you enter a new answer.
32:00 In the submission panel, it shows you what answer you submitted previously.
32:04 On the correct answer panel, it shows the correct answer.
32:08 Then there's stuff that we only show in the answer panel, which is like,
32:12 I'm giving you some feedback of this is what the correct answer is.
32:15 I guess this is one solution is correct answer.
32:19 You can see here this is what's called a template.
32:23 This is something that's being computed by a piece
32:26 of code that's being jammed into this hole.
32:32 This is a hole in the document that we're going
32:35 to fill with something that's being generated by running code,
32:37 similar to this feedback message.
32:40 I also have a link to a video.
32:43 If you want to see an example of me solving one instance of this problem,
32:47 you can collect this video, and it won't be the instance that you got,
32:52 but hopefully it'll give you the process so that when you do it the next time,
32:55 you can get it right.
32:57 The last piece is the logic of the question.
33:02 This is maybe one of the lesser impressive things,
33:05 but whenever a student asks for a new version of the question,
33:10 we'll call the generate function if you choose to implement it.
33:13 There's a bunch of functions you can choose to implement or not implement.
33:16 In this case, the generate function gets called,
33:19 and I have a piece of code that's in a library that I wrote.
33:23 I have a bunch of questions like this.
33:25 I wrote the code once, and I put it in a library.
33:28 This library says, Call this function that says,
33:31 Give me an expression that has three inputs and is of difficulty one.
33:36 The library that I wrote there's a bunch of if I said difficulty zero,
33:41 I would have gotten some simpler expressions than the ones here.
33:45 It's generating that expression, and then it's putting it into the database.
33:49 We pass in this data object.
33:52 Not all.
33:53 If you're overwhelmed by this, I would say don't be
33:57 because a lot of even when I write these questions,
34:00 I do a lot of cut-and-paste coding where I take
34:02 a question that is very much like what I want to do,
34:05 make a copy of it, and only edit the things that I want to change.
34:08 I never remember what the right format of this is.
34:12 But basically, we're taking this expression I generated,
34:14 and that's going to be the correct answer.
34:16 I'm sticking it in this data object,
34:18 which gets stuck in a database by PrairieLearn.
34:21 The correct answer, and apparently, for some reason,
34:24 I decided to hardcode this thing here.
34:29 When the HTML goes, and it tries
34:38 to load this figure because it's a type dynamic.
34:41 It's going to call my code.
34:47 It's going to ask for a file, and I say,
34:50 if the file you're asking for is this, what I'm going
34:53 to do is I'm going to reach back into that data object.
34:56 I'm going to get the expression I generated,
34:58 and I'm going to call a function that knows how to draw
35:02 that, and I'm going to send that picture that I generated back.
35:06 It's three input functions, the generated circuit.
35:12 I was lazy, and so I passed that information.
35:16 The last piece is the grading.
35:19 This is a question where there's multiple possible correct answers.
35:24 If there was one correct answer,
35:26 I could compute that correct answer in this generate function,
35:30 and I'll show you some examples of that.
35:32 I tell the element, here's the correct answer,
35:35 check to make sure they gave the correct answer.
35:38 A numeric input element would have a tolerance and check to say,
35:43 is their answer within some tolerance of the answer that you said was correct?
35:47 If it was symbolic, we would say,
35:49 can I manipulate this to get an equivalent expression.
35:53 In this case, there wasn't a built-in
35:56 can you manipulate this Boolean expression thing?
35:59 What I do is I take the data object that Plen hands me,
36:06 which now contains the student-submitted answer for that expression,
36:11 and I try to generate a truth table from that.
36:16 But the student may have written gobbledygook.
36:18 If they did, this function is going to fail, it's going to throw an exception,
36:22 and in which case, I give them a zero,
36:24 and I say, your submitted answer is garbage.
36:27 If that doesn't happen, then I take my correct answer,
36:30 and I generate a truth table for it,
36:32 and I say, if their truth table is the same as my truth table, they get 100%.
36:37 Otherwise, they give a zero.
36:38 By the way, here's some other information.
36:41 I guess the feedback message is zero in this case.
36:44 Speaker 5: The professor or the TAs or whatever
36:47 have to do all of this for every question?
36:50 Craig Zilles: Somebody or something has to do this.
36:56 Let me shift gears a little bit.
36:59 Do you have another question about [inaudible] question?
37:03 That gives you hopefully some idea of what the internals look like.
37:09 Let me shift gears a second, and talk about writing questions.
37:17 Best practices for when you're writing computer-based questions.
37:22 Implicit in a bunch of this is that we're
37:24 going to try to auto-grade as much as possible.
37:27 I'm not an auto-grading zealot.
37:29 I think you should autograde everything that you can autograde,
37:33 but you shouldn't limit yourself to only
37:36 asking questions that can be auto-graded.
37:38 But there's a big bang for the buck.
37:40 You get this fast feedback loop
37:42 for the students for things that are auto-graded.
37:44 Things that make auto-grading easier.
37:47 First is make each question independent.
37:51 Try to avoid things where you have seven parts to a question,
37:54 and Part B depends on Part A, and Part C depends on Part B.
37:57 Try to isolate learning objectives.
38:01 If you have a question that's testing, do they know three different things?
38:05 If they know two of them, but they don't know the third,
38:07 it's hard to correctly assign partial credit deterministically.
38:12 That's a good thing for trying to use AI if you want to.
38:16 Another recommendation is to try to minimize the amount of storytelling.
38:19 People often have these elaborate scenarios
38:21 where they're like, blah, blah, blah.
38:24 Two things.
38:24 One is, if you're going to have
38:27 many isolated questions that waste the student's time,
38:30 and I think it's also an equity issue
38:32 that if you have non-native English speakers taking your class,
38:35 that storytelling is going to slow them down,
38:38 and it may have references that they don't know.
38:41 If you want to know, can you do this thing?
38:43 Ask them to do the thing.
38:46 The question was, do your instructors, and TAs have to write these questions?
38:53 Five years ago, the answer was yes.
38:55 The answer now is maybe.
38:58 I'm a huge advocate of having AI do boring stuff that I don't want to do,
39:03 and it's good at writing questions,
39:07 my mental model for AI is it's a middle of the pack
39:12 TA in that, my experience is my best TA is,
39:16 I can say, go write a question on X, and they write a beautiful question.
39:19 I'm like, no notes.
39:20 Let's use it.
39:21 The middle-of-the pack TA can do some things well,
39:24 but they can do other things not so well.
39:26 Ideation of I like, I tend to write the same problems over and over again.
39:32 I like getting ideas from my core staff, like, what questions do we write?
39:36 The ideas are sometimes good enough to help
39:39 me think of questions I should write,
39:40 but rarely would a middle-of-the-pack TA come up with a fully
39:45 formed question that I could use out of the box.
39:48 AI is good at coming up with ideas.
39:51 As far as posing questions, sometimes, it doesn't ask it the right way.
39:58 There's a lot of pedagogical content knowledge.
40:02 If you've taught a class a bunch of times,
40:04 you know how to strip out all the stuff you
40:06 don't care about and isolate the thing that you care about.
40:09 Middle-of-the-pack TAs aren't very good at that.
40:13 They are very good at implementing questions.
40:15 If you tell them, if you spec out the question,
40:19 they can mechanically generate the question.
40:21 AI is very good at that.
40:23 I'm also very worried about the AI's ability to randomize,
40:27 and generate similar difficulty versions.
40:30 I think there's a lot of pedagogical
40:32 content knowledge that we as instructors have
40:34 that indicates what things would make a question
40:43 harder or easier or the same difficulty?
40:46 Speaker 6: Can I ask a quick question?
40:47 Craig Zilles: Yeah.
40:48 Speaker 6: This came up several slides ago.
40:50 But you said, check the answer and make sure
40:54 it's symbolically equivalent to the answer that you give.
40:58 The question often is, what prevents the student from copying the question,
41:03 and putting it in for the answer, and it seemed, no, wait.
41:07 I wanted you to simplify that expression.
41:09 Craig Zilles: That's not in the hands of the students.
41:12 Basically, they type in their expression, and then that gets sent to the server.
41:19 On the server, we're checking is their version
41:21 equivalent to what we said was the correct answer.
41:27 Speaker 6: The question is equivalent to that [inaudible] Craig Zilles:
41:41 I haven't taught a math class, so that's an interesting question.
41:47 In this case, this is an example of a non-math
41:52 person asking it to write a calculus question.
41:57 Many people use AI to write PrairieLearn questions outside of PrairieLearn.
42:02 We have added support insight of PrairieLearn for people that don't want
42:06 to do that, be like a one-stop shop to write questions using AI.
42:11 The prompt at the top right is what I told it.
42:14 I said, generate a calculus question where
42:16 students have to evaluate a definite integral.
42:18 The integration should be over X,
42:20 but the question should also include terms involving Y
42:22 so that they have to provide a symbolic answer.
42:28 Maybe one of you could tell me what the correct answer is.
42:31 We could test it, but it thought a while it generated this.
42:36 Speaker 6: My concern is if you write in that whitespace,
42:39 integral [inaudible], then it would say, yes, that's correct.
42:44 Craig Zilles: Certainly, you could do things where you could be like scan
42:50 the answer to see if there's any instances of X there,
42:54 and if there are, give it a zero write off.
42:59 Speaker 6: Or if it had an integral sign, and the answer.
43:03 Speaker 7: There are ways to fix it, and other ways to cheat too.
43:09 There's ways to fool AI.
43:12 Craig Zilles: This is an AI, this is being done deterministically.
43:16 There's a Python library called SimPy, which does symbolic math manipulations.
43:20 I don't know the specifics of it, but my guess is so, yes,
43:27 you do have to be careful about students abusing auto graders that typically,
43:32 when we write coding questions,
43:37 what we'll do is, so I'm going to say save grade,
43:43 we'll often have a couple of hard-coded test cases where,
43:52 the students can see, on a particular input, did they get the right answer.
43:58 If you only do these, then they can hard code your function to like,
44:02 if they submit one, you should give this answer.
44:04 If they submit seven, you should give this answer.
44:06 we'll typically give most of the points.
44:07 I would actually have 70% of the points being
44:11 done by random test cases that change every time,
44:15 and that the students can't see.
44:16 You do have to think about,
44:18 what are the ways that students might abuse my system?
44:21 But I'm pretty confident that we can we can deal
44:27 with that, you can review the students' work, and see stuff.
44:38 Does anybody want to give me a prompt,
44:40 either come up and type something or try to give a somewhat what?
44:45 Speaker 7: Can I ask a different question?
44:47 One of the challenges that one faces if one
44:52 wants to go to PrairieLearn is onboarding an existing class,
44:57 the writing the questions is non trivial.
45:01 could you talk about the process that one would go through over
45:06 maybe multiple semesters or quarters
45:09 of porting an existing class into PrairieLearn,
45:12 and how that plays out, and why it's maybe not as bad as it sounds?
45:18 Craig Zilles: The first thing that I would do,
45:20 if I had never used PrairieLearn before,
45:22 and I was teaching a new class in, I would go join our Slack community,
45:28 and I would go join the general channel, and I would introduce myself and say,
45:32 "Hi, I'm teaching X for the first time using PrairieLearn.
45:35 Does anybody have any questions that they could
45:38 share with me?" I've had a lot of experience.
45:42 I gave Leo Porter a bunch of my CS1
45:45 questions that he used in the CS1 that he revised,
45:48 and my machine organization questions that they've, I believe, used here.
45:53 Our experiences have been that when people write question generators,
45:57 they're not so concerned about their questions going out because what
46:01 makes my exams trustable isn't that I'm keeping the exam secret.
46:05 That's not the primary thing.
46:06 It's that there's so much randomization in the exam that you can't
46:09 fit all the stuff in your head without actually learning the material.
46:13 People are very willing to share stuff.
46:15 I think a second important thing is you don't need to shift all
46:20 of your stuff to PrairieLearn all in one semester that for many years,
46:26 we didn't know how to do our cash exam as well as we wanted on computers.
46:31 We had six computerized exams, and one paper exam.
46:36 you can piecemeal things.
46:38 A mix of auto grading, and manual grading,
46:41 I think, is also useful in our data structures class.
46:45 The last question on all their exams is, here's a situation?
46:49 What data structure would you use to solve this situation, and why?
46:53 The other thing is basically you start off with the easy stuff.
46:59 You can make some hard-coded questions.
47:01 You can make some questions where here's a picture.
47:03 I'm going to randomize the numbers.
47:06 You do the easiest stuff first.
47:09 What takes a lot of time is doing the first question of a given type.
47:15 For example, the first time I did one of these finite state machine questions,
47:23 it took me like two days to get the first
47:25 one working because I had to get all the pieces working.
47:27 Once I had one working,
47:28 then I could bang out more of them in half an hour or whatnot.
47:33 The big advantage is rather than rewriting all of your stuff every semester,
47:39 you can reuse the stuff you used last semester.
47:42 You don't have to rewrite that because
47:44 if you've written it up as question generators,
47:46 you can extend your question pool, make it bigger,
47:51 and bigger every semester because that effort you would have spent
47:54 by rewriting everything you could instead put to writing more stuff.
48:00 I don't know, Steve, is that.
48:02 Speaker 8: Is there any checking when people
48:04 go onto the board to make sure they're not
48:09 a student instead of a faculty or, can you
48:12 imagine I'm signed up for your class, and I like,
48:14 go on board and say, "I'm teaching a class on whatever it is,
48:17 can you send me the question?" Craig Zilles:
48:19 So far, I've shared it with people that I have a personal relationship with.
48:23 But there's a lot of, I'm also not super concerned about that.
48:33 Again, my mental model is, if my question pool is large enough,
48:39 I could put it in the public domain,
48:41 and it wouldn't matter because it's hard to memorize
48:46 a bunch of stuff that doesn't mean anything to you,
48:49 that I'm not trying to stop students from cheating.
48:52 I'm trying to make it not profitable,
48:54 that I want the rational strategy in my class to learn
48:57 the material because that's the easiest way to get the grade.
49:01 There are students that will still try to memorize all sorts of stuff.
49:05 They'll spend way too many hours,
49:07 and it's not as successful as if they sat down and, learned the stuff.
49:12 We see people trying to sneak notes into our testing facility of questions
49:17 that were on the practice exams that aren't actually on the exams.
49:20 It's like, what are you even doing?
49:22 But I understand your concern.
49:26 I think people would probably do a web
49:28 search for you before they shared the stuff.
49:33 But in practice, I'm not actually that worried about students getting
49:39 my stuff because they're going to have to demonstrate that they can do
49:42 the thing in our testing facility when they don't have access
49:45 to AI or, communicate with friends that can look stuff up for them.
49:50 Speaker 7: I would add that one of the things that we do,
49:53 also many of the professors in our department do is
49:56 our tests are questions that they've already seen in the homeworks.
50:00 We will have a massive testing bank with many questions,
50:05 and they'll have access to all of them for homeworks,
50:07 maybe two or 300 questions.
50:10 then the homework will say that the exam is going to be a subset of these..
50:15 Craig Zilles: They don't know which?
50:17 Speaker 7: They're going to be randomized, and so go practice the homeworks.
50:20 If you can do them all, then you're done.
50:23 It also means your exams no longer have to be hard,
50:28 you can make your homeworks as hard as you want the students to be able to do,
50:33 and then verify that they can do
50:34 that in a relatively straightforward way in the exams.
50:38 Speaker 9: Even if they have all the questions,
50:41 go, learn how to do all the questions.
50:43 That would be great.
50:44 Speaker 10: How realistic is that for actual university learning
50:48 when they get out in the world and they're like,
50:49 I haven't seen this question before.
50:51 I don't know what you did.
50:53 Craig Zilles: We will typically, again,
50:57 the instances of these questions are probably
51:00 not going to be ones they've seen before.
51:02 For example, the circuit, they won't have seen that circuit before.
51:07 For programming questions,
51:09 we will typically put slightly different programming questions on our exams
51:13 than the ones that are on the practice exams and homeworks.
51:17 Example, I teach a CS1, and so one of the things I ask
51:22 the students to do is given a collection of things,
51:24 give me a new collection of things
51:26 that contains the subset that all fit this property.
51:29 I can write many different versions of that question.
51:34 I know what you're saying is transfer is the thing that we care deeply about,
51:40 especially as academics.
51:42 I think usually that's the thing that's distinguishing maybe
51:45 a B level student from an A level student that.
51:48 Speaker 10: Even levels of knowledge.
51:51 There's a regurgitation question, and there's like simple application.
51:56 At the top, there's like actual level.
52:00 Craig Zilles: But be honest with yourself what fraction
52:03 of your students do you expect to do actual novel problem solving?
52:09 Those are the people that were typically saying you
52:11 should go to graduate school when we meet them.
52:14 Speaker 9: I think also you can you can also do manual grading.
52:19 You can draw pictures, you can upload the images.
52:23 Those aren't auto grade.
52:24 Those can be in a different exam.
52:26 You're not just bound to doing auto grading only,
52:31 and those questions you could not release.
52:34 Craig Zilles: We were talking at lunch
52:36 that you could auto grade a bunch of stuff
52:39 in the CBTF and then have a paper exam that everybody takes at the same time.
52:45 If you have one version of the question that's asking them to do
52:48 novel transfer that each semester you
52:50 write that question completely from scratch.
52:54 I think it doesn't have to be an all
52:57 or nothing to get benefit from using this approach.
53:01 Just quickly flip through the slides,
53:04 but people can pause the recording and read the slides,
53:10 or I can quickly in 10 minutes go over a bunch of best practices.
53:22 Speaker 9: There's a question in the chat.
53:25 Hold on.
53:25 I was just curious about if there were
53:28 any key features of the question generating scheme
53:31 that maybe you wanted to implement when creating
53:34 PrairieLearn but that didn't end up working out.
53:40 Craig Zilles: PrairieLearn is a living platform.
53:45 PrairieLearn is open source.
53:46 If there's a feature that you want that it doesn't implement,
53:50 you are encouraged to reach out to the maintainers.
53:55 I encourage you to reach out to them before you
53:58 write a bunch of code and submit it as a patch,
54:00 but because that'll hopefully save you some time.
54:02 But they definitely take patches from various people to add new features.
54:08 We were talking at lunch, one of the features that isn't there yet,
54:13 but seems to be more and more interesting is
54:16 the ability for a question to know things about its context,
54:21 that right now a pre-learning question doesn't know which student is doing it,
54:25 and whether the student is doing it as homework or on exams.
54:32 That second part is mostly good in that in general,
54:35 you'd like to be able to write a question
54:37 and then reuse that same question maybe on in class activities,
54:40 on homework, on practice exams, and exams.
54:43 But there's reasons why you may want
54:46 to have slightly different versions of that question.
54:48 rather than making two copies of it one for each context, if you could say,
54:52 hey, what context am I in, that could be a useful feature.
54:57 That might be a thing that comes to mind.
55:00 Speaker 11: You pointed that out with a nice line
55:03 there about the similarity in the complexity of the exam problems,
55:07 and you pointed out it's very important.
55:08 I also think it's important.
55:09 What do you do to make sure
55:12 that different students get exams of the similar complexity?
55:16 Craig Zilles: Mostly at this point,
55:22 I've been running the same exams for a bunch of semesters,
55:25 and so I'm pretty comfortable that they're of similar difficulty.
55:29 PrairieLearn has pretty good statistics about after you run an exam,
55:33 if you have a question pool,
55:34 it'll show you what the average correctness was for each question in that pool.
55:39 If you have an outlier or it's much higher or lower scoring,
55:43 typically, I would take that out of the pool.
55:46 The next semester that I ran it, I would feel a little bad,
55:48 and then I would go on with my life in the current
55:51 semester that my world is that I will give the students two things.
55:55 One is, I'm asking them many questions over the semester,
55:59 like those little things even out.
56:02 The second thing that I do is I do second chance testing.
56:05 They have two opportunities to do well on my exams.
56:08 If they're not doing well,
56:10 it's probably not because of the random difficulty variances,
56:14 it's probably because they just don't know the material.
56:17 I sleep very well.
56:19 This is not something that I spend a lot of time worrying about.
56:21 It's also not a thing that the students complain a lot about.
56:26 Only when there's a small number of problems that are worth
56:30 a lot of points and they don't have multiple attempts at them,
56:33 that seems to be like the thing to not do.
56:39 Speaker 12: Is it like picking, like four questions out of your bank of 200,
56:44 or is it like Question 1,
56:47 pick something from this group that's of approximate same difficulty.
56:51 Question 2, another group of approximates.
56:53 Craig Zilles: It's much the second of them.
56:55 My exams will typically, I call them slots on the exam.
57:00 Each slot is guided by a pool, and that pool usually is testing
57:04 a particular learning objective at a particular difficulty.
57:08 For example, if I go back to this circuit question,
57:14 so there's a very simple instance of it.
57:16 That's the one that I'm embarrassed by.
57:19 I have a whole collection of questions
57:21 that are testing the same learning objective, which is,
57:24 can you take a Boolean thing in one format and convert it to another format?
57:29 The formats I have are circuits, Boolean expressions, truth tables,
57:32 and [inaudible] Can I give you one of those and you give me back another one.
57:37 My question pool is,
57:39 for a certain difficulty it's like whatever nine or 10 different versions
57:44 of problems where I give you one version and ask for a different version.
57:48 Speaker 12: You'll have this one with different circuits,
57:51 and then you'll have a separate problem that also has variations.
57:55 Craig Zilles: I give you a random truth
57:57 table and ask you to write the Boolean expression,
57:59 or I give you a random expression and ask you to fill in the truth table.
58:01 I show you a circuit and ask you to fill in the truth table.
58:05 I want you to be fluent with this notion of like,
58:09 these are all representations for the same thing.
58:12 Speaker 12: If two students take it at different times,
58:15 because you are saying that students can take them at different times,
58:18 then the first one leaks that there's this question.
58:22 The second one doesn't get that same exact question,
58:24 but now they know they're going to get something that's still of the same.
58:28 Craig Zilles: I've already pre-leaked this.
58:30 What Steve was saying,
58:31 I will give them a practice exam generator that has this exact pool on it,
58:36 that when I have a big pool of question generators, I don't keep it secret.
58:40 I, in fact, want them to grind that pool of question
58:43 generators and be able to do all of the things.
58:46 I can be like, great, you can do all the things.
58:48 I'm not teaching to the test is only bad if the test is bad.
58:52 Once you have enough stuff you're just like, hey, go to town.
58:59 Initially, when you make this transition at first,
59:03 you may not have a lot of questions and you're worried about,
59:05 we use the AI term of overfitting where it's like
59:08 if your test set is the same as your train set,
59:11 they're going to do disproportionately well on that.
59:15 But once your test set is enormous,
59:20 if your test set covers all the things that you care about,
59:23 then you could train on the test set and be happy with the consequences.
59:29 Speaker 12: How long would it take
59:30 somebody getting into PrairieLearn from scratch to get
59:33 up to where they have a bank of 200 questions or whatever you need?
59:36 Craig Zilles: It depends a lot.
59:38 A, if you know somebody who can give you 200 questions, then a few minutes.
59:45 If there's already questions like what you want to do.
59:54 I basically built a CS1 course in real time one semester.
1:00:01 I've been using PrairieLearn for a long time
1:00:03 in my machine organization class and a couple of other classes.
1:00:05 I was teaching the CS1 for non majors for the first time.
1:00:09 One fall semester, I'm like, over the summer,
1:00:12 I'll write all the questions as we as faculty are want to do.
1:00:18 By the end of the summer, I had two weeks of material,
1:00:22 and I basically wrote the rest of the material as the semester was happening.
1:00:28 In addition to producing all the lectures and whatnot,
1:00:34 because I was completely revising the question.
1:00:36 It was taking me as a relatively experienced pray learned person key thing is
1:00:42 if there's a question that's the same form as what you want to do,
1:00:45 so that what's a good example?
1:00:49 Like this thing where it's just like, there's going to be a picture.
1:00:52 I'm going to randomize some numbers around it.
1:00:55 I can make a copy of this, and in 15 minutes, I can produce the new picture,
1:00:59 I can write the text, I can write the randomization,
1:01:02 and then I send it to my TA to check a bunch
1:01:05 of times to make sure it seems reasonably good, and I'm done.
1:01:08 You can blast out these questions pretty quickly.
1:01:12 What low is doing the first of a new question?
1:01:17 At this point, there's a lot more things,
1:01:25 like now you can do a grabbing question in relatively quickly
1:01:28 or one of these Parsons question or order block questions where
1:01:35 you just have to come up with the proof or the piece
1:01:38 of code that you're going to turn into a Parsons problem.
1:01:44 I guess you have to think about the randomization piece a little.
1:01:49 But maybe you don't have to randomize everything initially.
1:01:56 But it can be done by one person
1:02:00 in a slightly unhappy semester in real time as a bound.
1:02:08 Speaker 9: I was more of a novice
1:02:10 when I did this, and I PrairieLearned a course.
1:02:13 I got some materials from another professor.
1:02:16 I went through and did wrote translations of the exams
1:02:20 and the homeworks and went through and just
1:02:23 did the math once and then put in some
1:02:25 random numbers to make it a little bit harder,
1:02:27 and that took maybe like 15 minutes a question.
1:02:31 Then I would go through my slides,
1:02:35 and I wrote down all the vocabulary words and all the key concepts,
1:02:39 and I handed a list to a TA.
1:02:42 I said, go make questions about vocabulary words for these 10
1:02:46 terms and then we would go over them in our weekly meeting.
1:02:51 That got most of the way in another quarter.
1:02:54 Then by the third quarter in that quarter, I was being pretty careful about.
1:02:59 I didn't have enough questions,
1:03:00 so a big question bank and use them on the final.
1:03:03 But then by the quarter after that, I had
1:03:06 a big enough question bank that everything was pretty much running.
1:03:10 It took, I guess, by the third iteration, things were in good shape.
1:03:17 Speaker 12: What situation with a lot of textbooks
1:03:21 that offer question banks that instructors can use in online curriculums?
1:03:26 Speaker 9: I didn't have that situation in my class.
1:03:28 There's not a great textbook for the class I was teaching.
1:03:32 But I guess you could you could draw on those question
1:03:38 banks as well or maybe look at them for inspiration.
1:03:41 Speaker 12: Do those question banks exist for any person?
1:03:44 Speaker 9: I don't know.
1:03:45 Was anyone publishing PrairieLearn?
1:03:47 Craig Zilles: Not that I know of.
1:03:50 Again, definitely what I would recommend is if you have colleagues that you find
1:03:55 out that are teaching the thing
1:03:56 that you are teaching that have used PrairieLearn,
1:03:58 you should just ask them for their questions
1:04:00 because many people are willing to share that one of the big thrusts now
1:04:09 in PrairieLearn is to facilitate question sharing.
1:04:12 The old way of sharing is I give you read access
1:04:16 to my GitHub and you can just take whatever you want.
1:04:18 Now support inside of PrairieLearn where you can mark questions saying,
1:04:22 Hey, I'm willing to let other people use these.
1:04:24 if you know about them,
1:04:25 you can use other people's questions through the platform,
1:04:28 which the advantage is that you're not forking the source.
1:04:31 If they fix bugs, that you get the bug fixes.
1:04:34 There's not a good discovery tool yet of what
1:04:37 I want is the yelp where it's like, give me a question on Dijkstra's algorithm,
1:04:42 and there's all these upvoted questions and I see, Steve Swanson wrote this one,
1:04:46 so I know he's good, and so I'll take his question.
1:04:49 That's the future that we want to get to, and then somebody then curates
1:04:53 all of the best questions into here's a good assignment on such and such topic.
1:04:59 We're not there yet, but I think it's just a small matter of people rolling
1:05:06 up their sleeves and doing the work that I think again,
1:05:11 people are willing to share stuff.
1:05:13 It's just getting people together and putting stuff out there.
1:05:19 Speaker 11: May I ask how powerful is the AI generator?
1:05:23 I don't know that we saw it in action.
1:05:27 Craig Zilles: This is still a work in progress.
1:05:30 It seems pretty good at questions that involve a single
1:05:35 line of input either like you can have multiple choice questions
1:05:39 if you cared about those, but a single input
1:05:42 of a numeric question or some one line input symbolic or whatnot.
1:05:47 If you want to try typing something, it doesn't do programming questions yet,
1:05:53 at least the last time I test drove it,
1:05:55 and it was struggling at doing visual stuff that at that time,
1:06:04 the agent wasn't able to see the errors that were happening in the browser.
1:06:08 It was drawing a bunch of stuff that didn't
1:06:09 end up getting rendered because there were JavaScript errors.
1:06:13 But the developers know that.
1:06:16 But if you want to either type something or tell me what to type.
1:06:19 Speaker 11: The questions that you showed
1:06:20 that you developed yourself and coded up.
1:06:23 This AI cannot be used for such questions, is that right?
1:06:26 Craig Zilles: Right now.
1:06:29 But if you were comfortable using something like Claude Code,
1:06:32 though, and you had an example question,
1:06:35 I'm very confident that you could get something like Claude Code to say,
1:06:39 look at this directory to get the idea of the structure.
1:06:43 Here's what I want you to do.
1:06:45 Make a new question that does this using that structure.
1:06:49 Here's the PrairieLearn documentation link,
1:06:54 read the documentation if you need to.
1:06:59 You haven't if you are somebody who's willing to do programming
1:07:03 and you haven't played with Claude code or the like yet,
1:07:08 it's a trip that it's quite impressive.
1:07:19 That stuff will get integrated into this, but they
1:07:23 are only rolling out stuff that they know works.
1:07:26 Right now, they haven't set it up
1:07:31 to have the examples of doing the programming questions.
1:07:35 Speaker 9: Any more questions online or in the room?
1:07:41 Let's thank our speaker once more [Alarm]