Jacqueline Leighton's Reading List

"He argues that we can, but we have to make sure that we articulate the way that we think the mind works so that we can actually create test items that tap into parts of the way the mind might be measurable. I know that sounds circular, but the contents of the mind are not observed and so we need to find instruments to probe and make observable the contents of the mind—the way that it thinks or the way that it creates ideas, the way that it’s able to formulate ideas before they’re spoken. That can help us create tests that actually are the keys that can unlock the expression of the mind in written form. Get the weekly Five Books newsletter What I love most about this book is that it challenges assumptions about what we think about tests. Not only is it solidly conversant in psychometric theory, mathematically, psychometrically, it’s very good—certainly, an expert wrote it— but this expert is challenging the way that we’re thinking about tests, especially in terms of how we think about validity. One of the ways that we can better measure the mind, according to the book, is by developing theories of the way that we expect people to think about things—like mathematics, for example. So before we start creating a math test that, say, measures someone’s understanding of trigonometry, creating questions that we think are the best way to tap into what they know, it would be really useful to understand, first of all, how people think mathematically. How do people, especially those who do so correctly, solve problems in trigonometry? How do they think about numbers? How do they put this information together? How does the mind—with the limitations that we know it has, like working memory—select information, combine information, express information? So before we even begin to ask questions, he argues, we need to understand exactly what we’re measuring. So if we want to measure mathematical reasoning, problem solving, in trigonometry, how do people think about that? That’s the first question to ask. Then, once we understand that, we can then begin to ask questions about different parts of the measurement process."

"Well, that’s actually one of the other books on my list, Protocol Analysis (1993) . So one way that we can understand how people think is by conducting interviews for what are called ‘think aloud’ studies. You give somebody a problem—let’s say like a test item or a task—and say, ‘I want you now to think out loud as you solve this problem, and as you’re doing so, tell me everything that’s going through your mind as you try and figure it out. Now, that’s just one way of understanding because there are certain limitations with the approach. First of all, if the problem is too easy or too difficult, there won’t be anything to report. So the problem has to be of moderate difficulty for the individual to actually be able to report something. That assumes, as well, that they’re making use of their short-term working memory for the thoughts or ideas, that they’re then able to report. Another way is to use eye-tracking devices. This is done especially with reading comprehension problems. You give somebody a script to read or some text, and then you ask them some questions, and you follow their eye movements as they try and respond to the questions. So you track the way that they read the text and then you track the way that they go back to the text and look at certain words, look at certain parts, as they’re trying to respond to the question. Those kinds of behavioural indicators give clues as to the way that people might actually go about discriminating, selecting information, combining information ultimately to generate a response to a test item."

"This book had a big influence on me because it came out in 2001, just as I finished my post-doc and got my first job as a professor. I was a psychologist interested in testing, and I knew that I had a real uphill battle. There weren’t many psychologists, it seemed, who were actually focused on cognitive psychology really working in the field, in the sense of really wanting to understand what test items measured in students. Most were quantitative psychologists working on just the numbers, mainly measurement models. “I think that we should, as testing specialists, be very aware of what the unintended consequences might be of the tests that we administer to children and frankly all learners. What are they really learning?” This book was one of the first to clearly articulate that we really have to think about the cognitive psychology behind the tests that we create. So one of the important parts of the book is that it includes what is called the ‘assessment triangle.’ We have to think about the observations we make, the interpretations that we make, and also about the cognition that we observe. So it’s a book that very clearly emphasises the need for us, as testing specialists, not just to think about the quantitative measurement model aspects of creating tests but also the psychological aspects of exactly what we’re measuring. Certainly there were some people writing about it. Susan Embretson , for example, is a psychologist and a testing specialist that I have admired for decades. She was one of the first scholars who I began reading about who was making this point—that we really need to create tests that are closely linked to the psychological constructs that we’re interested in measuring. But she was a lone voice in many ways. It wasn’t part of the everyday conversation about testing. Now it is. This book really opened the floodgates for a lot of research focused on cognition and assessment, with an increasing number of scholars taking Susan’s work and really running with it."

"This is a book written by a schoolteacher back in the 1960s. It’s a really interesting book. It forces the reader to consider the unintended consequences of what we do in classrooms, including testing, and what we subject students to. One of the things that John Holt talks about is how children can learn to game the system, because they begin to realise what it will take to do well in school. So they can “fake” doing well, but that doesn’t mean they’re learning. They’re just learning to do well in the system, and the system rewards certain ways of thinking, certain ways of behaving, certain ways of communicating. That can actually turn children off the real learning that we need them to do. “I think there is virtue in understanding why there is a segment of the population that is against testing. ” The reason I think this is an important book for testing is because I think that we should, as testing specialists, be very aware of what the unintended consequences might be of the tests that we administer to children and frankly all learners. What are they really learning? What is the evidence that they are truly learning content or to think deeply? Or are they simply showing us what it is we want them to show us now but will soon forget as meaningless later? So they’re showing us what they know, but what might be the consequences or the unintended consequences of the way that we do it? Yes, he would. So this is just my opinion, but I think there is virtue in understanding why there is a segment of the population that is against testing. There is value for testing specialists, like myself, to understand their perspective because I think we have a lot to learn. That perspective is highlighting the things that tests might do that is actually not very good. How can we improve on that? Yes. And you know what I love about so many of the quotes from that book is that they also apply to adults. I’m a university professor, and I’m here in this great institution of higher learning. I love to learn, but even I don’t want to be taught, necessarily."

"They might have a much broader understanding of the way that this book has influenced teaching. The reason I have it on my list is because it’s influenced the way that I think about testing. Because even though the book is about learning, assessment is part of the way that we currently teach students. So you have teachers who obviously are teaching and hoping students learn, but are also part of a system of accountability and so we have assessments. This book, more than any other book that I have come across, articulates the importance of the social and the emotional variables that are important to consider not only when children are learning, but also when they’re being assessed. So for example, one of the great things about this book is that it involves issues of trust. As instructors, as teachers, are we trusted by our students? Are we trusted by them to have their best interests at heart? Are we trusted by them to help them when they make mistakes, to help them express themselves? Because trust is important if we are then going to give them feedback about where they need to improve and the methods by which they should improve. “Part of really feeling free to learn is that you feel supported in your entirety as a learner, not just cognitively but emotionally and socially.” And all of this is taking place within a social environment, with other children. Are those other children part of the learning process in a positive way? Is there empathy in that classroom? All these aspects, ultimately, all these variables, are important to consider at the end of the semester when children are completing their assessments or their final exams. Has a culture of learning truly been created in the classroom, or any kind of learning environment, that is supportive of learning, which then an assessment is designed to measure? I don’t think assessments can measure learning in isolation, although we try to. But it is really a social and collective process. Children don’t learn in isolation, certainly they are not isolated in classrooms, and we’re testing them in classrooms. This is an important part to consider. My understanding is that freedom to learn is the freedom to express yourself, and the freedom to make mistakes. Learning is a messy process. There are going to be ups and downs. So part of really feeling free to learn is that you feel supported in your entirety as a learner, not just cognitively but emotionally and socially. Children learn in different ways. Do we give all children the freedom to learn in their own way? Well, of course in classrooms it’s not possible to tailor instructions to each individual student, but is there at least recognition that there is not a one-size-fits-all and that we try and take steps to accommodate children who learn differently? We have to be careful with that. We don’t want to give children mixed messages. It’s very dangerous not to give children clear messages about where they are at, especially academically. Personally, I think there is nothing wrong with saying, ‘That answer is not correct. But let’s find out how you came up with that answer, and let’s see where you might have gone astray.’ Furthermore, if a student and a teacher have a trusting relationship, then there is room to be supportive but honest. The reason I became interested in educational testing is because my psychological research was focused on measuring logical reasoning. So my PhD research and also my post-doctoral research were all focused on uncovering why it was that in the 1960s and 1970s psychologists were using a certain task called the ‘Wason’ task to evaluate the level of logical reasoning competence that undergraduates had. One of the things that I found most intriguing with this research was that I considered the task to be very biased in the way it was prompting students to produce their responses. It assumed that these students who were responding to the task had a certain level of knowledge in the area of what is called ‘deductive logic.’ So I began to scrutinise and really think more deeply about the way that we assess knowledge and skills in the population. We think about these items and tasks, even in psychology, as things that are fairly direct measures. In fact, they’re anything but direct measures—especially when we are trying to measure things that are unobservable, like reasoning process, problem solving, evaluation, analysis, creativity. These are all unobservable constructs."