In this month’s session, guests discuss the logistics of beta testing an open textbook, including: strategies to recruit beta testers, mechanisms for collecting and implementing feedback, and marketing this process. If you’re curious about how open textbooks are tested in classrooms, or how and when student and instructor feedback is incorporated, read the recap below or watch the video recording.
This Office Hours session began with brief introductions to the Open Textbook Network, whose member institutions pool expertise and promote best practices in open education, and the Rebus Community, a collaborative resource of open textbook creators and users. We also solicited suggestions for topics for future Office Hours sessions, and if you have ideas that you would like to explore or revisit, please let us know.
This month, we were joined by Michael Laughy, Dianna Fisher, Linda Bruslind, and Elizabeth Mays to discuss the process of beta testing an open textbook. The opportunity for beta testing is one of the main competitive advantages of open textbooks, as they can be updated and revised based on classroom feedback more quickly than traditional textbooks. Watch the video recap of the session, or continue reading for a full summary.
Speaking first was Michael Laughy, who is an assistant professor of Classics at the Washington & Lee University. He recently co-authored an open textbook on Ancient Greek, and has been using it in his language courses. The book is also being beta-tested by faculty at Louisiana University and the University of Illinois. Laughy says he makes live changes to the book to incorporate students’ feedback during class. Laughy judges students’ reactions to material as he teaches it, and in so doing learns how to edit chapters in the book for the next time he and others teach the course. The experience of teaching with the book also gives him a better sense of how to partition the book – and how much material can actually be covered weekly during a semester. The faculty at Louisiana and Illinois also have editing rights on the book, and make similar adjustments based on their experiences in the classroom. However, Laughy acknowledged that the live changes can prove confusing for students who may be trying to look for a piece of information from earlier in the course, which has been altered or deleted.
Next, Dianna Fisher, director of Open Oregon State at Oregon State University, described their process for beta testing. She asks faculty to first pay attention to areas where students have historically had difficulty in understanding subject matter, and test these sections of the book with students. Dianna says that they keep two versions of each book – one that is “in use” in the class, and one that is being “edited,” so that they can easily restore complete versions of chapters if needed. She said that they use Basecamp for project management, including managing beta testing changes. On some books, Dianna notes that beta testing is done in stages with specific groups of students – for instance, first with doctoral students, next with masters students, and later with advanced undergraduate students. With the final group, she encourages faculty to find out what students need to know to fully digest or comprehend the information in the book, so that different versions of the book can be created for different levels.
Linda Bruslind, who is a senior instructor and lead advisor in Microbiology at Oregon State University, was next to offer her perspective. She authored an open textbook for a 300-level general microbiology course, as she noticed that students weren’t using the traditional texts she had assigned, and since she wanted a simpler text for students. Linda first tested this book in her summer 2016 course, and later tested it online through Oregon State University’s eCampus. Linda found that students in the in-person class would access the text on their phones, or print out specific chapters, at the same time as they were completing group activities. She invited students to give her feedback, identify areas where information was unclear or lacking, and point to any errors in the book. She then passes this feedback on to Dianna, whose team makes changes to the book. Linda noticed that the post-assessment scores in her courses went up dramatically after using the textbook.
Our final speaker was Elizabeth Mays, adjunct professor in the Walter Cronkite School of Journalism and Mass Communication at Arizona State University and marketing director at the Rebus Community. Elizabeth combined forces with lead editor Michelle Ferrier at Ohio State University to create an open textbook on Media Innovation and Entrepreneurship. Mays said Ferrier quickly recruited 12 beta testers for this book by reaching out to people in her network. They, along with the book’s authors and 38 others in the community of practice teaching these subjects, were encouraged to join a fortnightly call to discuss each chapter of the book. Beta testers were given guidelines and many mechanisms to provide input. Feedback was also solicited through Hypothesis, which was enabled on the book; through the Rebus Community forum; through a Google Form; and via email. Mays compiled all the feedback into one spreadsheet. The co-editors then made decisions on what changes to implement and how. Elizabeth discovered this process was quite laborious, and said she wished there was a better mechanism on the book itself through which to collect, track, and respond to feedback.
The floor was then opened to questions from other participants. Participants were curious as to whether students in online courses and in-person courses provided different feedback about the textbooks. Linda noted that feedback from the two groups was very similar. Other participants wondered what types of feedback were solicited and how. Linda welcomed all kinds of feedback from spelling errors to the clarity of a particular chapter. Elizabeth mentioned that a Guide for Beta Testers was created with prompt questions for faculty using the book in their courses. Karen wondered what the pros and cons were of framing the book as being in beta testing. Linda liked the idea of a more formalized feedback process, and would want to provide all students with an opportunity to give feedback on the book. Michael noted that students were hesitant to critique, challenge, or correct information in the book as it has been written by “the professor” – he actively sought to make students comfortable to state their opinions about the book and its content. Dianna noticed that students overcame this initial discomfort and later feel more invested in their learning, and felt that their contributions made an impact. Linda said that students seemed excited about being engaged in the process and having some control over the content of the book. Another question was about who implemented these changes – in Michael and Elizabeth’s case, they made changes themselves; in Linda’s case, the implementation was done by the publisher.
Beta testing is a valuable process to gain student and faculty insights on how open textbooks can be improved. Thanks to our guests, and to participants who attended and shared their thoughts! If you would like to have further conversations on these or related areas, please let us know on the Rebus Community forum!