American Society for Engineering Education                                                                Pacific Southwest Section

Published in Spring 2003 Conference Proceedings

Using On-Line Quizzes Outside the Classroom to Increase Student Engagement Inside the Classroom

Eileen M. Cashman, Elizabeth A. Eschenbach

Environmental Resources Engineering, Humboldt State University

Introduction

This paper describes the implementation of a web-based teaching tool at Humboldt State University (HSU) to teach Introduction to Environmental Resources Engineering to a large class of introductory environmental engineering and environmental science students.  The course instructors are adapting an approach called Just-in-Time Teaching (JiTT), to increase interaction between the students and the faculty in the classroom. In this adaptation, students complete on-line quizzes that are due by electronic submission before class begins.  The instructor reviews the responses “just in time” for class to assess the students’ acquired knowledge, as well as misconceptions, and so, better utilize active learning strategies in the classroom.  This work is part of a Course, Curriculum and Laboratory Improvement (CCLI) grant from NSF.

This paper provides examples that demonstrate how the instructors are adapting this web-based tool at HSU and discusses the strengths and drawbacks of the approach.  The instructors have observed that students come to class more engaged and prepared to discuss course material using JiTT.  Feedback from the students after one semester indicated that 92% thought the JiTT approach improved their learning. 

Course Description

Engineering 115:  Introduction to Environmental Resources Engineering is a required introductory course for both Environmental Resources Engineering students and Environmental Science students at HSU.  The course has 60-70 students enrolled each semester.  The course has two 50-minute recitation periods where all students meet together and a three-hour laboratory period with 24 students in each lab section.  Three lab sections are offered.

The course includes critical analyses of problems from both engineering and science perspectives through case studies in air resources, geotechnical resources, water resources and energy resources.  The course integrates lecture, discussion, student projects, computer labs, wet labs and outdoor field labs in the context of environmental engineering and science students working together on resource issues.

Students are introduced to a number of problem solving and research skills, including using the Internet, web page design, word processing, data collection using field instruments, and data analysis using spreadsheet modeling and Geographical Information Systems.  Students are expected to take this course their first or second semester on campus and the only requirement is a co-requisite of college algebra.

Just-in-Time Teaching - JiTT

JiTT blends web-based preparatory assignments with classroom learning.  The JiTT strategy has been pioneered by physics faculty and has been used by faculty at more than 50 institutions across the country (Novak et al. 1999).  There is documented increased attendance and decreased attrition with the implementation of JiTT (Novak et al. 1997, Rozycki 1999).

Immediate feedback is a critical component of JiTT.  Students complete web-based preparatory assignments before class.  Students know that their comments and questions will be incorporated into the content delivered in the classroom.  Students come to class motivated to learn what they have not been able to teach themselves.  Thus, instructors spend time on common misconceptions and stumbling blocks and do not spend time on material students have shown they have easily grasped.

Web-based Preparatory Assignment - On-line Quizzes

On-line quizzes are an important component of JiTT.  At HSU, once a week, students complete on-line quizzes that are due by electronic submission (via BlackBoard) one hour before class.  The instructor reviews the responses “just in time” for class and adjusts and organizes lessons based on those student responses.  The students come to class prepared and already engaged in the material.  The instructor has already assessed the students’ acquired knowledge, as well as misconceptions, and so can better utilize active learning strategies in the classroom.

The quiz generally consists of two to three short-answer questions and a feedback box.  There is usually a calculation question that requires that students make some assumptions, and a conceptual question that requires that students process a specific concept and synthesize it in a clear concise answer.  At HSU, the feedback box is heavily utilized and provides additional feedback on topics the instructor may not have covered in the quiz questions, but are still of concern to students.

Quiz answers are graded on the quality of the thought and the detail in the information provided.  Students can receive full-credit for a quiz answer that is incorrect, if they have articulated their assumptions and thought processes in a way that makes it clear to the instructor where the student’s error occurred.  Students receive no credit for an answer that is correct if they do not demonstrate how they arrived at their answer.  In this sense, the term “quiz” is a misnomer.  At other institutions, such as the Air Force Academy, these web-based assignments have been termed “Pre-Flights” (Novak 1999).

The instructor must download and review quiz answers during the hour prior to class.  Instructor preparation involves reading through the quiz answers and summarizing the main ideas submitted by the students.  The instructor must also be prepared to modify the planned lecture material or learning activities based on the answers (and questions) submitted, if necessary.  One hour prior to class has proven to be an adequate amount of time for an introductory course of 70 students.  Two examples provided below describe modifications made to lecture and recitation period activities based on student feedback.

On-line Quiz Examples

Two example on-line quiz questions are presented below.  Summaries of student answers are presented along with an analysis of student responses and a discussion of how the course content was adjusted based on student responses.  The first question illustrates a conceptual question and the second question a calculation question. 

The questions were posed as part of a unit on indoor and outdoor air resources.  The class discussion and course readings were materials related to indoor and outdoor air pollutants.  In the lab, students were in the middle of a ventilation rate analysis of rooms on campus using CO₂ as a tracer gas.  Students had collected CO₂ data in a classroom on campus and were just beginning to conduct a spreadsheet analysis of their data to calculate a ventilation rate.   Students were aware of the air emissions problems associated with fireplaces, wood-burning stoves and fossil fuel combustion.  Energy resources and technology had not yet been covered.

Question 1:  The authors of your text suggest electric heating as a means to reduce air emissions.  Do you think this is a good suggestion?  Explain your answer.

Summary of Student Responses

• Good Suggestion - A few students (~5% of the class) thought this solution was great and provided no further critical analysis of the question.

• No- Consider Source of Electricity - The majority of the students did not find this suggestion to be a good one.  Many of the students pointed out that one must first consider the generating source for the electric heating.  They discussed that the most common generating source is likely to be a fossil fuel combustion power plant.  While this electric heating technology would serve to centralize and very likely decrease air emissions due to centralized emissions, removal technologies and higher stack heights, they suggested that other alternatives might be better for providing heat. 

• Alternative Technology - Approximately half the students suggested passive solar technologies or district heating as a preferred solution to the emissions technology.  Nearly all the students recognized the need to look at the resource issue in the larger context of energy and air resources.

• Environmental Justice and Other Social Issues - Some students (~20%) raised issues related to social equity in terms of the siting of electric generating stations, human behavior issues related to comfort of fires and wood-burning stoves and the decreased attention to conservation of energy resources when all one has to do is “flip a switch.”

• Second Law of Thermodynamics - Although, a few students got close, only one student over the two semesters that this quiz question was used actually articulated the issue of the Second Law of Thermodynamics and the inefficiency inherent in using “high quality” energy (electricity) for a “low quality” use (heat).

Adjusting Course Content

In reviewing these answers, the instructor was able to better gauge at what point the students were ready to begin discussing the management of air resources.  From the answers it was clear that many students in the class had thought about alternative energy technologies and were able to think outside the air resource unit and make connections to other resource issues.   Some of the students were ready to discuss issues of social justice and equity in resource use and how the use of technology can exacerbate or resolve those issues.  As an instructor, it became clearer where to begin the conversation in class. 

From these answers, the instructor quickly developed a portion of the lecture addressing the issue of high quality and low quality energy and second law efficiencies.  The student feedback showed that they had a deeper understanding of the connections between air and energy resources and the associated technologies than the instructor had anticipated.  The instructor was able to skip introductory material and build on the some of the more clearly articulated student ideas.  The instructor pointed out some of the more complex and intriguing ideas that students were already thinking about, but maybe had not articulated or realized by introducing some very general concepts of thermodynamics in this portion of the class.

The connections made between air resources and energy resources also resulted in the instructor rethinking the sequencing of the course units.  In the original course syllabus, the next unit in the course is geotechnical resources (this is largely dictated by the sequencing of the chapters in the course textbook).  This information has been noted and will be used to possibly reorder the sequence of units for next semester. In this respect, the use of JiTT has provided feedback that will be useful for curriculum development at a larger scale (course planning). 

The recitation session following the quiz focused on helping students to better articulate their ideas about the connections between air resources and energy resources and begin to critically think about ways to approach and manage these issues.

Question 2:  In your reading of the Scientific American article, the authors reported on exposure to chloroform while in a steamy shower.  How much (grams) chloroform do you inhale in your average shower?  Based on OSHA’s recommendations of 50 ppm over an 8-hour day/ 40-hour work week, is this a concern to you?

The first part of this question requires students to complete several tasks.  They must read the article and extract an average exposure from the graphics presented by the authors.  The average exposure concentrations of chloroform are presented only in graphical form and the graphs are on a log scale.  The units provided in the article are in mg/m³.

Once they estimate the concentration in shower steam, they must make a series of estimates, such as: 1) how long are they are in a shower, 2) how much air do they breath while in the shower, 3) what is the volume of their lungs, and 4) how many breaths do they take per minute?

The final part of this question requires students to realize that the recommended standard is reported in ppm and they have a reported exposure in mg/m³.  The student must convert between the two units using the Ideal Gas Law (IGL).  The students have been referred to reading that summarizes the IGL and there is discussion of this conversion in the text and a table of conversion factors available from common air pollutants (chloroform is not one of them). The students had already participated in a discussion of units of air pollutants including conversions and how to interpret the concentration unit of ppm.

Summary of Student Responses

• Interesting Problem - Many commented that they really wanted to know how to solve this problem and several students wrote that they looked forward to figuring it out in class.  The answer to the problem related directly to their own lives (how much chloroform were they exposed to?) and motivated the students to want to find out the answer.

• Problems reading graphics - Nearly half the students in the class were not able to extract a reasonable concentration number from the logarithmic graphics. 

• Problems estimating lung capacity - Greater than half the students were very uncomfortable with making estimates about the volume of their lungs.  Although about a quarter of the class did come up with clever ways to estimate their lung capacity (e.g. blowing up balloons, searching on the internet and finding regression equations based on age and height, holding liter bottles next to their bodies and seeing if it would “fit”).  They also included information or corrections if they are asthmatics or smokers.  It should be noted that this question is the first time in the semester where the students are faced with making these types of estimates, and they do become much better and more confident in these types of problems as the semester progresses.

• Problems with units - Students were unclear on appropriate units.  They did not make the connection that, given concentration units in mg/m³, it might be best to estimate lung capacity in terms of m³ or liters (1 m³ = 1000 liters).  Lung capacity was reported in units of pints, gallons, in³, ft³, as well as inappropriate units, such as grams.  Many students did not remember how to convert between moles and grams because they did not remember what molecular weight is. (Most students learned this topic in high school chemistry). 

• Proficient with Algebraic Manipulation of the Ideal Gas Law - Most students were able to successfully manipulate the IGL and determine which variable to solve for.

• Volumes for chloroform and air - Many students became confused about whether they should be analyzing based on the volume of air or the volume of chloroform.  They did not set the IGL equation up properly to accommodate the appropriate volumes.

Adjusting Course Content

After reviewing this information, the instructor decided that more time should be spent in class on basic definitions for the terms used in the calculations.  Students needed to talk about the appropriate units for volume.  The class needed to be reminded of how to read a logarithmic graphic and of the definition of molecular weight.  Without this information, the instructor may have focused more on the mechanics of solving the IGL equation when in fact most of the students were not having trouble with the algebraic manipulations.  

In the recitation period immediately following this quiz, the instructor summarized the main areas of misconceptions brought out by the student answers and discussed each one.  The issues of how to read the graphics and the definition of molecular weight was addressed quickly as these were concepts about which most students just needed reminding.  The question of how to estimate lung size was addressed by presenting several of the student answers to the class.  Students were impressed (and amused) by their classmates’ creativity and by the variety of ways their colleagues estimated lung capacity.

Finally, it was clear that the entire calculation of using the IGL to convert between mg/m³ and ppm should be reviewed in class.  The students guided the instructor through the calculation and were able to ask very specific questions based on where they had hit roadblocks in completing this calculation on their own.  The students were clearly engaged in this calculation and not simply copying what the instructor wrote on the board.

This quiz question lead to further class discussion of the following questions:

• Why might ppm be a preferred unit for standards over mg/m³?
• What are different exposure pathways?  How is drinking different than inhalation?
• What is chronic versus acute exposure?
• What are regulations for indoor air pollutants?
• How can engineering and environmental science professionals contribute and work together on these types of air resource and management issues?

These questions provided material for several future recitation sessions and the students were engaged in the process. 

Benefits and Drawbacks of JiTT

The following section discusses general observations from the instructors with regard to the benefits and drawbacks of the JiTT approach.  There are also some comments on further components the instructors would like to implement in the context of this course.

Benefits

• Increased Student Confidence - A subset of the student’s quiz answers are typed into Powerpoint and projected to the class anonymously. The class is asked to discuss the quiz answers and point out the important concepts.  The discussion that ensues from these answers is essentially beginning with the students and they have more ownership over the conversation that happens in the classroom.  Presenting some of the more clearly articulated ideas verbatim from the students themselves instilled greater confidence in their knowledge and created a more open environment for discussion.  Pointing out some of the more complex and intriguing ideas that students are already thinking about, but haven’t articulated or realized, appears to give the students more confidence in their critical thinking skills and encouraged further discussion in class. Although a specific student’s name is not identified, the instructor has observed students sharing with others that their answer was chosen. 

• Increased In-Class Discussion – Increased student confidence leads to an increase in the amount of student-initiated class discussion.  There has been a significant increase in student participation in a large class with the implementation of this approach.

• Feedback Box is Useful – Providing students with a specific place to provide comments or ask questions on the material covered in class that week keeps instructors informed.  The feedback is immediate and would otherwise be forgotten at the end of the course when course evaluations are administered.  There is often feedback on that week’s lab activities, whether students had enough time to complete their work, whether they found certain activities helpful or useful learning tools and questions about external sources that are related to the course material.  The feedback box has also allowed the instructor to focus and provide additional guidance on the appropriate way to give feedback to allow students to be more successful at communicating with instructors and getting what they need from the class.  The use of this box has provided the instructor with feedback that has not been provided in any previous semester.  The instructor has also noted receiving feedback from students who do not offer comments in class.

• Increased Interaction Between Students and Instructor – Additional conversation occurs on-line between the students and the instructor and has resulted in an increased sense of interaction between the instructor and the students.  The instructors also perceive a better understanding of students’ misconceptions.  The instructors have engaged in conversations regarding learning styles and content delivery with the students creating more a sense of a partnership in the learning process.

• Multiple Attempt Quizzes – At HSU, the students are allowed to take quizzes multiple times.  Since the main goal of the on-line quizzes is to engage the students in the material prior to coming to class, allowing a student to go back to the material and review/reread a section they may have glossed over is a positive aspect. 

Drawbacks

• Reliance on Technology – The implementation of this approach requires reliable access to the Internet.  JiTT can be difficult for students without Internet access at home or easy access to a computer facility on campus.  This appears to be an issue for several students (<5%) at HSU each semester.  The instructor must also be prepared to be flexible in times when the university server goes down or the power goes out, which happens about once or twice a semester at HSU.

• Multiple Choice Quizzes - The authors of the JiTT technique suggest multiple-choice quizzes as a useful web-based preparatory tool for students.  Due to the limitations of the technology at HSU, it is easiest to allow students to take the quiz multiple times.  However, multiple-attempt quizzes rule out the use of multiple-choice quizzes.  Most students are smart enough to discover they can correctly answer a multiple-choice quiz by simply taking it enough times, thus avoiding going back and review any material!

• Preparing “Just-In-Time” - Initially, the idea of prepping for a class one hour before hand (particularly for newer faculty) is an uncomfortable idea.  The instructor must be willing and able to close the office door and dedicate a solid hour of prep time prior to class.  The instructor must also be willing to throw out material previously prepared and substitute different material.  In this specific case, the introductory nature of the material lowers the risk of not being able to prepare adequately.  (However, this approach is used in calculus based physics courses) (Novack 1997).  The risk of inadequate preparation time is also lowered as an instructor becomes more experienced in teaching the material.

• Increased Preparation Time - Just as this approach requires students to stay engaged and consistently involved in the material, this approach also requires instructors to do so.  The on-line quizzes must be reviewed and graded every week (or every class depending on how often they are implemented).  For this reason, the on-line quiz is only given once per week, rather than once per class period as suggested by the JiTT authors.  Additionally, the instructors do not attempt to grade the quizzes in the one-hour period before class, thus the instructor must return to the quizzes later in the week to grade them.

• Appropriate Quiz Questions - The most difficult task is writing quiz questions that are complex enough to generate significant thought about certain concepts without discouraging students from even addressing the questions.  A quiz question provides little information if it is too easy (everyone answers the entire question correctly) or too difficult (no one attempts the question).

Conclusion

The implementation of JiTT quizzes in the introductory survey course in Environmental Resources Engineering has been successful in terms of engaging the students via the Blackboard delivered online quizzes. The instructors perceive a better understanding of the students’ misconceptions and have an increased sense of interaction in a class of 70 students.

This technique continues to be implemented at HSU.  Improvements will come about as more detailed and quantitative assessment results become available and other faculty members implement the course.

References:

1. Novak, Gregor, Evelyn Patterson  (1997).  World Wide Web Technology as a New Teaching and Learning Environment. International Journal of Modern Physics, Vol 8, No. 1: 19-39.

2.  Novak, Gregor, Andrew Gavrin, Wolfgang Christian, Evelyn Patterson  (1999).  Just-In-Time-Teaching: Blending Active Learning with Web Technology. ISBN 0-13-085034-9, 1/e. Prentice Hall.

3.  Rozycki, William  (1999).  “Just In Time Teaching.”  Research and Creative Activity.  Office of the University Graduate School at Indiana University. April.

Biographical Information:

Eileen M. Cashman is an assistant professor in the Environmental Resources Engineering Department at Humboldt State University.  She teaches introductory engineering courses and upper division courses in fluid mechanics, water quality and river hydraulics. 

Elizabeth A. Eschenbach is an associate professor in the Environmental Resources Engineering Department at Humboldt State University.  She teaches introductory engineering courses and upper division courses in probability, environmental impact assessment and water resources. 

Dr. Cashman and Dr. Eschenbach are Co-PIs on an NSF funded grant for Course Curriculum and Laboratory Improvement grant (CCLI) award 0127139 and the work described above has been supported by this grant.  Both authors are members of ASEE and Faculty-21 in Project Kaleidoscope.