VIBE Education Edition

Project Summary

1. Project Design

Throughout the project, INCOGEN collaborated closely with a team of professionals from the College of William & Mary (W&M) and other institutions to develop an effective plan for assessment and design of the educational software and its introduction into the classroom. This team was comprised of bioinformatics educators from the Departments of Biology, Physics, and Applied Science; the Dean of and a professor in the Department of Education; the Director of Assessment, the Director of Academic Information Services, and an independent educational consultant. 

The contributions of the team members were gathered through large meetings involving the entire team and more frequent and focused meetings involving specific subsets of the team. First, based on feedback from bioinformatics educators, we defined the role that an educational software product would play in the bioinformatics classroom setting. Second, the scope of the software was discussed to ensure a pedagogically sound environment that encourages deep learning for the students. Education professionals and faculty contributed their expertise to determine the level of instruction needed in a teaching tool, especially in the specialized context of bioinformatics classes at universities and at community colleges and high schools. Third, we examined the importance of assessment in developing tools for learning, and developed methods for assessing the software and evaluating student learning.

2. Preparation for Classroom Use

In order to design and develop an application that performs its intended function well, it is critical to understand the mindset of the target audience as they use the product. Software users, however, are often inaccurate and/or incomplete in describing their current use and their desired use of the application. This is particularly true for non-expert computer users. To be able to efficiently gather useful and reliable feedback from students, we designed additional tracking and logging features that were incorporated into VIBE. The logs tracked key information about each student's use of the software. They allowed the project management team and the developers to more closely identify with the mindset of the students and to recognize certain actions and patterns of actions that may have seemed counterintuitive to developers, but were reasonable to the students. This information was also helpful in pinpointing specific errors that were common among students, which could highlight a particularly difficult concept in the use of the software or bioinformatics analysis.

The tracking system was designed with the expert input of educational professionals and software developers. Dr. George Bass from the Department of Education at W&M provided the development team with key behaviors or actions that should be observed in order to understand and assess the students' learning process. Armed with this information, INCOGEN's software developers created a technical plan for intelligently extracting relevant information from VIBE's existing logging system. The design of this tracking system was iteratively refined throughout the project, as we determined the most relevant information to be captured.

3. Collecting Feedback and Gathering Requirements

One of the most important aspects of our work was to gather information about critical features necessary for an educational tool for bioinformatics. In addition to the valuable feedback from educators and education professionals, extensive feedback was gathered from the students. This allowed us to develop a design plan for a software product that was based on the needs of its intended users, as determined by direct observation and analysis. We introduced the existing VIBE research tool to three groups of students at W&M, who used the software to learn bioinformatics in various settings: 

Group 1:
VIBE was used as part of a bioinformatics class taught at W&M by the INCOGEN PI, Dr. Maciek Sasinowski, who is an adjunct professor in the Department of Applied Science. The class included a mix of undergraduate and graduate students from a variety of backgrounds, including biology, computer science, and chemistry, which is representative of a "typical" bioinformatics class. At the beginning of the semester, students completed a diagnostic questionnaire about their experience and knowledge of bioinformatics, computational tools, and software use for education. This survey, designed with the help of Dr. Bass, served as a baseline for assessing the students' performance and learning throughout the course.

Upon their first exposure to VIBE, the students were asked to complete an assignment that guided them through an exploration of the software. In this assignment, students were introduced to bioinformatics analysis in the VIBE environment and were asked to answer questions about these concepts. This was designed to reveal the first impression and instructional quality of VIBE to a new student user. In this assignment as well as throughout the course, students were strongly encouraged to provide feedback and pose questions concerning VIBE's current and potential educational value, especially as they became familiar with the tools available in VIBE through class lectures. 

Dr. Sasinowski's lectures covered a wide range of bioinformatics topics including pair-wise sequence comparison, scoring matrices, models and profiles for sequence comparison, multiple sequence alignment, phylogenetics, proteomics, and mass spectrometry in cancer diagnosis. In addition to lectures covering these topics, each student in the class gave a presentation on a selected topic in bioinformatics. In conjunction with the class lectures, students explored the topics further through homework assignments using VIBE. VIBE-Ed project lead Kristina Gleason served as the teaching assistant for the bioinformatics class and interacted one-on-one with many of the students throughout the entire semester. This allowed her and the project team to gather detailed first-hand information about experiences the students had with the concepts in bioinformatics as well as their use of the software.

Group 2:
The second group of students was involved in independent research in molecular biology at W&M. Dr. Margaret Saha, associate professor of biology, collaborated with INCOGEN to incorporate VIBE into each student's research project. Each student in this group used VIBE to study the gene that was the focus of his or her research project. As the students used VIBE for the analysis, they kept personal logs of their experience. Students were asked to record their use of VIBE, including questions or problems, analyses performed, and feedback about their experience.

Group 3:
The third group allowed us to investigate tools that are currently being used for bioinformatics education and directly compare those tools with VIBE, the existing foundation for VIBE-Ed. To this end, we designed a controlled study in which we observed students using two distinct methods to learn about bioinformatics; one method involved VIBE, the other did not. This study gave us directly comparable and quantifiable data that we could use to 1) demonstrate the potential of VIBE as a teaching tool, 2) compare VIBE's current value in learning bioinformatics compared with existing methods, and 3) directly extrapolate important components of existing methods that need to be included in or adapted to VIBE-Ed to make it an effective teaching tool. Several members of the team, led by Dr. Bass, designed a crossover study with these objectives in mind. Dr. Bass has extensive experience in studying learning, including some involving computer software. His input into the design ensured that several key components were included to make the study as valuable as possible to both the VIBE-Ed project team and the students involved. 

Twenty-five students were recruited from W&M to participate in the study. The recruitment was aimed at biology majors in order to target students who met our criterion of having basic knowledge of molecular biology. Eligibility requirements included successful completion of the introductory cell and molecular biology class taken by all biology majors at W&M (BIO 203). In addition, students from groups 1 or 2 were ineligible to participate in group 3. All students were required to attend a general session in which they 1) received information about the purpose of the study; 2) completed an entry survey to assess their educational background and level, experience with software use in education, and baseline knowledge of bioinformatics; and 3) received a lecture given by Dr. Sasinowski. The lecture delivered an overview of bioinformatics and a general introduction to key concepts in sequence similarity searching and comparison, the particular subject in bioinformatics on which the students in the study were to concentrate. No specific tools were discussed in this lecture. Following the general introductory session, the students were rated on a scale from 1 to 5. The rating was based on their responses to the entry survey, their GPA, the biology and computer science classes completed, their prior computer usage, their experience with software for learning, and their correct responses to questions about bioinformatics. Based on these ratings the students were divided into two clusters containing a balance of ability and experience. Students were given a set of problems about sequence similarity searching and comparison that they completed using two separate methods. Cluster 1 used VIBE to answer the questions and cluster 2 used NCBI and other web tools. After each student completed the questions using one method, the clusters switched (crossed-over) and answered the questions using the other method. The crossover component of the study design ensured that all students were given an equal educational experience and had the opportunity to learn the same amounts of information. It also gave us information of how the students' experience with one tool enhanced their use of the subsequent tool. 

Students using web tools to complete the questions were directed first to tutorials at NCBI that explained sequence similarity searching, including the use of the BLAST tools. They were allowed to use any available web resources that they could find as long as they reported those resources and how they found them. A two-hour time limit was set for those students using the web to answer the questions. 

The VIBE session began with an introductory tutorial video that covered basic use of VIBE. The VIBE session was limited to ninety minutes to make the VIBE experience comparable to the web-based portion of the study. Students using VIBE were not permitted to use other resources on the web. They were encouraged to speak when they were confused, frustrated, or pleased. The data we collected from the students' use of VIBE allowed us to observe and determine exactly which parts of the software or which concepts in bioinformatics were most challenging to the students and, often, what specifically made them challenging.

During the VIBE segments, we wanted to learn as much as possible about the students' use of VIBE to explore problems in bioinformatics. To accomplish this, Dr. Bass suggested a practice commonly used in the education field in which the students' actions are recorded as they complete a task. As students used VIBE, their on-screen actions were captured using a software program to record the activity of the mouse and the computer screen. In addition, a video camera captured the facial and verbal expressions that corresponded to the on-screen activity. By carefully synchronizing these two recordings, we could observe each move a student made in VIBE and their corresponding body language, facial expressions, and, in some cases, verbal comments.