Introduction
Basic science discoveries, clinical knowledge and information technology are expanding rapidly. The current generation has adapted to this by connecting to information resources using technologies from desktop computers to tablets and smartphones. Despite these changes, there has not been an accompanied shift in how physicians are educated. Most medical schools utilize the traditional lecture-centric teaching model, which emphasizes attendance and committing to memory basic science and clinical concepts. They primarily deliver these lectures in large auditoriums, assuming that students attend, listen and remember what is presented. Many of today's students prefer to view lectures on their computers or other devices at a place and time of their own choosing. They access learning resources according to their individual learning preferences. Students continue to be tested on their memorization ability rather than on their development of thinking and search skills, problem-solving skills and teamwork. Memory-driven learning is problematic as Ebbinghaus demonstrated that infrequently used memorized material can be rapidly forgotten within hours.¹

The past several decades have brought substantial changes in medical practice. The long-dominant model of patient care by solo practitioners is increasingly replaced by care through collaborative teams of healthcare providers that include doctors, nurses, social workers, pharmacists and others. Thus an important component of modern medical practice is the ability to work both within as well as lead teams of health care providers in partnership with patients and their families. Unfortunately our student’s educational success is typically achieved through individual efforts, in competition rather than in collaboration with other students. Given all of the above information, it is clear that the current learning model is not well aligned with the way people access and update themselves, promote retention of concepts, or prepare for the current model of real-world medical practice.

In planning for our new medical school (Duke-NUS), we wanted to align our learning strategy to these changes. We wanted to emphasize critical and creative-thinking skills, self-directed learning, and teamwork. We sought to create in-class sessions where students solve problems that reinforce the concepts that are first delivered by electronic means outside of the classroom. We built an entire approach, called TeamLEAD, based on these principles. This paper presents TeamLEAD and compares test scores from our first three classes of students with the National Board of Medical Examiners [NBME] Comprehensive Basic Science Examination [CBSE], and the first two classes with United States Medical Licensing Examination [USMLE] Step 1 to evaluate whether this novel learning strategy successfully conveyed basic science core concepts to students.

Methods
The Duke-NUS Program
Duke-NUS was established in 2005 as a collaboration between the Duke University School of Medicine (DSOM) and the National University of Singapore (NUS) to train and cultivate a cadre of physician leaders and scientists to support and further develop Singapore’s biomedical science initiative.² Creating this new school provided a rare opportunity to review, modernize and align the educational process with the prevalent working and learning styles of today’s students and physicians.

The Duke-NUS and DSOM programs cover the core basic science concepts over the first year (Figure 1), rather than providing the more detailed coverage of basic science content that is typically taught over the first 2 years. Duke-NUS and DSOM second-year students instead rotate through clinical clerkships. During the third year, students engage in independent research, under the mentorship of experienced researchers. The fourth year builds on the first three years with clinical experiences that round out each student’s educational program in preparation for residency training.

In designing our curriculum, we had to ensure that it covered the same content as the DSOM curriculum.³ We also had to ensure that the process of trans-locating a new educational philosophy into a different cultural and physical environment did not adversely affect student learning outcomes. In addition, we sought to provide our students with skills, such as collaborative practice, self-directed learning and problem solving. We hoped that the approach we chose would also inspire their curiosity.

Building upon the existing DSOM curriculum, we chose cooperative learning as our primary pedagogic delivery method for basic science education. Cooperative learning refers to any educational environment in which groups of students work together to achieve a common learning objective. It has resulted in higher academic achievement, increased student self-esteem, and greater levels of student mutual support across a variety of educational settings.^4-9

Team-based learning (TBL) is a form of cooperative learning developed by Dr. Larry Michaelsen.^10,11 When we began our school, TBL had been rarely implemented throughout an entire medical basic science curriculum.^12-20 TBL is a highly structured format that starts with students coming to class prepared, incorporates individual and team assessment of that preparation (Individual and Group Readiness Assessment or IRA/GRA), a team-based practical application (Application) of the content to solve a clinical problem and peer assessment.


Figure 1: The first year curriculum at Duke NUS: 4 integrated basic science courses. Molecules and Cells, (Cell Biology, Biochemistry, Genetics) 6 weeks; Normal Body, (Histology, Anatomy, Physiology) 12 weeks; Brain and Behavior, (Neurobiology, Neuroanatomy, Behavioral Medicine) 5 weeks; Body and Disease, (Pathology, Pharmacology, Immunology) 20 week; and two 1 year long longitudinal courses: Practice course and Investigative Methods and Tools.


Figure 2. Components of TeamLEAD (Adapted from Michaelsen10)
-SDL Self-Directed Learning: Duke lectures and faculty identified objectives and supplemental material for pre class preparation
-RAP Readiness Assessment Phase consists of three major elements: IRA - Individual Readiness Assessment; GRA - Group Readiness Assessment; Faculty led discussion
-AP Application Phase consists of two elements: Application Exercise; Faculty led discussion
-CEL Content Expert Lecture

We chose TBL for a variety of reasons. It requires, drives, and rewards self-directed learning. It encourages and rewards student peer teaching and collaborative behavior. Inherent in its structure is a learning efficiency that leverages on several hours of student study time done prior to faculty contact time. It does not require a large number of faculty members to teach any given session for the whole class. The critical analysis of information and problem solving are a vital part of the “open book/open Internet” Application portion. Finally, we hoped that as an active learning strategy with immediate feedback, it would be engaging and enjoyable for the students and faculty. We adapted the TBL format to meet our challenges and objectives. We call our adaptation “TeamLEAD” (Learn, Engage, Apply, and Develop).

TeamLEAD Development
The first challenge for the faculty was the development of the TeamLEAD materials. The core concepts for each course were reflected in the recorded lectures and exam materials developed by the DSOM faculty and made available to the Duke-NUS faculty. To develop the TeamLEAD format, the Duke-NUS faculty articulated learning objectives, identified learning resources (Web sites, journal articles, books, and DSOM video-taped lectures) for the students. Duke-NUS faculty wrote the multiple-choice questions (MCQs) used to assess core learning material as well as application questions that were designed to enable students to apply important concepts in the assigned learning materials.

Description of TeamLEAD
The components of TeamLEAD are summarized in Figure 2. Each student is individually responsible for learning the core concepts and principles prior to coming to class, using learning materials made available to them by the faculty. This learning is reinforced by the Readiness Assurance Process (RAP) which includes individual assessments (IRA), usually given in the form of MCQ, and followed by team assessments (GRA) in which students repeat the same MCQs but answer as a team. The IRA/GRA MCQs were written to the level of difficulty that our students as individuals obtained approximately 65-75% items correct. When these same problem sets were re-addressed as a team, they typically scored at 90-95% correct. The IRA process is designed to evaluate the student’s understanding and retention of the core concepts and principles, but is not the end of the learning process. The GRA permits the students to learn from each other and as a team identify any gaps or uncertainties – which opens the student’s minds for further learning. Both IRA and GRA assessments contribute to each student’s individual final grade. After the RAP is completed, the students proceed through the open-book/open Internet Application exercises which require critical analysis, problem solving and creativity and are all a part of their grade. The problem sets in this portion are addressed as a team and require the use of core content covered earlier (often directly linked to immediate RAP, but can be from earlier sessions as well). The team score for Applications generally runs between 75-85% correct.

All student teams meet in the same room and the entire class participates in discussions facilitated by a faculty member. Therefore, the learning goes from the individual student, to their team of seven, to the entire classroom. This strategy does not require an individual faculty preceptor for each team; instead TeamLEAD has a faculty facilitator who guides the learning for the entire class and is usually assisted by several other faculty members, each with different content/subject expertise. They all work together to determine the key learning points and related preparatory content. Then, they co-develop MCQs, applications. Finally, the content experts provide clarity to core principles during class and final summary of key points.

TeamLEAD sessions are generally delivered on average twice a week. Two hours are devoted to the readiness assurance process, consisting of an average of 25 MCQs based on the prior preparation (consisting of 5-7 Duke pre-recorded lectures and some supplemental readings). After a break, two-three hours are devoted to an application exercise which consists of a series of problem statements accompanied by several challenging evaluative questions.

The Current Study
We were concerned about adversely affecting the learning and understanding of the material through our use of TeamLEAD. We assessed the impact of our strategy by requiring our students to sit for the National Board of Medical Examiner’s (NBME) Comprehensive Basic Science Exam (CBSE), as an independent assessment of the student’s understanding of the basic science content needed to pass the required, high stakes USMLE Step 1 exam. We administered the CBSE as a formative exam at the end of their first and second years and students take USMLE Step 1 in their third year. These test results served as a form of comparison to national results.

The objective of the current study was to use these test scores to conduct a comparison of all Duke-NUS Graduate Medical School students who took CBSE during years 2008-2010, with national data obtained from NBME on US medical school students who took same exam during years 2006-2009 to determine whether the TeamLEAD strategy successfully conveyed basic science core concepts to students. The study protocol was approved by the National University of Singapore Institutional Review Board.

Study Participants
The study’s participants were the medical students who matriculated at Duke-NUS during the academic years 2007-2009 (n=130). The data represents the year in which they took the exams (i.e, First year exams administered in 2008-10; second year exams administered in 2009-11). Data for USMLE Step 1 exam came from the first two matriculating classes and was obtained during 2010 and 2011. Demographic data for the Duke-NUS students were extracted from de-identified admissions records. Informed consent was not required and Duke-NUS students received no compensation from Duke-NUS for the purposes of this study.

The comparison group provided by the NBME consisted of 6,421 medical students from LCME-accredited medical schools in the US who took the CBSE from 2006-2009. Demographic, Medical College Admission Test (MCAT) and undergraduate Grade Point Average (GPA) data for these test takers were not available from the NBME, so we used data from all students entering US medical schools in 2008 (n=18,390).

Measures
Comprehensive Basic Science Exam
Duke-NUS students take the CBSE at the end of their first and second years. The exam, written by the NBME, closely reflects curricular content covered in the USMLE Step 1. Students had no more than 3 days of free time to prepare specifically for this assessment. Because most other participating medical schools have a two year basic science curriculum, it was assumed that the CBSE would be most commonly administered after the first 2 years of basic science courses. These schools use a variety of educational strategies to teach this material, but we are unable to identify any particular strategy – so we used the entire population for comparison.

Since demographic information is not available for the national comparison group from the NBME, we also made the assumption that the demographic, MCAT and undergraduate GPA data of students taking the CBSE would approximate that of students who were entering all LCME-accredited schools in the US.^21-23

USMLE Step 1
The USMLE Step 1 examination is sponsored by the NBME and the Federation of State Medical Boards. All Duke-NUS students must sit for this examination and pass prior to graduation. Two years of Duke-NUS student data was available at the time of manuscript for analysis (n=68). US medical students are also required to sit for this examination prior to graduation and must pass it for licensure.

Data Analysis
Demographic data (age, gender, MCAT, undergraduate GPA) for the Duke-NUS students (matriculating classes 2007-9) were compared with the US national student demographic data (2008) using a t-test (corrected for multiple comparisons) and their 95% confidence intervals determined with the US national means as reference.

The mean CBSE scores from the Duke-NUS students at the end of their first year (2008-10), and the mean CBSE scores from the same students at the end of their second year (2009-11), were compared with the mean score of CBSE US national administrations provided by the NBME (2006-09), using a t-test. The mean USMLE Step 1 scores from Duke-NUS (2010-11) were compared to the mean score of USMLE Step 1 US national administrations (2010), provided by the NBME, using a t-test. For each of these comparisons, the means and 95% confidence intervals for the Duke-NUS students were determined and plotted with the respective US national mean score as a reference to provide evidence that TeamLEAD did not cause harm. In addition, we computed effect size of these differences (Cohen’s d) to determine the magnitude of mean differences.

Results
Demographic, MCAT, and undergraduate GPA data for the Duke-NUS student and for students entering US medical schools, are shown in Table 1. Duke-NUS undergraduate GPA and MCAT Verbal Reasoning subscores were significantly lower than US medical students and significantly higher on MCAT Physical Science (PS) subscores. There was no significant difference in the MCAT total score and Gender mix. We do not have the standard deviations in the US data to be able to statistically compare age differences between the two groups. Given the significantly higher PS subscore, we computed Cohen’s d to determine effect size of the difference and examined the correlation of the PS with CBSE and USMLE Step1 to see if how strong of a relationship it had to student performance on our outcome measures. The effect size was 0.4. The correlations of PS subscale were 0.39 with CBSE Y1; 0.34 with CBSE Y2, and 0.42 with USMLE Step 1.

The mean score result reported by the NBME for 2006-2009 US CBSE examinees was 61.0 (±11.0). At the end of the first year of basic science courses, the Duke-NUS students combined achieved a mean CBSE score of 61.4 (±8.7), which was not significantly different from the NBME comparison group. At the end of their second year, the Duke-NUS achieved a mean CBSE score of 66.5 (±7.8), which was significantly higher than the NBME comparison group (pTable 1: Comparisons between entry demographics and criteria of medical students matriculating Duke-NUS compared with US medical schools.^21-23


Figure 3. NBME Comprehensive Basic Science Exam (CBSE), normed data from US medical students from 2006-2009 compared with Duke-NUS students in years 2008-2010 at the end of their first and second year with 95% Confidence Interval.


Figure 4. USMLE Step 1 Duke-NUS 2007 & 2008 at the end of their third year with 95% Confidence Interval.

Study Limitations
We recognized that our class size is smaller than most US medical schools, which may have contributed to our results. At this point in time we cannot directly compare our results with specific learning strategies used to convey basic science courses in the US, such as PBL, lectures, or small group discussions. In fact, few medical educational strategy has yet been clearly shown to be superior for preparing students for knowledge-based examinations.^24-27 However, that was not the purpose of this study – we wanted to make sure we did no harm in their learning.

Discussion
The CBSE and USMLE Step 1 were chosen as objective measures of performance rather than faculty developed questions. We feel that the CBSE administered at the end of the first year of basic sciences more closely measures the academic outcome of our new learning strategy than does the USMLE Step 1. The CBSE was given as formative assessments and not impacted by individual student efforts at studying.

However, one could argue that our results were because our students came into medical school better prepared in the basic sciences. We examined the matriculating data to determine if that assumption was true. Due to the absence of pre-admission information for US examinees taking the CBSE and USMLE Step 1, we chose to use US matriculating student data as a best estimate for the demographics of the US examinees taking these exams.

Compared to the published demographics, we found our student’s GPA and MCAT VR subscores were lower; the Physical Science MCAT subscores was higher; and the total MCAT was not significantly different. The literature reports undergraduate GPA, MCAT Total and BS subscore to be predictive of USMLE step 1 performance, however, it is less clear on the predictive value of VR and PS subscores.^24,28,29 As our PS subscore was significantly higher we correlated the PS subscore with the CBSE and Step 1 scores and found the correlation modest (r=.34-.42). This is not surprising, given the limited relationship between PS subscore and achievement reported in the literature. Since the demographic items were both higher and lower in some areas and the modest correlation of the PS subscale, we are unable to clearly state whether or not our students are better or worse prepared for our first year basic science curriculum, and therefore we made the assumption that at entry our students are not different from typical US medical students.

An early concern for the newly-established Duke-NUS Graduate Medical School was whether we could successfully implement a major shift in learning strategy without doing “harm” to our students. Our students performed comparably to the US national average on the CBSE at the end of their first and only year of basic science instruction. This was an encouraging result, given that our curriculum focuses on learning major concepts over a single year instead of covering more detailed content over 2 years. We were further encouraged by our students’ performance on the CBSE at the end of their second year that focused on clinical instruction and on the USMLE Step 1; they scored significantly better than the US national average at comparable points in time. In short, we did not find any negative impact of delivering the DSOM curriculum using TeamLEAD as an alternative educational strategy. Although we cannot directly relate their USMLE Step 1 performance to the first year learning environment, we believe that the TeamLEAD process gave them a solid foundation of knowledge that permitted them to do well on Step 1.

Our experience and data lead us to conclude that TeamLEAD is an effective strategy for instructing medical students in the basic science core concepts. This may be due, in part, because it incorporates elements of active learning and test enhanced learning, which has been shown to improve student performance.30-33 We also believe the TeamLEAD strategy provides our students with teamwork skills, student mutual support, the ability to engage one’s curiosity, and manage their own self-directed learning to master the core concepts.

The advantages for our faculty include the ability to continuously assess student progress throughout the courses, rather than just after midterm and final examinations. This also permits faculty to have time to identify students’ gaps, correct misunderstandings, and foster critical thinking. While TeamLEAD provides students with a small-group learning experience, we suspect that at the same time it provides considerable cost savings over other educational strategies that require a faculty member to be assigned to each small group.

Over the next few years, we plan to continue to evaluate how well our students retain and utilize the learned information through their performance on standardized examinations of knowledge, as well as exploring the other elements of TeamLEAD such as problem solving, critical and creative thinking skills and how it relates to their development of clinical reasoning skills.

In summary, the TeamLEAD strategy appears to have successfully exported the Duke curriculum to Singapore with no diminution of knowledge acquisition compared to the general US population, and fits well with our understanding of how students learn and the models of how physicians work. Our positive outcomes have convinced the educational leadership at DSOM to begin to implement a learning strategy similar to TeamLEAD for their students in Durham, North Carolina.


Notes on the Contributors
ROBERT K. KAMEI, MD, is Vice Dean of Education, Duke-National University of Singapore Graduate Medical School, Singapore, and Professor of Pediatrics, Duke University School of Medicine, Durham, North Carolina, USA.
SANDY COOK, PhD, is Senior Associate Dean for Curriculum Development, Duke-National University of Singapore Graduate Medical School, Singapore.
JANIL PUTHUCHEARY, MB BCh, is Director of Faculty Development, Duke-National University of Singapore Graduate Medical School, Singapore.
C. FRANK STARMER, PhD, is Associate Dean for Learning and Information Technology, Duke-National University of Singapore Graduate Medical School, Singapore, and Professor of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, North Carolina, USA.




Keywords
Basic Science, United States Medical Licensing Examinations, Learning Strategies, Pre-Clinical Curriculum, Team Based Learning


References

1. Ebbinghaus H, Ruger HA, Bussenius CE. Memory: A contribution to experimental psychology: Teachers College, Columbia University; 1913.
   
2. Williams RS, Casey PJ, Kamei RK, et al. A global partnership in medical education between Duke University and the National University of Singapore. Academic Medicine. Feb 2008;83(2):122-127.
   
3. Grochowski CO, Halperin EC, Buckley EG. A curricula rmodel for the training of physician scientists: the evolution of the Duke University School of Medicine curriculum. Academic Medicine. 2007;82(4):375.
   
4. Johnson DW, Johnson RT, Stanne MB. Cooperative learning methods: A meta-analysis. Minneapolis, MN: University of Minnesota. May. 2000; http://www.tablelearning.com/uploads/File/EXHIBIT-B.pdf (Retrieved 27 March 2012).
   
5. Johnson DW, Johnson RT, Smith KA. Cooperative learning returns to college: What evidence is there that it works? Change. 1998:26-35.
   
6. Smith KA, Sheppard SD, Johnson DW, Johnson RT. Pedagogies of engagement: classroom-based practices. Journal of Engineering Education. 2005;94(1):87-101.
   
7. Slavin RE. Cooperative learning. Review of educational research. 1980;50(2):315.
   
8. Boling NC, Robinson DH. Individual Study, Interactive Multimedia, or Cooperative Learning: Which Activity Best Supplements Lecture-Based Distance Education? Journal of Educational Psychology. 1999;91(1):169-174.
   
9. Lord TR. 101 reasons for using cooperative learning in biology teaching. The American Biology Teacher. 2001;63(1):30-38.
   
10. Michaelsen LK, Parmelee DX, McMahon KK. Team-Based Learning for Health Professions Education, A Guide to Using Small Groups for Improving Learning. 1st edition ed. Sterling, Virginia: Stylus Publishing, LLC; 2008.
    
11. Michaelsen LK, Knight AB, Fink LD. Team-Based Learning, A Transformative Use of Small Groups in College Teaching. 1st edition. Sterling, Virginia: Stylus Publishing, LLC; 2002.
    
12. Vasan NS, Defouw D. Team learning in a medical gross anatomy course. Medical Education. 2005;39(5):524-524.
    
13. Thompson BM, Schneider VF, Haidet P, Perkowski LC, Richards BF. Factors Influencing Implementation of Team-Based Learning in Health Sciences Education. Academic Medicine RIME: Proceedings of the Forty-Sixth Annual Conference November 4-November 7, 2007. October 2007;82(10):S53-S56.
    
14. Searle NS, Haidet P, Kelly PA, Schneider VF, Seidel CL, Richards BF. Team learning in medical education: initial experiences at ten institutions. Academic Medicine. 2003;78(10):S55.
    
15. Nieder GL, Parmelee DX, Stolfi A, Hudes PD. Team-based learning in a medical gross anatomy and embryology course. Clin Anat. Jan 2005;18(1):56-63.
    
16. Levine RE, O'Boyle M, Haidet P, et al. Transforming a clinical clerkship with team learning. Teach Learn Med. Summer 2004;16(3):270-275.
    <li
17. Haidet P, O'Malley K, Richards B. An initial experience with "team learning" in medical education. Acad Med. Jan 2002;77(1):40-44.
    
18. Goldberg HR, Dintzis R. The positive impact of team-based virtual microscopy on student learning in physiology and histology. Advances in Physiology Education. 2007;31(3):261.
    
19. Michaelsen L, Richards B. COMMENTARY: Drawing Conclusions from the Team-Learning Literature in Health-Sciences Education: A Commentary. Teaching and Learning in Medicine. 2005;17(1):85 - 88.
    
20. MCAT Scores and GPAs for Applicant and Matriculants to US Medical Schools by Sex, 2002-2009. 2009. http://www.aamc.org/ data/facts/applicantmatriculant/table23-mcatgpabysex09mat.pdf. Accessed December 22, 2009.
    
21. Age of Applicants to U.S. Medical Schools at Anticipated Matriculation by Sex and Race and Ethnicity. 2009. http://www.aamc.org/ data/facts/applicantmatriculant/table6-facts2009age-web.pdf. Accessed December 22, 2009.
    
22. MCAT Scores and GPAs for Applicant and Matriculants to US Medical Schools, 1998-2009. 2009. http://www.aamc.org/data/ facts/applicantmatriculant/table17-fact2009mcatgpa98-09-web.pdf. Accessed December 22, 2009.
    
23. Callahan CA, Hojat M, Veloski J, Erdmann JB, Gonnella JS. The Predictive Validity of Three Versions of the MCAT in Relations to Performance in Medical School, Residency, and Licensing Examinations: A Longitudinal Study of 36 classes of Jefferson Medical College. Academic Medicine. 2010;85(6):980-987.
    
24. Colliver JA. Effectiveness of problem-based learning curricula: research and theory. Academic Medicine. 2000;75(3):259-266.
    
25. Vernon DT, Blake RL. Does problem-based learning work? A meta-analysis of evaluative research. Academic Medicine. 1993;68(7):550.
    
26. Kirschner PA, Sweller J, Clark RE. Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching. Educational Psychologist. 2006;41(2):75-86.
    
27. Donnon T, Paolucci EO, Violato C. The predictive validity of the MCAT for medical school performance and medical board licensing examinations: a meta-analysis of the published research. Academic Medicine. 2007;82(1):100.
    
28. Julian ER. Validity of the Medical College Admission Test for predicting medical school performance. Academic Medicine. 2005;80(10):910.
    
29. Prince M. Does active learning work? A review of the research. Journal of Engineering Education. 2004;93:223-232.
    
30. Donovan JJ, Radosevich DJ. A meta-analytic review of the distribution of practice effect: Now you see it, now you don't. Journal of Applied Psychology. 1999;84(5):795.
    
31. Larsen DP, Butler AC, Roediger III HL. Test enhanced learning in medical education. Medical Education. 2008;42(10):959-966.
    
32. Larsen DP, Butler AC, Roediger III HL. Repeated testing improves long term retention relative to repeated study: a randomised controlled trial. Medical Education. 2009;43(12):1174-1181.