Provide evidence of the ability to construct and implement reliable and valid assessments of student performance over a range of methods (e.g. written examinations, Objective Structured Clinical Examinations, oral examinations, chart reviews). Two essential qualities are:

1. Reliability: consistency, precision and dependability of the measuremement
   • Test-retest reliability
   • Inter-rater reliability
2. Validity:
   • Face validity: student ratings of the "fairness of the evaluation"
   • Content validity: appropriately samples the instructional experience
   • Criterion validity: How do your evaluations compare with other faculty's?
   • Authorship of administrative reports

Assessment of Learner Performance Example:

Associate Professor (Clinical Track)

My most valuable experience in developing valid tools for the assessment of learner performance was in the development of the program to teach arthrocentesis. I initially designed a set of educational objectives, a written test and a practical test to measure the knowledge and skill level of the residents and students. A self-assessment (by the students and residents) of confidence in performing arthrocentesis was also included. A panel of experts reviewed the objectives, written test and practical test for applicability to medical practice and feasibility of teaching. These experts included specialists in Rheumatology, Orthopedics, Internal Medicine and Family Medicine as well as a Biostatistician. Modifications were made from the suggestions of the expert panel, a brief pilot study was performed, and the testing period was initiated. Residents and students were assigned to different groups: one group was tested at the beginning of their month rotation to give us an idea where residents knowledge begins; another group rotated through rheumatology without specific instruction in arthrocentesis; another group was given only the didactic lecture about arthrocentesis; another group was given the didactic lecture and the workshop; and finally some of the last group were retested 6 months later to evaluate recall. The test period is complete and the results are very encouraging. As expected, the residents and students who participated in the lecture and the workshop performed significantly better in both the written and practical tests. In addition, they had significantly more confidence in their ability to perform arthrocentesis.

I have participated in a workshop on test question writing focusing on extended matching questions. As we reviewed and integrated the curriculum for the rotating residents and students, we incorporated extended matching questions into the self-assessment tests.

The Educator's Portfolio (1) from the Medical College of Wisconsin will be used as a model.

References

1. Simpson DE, Beecher AC, Lindemann JC, Morzinski JA. The Educator's Portfolio. 4th Edition. Medical College of Wisconsin. 1998.