Skip to main content
Sign In
 

Teaching Tips

Tips on Evaluating Student Competencies


Joel Yager, M.D.​

​​There are positive and negative aspects associated with the lists of general competencies and specialty-specific milestones set forth by the Accreditation Council for Graduate Medical Education (ACGME).

On the plus side, these well intended, high-minded and honorable lists of competencies ratify and draw attention to important areas of professional performance. As educators, we should be accountable to the public for assuring that trainees bearing our imprimaturs have achieved necessary levels of competence prior to independent practice.

On the minus side, a “competency-industrial complex” requiring formal assessment has emerged, generating large numbers of bureaucratic documentation requirements, chock-full of unintended consequences and unfunded mandates with regard to effort, time and staff resources.

The trick for educators and program staff is to (competently) conduct the necessary appraisals and fulfill documentation requirements without killing the spirit of educating trainees in the process. Competency assessment should not turn into mind-numbing, cynical exercises devoted to filling out valueless forms. Nor should formal evaluation stand in the way of (or stand in for) actually guiding progressive competence-building in trainees.

The following tips are provided with two goals in mind: 1) Helping trainees grow their competence AND 2) Satisfactorily documenting their progress.

5 Tips for Helping Trainees Increase Competence

  1. Appreciate that the majority of trainees accepted into our programs ultimately meet expected levels of competence. When trainees know what’s expected of them (i.e., are given the lists of expectations at the outset), most can acquire these abilities and, when necessary for Board certification, document that they’ve competently performed them. As educators we should approach trainees with the attitude and expectation that they will meet these competencies. Moreover, we should know what these expectations are in order to competently guide them toward achievement.
  2. Try not to generalize or set expectations about knowledge or skill-based competence from single observations, especially early in the trainee’s experiences. Appreciate that with repeated practice, demonstration, and feedback based on close faculty observation and encouragement, most trainees who seem initially to be inept ultimately turn out “ept.”
  3. Use your specialty’s competency milestone statements as the basis for offering trainees constructive comments for improvement as well as positive feedback. Do so immediately after you observe a trainee’s performance or as soon thereafter as feasible.
  4. Understand that a single observation of frank unprofessional behavior is worrisome. Acts of frank unprofessionalism may reflect enduring personality issues, insecurity, or counterproductive responses to clinical pressures or workplaces stresses. Any act should be discussed with the trainee immediately and should serve as the basis for close scrutiny in the future.
  5. Understand that medical educators must act responsibility concerning the small number of trainees who are unable to master the requisite competencies and who act unprofessionally. The ultimate public trust mandates that medical educators protect patients and their families from incompetent practitioners. There are resources available to support remediation of struggling learners through the Office of Lifelong Learning (contact Jeannette Guerrasio​). The earlier you identify the issue, the better we are able to help improve the long-term outcome.

Tips for Formally Evaluating Competencies

  1. Unless these are “high stakes” ratings, don’t waste much time developing your own competency or milestones assessment forms. (Unless you’re planning to devote yourself to competence assessment as a major, time-consuming scholarly project!). Instead, use forms that have been developed and disseminated by professional educational associations or other educators in your specialty. Of note, if you have the opportunity to engage in cross-institution projects to develop and test competence assessment tools, by all means, participate if the project will not take up an inordinate amount of time and you are interested in the professional development and community-building aspects. You may benefit professionally as a result, while enjoying the company of educators from other institutions participating in the project. 
  2. Remember that detailed narrative descriptions are generally more valuable than numerical rating systems. However, many busy faculty fail to take sufficient time, or are insufficiently articulate, to offer nuanced accounts. Of note: faculty provided with detailed, descriptive “drop down” menus of complete sentences/paragraphs that capture important performance characteristics may help raters save time and ultimately provide program directors with higher quality information on which to act. Regardless, your comments and specific examples as a faculty member who has directly observed the behavior of the resident are INVALUABLE.
  3. Use the competency-based forms and milestone instruments as prompts for helping to organize your thinking about residents’ performances. In day to day activities, think about how each trainee’s performance aligns with the various expected competencies and milestones.
  4. Understand that, except for specific procedural skills and at the extremes, most rating schemes of general  domains (e.g., “medical knowledge”, “systems-based care”) on which faculty are asked to evaluate trainees have limited reliability and validity because they are broadly stated and poorly operationally-defined. Realize that most assessments are inexact. Except in cases of frank incompetence or exceptionally brilliant performance, you’re unlikely to encounter strong inter-rater reliability (not that such reliability is guaranteed even at the extremes). Whereas gross assessments of “safe”, “unsafe,” “smart” and “dumb” may be replicated, inter-faculty agreement on finer-grained distinctions within each area of competency may be much harder to achieve.
  5. Appreciate that competency-based evaluation forms offer trainees important opportunities for self-reflection and self-assessment. Trainees should be asked to self-evaluate using these forms and to discuss their self-evaluations with faculty. Many trainees humbly undervalue themselves, often rating their competence as lower than perceived by faculty. On the other hand, finding trainees with an inflated sense of competence is potentially worrisome and may warrant closer future scrutiny.

Before Completing a Rating Form

  1. Know exactly who will see the form and how it will likely be used—now and in the future. What are the intended consequences of their use, and what unintended consequences might you foresee – on the trainee’s self-esteem, reputation, future recommendations and future prospects? How might any of your comments come back to you in the future? Before you err on the side of harsh evaluations, for example, do you feel sufficiently justified in the strength of the evidence on which you base your opinions (e.g., repeated observations or an egregious sentinel event) so as to rate someone in a strong, negative manner? Have you sufficiently documented the specifics involved so that the person receiving the evaluation can appropriately use the information to act on the information provided?
  2. Know your own rating biases. Are you a generally hard rater or a generally soft rater?  Are you taking these biases into account?
  3. Think about the rating biases of your peers and try not to succumb to “groupthink.” Don’t allow yourself to be subconsciously biased or swayed because you’ve heard other faculty badmouthing or excessively praising a trainee. For your own assessments, do your best to discount others’ comments and base your assessments entirely on your own observations and interactions. Be honest when you’ve had insufficient experience with a trainee to offer a useful appraisal.

We hope you find these tips useful as you work on the assessments required of you. Note that the Academy of Medical Educators is happy to come to your department, division, section or program meetings to provide faculty development training specific to your assessment framework. If you are interested, please contact Eva Aagaard.​