Frequently Asked Questions Page 3

Questions regarding Policies and Procedures

  1. Are my results considered personal and private information? What is available to third parties (for e.g. my employer or clients?)
  2. If I don’t pass the RCA, will I have to stop practicing?
  3. Is there an ‘expected’ pass or fail rate for the RCA?
  4. What is the cost to registrants for taking the RCA?
  5. Is anyone exempt from writing the RCA (ie. new graduates)?
  6. Will the RCA be held online?
  7. What if I wanted to look at the literature on performance assessments, Key Features tests and other topics related to the Quality Assurance Program?



1. Are my results considered personal and private information? What is available to third parties (for e.g. to my employer or clients?)

Section 26.2 of the Health Professions Act (HPA) protects the information gathered for the purposes of quality assurance. In other words, it cannot be released to any third party or used in the investigation of a complaint and/or civil procedure. Click HERE for further information and an excerpt from the HPA.

The Quality Assurance Committee is currently drafting Policies & Procedures related to the program. There is a draft policy on confidentiality that is pending the Board approval.

All volunteers (physical therapists) and College employees involved in the development and implementation of the program sign a confidentiality agreement.




2. If I don’t pass the RCA, will I have to stop practicing?

The Registrant Competence Assessment is intended to assess if you are meeting the standards, and if not, to help support and direct how you can meet those standards. This non-punitive approach means that if you are unsuccessful you will not lose your registration. The College will set up an individualized plan to assist you fill the gap(s) in your competence and improve your practice.

If not successful the registrant will receive a summary report to understand areas of weakness on the test. The policy on permission for repeated attempts at the test is under review by the Board and will be released as available.

If not successful in the RCA, then the College will help support those registrants by providing customized directions on how each of them can successfully meet the CPTBC competence standards.




3. Is there an ‘expected’ pass or fail rate for the RCA?

No.

The Registrant Competence Assessment is a ‘criterion referenced’ exam where all those that meet the standards pass.

Performance assessment literature and the experiences recorded from continuing competence programs over the past 20 years – regardless of whether a written test, portfolio or on-site assessment is used – has revealed that somewhere between 90 - 95% of examinees are successful the first time with almost all (i.e. 98-99%) being successful after a repeat assessment. Generally, only 1-2 % have been found to need additional remedial education or support.

An overview of literature about pass/fail rates is found in a summary table.




4. What is the cost to registrants for taking the RCA?

Containing the cost, while developing and implementing the Quality Assurance Program, is a HIGH priority for CPTBC, and was one of the criteria used in choosing the format for the Registrant Competence Assessment.

The cost for the Quality Assurance Program is included in the annual registration fees.




5. Is anyone exempt from writing the RCA (i.e. new graduates)?

Inactive registrants will not be required to complete the Registrant Competence Assessment for the period they are inactive. Those who have recently passed the entry to practice examination (i.e. Physiotherapy Competency Exam (PCE) will be exempt from writing the Registrant Competence Assessment for a period of 2-5 years. The Quality Assurance Committee is working on policies and procedures related to this issue.




6. Will the RCA be held online?

The College continues to investigate online and paper-and-pencil options considering such issues as logistical, administrative and cost implications. A decision on the final format is planned for Fall 2011.




7. What if I wanted to look at the literature on performance assessments, Key Features tests and other topics related to the Quality Assurance Program:

Selected readings on performance assessments, Key Features tests and other topics related to the Quality Assurance Program:

  1. American Educational Research Association, American Psychological Association, et al. (1999). Standards for Educational and Psychological Tests. Washington: Author.
  2. Austin, Z., Marini, A., et al. (2004). Assessment of pharmacists' patient care competencies: Validity evidence from Ontario (Canada)'s Quality Assurance and Peer Review Process. Pharmacy Education 4(1), 23-32.
  3. Bordage, G., Brailovsky, C., Carretier, H., & Page, G. (1995). Content validation of key features on a national examination of clinical decision-making skills. Academic Medicine: Journal of the Association of American Medical Colleges, 70(4), 276-281.
  4. Caulford, P., Lamb, S., et al. (1994). Physician Incompetence: Specific Problems and Predictors. Academic Medicine 69(10).
  5. Cizek, G. J. (1996). Setting passing scores. Educational Measurement: Issues and Practice, 15(2): 20-31.
  6. Cizek, G. J. (1996). Standard Setting Guidelines. Educational Measurement: Issues and Practice, 15(1). 12-21.
  7. Cizek, G. J., Ed. (2001). Setting Performance Standards. Mahwah, NY, Lawrence Erlbaum Associates.
  8. Colthart, I, Bagnall, G., Evans, A., Allbutt, H. Haig A., Illing, J. and McKinstry, B. (2008). The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. 2008, Medical Teacher. Vol. 30, No. 2 , Pages 124-145
  9. Council on Licensure Enforcement and Regulation (CLEAR) (1993). Development, Administration, Scoring and Reporting of Credentialing Examinations: Recommendations for Board Members. Lexington, KY, The Council of State Governments: 41.
  10. Cummings, H., Lipski, U., et al. (1997). A Framework for Evaluating the Quality Assurance Programs of the Colleges of Health Professions in Ontario. Toronto: Harry Cummings & Associates. 111.
  11. Cunnington, J. P., Hanna, E., et al. (1997). Defensible assessment of the competency of the practicing physician. Academic Medicine, 72(1), 9-12.
  12. Davis, D. A., P. E. Mazmanian, et al. (2006). "Accuracy of physician self-assessment compared with observed measures of competence." JAMA 296(9): 1094-1102.
  13. Dudek, N. L., Marks, M. B., & Regehr, G. (2005). Failure to fail: the perspectives of clinical supervisors. Academic Medicine, 80(10), S84.
  14. Eva, K. W. & Regehr, G. (2005). Self-assessment in the health professions: A reformulation and research agenda. Academic Medicine, 80(10), S46.
  15. Farmer, E. A., & Page, G. (2005). A practical guide to assessing clinical decision-making skills using the key features approach. Medical Education, 39(12), 1188-1194.
  16. Fischer, M. R., Kopp, V., Holzer, M., Ruderich, F., & Junger, J. (2005). A modified electronic key feature examination for undergraduate medical students: Validation threats and opportunities. Medical Teacher, 27(5), 450-455.
  17. Goldman, C. (1992). The Reliability of Peer Assessments of Quality of Care. Journal of the American Medical Association, 267(7), 958-960.
  18. Goldman, R. L., & Ciesco. E. (1996). Improving Peer Review: Alternatives to Unstructured Judgements by a Single Reviewer. Journal on Quality Improvement, 22(11), 762-769.
  19. Gormley, G. J., Collins, K., Boohan, M., Bickle, I. C., & Stevenson, M. (2009). Is there a place for e-learning in clinical skills? A survey of undergraduate medical students' experiences and attitudes. Medical Teacher, 31(1), e6-12.
  20. Hanna, E., Premi, J., et al. (2000). Results of Remedial Continuing Medical Education in Dyscompetent Physicians. Academic Medicine, 75(2), 174-176.
  21. Hatala, R., & Norman, G. R. (2002). Adapting the key features examination for a clinical clerkship. Medical Education, 36(2), 160-165.
  22. Hertz, N. R., & Chinn, R. N. (1999). Resource Briefs: Licensure Examinations. Lexington, KY: The Council on Licensure, Enforcement and Regulation (CLEAR).
  23. interpretations and decisions: A response. Applied Measurement in Education, 10(1), 97-104.
  24. Litchfield, R. E., Oakland, M. J., & Anderson, J. A. (2001). Key feature exams: A tool for assessing clinical competency. Journal of the American Dietetic Association, 101(9, suppl. Supplement 1, pp. A-62), September.
  25. Livingston, S. A., & Zieky, M. J. (1982). Passing Scores: A manual for setting standards of performance on educational and occupational tests. Livingston, NJ: Educational Testing Service.
  26. McAuley, R. G., & Henderson, H. W. (1984). Results of the peer assessment program of the College of Physicians and Surgeons of Ontario" Canadian Medical Association Journal, 131, 557-561.
  27. McAuley, R. G., Paul, W. M., et al. (1990). Five-year results of the peer assessment program of the College of Physicians and Surgeons of Ontario. Canadian Medical Association Journal, 143(11), 1193-1199.
  28. Mehrens, W. A. (1992). How to evaluate the legal defensibility of high-stakes tests. Applied Measurement in Educatio,n 5(3), 267-282.
  29. Mehrens, W. A. (1997). Validating licensing and certification test score
  30. Nayer, M. (2003). Quality Management Program Evaluation Report. Toronto, ON, College of Physiotherapists of Ontario: 62-65.
  31. Nayer, M. (2006). Onsite Assessment Pilot Test Report. Toronto, ON, College of Physiotherapists of Ontario: 146.
  32. Nikendei, C., Mennin, S., Weyrich, P., Kraus, B., Zipfel, S., Schrauth, M., et al. (2009). Effects of a supplementary final year curriculum on students' clinical reasoning skills as assessed by key-feature examination. Medical Teacher, 31(9), e438-42.
  33. Norman, G. (2005). Research in clinical reasoning: Past history and current trends. Medical Education, 39(4), 418-427.
  34. Norman, G. R., Davis, D. A., et al. (1993). Diagnostic assessment of primary care physicians as a component of a peer review program. Journal of the American Medical Association, 270, 1046-1052.
  35. Norton, P. G., Dunn, E. V., et al. (1998). Long-Term Follow-up in the Peer Assessment Program for Nonspecialist Physicians in Ontario, Canada. The Joint Commission Journal on Quality Improvement, 24(6), 334-341.
  36. Page G, Bordage G, Allen T. (1995) Developing key-feature problems and examinations to assess clinical decision-making skills. Academic Medicine 70:194-201.
  37. Page, G & Bordage, G (1995). The Medical Council of Canada’s Key Features Project: A more valid written examination of clinical decision-making skills. Academic Medicine 70(2):104-110.
  38. Page, G, Farmer, E, Spike, N and McDonald, E (2000). The use of short answer questions in the key features problems in the Royal College of General Practitioners Fellowship examination. Proceedings of the 9th International Ottawa Conference on Medical Education. Cape Town, South Africa
  39. Page, G. G., Bates, J., et al. (1995). Physician-assessment and physician enhancement programs in Canada. Canadian Medical Association Journal, 153(12), 1723-1728.
  40. Paget, N. S., Newble, D. I., et al. (1996). Physician Assessment Pilot Study for the Royal Australasian College of Physicians. Journal of Continuing Education in the Health Profession 16, 103-111.
  41. Pamphlett, R. (2005). It takes only 100 true-false items to test medical students: True or false? Medical Teacher, 27(5), 468-472.
  42. Patterson, F., Baron, H. Carr, V., Plint, S. and Lane, P. Evaluation of three short-listing methodologies for selection into postgraduate training in general practice. Medical Education 2009: 43: 50–57.
  43. Pleguezuelos, E; Brailovsky, Carlos A; Hornos, E; Martinez, M. May 2010. Script Concordance Test as a new CME model focused on clinical reasoning: An international validation in Spanish speaking countries. Ottawa Conference on Competence in Health Professionals. Miami, Florida.
  44. Raupach, T., Muenscher, C., Anders, S., Steinbach, R., Pukrop, T., Hege, I., et al. (2009). Web-based collaborative training of clinical reasoning: A randomized trial. Medical Teacher, 31(9), e431-7.
  45. Regehr & Eva, 2006 Self-assessment, self-direction, and the self-regulating professional. Clin Orthop Relat Res. 2006 Aug;449:34-8
  46. Richards, B. F., Ober, K. P., Cariaga-Lo, L., Camp, M. G., Philp, J., McFarlane, M., et al. (1996). Ratings of students' performances in a third-year internal medicine clerkship: A comparison between problem-based and lecture-based curricula. Academic Medicine : Journal of the Association of American Medical Colleges, 71(2), 187-189.
  47. Rotthoff, T., Baehring, T., Dicken, H. D., Fahron, U., Richter, B., Fischer, M. R., et al. (2006). Comparison between long-menu and open-ended questions in computerized medical assessments. A randomized controlled trial. BMC Medical Education, 6, 50.
  48. Sargeant, J., Mann, K., Van Der Vleuten, C., Metsemakers, J. (2008). “Directed” Self-Assessment: Practice and Feedback Within a Social Context. Journal Of Continuing Education In The Health Professions, 28(1):47–54, 2008
  49. Schuwirth, L. W., & van der Vleuten, C. P. (2004). Different written assessment methods: What can be said about their strengths and weaknesses? Medical Education, 38(9), 974-979.
  50. Showers, B. (1999). Resource Briefs: Reference Guide for Auditing a Credentialing Examination Program. Lexington, KY: Council on Licensure, Enforcement and Regulation (CLEAR).
  51. Southgate, L., Campbell, C. M., et al. (2001). The General Medical Council's Performance Procedures: the development and implementation of tests of competence with examples from general practice. Medical Education, 35(1), 20-28.
  52. Sturmberg, J. P., Atkinson, K., Farmer, E. A., & Research and Development Subcommittee, Board of Examiners, The Royal Australian College of General Practitioners. (2005). Standards and performance--attainment and maintenance of professional capabilities. Australian Family Physician, 34(5), 371-373.
  53. Tamblyn, R, Abrahamowicz, M, Dauphinee, D, Wenghofer, E, Jacques, A, Klass, D, et al. (2007). Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. Journal of the American Medical Association 298(9): 993-1001.
  54. The Council on Licensure Enforcement and Regulation. (1993). Development, administration, scoring and reporting of credentialing examinations. Lexington, KY: The Council of State Governments.
  55. The Joint Committee on Standards for Educational Evaluation, Ed. (1994). The Program Evaluation Standards: How to assess evaluations of educational programs. Thousand Oaks, CA: Sage Publications.
  56. Violato, C. and J. Lockyer (2006). "Self and peer assessment of pediatricians, psychiatrists and medicine specialists: implications for self-directed learning." Advances in Health Sciences Education 11(3): 235-244.
  57. Williams, B. W. (2006). "The prevalence and special educational requirements of dyscompetent physicians." Journal of Continuing Education in the Health Professions 26(3): 173-91.

TABLE: 1 Overview of literature about pass/fail rates in competence assessments

College Process % successful
College of Massage Therapists of Ontario (CMTO) (College of Massage Therapists of Ontario 2011)
  • Portfolio
  • CE credits
  • Peer assessment – up to 20% of registrants selected annually for a peer assessment visit
 98-99% (College of Massage Therapists of Ontario 2008; College of Massage Therapists of Ontario 2010)
College of Occupational Therapists of Ontario (College of Occupational Therapists of Ontario 2011)
  • Self-directed learning modules
  • Self-assessment tool – completed every 2 years
  • Professional development plan – document their own goals, activities, results of learning; completed annually
  • Competency Review and Evaluation

    •      -Submit copies of the above tools

         -Complete multi-source feedback with peers, co-workers
          and clients

         -Review of documents

         -May be identified for a more detailed Competency
          Evaluation

         -Peer assessor reviews & discusses records and
          interviews the OT using 4 tools

                   -Pre-questionnaire, case-based questions,
                    situation-based questions, chart review

         -Depending on outcome, continuing competency
           activities may be required
 93-95% (College of Occupational Therapists of Ontario 2009; College of Occupational Therapists of Ontario 2010)
Ontario College of Pharmacists (Ontario College of Pharmacists 2011)
  • 20% annually complete self-assessment tool
  • 240 registrants participate in Practice Review, a 6-hr process which includes a discussion of portfolio, a written test and an OSCE
 86% meet standards (Austin, Marini et al. 2004)
Physicians
  • Review article: Age-related cognitive decline, impairment due to substance use disorders, and other psychiatric illness can contribute to underperformance, diminishing physicians' insight into their level of performance as well as their ability to benefit from an educational experience.
99.4 – 50% depending on method of assessment; reasonable estimate of dyscompetence 6-12% (Williams 2006)
College of Physicians and Surgeons of Ontario (College of Physicians and Surgeons of Ontario 2011)
  • Self-assessment
  • Develop a personal learning plan
  • Onsite assessment for chart review & discussion of patients
 92% (McAuley and Henderson 1984)
88% (Norton, Dunn et al. 1998)
College of Physiotherapists of Ontario (College of Physiotherpaists of Ontario 2011)
  • Practice Reflection: a portfolio is to be kept up to date throughout PT’s career
  • Practice Assessment, where a peer assessor comes to the PTs place of employment
  • Includes Facility/Practice Evaluation, Billing Practices Evaluation, Record Keeping Evaluation, Portfolio Review, and Chart Stimulated Recall
 98.9% (Nayer 2003)
94% (Nayer 2006)
88% (Miller, Nayer et al. 2010)
College of Respiratory Therapists of Ontario (College of Respiratory Therapists of Ontario 2011)
  • 50 question open-book Professional Standards Assessment
  • 70% of above 6th percentile to pass
  • Personal communication (2011)
  • If fail rewrite
  • If not successful on rewrite write again
  • If still fail after 3 times move to mentoring (only happened 2x since 2005)
 91.8-93.7 after 3 attempts (College of Respiratory Therapists of Ontario 2011)

References for Table 1

  • Austin, Z., A. Marini, et al. (2004). "Assessment of pharmacists' patient care competencies: validity evidence from Ontario (Canada)'s Quality Assurance and Peer Review Process." Pharmacy Education 4(1): 23-32.
  • College of Massage Therapists of Ontario (2008). Annual Report. Toronto, ON, College of Massage Therapists of Ontario.
  • College of Massage Therapists of Ontario (2010). Annual Report. Toronto, ON, College of Massage Therapists of Ontario.
  • College of Massage Therapists of Ontario. (2011). "Quality Assurance Program Description." Retrieved August 21, 2011, from http://www.cmto.com/member/cont.htm.
  • College of Occupational Therapists of Ontario (2009). Annual Report. Toronto, ON, College of Occupational Therapists of Ontario.
  • College of Occupational Therapists of Ontario (2010). Annual Report. Toronto, ON, College of Occupational Therapists of Ontario.
  • College of Occupational Therapists of Ontario. (2011). "Competency Review and Evaluation." Retrieved August 21, 2011, from http://www.coto.org/quality/review.asp
  • College of Physicians and Surgeons of Ontario. (2011). "Preparing for a Peer Assessment." Retrieved August 21, 2011, from http://www.cpso.on.ca/members/peerassessment/default.aspx?id=1952
  • College of Physiotherpaists of Ontario. (2011). "Practice Reflection." Retrieved August 21, 2011, from http://www.collegept.org/Physiotherapists/Quality%20Management/Practice%20Reflection/PracticeReflection
  • College of Respiratory Therapists of Ontario. (2011). "Professional Standards Assessment." Retrieved August 21, 2011, from http://www.crto.oProfessioan.ca/psa.aspx.
  • College of Respiratory Therapists of Ontario (2011). Results of Professional Standards Assessment. M. Nayer. Toronto, Ontario.
  • McAuley, R. G. and H. W. Henderson (1984). "Results of the peer assessment program of the College of Physicians and Surgeons of Ontario." Canadian Medical Association Journal 131: 557-561.
  • Miller, P., M. Nayer, et al. (2010). "Psychometric properties of a peer assessment program to assess continuing competence in physical therapy." Physical Therapy 90(7): 1026-1038.
  • Nayer, M. (2003). Quality Management Program Evaluation Report. Toronto, ON, College of Physiotherapists of Ontario: 397.
  • Nayer, M. (2006). Onsite Assessment Pilot Test Report. Toronto, ON, College of Physiotherapists of Ontario: 146.
  • Norton, P. G., E. V. Dunn, et al. (1998). "Long-Term Follow-up in the Peer Assessment Program for Nonspecialist Physicians in Ontario, Canada." The Joint Commission Journal on Quality Improvement 24(6): 334 - 341.
  • Ontario College of Pharmacists. (2011). "Quality Assurance Program Description." Retrieved August 21, 2011, from http://www.ocpinfo.com/client/ocp/OCPHome.nsf/web/Introduction+to+the+Practice+Review

Copyright 1998 - College of Physical Therapists of BC - All rights reserved.

 
LAST UPDATED: May 7, 2012