Huang et al. BMC Medical Education (2022) 22:76 https://doi.org/10.1186/s12909-022-03140-0 RESEARCH Open Access The ASK-SEAT: a competency-based assessment scale for students majoring in clinical medicine Linxiang Huang1†, Zihua Li1†, Zeting Huang1, Weijie Zhan1, Xiaoqing Huang1, Haijie Xu1, Chibin Cheng1, Yingying Zheng2, Gang Xin3*, Shaoyan Zheng4* and Pi Guo5* Abstract Background: To validate a competency-based assessment scale for students majoring in clinical medicine, ASK-SEAT. Students’ competency growth across grade years was also examined for trends and gaps. Methods: Questionnaires were distributed online from May through August in 2018 to Year-2 to Year-6 students who majored in clinical medicine at the Shantou University Medical College (China). Cronbach alpha values were calculated for reliability of the scale, and exploratory factor analysis employed for structural validity. Predictive validity was explored by correlating Year-4 students’ self-assessed competency ratings with their licensing examination scores (based on Kendall’s tau-b values). All students’ competency development over time was examined using the Mann- Whitney U test. Results: A total of 760 questionnaires meeting the inclusion criteria were analyzed. The overall Cronbach’s alpha value was 0.964, and the item-total correlations were all greater than 0.520. The overall KMO measure was 0.966 and the KMO measure for each item was greater than 0.930 (P < 0.001). The eigenvalues of the top 3 components extracted were all greater than 1, explaining 55.351, 7.382, and 5.316% of data variance respectively, and 68.048% cumulatively. These components were aligned with the competency dimensions of skills (S), knowledge (K), and attitude (A). Significant and positive correlations (0.135 < Kendall’s tau-b < 0.276, p < 0.05) were found between Year-4 students’ self-rated competency levels and their scores for the licensing examination. Steady competency growth was associated with almost all indicators, with the most pronounced growth in the domain of skills. A lack of steady growth was seen in the indicators of “applying the English language” and “conducting scientific research & innovating”. Conclusions: The ASK-SEAT, a competency-based assessment scale developed to measure medical students’ com- petency development shows good reliability and structural validity. For predictive validity, weak-to-moderate correla- tions are found between Year-4 students’ self-assessment and their performance at the national licensing examina- tion (Year-4 students start their clinical clerkship during the 2nd semester of their 4th year of study). Year-2 to Year-6 *Correspondence: gxin@stu.edu.cn; syzheng@stu.edu.cn; pguo@stu.edu.cn †Linxiang Huang and Zihua Li contributed equally to this work. 3 Department of Microbiology and Immunology, Shantou University Medical College, Shantou 515041, China 4 Department of Higher Medical Education, Shantou University Medical College, Shantou 515041, China 5 Department of Public Health and Preventive Medicine, Shantou University Medical College, Shantou 515041, China Full list of author information is available at the end of the article © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licens es/ by/4. 0/. The Creative Commons Public Domain Dedication waiver (http:// creat iveco mmons. org/p ubli cdoma in/ zero/1. 0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Huang et al. BMC Medical Education (2022) 22:76 Page 2 of 10 students demonstrate steady improvement in the great majority of clinical competency indicators, except in the indicators of “applying the English language” and “conducting scientific research & innovating”. Keywords: Competency-based assessment, Self-assessment, Medical education, Licensing examination, Clinical medicine Background semination, translation, and creation of knowledge In 1978, McGathie et  al. prepared a report for the and practices”; and. World Health Organization (WHO), advocating for g) Professional—“being committed to ethical practice, cultivating medical talents through competency-based accountability to the profession and society” and medical education (CBME) in order to meet the health- maintaining personal health. care needs of local populations worldwide [1]. Three decades later, a group of international educators refined In 2014, Sun et  al. constructed the Chinese Doc- CBME as “an outcomes-based approach to the design, tors’ Common Competency Model [7, 8], an initiative implementation, assessment, and evaluation of medical jointly approved by the National Medical Examina- education programs, using an organizing framework of tion Center and the Ministry of Education. The model competencies” [2]. has since served as an important reference and stand- Developed countries such as U.K., U.S., and Canada ard for the training of Chinese medical professionals. have developed more comprehensive competency- In July 2017, the General Office of the State Coun- based frameworks [3–6]. For instance, the Accredita- cil issued a policy entitled “Deepening the Synergy tion Council for Graduate Medical Education in U.S. Between Education and Healthcare System to Further expects residents to obtain competencies in 6 areas: Promote Reforms and Development of Medical Educa- patient care, medical knowledge, interpersonal & com- tion in China” [9], and highlighted the pressing need munication skills, professionalism, practice-based to establish a system for the evaluation of medical learning & improvement, and system-based prac- education. tice [4]. The General Medical Council (GMC) in U.K. The medical education in China is administered has outlined, in its Good Medical Practice (GMP), the through a variety of programs. From Year 2 to Year 4, standards which practitioners shall meet and they span students take courses on medical fundamentals. Year-4 4 domains: knowledge, skills & performance; safety & students start their clinical clerkship in the 2nd semester. quality; communication, partnership & teamwork; and At Year 5, students attend clinical rotations at teaching maintaining trust [5]. At CanMEDS 2015, a physician hospitals, and receive their bachelor’s degree in medicine competency framework endorsed by 12 Canadian med- at the end of their 5th year of study. Year 6 marks the 1st ical organizations was presented which identified mul- year of 3 years of standardized resident training. With the tiple key roles played by a competent physician [6]: “5 + 3” program, students receive both the bachelor’s and master’s degrees when completing their study. With the a) Medical expert— applying medical knowledge, clini- 8-year track, students are awarded bachelor’s and doc- cal skills, and professional values to provide quality tor’s degrees when they graduate. patient-centered care; b) Communicator—forming relationships with patients and their families which facilitate sharing essential National Medical Licensing Examination (NMLE) information for the delivery of effective health care; In April 2015, the National Medical Examination Center c) Collaborator—working effectively with other health in China reformed the administration of the NMLE care professionals to provide quality patient-centered into two phases. The Phase-I examination (hereinafter, care; referred to as “NMLE-Phase I”) contains two sections: d) Leader—engaging with others to contribute to real- basic medical knowledge (hereinafter, referred to as izing visions of quality health care systems; “theory examination”) and basic clinical skills (hereinaf- e) Health advocate—contributing expertise and influ- ter, referred to as “skills examination”), while the Phase- ence to improve healthcare when partnering with II examination (hereinafter, referred to as “NMLE-Phase communities or patient populations; II”) tests candidates’ comprehensive clinical knowledge f ) Scholar—demonstrating a commitment to continu- and skills. The clinical skills portion of the examination ous learning and “contributing to the application, dis- is modeled after the Objective Structured Clinical Exami- nation (OSCE). Medical students are eligible to take the NMLE-Phase I at the end of their 4th year of study and Huang et al. BMC Medical Education (2022) 22:76 Page 3 of 10 Fig. 1 ASK-SEAT: a competency-based assessment scale Phase-II at the end of Year 6 [10]. Unlike most standard frameworks created in developed countries as illustrated tests administered in medical schools, the NMLE evalu- in the above. To enable a more granular assessment of ates multiple dimensions of candidates’ clinical compe- students’ competencies, a matrix design was adopted. tency—knowledge and clinical skills—and hence a closer Four aspects of mastery—state (S), explain (E), apply (A), approximation to a more rounded competency-based and transfer (T)—were used to characterize the 4 levels assessment. of competency for each indicator, reflecting the progres- sion of competency in Miller’s Pyramid. A 5-point Lik- ASK?SEAT: a competency?based assessment scale ert scale—I (not at all), II (somewhat), III (moderately), In the 1990s, drawing from the process of cognitive IV (mostly), and V (completely)—was added to further development, George Miller, an American medical edu- quantify each SEAT level. A total of 96 textual descrip- cator, proposed “Miller’s Pyramid” for assessing the tions (for 24 indicators and 4 competency levels) were clinical competencies of medical students and resident also drafted. physicians [11]. The pyramid illustrates how the ultimate Inspired by Miller’s framework, the tiers of competency mastery of each competency progresses from the level of in the ASK-SEAT did diverge somewhat from Pyramid, cognition to clinical practice, and how different levels of mainly the top 2 tiers. While Miller separated performing mastery can be measured. The 4 tiers of Miller’s Pyramid in a conditioned setting (“Shows How”) from performing comprise the following: 1) Knows (knowledge)—“knows in the real world (“Does”), the ASK-SEAT collapsed these what’s required in order to carry out professional func- two into one tier (“Apply”) and created an additional tier tions effectively”; 2) Knows How (competence)—knows of “Transfer”. The creation of this tier was underscored how to use the knowledge acquired (e.g. formulating by 2 contributing factors related to the mission and focus diagnosis and treatment plans); 3) Shows How (perfor- of medical education in China. First, as presented in the mance)—shows how to perform when facing a patient; 2015 Global Standards for Quality Improvement: Basic and 4) Does (action)—how to act when “functioning Medical Education by the World Federation for Medi- independently in a clinical practice”. cal Education (WFME), medical students upon gradua- However, to the best of our knowledge, there have tion are expected to be able to perform competently the been few standardized assessment systems, in China or roles of, among others, “teacher” and “scholar” [12]. Sec- abroad, to evaluate the competency development of stu- ond, the ability to transfer prepares medical graduates for dents majoring in clinical medicine. Hence, based on the “participatory learning” that will be emphasized in the Chinese Doctors’ Common Competency Model created subsequent standardized resident training. by Sun et al. [7, 8], we created 24 competency indicators As a pilot study, the ASK-SEAT scale was used in for students majoring in clinical medicine which reflect 2018 to assess the core competencies of 155 Year-5 stu- 3 domains of clinicians’ competencies: attitude (A), dents (“new graduates”) majoring in clinical medicine skills (S), and knowledge (K) (Fig.  1). These indicators at the Shantou University Medical College (SUMC) in broadly reflected the competencies enumerated in the China [13]. Therefore, the goal of the current study was Huang et al. BMC Medical Education (2022) 22:76 Page 4 of 10 Table 1 Basic information of questionnaire respondents Year Item Gender Age Male Female Total 18 ~ 20 21 ~ 23 24 ~ 27 Unknown Total 6 No. 55 49 104 – 2 102 – 104 Percentage 52.90% 47.10% 100.00% – 1.90% 98.10% – 100.00% 5 No. 80 71 151 – 64 87 – 151 Percentage 53.00% 47.00% 100.00% – 42.40% 57.60% – 100.00% 4 No. 80 73 153 – 135 12 6 153 Percentage 52.30% 47.70% 100.00% – 88.20% 7.80% 3.90% 100.00% 3 No. 91 88 179 10 166 1 2 179 Percentage 50.80% 49.20% 100.00% 5.60% 92.70% 0.60% 1.10% 100.00% 2 No. 88 85 173 84 85 – 4 173 Percentage 50.90% 49.10% 100.00% 48.60% 49.10% – 2.30% 100.00% Total No. 394 366 760 94 452 202 12 760 Percentage 51.80% 48.20% 100% 12.40% 59.50% 26.60% 1.60% 100.00% to validate the results from the pilot study by survey- self-assessed ASK-SEAT ratings (students’ NMLE-Phase ing a larger group of students. Predictive validity of I scores were collected from the Academic Affairs Office the scale would be tested by correlating students’ self- of the SUMC). Statistical significance was set at 0.05. The assessed competency ratings with their performance at Mann-Whitney U test was employed to identify compe- the NMLE-Phase I. Participating students’ competency tency differentials between adjacent grades for students’ growth across grade years would also be examined for competency development over time. Except the correla- trends and gaps. tion analysis which relied solely on information related to Year-4 students, the remaining analyses were carried out Methods using the questionnaires from all participating students. Questionnaire Questionnaires created based on the 24 indicators were Results distributed via an online platform from May through Respondents August 2018 to Year-2 to Year-6 students at the SUMC. A Out of 960 questionnaires collected, 760 met the inclu- questionnaire response was excluded if it met one of the sion criteria and were analyzed (366 from female stu- following criteria: 1) from respondents majoring in clini- dents, accounting for 48.2% of the total). The number of cal medicine at the SUMC but outside the grade years responses in each grade-year group exceeded 150, except specified; 2) from respondents who supplied identical the group of Year 6 of slightly more than 100 responses. answers to all questions; 3) from respondents who sub- Participating students’ basic information is summarized mitted multiple questionnaires using the same IP address in Table 1. (in this case, only the last questionnaire submitted would be accepted, with the rest, discarded). The questionnaire ASK?SEAT: reliability includes 13 items on personal background and 24 items The overall Cronbach’s alpha value was 0.964. The item- on competency, all of which are mandatory. total correlations were all greater than 0.520, within an acceptable range. Hence, all items were retained, as Data analysis shown in Table 2. Statistical analyses were conducted using SPSS 21.0 (IBM Corp. Released 2012. IBM SPSS Statistics for Windows, ASK?SEAT: structural validity & predictive validity Version 21.0. Armonk, NY: IBM Corp.). Cronbach alpha EFA based on varimax rotation was first performed values were calculated to examine the reliability of the without a limit to the number of factors to be extracted. scale, and exploratory factor analysis (EFA) performed The data variance explained by the 4 factors extracted for structural validity. For predictive validity, correlation was 55.351, 7.382, 5.316, and 4.523% respectively, and analysis was carried out (based on Kendall’s tau-b val- 72.572% cumulatively. In this round, only 2 indicators ues), using Year-4 students’ NMLE-Phase I scores (con- loaded on the 4th factor: taking the medical history, sisting of 3 sections: theory, skills, and total) and their and conducting the physical examination. Hence, a 2nd H uang et al. BMC Medical Education (2022) 22:76 Page 5 of 10 Table 2 Reliability and validity of the ASK-SEAT assessment scale Competency indicators Reliability Factor Loadinga after Varimax Rotation Corrected item?total Cronbach’s alpha if Domains correlation item deleted S K A S-1 Taking the medical history .527 .964 .449 – .301 S-2 Conducting the physical examination .616 .963 .602 – – S-3 Applying basic operational skills .706 .963 .832 – – S-4 Presenting clinical cases verbally .776 .962 .800 – – S-5 Formulating the treatment plan .776 .962 .850 – – S-6 Ensuring patient safety .778 .962 .700 – .340 S-7 Managing chronic illnesses .798 .962 .748 .302 – S-8 Participating in education & promotion of health .725 .962 .564 – .427 S-9 Conducting emergency rescue .777 .962 .785 – – S-10 Selecting lab tests & medical examinations .813 .961 .774 .324 – K-1 Understanding the healthcare system .775 .962 .642 .464 – K-2 Retrieving, organizing, & analyzing medical information .727 .962 .364 .734 – K-3 Applying the English language to knowledge acquisition, .567 .964 – .781 – professional exchange & clinical practice K-4 Acquiring & applying basic biomedical knowledge .711 .962 .387 .701 – K-5 Acquiring & applying knowledge of social science .767 .962 .445 .615 .314 K-6 Acquiring & applying clinical knowledge .845 .961 .646 .469 .336 K-7 Updating knowledge and skills .718 .962 – .639 .459 K-8 Applying critical thinking .730 .962 – .664 .484 K-9 Conducting scientific research & innovating .717 .962 .311 .737 – A-1 Controlling patient’s medical expenses .703 .963 .543 .402 – A-2 Maintaining psychological health .655 .963 – – .719 A-3 Communicating & cooperating with clients .744 .962 .406 – .727 A-4 Protecting patient confidentiality .578 .964 – – .821 A-5 Teamwork .601 .963 – – .787 a The factor loading refers to the correlation coefficient between the indicator and the relevant domains, ranging from 0 (weakest) to 1 (strongest). Indicators with a factor loading below 0.3 are excluded from the table round of EFA was performed where the number of fac- of these 3 indicators to maximize the utility of the scale tors to be extracted was limited to 3. The 2nd EFA yielded (Table 2). a linear correlation among the variables (24 items) and The correlation between Year-4 students’ self-assessed an adequate data structure (overall KMO = 0.966; KMOs ASK-SEAT ratings and their performance at the NMLE- for items > 0.930; P  < 0.001). Hence, principal compo- Phase I is presented in Fig.  2 where significant correla- nent extraction was deemed suitable. The eigenvalues of tions are in bold. Significant and positive correlations the top 3 components extracted were all greater than 1 (0.135 < Kendall’s tau-b < 0.276, P < 0.05) spread generally (explaining 55.351, 7.382, and 5.316% of data variance evenly across 3 domains of attitude (A), skills (S), and respectively, and 68.048% cumulatively). These compo- knowledge (K) for the theory as well as the combined nents corresponded to the 3 competency dimensions of total portion (theory plus skills). In the skills portion, skills (S), knowledge (K), and attitude (A). Three indica- more correlations were associated with the domains of tors were not aligned as expected: K-1 (“understand- attitude (A) and knowledge (K). ing the healthcare system”), K-6 (“acquiring & applying clinical knowledge”), and A-1 (“controlling patient’s Competency growth medical expenses”). After taking into consideration the The mean ratings (with standard errors) of competency grade-specific results where selected indicators were by students (Year-2 to Year-6) are graphed in Fig. 3-1 (by also aligned differently, a decision was made not to make domain) and Fig. 3-2 (by competency level). The highest further adjustment and to retain the initial alignments rating for each grade year was in the domain of attitude Huang et al. BMC Medical Education (2022) 22:76 Page 6 of 10 Fig. 2 Correlations between Year-4 students’ self-assessed competency levels and their scorings for the NMLE-Phase I (A), and the most improvement was in the domain of Positive correlations between students’ self-assessment skills (S) (Fig. 3-1). For the level of competency, students’ and their performance in the skills portion of the exami- performance trended steadily upward across grade years, nation were also seen in 10 indicators, but only 1 correla- with the highest rating associated with the level of “state” tion pertained to the “skills domain” (i.e. S1: Taking the followed by the levels of “explain”, “apply”, and “transfer”, medical history). To ace the skills section of the exami- in that order and for each grade year (Fig. 3-2). nation (modeled after the OSCE), students needed to The indicators with significant and positive changes in draw from their capabilities in all 3 domains. Students’ competency level between adjacent grades are marked in strong foundation in “attitude” and “knowledge” domains blue in Fig. 4. Growths were reported by students for all (as evidenced in their scoring for the theory-knowledge indicators except 2, with the most pronounced growth portion of the NMLE) contributed meaningfully to their in the domain of skills. The 2 indicators where no steady overall scoring in the skills portion. On the other hand, growth was seen were “applying the English language” the correlation in only 1 indicator which pertained to and “conducting scientific research & innovating”. the skills domain might be attributed to the fact that Year-4 students just started their clinical clerkship during Discussion the 2nd semester of their 4th year of study (who would ASK?SEAT: a competency?based assessment scale receive additional clinical exposure and training during The overall Cronbach’s alpha value (0.964) and the item- subsequent clinical rotations during Year 5 and resident total correlations (all greater than 0.520) demonstrated training from Year 6 through Year 8). Therefore, it is not good reliability of the ASK-SEAT scale. The three fac- entirely surprising that the correlation between the self- tors extracted in the pilot study—attitude (A), skills (S), assessment and the skills portion skewed toward the and knowledge (K)—were also confirmed through EFA. domains of “attitude” and “knowledge”. Meanwhile, as presented in Table 2, most indicators had At the same time, the correlations ranged from weak loadings of more than .30 (the cutoff) on 2 and some- to moderate, even though they met the statistical sig- times 3 of the factors extracted. The mastery level of one nificance set for the study. In order to more definitively dimension of a competency can have an additive effect characterize how medical students’ self-assessed compe- on the mastery level of another dimension. For example, tencies correlate with their performance in the licensing a more clinically-skilled student is likely to give a higher examination, follow-on studies can replicate the correla- rating for his/her mastery level in both knowledge and tion analysis (proposed in this study) among students of skills dimensions, since knowledge serves as the foun- more advanced grade years (i.e. Year 5 to Year 8) as well dation of skills and, as skills are developed, the relevant as between students’ self-assessed ratings and their scor- knowledge is also enforced. Hence, a less “clean” loading ings for the NMLE-Phase II taken at the end of Year 6 between indicators and factors could be the reflection of (which evaluates candidates’ comprehensive knowledge this unique property of competency. If the loading cutoff and clinical skills). By then, students will have accumu- were raised (to higher than .30), we would have obtained lated more clinical experience and are more cognitively a cleaner set of loadings. However, the unique additive equipped to rate their clinical skills levels. Different cor- effect between competency dimensions would have also relation patterns could well emerge in these investiga- been masked. Further exploration of this topic in future tions. Nevertheless, the positive correlations between studies may help shed more illuminating lights. students’ self-assessed competencies in “attitude” and In the current study, positive correlations between “knowledge” domains and their scorings for both the the- Year-4 students’ self-assessed competency ratings and ory and overall skills portions of the NMLE did testify to their scorings for the theory-knowledge portion of the the SUMC’s investment in building students’ capabilities licensing examination were found in 14 (out of 24) indi- in these 2 domains. cators, and the correlations spread generally evenly In China, emphasis has been increasingly placed across the 3 domains (attitude, skills, knowledge). on physicians’ professionalism as well as clinical H uang et al. BMC Medical Education (2022) 22:76 Page 7 of 10 Fig. 3 -1 Competency growth by domain—attitude (A), skills (S), and knowledge (K)—among Year-2 to Year-6 students. 3–2. Competency growth by competency level—state (S), explain (E), apply (A), transfer (T)—among Year-2 to Year-6 students skills—dimensions vividly captured by the ASK-SEAT NMLE, so the sensitivity of the licensing examination as which measures multiple domains. The 24 indicators in an assessment process can be enhanced. the ASK-SEAT scale can thus serve as a more detailed During our research, we did locate a system in Ger- reference to assist fine-tuning and redesigning the many which was designed by Prediger et al. to assess the Huang et al. BMC Medical Education (2022) 22:76 Page 8 of 10 Fig. 4 Significant competency growth between adjacent years (Year-2 to Year-6) by indicator based on the Mann-Whitney U test competencies of medical students [14]. While both the scientifically and empirically grounded method of work- ASK-SEAT and the system developed by Prediger et  al. ing; and verbal communication with colleagues and are drawn from the framework of Miller and examine supervisors). The ASK-SEAT stresses, instead, on compe- competencies beyond knowledge, the two systems differ tencies specific to clinical medicine (Table 2). in a number of aspects. First, the system by Prediger et al. targets students of advanced grade years (i.e. Year-5 and Competency development continuum & discriminating Year-6 students in a 6-year curriculum which consists competencies of 2 years of pre-clinical and 3–4 years of clinical expo- Steady improvements in all 3 competency domains were sure) by simulating the 1st working day of a resident in seen across grade years, with an accelerated improve- a hospital. The ASK-SEAT, at least for the current study, ment in the domain of skills (Fig. 3-1). At the SUMC, the has been conducted among a broader group of students, curriculum of “Basic Clinical Skills” is taught to students including those in their earlier years of study (Year-2 to of Year 2 to Year 3. More importantly, students start Year-6). Second, the system by Prediger et  al. aims at a clinical clerkship in the 2nd semester of Year 4, before more fine-tuned and deeper assessment of competencies advancing to Year 5 when they are exposed to more clini- of a more targeted group of students. The ASK-SEAT is, cal practice on the rotation basis. From Year 6 onward, on the other hand, a standardized tool that requires less students start receiving resident training which will last time and resources to perform and can be administered for 3 years. Students’ increasing exposure to clinical prac- to a larger and more diverse set of students. Third, con- tice from Year 4 to Year 6 might have thus contributed to trasted with the ASK-SEAT which is an assessment scale, the accelerated growth trajectory in the domain of skills. the system by Prediger et al. contains a checklist of com- Furthermore, as shown in Fig. 3–2, the gaps also grew petencies and each indicator is measured using different narrower between the competency level of “apply” and instruments (e.g. questionnaire, case vignette). Fourth, the levels of “state” and “explain” from Year 3 to Year the system by Prediger et  al. focuses more on applying 6, indicating a faster improvement in students’ ability “generic” skills in a clinical setting (e.g. teamwork and to apply what they acquired. In an invited review pub- collegiality; structure, work planning, and priorities; lished in 1990 where Miller presented his framework of H uang et al. BMC Medical Education (2022) 22:76 Page 9 of 10 Pyramid [11], he noted that the higher tiers of competen- was expanded in the ASK-SEAT to include a competency cies (“Shows How” and “Does”) might “imply” the mas- level of “transfer”. This top layer of competency captures tery of the infrastructure level of competencies (“Knows” the spirit of the role of “scholar” declared at CanMEDS, and “Knows How”). The narrowing gap portrayed in the role of teaching, training, and mentoring expected of Fig. 3–2 appears to provide empirical evidence to support a practitioner in the GMC’s GMP, and the role of “edu- this reasoning. Students’ increasing ability to apply could cating patients, families, students, residents, and other be attributable to not only more hands-on opportunities health professionals” outlined in the ACGME’s compe- (through clinical clerkship and rotations), but also poten- tency framework. tially the cumulative mastery of “stating” and “explaining” As a tool that is less elaborate to implement (compared over time (gains from competency acquisition require to, for instance, the system by Prediger et  al.) and vali- time to come to full fruition). Follow-on research can fur- dated across multiple grade years, the ASK-SEAT can be ther validate these attributions. integrated into formative assessment of a diverse base of On the other hand, students also reported a lack of medical students to facilitate more frequent “check-ins” steady growth in 2 indicators: “applying the English lan- of students’ ongoing development through their years of guage” and “conducting scientific research & innovating”. study. The scale can be completed by students themselves Interestingly, these 2 indicators were also the discriminat- (self-assessment) or by other stakeholders with a vested ing competencies identified in an earlier study—compe- interest in medical education (such as instructors and tencies which differentiated the high-performer from the supervising physicians). typical-performer [15], although the discriminating indi- cators found in that study were derived only from Year-5 students (students receive a bachelor’s degree at the end Study limitations of the 5th year—the 1st milestone in their medical study). Due to time and manpower constraints, self-assessment Future studies can explore discriminating competencies was sampled from students of 5 grade years as proxies during different “milestone” years, for example, Year 3 to measure the competency development continuum. A (before students start receiving formalized clinical expo- longitudinal follow-up of same groups of students over a sure) and Year 8 (when students complete the full length longer period of time (as students gain more confidence of their study). The insights uncovered can be converted from additional coursework and clinical rotations) will be to actionable strategies to augment the current curricu- needed to confirm the findings from the current study. lum, so students can be better prepared to master not Secondly, there was no input collected from stakeholders only the clinical fundamentals but also capabilities that such as instructors, supervising physicians, and student will catapult them to becoming high-performers. peers to corroborate students’ self-assessment. Future research can also test the ASK-SEAT scale among students of advanced grade-years, particularly those who are more deeply immersed in the resident Conclusions training, to contrast and compare with the current find- The ASK-SEAT, a competency-based assessment scale ings derived from students in earlier years of their study. developed to measure medical students’ competency development, shows good reliability and structural valid- ity. For predictive validity, weak-to-moderate correlations Implications are found between Year-4 students’ self-assessment and Given the scant availability, in China and abroad, of their performance at the national licensing examina- standardized competency-based assessment measures tion (Year-4 students start their clinical clerkship dur- to gauge the progress of students majoring in clinical ing the 2nd semester of their 4th year of study). Year-2 medicine through their education, the development and to Year-6 students also demonstrate steady improve- validation of the ASK-SEAT scale offer valuable learn- ment in the great majority of clinical competency indica- ings. The ASK-SEAT is relatively straightforward and less tors measured, except in the indicators of “applying the time- and resource-intensive to implement, and can also English language” and “conducting scientific research & be modified in accordance with particular competency innovating”. requirements by individual medical education institutes. Its applicability among not only the Chinese medical stu- dents is supported by the scale mirroring the competency Abbreviations frameworks endorsed by governing institutes outside WHO: World Health Organization; CBME: Competency-based medical educa-tion; SUMC: Shantou University Medical College; NMLE: National Medical China and in developed countries (as referenced earlier Licensing Examination; EFA: Exploratory factor analysis. in Background of this report) and the broadly recog- nized Miller’s Pyramid. In the meantime, Miller’s model Huang et al. BMC Medical Education (2022) 22:76 Page 10 of 10 Acknowledgments 6. Frank JR, Snell L, Sherbino J, editors. CanMEDS 2015 physician compe- The authors would like to thank all the participants of this study. We would tency framework. Ottawa: Royal College of Physicians and Surgeons of also like to direct special gratitude towards Professor Junhui Bian—the former Canada; 2015. Dean of Shantou University Medical College (SUMC)—for his stewardship in 7. Sun BZ, Li JG, Wang QM. Zhong guo lin chuang yi sheng gang wei sheng guiding the research, as well as Beiyan Wu—Chief Physician in the Depart- ren li mo xing gou jian yu ying yong [construction and application of ment of Pediatrics at the First Affiliated Hospital of SUMC—and Yao Gong—a Chinese doctors’ common competency model]. Beijing: People’s medical physician in the Department of Rheumatology at the First Affiliated Hospital of publishing house; 2015. p. 97–102. Chinese SUMC—for contributing to the research concepts. 8. Liu Z, Tian L, Chang Q, Sun B, Zhao Y. A competency model for clinical physicians in China: a cross-sectional survey. PLoS One. Authors’ contributions 2016;11(12):e0166252. GX and SZ directed the project from conception to completion. LH and ZL 9. General Office of the State Council of the People’s Republic of China: designed the study. CC, ZH, and WZ drafted the questionnaires. LH, ZL, HX, ZH, Deepening the synergy between education and healthcare system to WZ and PG collected, analyzed, and interpreted the data. CC, ZH, WZ, LH, and further promote reforms and development of medical education in ZL drafted the manuscript. XH and YZ provided intellectual input for the revi- China. http://w ww.g ov. cn/z heng ce/c onten t/ 2017-0 7/ 11/ conte nt_ 52096 sion of the manuscript. All authors read and approved the final manuscript. 61.h tm. Accessed 10 Nov 2021. 10. Shi HC, Gong WJ, Zheng Y, Wang ZB, Wang JS. “5+3” Yi xue jiao yu yu Funding zhi ye yi shi zi ge fen jie duan kao shi gai ge bei jing xia yi xue ren cai This project was funded by the Humanities and Social Sciences Research pei yang mo shi de tan suo yu shi jian [reflection on the dinical medical Project, Ministry of Education (17YJA880107) and the Shantou University talents training model from both the reforms of“5+3”-oriented medical Medical College Student Innovation and Entrepreneurship Training Program education system and two-period national medical licensing examina- (20180022). tion]. Chinese J Med Educ. 2015;5:661–3. 11. Miller GE. The assessment of clinical skills/competence/performance. Availability of data and materials Acad Med. 1990;65(9 Suppl):S63–7. The datasets generated and analyzed during the current study are available 12. Global Standards for Quality Improvement: Basic Medical Education. Mid- from the corresponding authors on reasonable request. dlesex (UK): World Federation for Medical Education (WFME). Available at: https://w fme. org/ downl oad/ wfme- global- stand ards- for- qualit y- impro vement-b me/? wpdmdl= 831& refre sh=6 18bd 61fa74 0c16 365542 71. Declarations Accessed 10 Nov 2021. 13. Huang LX, Li ZH, Zhan WJ, et al. Yi xue bi ye sheng zong he su zhi sheng Ethics approval and consent to participate ren li ping jia liang biao de gou jian fen xi [the construction and analysis This study was approved by the Ethics Committee of Shantou University of medical graduates’ comprehensive quality evaluation competency Medical College (approval number: SUMC-2018-03). The study protocol and scale]. Chinese J Med Educ Res. 2021;20(01):66–70. survey contents satisfied the relevant guidelines and regulations. Informed 14. Prediger S, Schick K, Fincke F, et al. Validation of a competence-based consent was obtained from all participants. Participation was voluntary, and assessment of medical students’ performance in the physician’s role. BMC students consented to participate through completion of the questionnaires Med Educ. 2021;20(6):1–12. of the study. 15. Huang LX, Zhan WJ, Li ZH, et al. SEAT neng li mo xing zai yi xue bi ye sheng xing wei shi jian fang tan zhong de yun yong [application of SEAT Consent for publication competency model in medical graduate behavior event interview]. Not applicable. Health Vocational Educ. 2019;037(011):121–3. Competing interests The authors declare that they have no competing interests. Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in pub- Author details lished maps and institutional affiliations. 1 Department of Clinical Medicine, Shantou University Medical College, Shan- tou 515041, China. 2 Department of Pediatrics, East Carolina University and Vid- ant Medical Center, Greenville, NC 27834, USA. 3 Department of Microbiology and Immunology, Shantou University Medical College, Shantou 515041, China. 4 Department of Higher Medical Education, Shantou University Medical College, Shantou 515041, China. 5 Department of Public Health and Preventive Medicine, Shantou University Medical College, Shantou 515041, China. Received: 25 February 2021 Accepted: 25 January 2022 References Ready to submit your research ? Choose BMC and benefit from: 1. Mcgaghie WC, Miller GE, Sajid AW, Telder TV. Competency-based curricu- lum development on medical education: an introduction. Public Health • fast, convenient online submission Pap. 1978;68:11–91. • thorough peer review by experienced rese archers in your field 2. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: • rapid publication on acceptance theory to practice. Med Teach. 2010;32(8):638–45. 3. Frank JR, Snell L, Englander R, et al. Implementing competency-based • support for research data, including large and complex data types medical education: moving forward. Med Teach. 2017;39(6):568–73. • gold Open Access which fosters wider collaboration and increased citations 4. Swing SR. The ACGME outcome project: retrospective and prospective. • maximum visibility for your research: over 100M website views per year Med Teach. 2007;29(7):648–54. 5. Good medical practice. Manchester (UK): General Medical Council. Avail- At BMC, research is always in progress. able at: https:// www. gmc- uk. org/ ethica l-g uida nce/ ethic al-g uida nce-f or- docto rs/g ood-m edic al- pract ice. Accessed 10 Nov 2021. Learn more biomedcentral.com/submissions