Research methods
We conducted a literature research and two expert letter consultation rounds following the Delphi method [22] to establish entrustable professional activities for psychiatry residents in China. The study was performed in accordance with the latest version of the Declaration of Helsinki and approved by the ethics committee of Beijing Huilongguan Hospital.
Composition of the research group
The research group was composed of ten people, one with a senior professional title, four with a deputy senior professional title, and five with an intermediate professional title. One had a doctoral degree and nine had a master’s degree. Specifically, the team included the director of the education department, the ward director, and eight senior clinical front-line teachers, all with over ten years of teaching experience. The group members are all psychiatrists who have undergone two rounds of training in the early stages and are familiar with the EPA concept. The main team responsibilities included literature research, formulating the consultation form, expert selection, preparing an online Questionnaire Star version, distributing and recovering the consultation forms, index screening and revision, statistically analyzing the data, and writing this manuscript.
Literature research
We searched five electronic databases (Medline, CINAHL, Scopus, PsychINFO, and Web of Science) in January 2020 for publications from 2011 to 2020 using the following keywords: “entrustability and psychiatry” and “entrustable professional activit* and psychiatry,” retrieving 34 publications. We screened them for suitability based on their titles and abstracts, searching for articles on “confidence in professional behavior (EPA).” Twenty-eight papers remained after the first screening step. We assessed their suitability based on a full-text review, searching for publications with specific EPAs. Ten articles remained after this screening(Fig. 1). We limited the search to articles written in English, the type of literature is narrative.
The ten articles contained 61 EPAs, including duplicates. Following the 13 EPAs mentioned in a published guide from 2020 [19], we removed duplicates, summarized, and merged items, leading to the selection of 12 EPAs.
Preparing the expert inquiry form
Based on the literature review, referring to the 13 EPAs mentioned by the Association of American Medical Colleges Core Entrustable Professional Activities for Entering Residency, the 20 EPAs suggested by the Royal College of Physicians and Surgeons of Canada, and the rotation mode of standardized psychiatry training detailed in the Rules for Standardized Resident Training in Beijing (2013 version) [23], we developed a phase-based modular EPAs suitable for the three rotation stages, that is, the general, specialized, and mobile stages. The first draft expert consultation form for EPAs of psychiatry residents was formed after several discussion rounds within the research group and included 19 EPA indicators.
Selecting the consulting experts
We adopted an objective sampling method to select 44 experts from nine Grade-A psychiatry departments in Beijing, Tianjin, Shanxi, Heilongjiang, Henan, Zhejiang, Sichuan, and Yunnan Provinces. The expert inclusion criteria were: (i) bachelor’s degree or higher; (ii) a deputy senior professional title or higher; (iii) over ten years of clinical work and teaching experience; (iv) participated in the standardized training of psychiatry residents; and (v) able to complete at least two expert consultation rounds. All experts provided their informed consent for participation in this study.
Implementing the expert consultation
We distributed the consultation questionnaires as Questionnaire Star and conducted two consultation rounds, 2 to 3 weeks apart. In the first consultation round, The first part of the questionnaire is to introduce to the experts EPA concepts, the study background information, its purpose, and a preliminarily constructed evaluation index system and asked them to score the importance and familiarity levels of the indicators. We used a Likert 5-level scoring method, assigning scores from one point (very unimportant/unfamiliar) to five points (very important/familiar). The questionnaire also collected basic information on the experts, their degree of familiarity with the subject, and the basis for their judgment. We sorted, summarized, and statistically analyzed the returned expert opinions of the first round to generate the form for the second round of expert consultation, which was based on the results of the first round. We asked experts to evaluate the observability, authenticity, multiple competencies, effectiveness, repeatability, and feasibility of the indicators. Experts graded the entrustable level that should be achieved at the end of each stage and at the end of the course, referring to the five-level entrustable assessment scale [5] and Twelve tips to develop entrustable professional activities [20] proposed by ten Cate. Level 1 is “no permission to execute the EPA,” indicating that the resident’s knowledge and skills are not enough to implement the EPA and must be completed by the supervisor. Level 2 is “execute EPA under full supervision,” indicating that the resident could execute the EPA but required full supervision and guidance. Level 3 is “execute EPA under occasional guidance” indicating that the resident could execute the EPA, but the supervisor needs to respond to the resident’s needs, provide timely help, and review the outcomes. Level 4 is “execute EPA independently, without supervision,” indicating that the resident can work independently, knows the risks, and can perform safely without supervision. Level 5 is “guide others to execute EPA,” indicating that the resident can supervise others when implementing the EPA. We summarized and statistically analyzed the returned expert opinions of the second round. As the statistical analysis results showed consistent feedback results after the two expert consultation rounds, the consultation ended.
Index screening criteria
We calculated the mean and coefficient of variation (CV) of each index following a expert opinions. The index screening criteria were importance score > 3.75 and CV < 0.25. The research group revised the EPAs following the questionnaire results and expert opinions.
Expert enthusiasm, authority, coordination, and concentration
The effective questionnaire recovery rate expressed the expert enthusiasm degree.
The authority coefficient, determined by the experts’ judgment basis and familiarity degree coefficients, expressed the expert authority degree. We used the following formula: authority coefficient = (familiarity degree coefficient + judgment basis coefficient) / 2 [24]. Values above 0.7 indicated good expert authority.
The CV and Kendall harmony coefficient expressed the degree of coordination among the expert opinions. The smaller the CV, the more consistent the expert opinions. The greater the Kendall harmony coefficient, the better the coordination among the expert opinions.
The arithmetic mean and the full score rate reflected the concentration of expert opinions.
Statistical methods
For statistical analysis, we used IBM SPSS Statistics for Windows, Version 26.0 (IBM Corp., Armonk, NY, USA). The measurement data are expressed as means (standard deviations), and counting data are expressed as frequencies and percentages. The mean, full score ratio, and CV described the index scores. We considered differences with P < .01 to be statistically significant.
