Evaluating Prospective Clinical Internship Sites

David J. A. Dozois, University of Calgary, Calgary, Alberta

Paper presented at the Canadian Psychological Association Annual Convention, Toronto, Ontario, June 1997

Abstract

The selection of a clinical psychology internship is one of the most important and exciting aspects of graduate training. Too often, however, the decision-making process is confusing and anxiety provoking. Although several books and articles have been written to help graduate students obtain a desired internship and satisfy their professional goals, other considerations of applicants have only recently been addressed in the literature (Gloria & Robinson, 1994; Stewart & Stewart, 1996b). This presentation will focus on how applicants can ensure that their choice of internship placements will maximize their professional, practical and personal goals. The paired-comparison ranking technique (Stewart & Stewart, 1996a) will be reviewed as a new strategy for choosing an internship and minimizing decision-making biases. A step-by-step discussion of this method will include 1) selecting criteria (e.g., quality of supervision, special training, breadth of population, choice of rotations, location, finances, work opportunities for one's partner); 2) establishing the importance of each criterion; 3) choosing sites for comparison; 4) evaluating internship sites on the basis of predetermined criteria; and, 5) interpreting the results and determining the optimal internship setting.

Introduction

Several articles and books have been devoted to the topic of applying for predoctoral internships in clinical psychology. This literature has addressed a plethora of issues and advocated a number of strategies to help applicants obtain quality internship placements. Until recently, however, there has been a paucity of conceptually- or empirically-based articles which have focused on the perspectives of internship applicants, their professional, personal and practical goals, or the decision-making processes they use to determine which internship placement is optimal (Gloria & Robinson, 1994; Stewart & Stewart, 1996a, 1996b).

The majority of papers have emphasized training concerns (e.g., whether or not there are shortages of internships, and how to deal with this potential problem; e.g., Clay, 1997), issues regarding the standardization of application forms (Holaday & McPhearson, 1996) or compliance with APPIC rules (Constantine & Keilin, 1996), the criteria that internship directors employ in their selection of an intern (Gloria & Robinson, 1994; Lopez, Oehlert, & Moberly, 1996; Tipton, Watkins & Ritz, 1991), and a variety of techniques to assist students with the application process (Barnes, 1982; Brill et a., 1985; Fitzpatrick, 1996; Grace, 1985; Mellot, Arder, & Cho, 1997; Mitchell, 1996; Plante, 1996). The 1995 text entitled, Everything you need to get a psychology internship illustrates this lack of attention to what students should look for in an internship setting. Although this book serves as an excellent resource for internship candidates, there is no mention of the factors that influence an intern's decision-making or what variables might be worth considering. Moreover, as Stewart & Stewart (1996b) correctly assert, most of the "how to" guides for securing an internship focus on the best-fit in terms of professional goals, virtually ignoring the subjective, emotional and practical dimensions of the decision.

Taking care of one's career in all respects (i.e., professionally, personally and practically) is extremely important for the longevity and productivity of his/her career, and overall satisfaction. The main objective of this presentation is to provide some information that I hope will help intern applicants to find the optimal internship setting. After reviewing some of the criteria that interns have used in their selection of an internship, I will discuss strategies for gathering information. Following this, I will present the paired-comparison approach (Stewart & Stewart, 1996a) as a technique for reducing decision-making biases when ranking internship sites.

* Establishing Criteria

Criteria Used by Other Interns

A few studies have examined which variables are important to intern applicants in the determination of their ideal placements (Burnstein, 1981; Gloria and Robinson, 1994; Matthews & Walker, 1997; Stedman et al., 1995). Tedesco (1979; cited in Stedman et al., 1995) examined 22 variables influencing an applicant's decision to apply to a given internship program and found that APA accreditation, geographical location, focus on adult work, the theoretical orientation of supervisors, and multidisciplinary settings to be among the highest rated variables. Burnstein et al. (1981) found that geographical location was rated highest, followed by program diversity, theoretical orientation, type of faculty and reputation. More recently, Stedman and his colleagues (1995) surveyed 148 predoctoral internship applicants 1 month after Uniform Notification Day. Participants were asked to rate their first, intermediate and last choice internship program on 25 variables in order to understand which variables go into a most desirable internship choice. The results indicated that 8 variables clearly differentiated between an applicant's first, intermediate and last choice: the match between interests and program, an affective "gut" feeling, the opportunity for specialty training, the breadth of population, the opportunity and desire to reside in the area following internship, and the quality of supervision were the variables that distinguished most between first- and intermediate-choice decisions and between top and last choices.

Gloria and Robinson (1994) found that intern applicants decision to select and accept internship positions was based primarily on the focus of professional training and geographical location. Other reasons influencing the choice of a site included one's personal life, problems with relocating the family, and not wanting to disrupt a spouse's career. Clientele, reputation, supervision, APA accreditation and stipend did not emerge as important factors in the overall decision.

Specific Criteria

Location.

Several investigations of internship applicants have identified geographical location as one of the most important considerations. In addition to being rated among the top criteria, location also relates to one of APPIC's rationales for why so few spots appear to be available. Aside from the problems associated with academic programs producing too many internship candidates each year, many internship applicants (for a variety of legitimate reasons) feel constrained to continue training in their own city. Another difficulty is that most students are attracted to the large city centers for internship, making the ratio of available applicants to positions seem worse than it really is (Clay, 1997).

There are many things to consider in terms of location, including: a) responsibility to one's spouse or family (e.g., will it be possible for your spouse to move with you?; what locations maximize your partner's needs as well?); b) unique training opportunities (e.g., having had several practice in your home city may make taking an internship elsewhere seem more appealing); c) the extent to which one may need to remain in contact with his/her academic program during the year (e.g., if one of your goals is to complete your dissertation, then having internship in close proximity may be preferable; conversely getting away to a more socially isolated spot may help to get the work completed; d) opportunities that a particular location might offer (e.g., you might like to live in a particular geographical region once you have completed your training); and, e) the financial ramifications of a given location (e.g., cost of interviewing, cost of relocating, cost of living, opportunities for employment for yourself and your partner). In order to use location as a criterion for ranking internship sites, it will be important to decipher which of these (or other) specific aspects of location are important to each individual.

Future Job Prospects.

Related to the issue of location is that of future job prospects. Some relevant questions to ask may include "will this internship experience put me in a good competitive standing for a practice or research position following graduation?"; "what is the job market like in this city?"; "have past interns from this placement been able to find work?"; "will this internship prepare me for program-based management?"; "will I be trained in empirically-validated (or evidence-based) therapy as well as other modalities so that I can be in a good position for the future trends of psychology and third-party payers?"

Additional Training/ Specialty Training.

Another important consideration relates to whether the internship site in question provides additional training in unique areas of practice. One may want to ask about the specific competencies that graduates of different internship programs develop. For which types of career settings are different internship programs best preparing their interns? These questions not only relate to skills-development but also to marketability. Tipton, Watkins and Ritz (1991) found significant differences among internships on a number of variables including child-family therapy, special population applications, psychological assessment, psychopathology, short-term therapy, long-term therapy and consultation. These researchers also found that internship programs differed on how well their interns were prepared for various career settings. Thus, it is pertinent to understand what the potential internship site offers in terms of training, and figure out how well that maps on to your own professional goals.

Two factors currently affecting internship training programs (especially in the U.S.) is the proliferation of empirically-validated treatments (EVTS; Chambless et al., 1996; Crits-Christoph, Frank, Chambless, Brody, & Karp, 1995) and the concern over health care costs (Koch, 1997). Due to these factors, it seems very plausible that future training will require such skills as program evaluation, high level consultation and the treatment of more complicated cases.

The APA Task Force on Promotion and Dissemination of Psychological Procedures published its 1995 report which listed what it considered to be either well-established or probably efficacious treatments. Although there is debate as to the pros and cons of this movement, it is clear that demonstrating therapeutic efficacy and the use of EVTS are entirely consistent with managed-care. It is quite plausible that insurance companies and other third-party payers will someday fund only those treatments that have evidence to support their efficacy. Thus, students would be wise to assess the extent to which various internship placements train interns in the use of EVTS. This is not to suggest that you should only obtain training in these modalities, but to recommend that you receive at least some training in these areas.

Rotations.

Generally all settings offer training in work with adults; however, only 32% of programs provide training with adolescents and even fewer settings offer training with child, geriatric, or forensic populations (Howes et al., 1996). Therefore, it is important to be clear on the type of population you would ultimately like to work with and seek out those opportunities.

Internships also usually provide a variety of assessment and therapy experiences. Most internships provide training in general clinical, personality and neuropsychological assessment, while very few provide opportunities for competency, forensic or custody assessments. Internships also usually provide a combination of long- and short-term rotations in individual, group and family psychotherapy yet, the emphasis on clinical duties, administrative responsibilities and research opportunities can vary across programs and across rotations within programs. Obtaining as much detail as possible about the various rotations offered (e.g., inquiring about the theoretical orientations of rotations and supervisors) is one way to ensure that one's training needs will be met (Grace, 1985).

Intern applicants may also want inquire about the manner in which supervision is conducted on rotations. One might ask, for instance, how many individuals will be supervising him/her, what their styles are (e.g., video, audio, direct), and (to ensure availability) how many of these supervisors are full-time employees at the setting (Romans, et al., 1995).

Diversity of Experience.

Diversity of experience and clientele tend to emerge as important factors in the decision-making processes of interns. There are two main reasons why I believe that diversity of experience is an important enough reason to seek clinical internships outside of your own university city. First, depending on your academic program, you may have already had 5 or 6 practicum placements throughout the city by the time you reach internship. Unless familiarity is comforting, one might find a novel experience more challenging and invigorating. Second, going outside of one's home-base during internship may provide an opportunity to be exposed to a different sample of individuals (cf., Mintz et al., 1995).

Stipend.

The stipend, though not one of the highest rated variables in the decision-making process, is important. Howes and her colleagues (1996) recently surveyed the internship directors from several settings and found that the average number of funded positions per internship site in Canada is 2.28 with an average stipend of $20,269 (Howes, Allis, Wilson, Ross, & Louis, 1996). On a province by province comparison, Manitoba offersSalaries have increased somewhat in BC, Alberta and Newfoundland; decreased in Manitoba, Ontario and Nova Scotia; and, like the time-zone, have not changed in Saskatchewan (see the CCPPP and APPIC internship directories for specific salaries). Apart from the stipend itself, you may also want to factor in the differences in the cost of living among various locations, financial support for relocation, conference travel, holidays, medical and dental coverage, and other fringe benefits that may be offered

CPA/APA Accreditation.

Although the literature is inconsistent as to whether intern applicants use APA and CPA accreditation as a decision-making criterion, applicants are frequently concerned about such approval. Currently there are 45 internship programs listed as members of the Canadian Council of Professional Psychology Programs (CCPPP) (Howes et al., 1996). Sixty percent are accredited (13% by CPA, 9% by APA and almost 38% by both organizations). When employing this criterion, one should be cognizant that APA and CPA accreditation only ensures a minimum degree of standards and does not guarantee that a given program will meet your individual training needs (Grace, 1985).

* Collection of Data

One excellent resource for intern applicants is an article by Grace (1985) in which he provided several important tips for evaluating internship sites. Grace argued that the intern applicant "should not accept on faith that the internship will meet his or her needs or even that it is primarily designed to do so" (p. 475). The process of evaluating internship settings is akin to conducting a good clinical assessment (Grace, 1985). Learning all that one can from multiple sources of data will increase one's ability to make effective decisions. Some of these multiple sources include the program brochures, the CCPPP and APPIC directories, former interns, academic faculty and staff from the internship site.

Internship brochures

Internship brochure descriptions are quite informative, especially at the beginning of your search. The disadvantage of brochures is that they tend to be very similar and, after reading 3 or 4, an individual can start to feel as though he/she is reading the same description again and again (Schmidt, 1989). Applicants should not assume that all of the information presented in program descriptions is accurate (Grace, 1985). Given that many programs have undergone restructuring over the last couple of years, due to federal and provincial cutbacks (cf. Alden, Mothersill, Steffy, et al., 1996), a program description may change by the time one reaches the interview stage or the start of his/her internship. When arriving for their interviews, applicants might wish to include questions about the stability of the current structure and whether there are any changes to the details listed in the brochure.

Informal Interactions

A worthwhile way to gather information and informally evaluate internship sites is to attend conferences and ask questions. The behavior one observes during conventions can provide a wealth of data. Most internship directors are willing to take a few minutes to talk and it may be helpful to note who gets back to a request for information promptly, who acted friendly and collegial and who appeared not to have the time or inclination to speak with students. These less formally observed behaviors can suggest the type of behaviors you may be able to predict would be evident during your internship (Tolin, 1997).

Current and Past Interns

Speaking with current and past interns can also be highly informative. Inquire as to where the program draws its interns (e.g., do they come locally or from far away, why?; If they are all from out of town, why is that? Is it because the local students do not respect the internship setting?). When talking with interns, it is important to remember that some sites use their current interns in the selection process and may not be very candid with an applicant. Asking for the names, phone numbers and email addresses of past interns can help to bypass this difficulty (Stewart & Stewart, 1996b).

Academic Faculty

As Grace (1985) points out, the information one can glean from his/her advisors and other faculty should not be underestimated. Through their connections with other colleagues, faculty members often have a very good idea about the quality of various programs and supervisors.

The Site Visit (Interview)

The site visit and interview is a vital means of collecting data. Aside from helping the internship director to determine an applicant's suitability for their program, the intervits. During the interview process, applicants should try to ensure that they have the opportunity to meet with specific supervisors rather than just the general mass of staff. If one has the chance, it may also be informative to ask some of the other disciplines (e.g., social workers) how they perceive the psychology department.

Approach the site by listening for what is there and what is perhaps not there. Do interactions with the internship staff feel open and candid or defensive and evasive? Pay attention to how the faculty interact with you and with each other (Stewart & Stewart, 1996b). After the interview, take some time to record observations and perceptions. This summary can be important later on, especially if you have gone through several interviews (Matthews & Walker, 1997).

Ranking Selection Criteria

After examining different sites, it is important to review each of their features and compare them systematically on the criteria that are individually important (Stewart & Stewart, 1996b). The first step in doing this is to determine the criteria that will be used for ranking internship placements. These criteria should be consistent with one's professional, practical and personal needs and goals. After this, it is helpful to rank order one's criteria and internship sites.

Although there are several ways in which to rank criteria and internship sites, the paired-comparison approach developed by Stewart and Stewart (1996b) is advantageous because it helps to minimize possible decision-making biases (cf., Tversky & Kahneman, 1981). For example, an incredibly interesting and stimulating conversation with one internship director might positively bias one's opinion of the placement.

The first step in using the paired-comparison approach is to prioritize the criteria. Stewart and Stewart (1996a) recommend writing the site features and the criteria that are important on index cards. Then, in a pairwise manner, each criterion is compared against every other criterion individually (i.e., criterion 1 & 2, 1 & 3 etc.), and a tally of the results is kept. The summary score for each criteria then determines the order of one's ranking priorities.

* Ranking Sites

The next step is to create a grid on a piece of paper and write the selection criteria at the top of the columns, with the sites on the left of the grid. Each site is then ranked, starting with the most important criteria and ending with the least important (see Stewart & Stewart, 1996a). The final step involves interpreting the grid. This can be conducted in a number of ways - summing the cells in each row will provide an overall ranking of the site that meets most criteria. One can also interpret the grid by paying particular attention to the first 2 or 3 columns, as these are the criteria that were deemed most important. A weighting procedure capturing both the summary score and the relative ranking of the criteria list might also be helpful.

To summarize, Stewart & Stewart's (1996a) pair-comparison ranking technique requires that students evaluate all possible pairs of sites on the criteria they select. Although this process is not much different from a series of "just noticeable differences", it is a valuable exercise which can provide applicants with a stable and objective set of decisions regarding which internships optimize their professional, personal and practical goals.

References and Suggested Readings

* Alden, L., Mothersill, K., Steffy, R., McIlwraith, R., Steinberg, R., McMullen, L., & Tasca, G. (1996). Priorities for professional training in the 90s: Perspectives of directors of psychology training programs. Canadian Psychology, 37, 221-228.

Barnes, B. (1982). Do intern applicants really need a survival guide? Professional Psychology, 13, 342-344.

Brill, R., Wolkin, J., & McKeel, N. (1985). Strategies for selecting and securing the predoctoral clinical internship of choice. Professional Psychology, 16, 3-7.

Clay, R. A. (1997, April). Concerns about internships lead APA to conduct survey. The APA Monitor, 28 (4), 51.

Clements, C. B. (1992). Training in human service management for future practitioner-managers. Professional Psychology: Research and Practice, 23, 146-150.

Constantine, M. G., & Keilin, W. G. (1996). Association of psychology postdoctoral and internship centers' guidelines and the internship selection process: A survey of applicants and academic and internship training directors. Professional Psychology: Research and Practice, 27, 308-314.

Fitzpatrick, C. M. (1996). The APA internship application countdown: What you can start doing now to prepare. The Newsletter of the American Psychological Association of Graduate Students, 8 (3), 8-9.

Gloria, A. M., & Robinson, S. E. (1994). The internship application process: A survey of program training directors and intern candidates. The Counseling Psychologist, 22, 474-488.

* Gloria, A. M., & Robinson, S. E. (1994). The internship application process: A survey of program training directors and intern candidates. The Counseling Psychologist, 22, 474-488.

Grace, W. C. (1985). Evaluating a prospective clinical internship: Tips for the applicant. Professional Psychology: Research and Practice, 16, 475-480.

Holoday, M., & McPhearson, R. (1996). Standardization of APPIC predoctoral psychology internship application forms. Professional Psychology: Research and Practice, 27, 508-513.

Howes, J. L., Vallis, T. M., Wilson, A. R., Ross, M. A., & Louisy, H. J. (1996). Predoctoral internship training in Canada - I: Internship settings and supervisory issues. Canadian Psychology, 37, 173-179.

Koch, W. J. (1997). Psychology internship training at Vancouver hospital and health sciences centre and the university of British Columbia. The Behavior Therapist, 20, 59-62.

LaFromboise, T. D., & Foster, S. L. (1992). Cross-cultural training: Scientist-practitioner model and methods. The Counseling Psychologist, 20, 472-489.

Levinger, C., & Schefres, I. (1995). Everything you need to get a psychology internship. Los Angeles: Internship Publishers.

Lopez, S. J., Oehlert, M. E., & Moberly, R. L. (1996). Selection criteria for American Psychological Association-accredited internship programs: A survey of training directors. Professional Psychology: Research and Practice, 27, 518-520.

Matthews, J. R., & Walker, C. E. (1997). Basic skills and professional issues in clinical psychology. Needham Heights, MA: Allyn & Bacon.

Mellot, R. N., Arden, I. A., & Cho, M. E. (1997). Preparing for internship: Tips for the prospective client. Professional Psychology: Research and Practice, 28, 190-196.

Mintz, L. B., Bartels, K. M., & Rideout, C. A. (1995). Training in counseling ethnic minorities and race-based availability of graduate school resources. Professional Psychology: Research and Practice, 26, 316-321.

Mintz, L. B., Rideout, C. A., & Bartels, K. M. (1994). A national survey of intern's perceptions of their preparation for counseling women and of the atmosphere of their graduate education. Professional Psychology: Research and Practice, 25, 221-227.

Mitchell, S. L. (1996). Getting a foot in the door: The written internship application. Professional Psychology: Research and Practice, 27, 90-92.

Perri, M. G. (1996). Thirteen recommendations for internship applicants. The Behavior Therapist, 19, 154.

Plante, T. G. (1996). Ten principles of success for psychology trainees embarking on their careers. Professional Psychology: Research and Practice, 27, 304-307.

Rodolfa, E., Haynes, S., & Kaplan, D. (1995).. To apply or not to apply: That is the intern applicant's first question. Professional Psychology: Research and Practice, 26, 393-395.

Romans, J. S. C., Boswell, D. L., Carlozzi, A. F., & Ferguson, D. B. (1995). Training and supervision practices in clinical, counseling, and school psychology programs. Professional Psychology: Research and Practice, 26, 407-412.

Schmidt, J. P. (1989). Selecting an internship: Some guidelines. AABT workshop presentation.

Stedman, J. M., Neff, J. A., Donohoe, C. P., Kopel, K., & Hays, J. R. (1995). Applicant characterization of the most desirable internship training program. Professional Psychology: Research and Practice, 26, 396-400.

Stewart, A. E., & Stewart, E. A. (1996). A decision-making technique for choosing a psychology internship. Professional Psychology: Research and Practice, 27, 521-526.

Stewart, A. E., & Stewart, E. A. (1996). Personal and practical considerations in selecting a psychology internship. Professional Psychology: Research and Practice, 27, 295-303.

Tipton, R. M., Watkins, C. E., & Ritz, S. (1991). Selection, training and career preparation of predoctoral interns in psychology. Professional Psychology: Research and Practice, 22, 60-67.

Tolin, D. F. (1996). On fine-tuning the search. The Behavior Therapist, 19, 153-154