Introduction to Program Evaluation:

• Mention the timing in June for APPE students.
• Six months away from most deadlines for residencies, fellowships, and graduate programs.
• Previous episode discussed setting aside protected time for postgraduate training.
• Today's deep dive: Program Evaluation.

Self-Reflection:

• Importance of self-reflection before diving into program evaluation.
• Questions candidates should ask themselves.
• Decisions to make early: geographic limitations, program structure preferences, class size preferences, mentor expectations, and specialization preferences.

Basics of Program Evaluation:

• Where to find programs: ASHP Residency Directory, ACCP Residency, Fellowship, and Graduate Program Directory.
• Specialty residencies may have their own directories.
• Filters and parameters for narrowing down the search.

Starting the Search:

• Use geographic filters and program type filters.
• Consider preferences for program structure and class size.
• Example of a combined PGY1, PGY2 program.

Using Program Websites:

• Transition from basics to self-reflection.
• Importance of program evaluation as an ongoing process.
• Program directories as a starting point.
• Moving to program websites for more in-depth information.
• Importance of looking beyond program descriptions.
• Program websites often provide detailed information about rotations, staffing, mentors, and more.
• Checking for updated information on program websites.
• Exploring teaching certificates, research opportunities, and other unique features.
• Utilizing information about current and past residents.

Continuation of Using Program Websites:

• Investigating preceptor qualifications.
• Checking for teaching certificates and research opportunities.
• Consideration of the potential for a residency to lead to a job opportunity.

Sweet Spot in the Program Evaluation Process:

• Discussion of the stepwise process.
• Sweet spot for self-reflection, initial program search, and in-depth website evaluation.
• Emphasis on feeling comfortable with each step.

Tracking Information:

• Importance of tracking information during the program evaluation process.
• Utilizing Word documents, Excel sheets, notepads, or specialized tools.
• Announcement of an upcoming program evaluation tool from the hosts.

Check out our website and sign-up to join the SASO (separate and stand out) squad. Check out our blog. If you like the show, support us by telling your friends or colleagues about it. You can also support us by clicking the coffee button on the website and buying us a cup of coffee or getting yourself some of our premium merch.

Follow us on twitter @PGPharmacist or on Facebook, Instagram, or LinkedIn @ThePostGraduatePharmacist. What questions did we not answer? What did you think of the show?

Music | "Sweet" by LiQWYD

Watch: https://youtu.be/eIYlaVPdNYM

License: https://www.liqwydmusic.com/how-to-use

Download/Stream: https://hypeddit.com/link/un7fp7

  Hi, this is Taylor Stuber. And this is Sean Smithgall. We are both clinical pharmacists and faculty members at Auburn University Harrison School of Pharmacy, and we are your hosts for The Postgraduate Pharmacist.

On the Postgraduate Pharmacist, we focus on preparing and obtaining postgraduate training positions from current events to expert advice. You'll have UpToDate content related to postgraduate training. New episodes are released every other Monday, so don't forget to like or subscribe. Follow us on Twitter at pg Pharmacist or LinkedIn at the Postgraduate Pharmacist for additional discussions or insights into our topic.

It's June, APPE students are entering their first or second rotation, and we're a little over six months away from most deadlines for residencies, fellowships, and graduate programs. A few episodes ago, we talked about setting aside reoccurring protected time for postgraduate training. Today, I want to take a deep dive into one of the things you can do during that time.

Program evaluation. Yeah, and program evaluation is a topic that might seem a little overwhelming because there's really no limit to how many programs you can investigate or how much information you can investigate about that program. There's thousands of them out there, so I think it's important to find that sweet spot.

And that's what we'll talk about today. I agree. I think it's easy to define what is too little. And it's easy to define what's too much. Like if a student's like, well, what's not enough? You know, you and I could probably answer that. And if they were like, what's spending too much time doing program evaluation?

We could say, well, here's an example. It's spending too much time. But I think it's, there's so much middle ground. Like you said, sweet spot. It's hard to find that sweet spot, that middle ground. Yeah. And we'll talk about a few things to help guide your search a little bit. Cause I think once you have parameters that you're looking for, I think it can be helpful.

Should we start with the basics first? Yeah, I think we should start by taking a step back from the basics and talk about self reflection a little bit before we jump in and talk about the. Program directories and everything that goes along with that. Yeah, I like that. So what questions should candidates be asking themselves and what decisions should they make now versus later?

Do you think? Yeah, I think it's a good time to start thinking about what is going to be important to you in a program. What types of things are you going to be looking for to guide your initial search for programs? Examples would be, are you geographically limited? Do you want a certain region of the country?

Do you not mind traveling further? What are you looking for in terms of the program itself? Do you want it to be more customizable with more flexibility in the types of rotations that are offered? Or do you want something that's a little bit more structured? Do you want to be part of a large or small residency class size with there being advantages and disadvantages to both?

What types of things are you looking for in your mentors or preceptors? Other things would be like, do you want to do a PGY1? with a PGY 2 in a certain area? Do you want to do a combined PGY 1, PGY 2? Do you want to just do a PGY 1 that's specialized or something that's broad? So these are all things that you should be asking yourself and I would encourage you to write these down to identify those things that are going to be important to you when you're Go to evaluate programs and Taylor, you did a combined PGY one PGY two pharmacotherapy.

Did you do that this preliminary step? Did you ask yourself that come up as one of the things you were interested in before you started looking at programs? Yeah, that was definitely something that I did look at. So I was not super geographically limited, but I didn't want to go somewhere that was. Like 20 hours away, for example.

And I'd heard about these combined PGY1, PGY2 programs through one of my mentors. And I was very interested in them once I did more research into that. So that's something that went into my search when I was thinking about, okay, I want to look in a, maybe an eight hour radius and which programs are going to be available in that area.

So that's something that I did. Think about when going into this, I had a similar conversations. I was actually engaged to get married, but my wife had a job and she and my family lived where she lived. So we had the discussion about would she be okay if I went away for a year to a place to do the residency and it was, that was okay.

We knew that was gonna be hard, but I was gonna be so busy with the residency, it went really. Make a big difference for me, it would make more of a difference for her. So it was ultimately her choice to allow me to go that far away. So I looked at, I, I did both. I did geographically restricted 'cause I wanted to stay in Tennessee or around Tennessee.

But I also looked at some programs outside since it was gonna be just that one year experience. 'cause my plan was to do a PGY two and to come back. And with my PGY two, I knew I would be regionally restricted. So it was kind of like, well, I won't be regionally restricted for my first one, but I will be regionally restricted for my second one, since that is more likely to be where I would stay on or, or have a position.

Yeah. And I think this just kind of highlights that every situation is going to be unique and what is important to you might not be as important to someone else. So. Make sure that you're thinking about those things. And if you have significant others or certain family members that you need to stick around, or if you need to, like you said, go away for a year, have those conversations now, rather than when you're applying and you're like, Hey, I've got an interview, uh, it's up in Washington state and we're in Alabama.

So try those conversations now. Yeah, exactly. It's a huge time waster. If you don't have those conversations and you jump on, like you were just saying with Washington state, you jump onto that directory and you start looking at programs all across the country. I mean, you're going to, you're just going to waste so much time if you don't have any desire to be there.

You know, you should still look at programs outside of your region, maybe once or twice just to get an idea of like, like a comparator thing. You know what I mean? Yeah. What else is that? What else is out there? What are they offering in other states? So you're not, you're not just like in your own state and that's all you ever see, but you don't want to spend 20 hours looking at programs across the West coast when you have no interest in, or you're going to have no real, real practical way to go to the West coast.

Absolutely. And so now that we've kind of covered the self evaluation piece a little bit and the importance of identifying what's important to you in a program, let's talk about some of the other basics. Where do you find these programs? So after you've had that conversation, the place to go next would be the directories.

There's two main ones that when I talk to students, that's why I usually tell them about it's the ASHP residency directory. You have ACCP's residency, fellowship, and graduate program directory. So it's going to have fellowships and PhD programs in there, but that's not all. There's some specialty residencies that have their own directories.

Some of these programs will exist within those larger directories, but some organizations, and the one that comes to the top of my head is like AMCP, the managed care organization, they have a managed care listing for all their programs. So it's possible that you might find even unaccredited programs in some of these organizations, specialty organizations, because that's where most likely those unaccredited programs are gonna advertise for their, uh, their unaccredited or pre accredited or fellowships or things like that.

So with these specific tools, how do you narrow down. So once you have those questions, like say you gave the geographic thing, say you, you have like a 300 mile radius, let's just use central Alabama. That's, that's our most listeners come from, most of our listeners come from Alabama, believe it or not, if you have Alabama, you can look at Mississippi, Georgia, Tennessee, and even Florida because of the panhandle.

So let's say you have like a 400 mile radius that encompasses like the little bit of those States. Sometimes you can even do things like that. Pick the Southeast. So the ACCP will allow you to pick the Southeast, which would kind of encompass those States. And you could look at programs within the Southeast region.

So with ASHP, it's by state. So, but you can one at a time, look at Alabama, Mississippi, Tennessee, Georgia, Florida, and then evaluate those programs. So there's filters that you can do. Like geographical filters, there's filters for the program type, PGY 2 combined. And then there's specific type of program.

So you can do pharmacotherapy, ambulatory care, managed care. So you can filter it down. And that's why, going back to what you were saying earlier, Taylor, I mean, that's why you've got to, you got to have these ideas in place because you're going to spend a. So much time and get so overwhelmed when you don't apply the right filters, just with the number of programs that are going to come out and how you start looking at all those programs.

Yeah. And I think once you've kind of identified those through those filters, then you can, they'll usually have a brief synopsis of the program right there on that. directory that you can look at to answer some of those other questions, such as how many co residents would I have if I went to this program.

And so that can help you narrow down your search even further before you jump into their program website and start reading everything about that program, that hospital system, or, or what have you. Would you say that that description is enough? Like if I was a candidate and I was creating my checklist of what I find about a program, so whether or not I like them, do you think I would find everything I need in that program description?

I think initially when you're filtering out programs that you want to maybe talk to at showcases or mid year or find out more information about, I think that those descriptions are helpful, but you know, you're not going to get a full sense of the preceptors, the RPD. What their personalities are like until you actually go and talk to them.

So I'd say initially at least, yes, but I wouldn't use just that to kind of make your final decision on programs to apply to. So let's stop there. Cause I have a question that's burning a hole in my pocket. I want to get to some PGP trivia. Let's do it. So, you know, we're, you know, I got two girls. So Disney's become like a staple in our lives.

I know you're a Star Wars fan. Are you a Disney? Obviously, they acquired Star Wars, so it's a separate conversation. Are you a Disney fan, too? Yeah, we've got Disney plus subscription in our household. Not to watch the Mandalorian or Marvel shows. Well, that's part of it. That's part of it. We now have a newborn, so she kind of.

She's, she's not really watching TV or anything yet, but, but I know she will soon. So, but you're on the clock then to go, uh, for your first Disney trip. It's coming, right? It's right around the corner. I'm not looking forward to it, but I know it is. We go to Disney. We love Disney cause it's, it's so much fun.

And obviously Disney doesn't promote this podcast. If they did, we'd be rich, but, um, we, uh, we love Disney trips, uh, and. The girls absolutely, their favorite park at Disney World is Magic Kingdom. Have you ever been to Magic Kingdom? Yes, it's been years since I've been there, but I have been to Magic Kingdom.

Okay, so you know a little bit about Magic Kingdom. I'm gonna give you multiple choice. What was the first ride that was at Magic Kingdom? Your options are, It's a Small World, with the lovely song. The Prince Charming carousel, the one that's right as you go through the castle, the carousel that's sitting there.

Which, Small World's only just a few feet away to the left. Or is it Space Mountain, which is the indoor, pitch black, futuristic rollercoaster ride? Yeah, yeah, so I'm a big fan of Space Mountain. That's pretty much the only ride I like in that park, specifically. And I'm pretty sure that it was the first ride.

Yeah, the company. Yeah, I didn't know it was that old. That's that's Addie's favorite ride. Every time we pass it, she wants to ride the carousel. Shows you how much I know about the magic kingdom of Disney. Oh, just you wait. You'll be an expert. I've got kind of a layup question for you to try to get you back at least competitive in this Trivia, um competition, but so do you ever play darts?

I do play darts Okay, so this is even easier for you. Maybe I don't play competitive darts Well, I don't either but you know, sometimes it's fun to throw a few darts here and there so it's not gonna be multiple choice Because I think you'll get it, but so what on a dartboard, if you're looking at a dartboard, what is the number that is directly across from the number one on the dartboard?

I thought you were gonna ask me how many points was a bullseye. Would you get that right? Uh, I don't know. I thought it was, um, I thought it was 50, but it might be 40. I might be thinking of something else. Uh, I think it's 15 is the number that's across from the one is 15. Is that your final answer? It's by you saying it like that, it's wrong.

But yes, that's my final answer. It's 15. No, it's not. It is 19. Actually, I was pretty close. Yeah, but you don't get. Partial credit. I don't get partial credit now. So if you so if you're looking at the dartboard, 20 is on top and then right next to it is the number one. And if you look down the bottom to the left a little bit, it's number 19.

How did those. So this is what I can't remember. How does the numbers like how do you use them in the game? Well, I think it depends on, well, they obviously count towards points if you're depending on the game that you're playing, but then there's also, um, certain games. I think that you hit, you have to hit like each number in sequence too.

Ah, cause that's the, cause they, they each have the little pie slice, right? There you're talking about the numbers around for me. Yeah. All right. So you, you, you left us off with talking about the description and I, and I like the way you kind of explained that. It's very useful as a very, as a snapshot, but it's not end all be all.

It's not where you should stop. What would be the, like, what, what would you do beyond that though? How would you investigate a program? Like. In your own home without having to like go and actually talk with them. How do you, where do you go next? Right. And so in the program description of the directories, they'll have, like I said, the basics, they'll have a basic program overview, maybe the rotations they offer.

Some of the specifics about maybe staffing, the number of co residents, the stipend. things of that nature, but they'll also have a link usually to the program website. So this program website will be different based on each program, but a lot of times it'll contain information about current residents, about what past residents have gone on to do, about the program preceptors and things of that nature.

So you'll get a lot more information. for the most part, in most circumstances, by looking at that program website. You mentioned the preceptors. One thing I like to look at, getting into like the things you can look at, are like preceptor qualifications. Are they just pharmacists? Are they board certified pharmacists?

Do they have, like, research experience in topics that you're interested in? Like, are there, are there CVs online? Can you look at that? Can you see what they've done? Or is there presentations online? So you get, like, a caliber. Like, if you have an infectious disease rotation, who is the infectious disease pharmacist?

How qualified are they infectious disease, you know, how what's their national recognition or local regional recognition in it? And so what kind of experience would you probably get as a as a resident? Yeah, that's an excellent point. And then other things that That just come to mind that the description might not have but that the program website will maybe things Such as like a teaching certificate some details about that that the program offers maybe they offer a Research 101 course for their residents.

And so those types of things might be on the program website as well. There's really no telling or no restriction as to what they put on those websites. And some will just kind of repeat from the program description on the directories. But, but that's okay. One other thing just to point out is sometimes that the program description is not always up to date.

So you want to make sure that you might want to peek at those program websites just to see, to make sure that everything is up to date. One other thing that the, like I mentioned about the past residents and what they're doing now, that could be a good gauge as to what type of residents and the caliber of residents that the program is producing.

Are they going on to be successful in their fields? Are those fields aligned with what you want to? eventually go into as far as your career goals are concerned. And so those are all good things to, that you can look at as well. Yeah. And are they hiring them? Meaning, is there a potential for this residency to be a year long job interview to where you actually stay on in that position moving forward?

I think, I guess in summary, The program website can potentially contain a lot more useful information beyond just what is in the program description, I think is what we're kind of getting at. What if there's like nothing on the website? And I think you just have to rely on what's on the program description, or this is where we can kind of maybe segue into talking a little bit about showcases.

You can get some information from the showcases. So that kind of. Brings me to a question for you, Sean. Do you think this process of program evaluation that we've been talking about replaces the need to attend showcases? I think that really depends. So you're like me, you have some regional places that you like, then you have some destination residencies.

Places you'd always want, you know, like how often are you going to get to go live somewhere for a year and not have that awkward, uh, you know, I'm quitting or I'm being fired, but you just know you're going to be terminated after a year, you know, that's pretty cool to know that you can just leave with a good clean slate, no bridges burned after a year.

So you want to look at something like destination places. What are some cool little cities or school places, or, you know, I have the opportunity to go live somewhere I've never been, or get out of my comfort zone. With evaluating the program. On their, in their website be good enough to where you wouldn't have to go to like mid year just to talk to that program to just get a little information in that situation, I would say, yes, program evaluation does replace the need to go to the showcase.

But if we're talking about your regional programs, if there's, if there's more than four. And that's very arbitrary number, but if there's several of the programs at a regional showcase that you're investigating, then no, just evaluating doesn't replace that need because you, you still don't have that red flag feeling.

You don't have the warm and fuzzies when you're talking to them versus the, Oh my gosh, I, this is not where I want to be. You can't get that from reading a page or looking at a website. So I think it, it does not, it just supplements. What do you think? Yeah. So I. Kind of agree with you, but then I also have a kind of don't, but no, it, it, I think it's just kind of unique to the situation.

So let's say for that, that example, when you're looking at a destination place and you don't know anybody up there, are you going to get along with? The people, the preceptors, everybody that is part of that program. And I don't think you can really do that by just looking at the pro at the website or the directory.

I think you need to actually talk to those people and kind of get a sense of what they're all about. And I think that's where showcases do help. Now you don't necessarily need to. Talk to them at a showcase. Cause if you do feel good enough to apply there, and if you're interview, if you interview there, you talk to them before interviews, that could be a time to get a sense of that, but it might save, save you a few bucks if you're, if you talk to them at mid year and you're like, Oh no, I don't, I, I don't see myself getting along with these people or I'm not vibing with them very well.

So that would just be my only caveat to your explanation. Well, yeah. And that's, that's where I was. That's kind of what I was getting at was the cost for the cost benefit. Of being able to talk to them, which, you know, I just thought of you can, you could email them. It's not like you can't have some interaction.

You could still email the program and email your questions to like the, the residents or the program director and ask whether or not the program director is going to respond to you is another question. But usually, if, especially if it's before mid year, you have a really good chance of that happening.

So Sean, we've kind of talked about. Evaluating what's important to you, looking at those program websites and directories, and evaluating. Those programs based on that information. And we kind of talked about what's this sweet spot where we're not putting too much, investing too much time into this, but still devoting enough time to adequately evaluate the program.

So what's that sweet spot that you mentioned? Just so we can give our listeners some sort of bearing on what that means. Yeah. And I think this is going to segue, I want to answer this and I think it's going to segue well into my next question for you, which is where to put all this information that you're collecting throughout this process.

But I think the sweet spot, like we were talking about earlier, it's a process. This is not just one single thing. We've almost broken it up into three steps. Step one. Self reflection step two, initial program, like looking at program descriptions and just getting a list of programs you like. And then step three would be actually diving in and looking at the websites of the programs that you're most interested in.

And like, it may be an order of most interest. And then step four would be attending showcases and using all that information. Is that kind of what you're seeing with a stepwise process? Yes. I think that's a really good way to put it is thinking about it in those steps. In those steps. So then the sweet spot that would be like what we talked about step one sweet spot would be making sure that you feel comfortable enough with what you want through that self reflection.

To then move on to step two, when you're like, you get a good sense of, okay, I think I know what I want. I think I know where to start. If you don't know where to start, you haven't done step one. Right. If you're still like, I don't, I don't know what filters to use on these directories, I don't even know where to start searching.

You need to go back and have a conversation with yourself about what's going to make you more comfortable. But once you, once you know where to start, you get a good feeling. You know what? I know where to start. Step two, that sweet spot is probably going to be. I mean, the sweet spot for step two is really easy.

It's just reading the descriptions. Now that you've searched and, and when you, you, you can just use the directory to get like a sense of how many pro it shows you at the bottom, how many programs are in that search. So it'll be like, you have 10 pages for 300 programs. That's not a sweet spot. Cause you can't look at 300 programs, but a sweet spot would probably be, cause you're just looking at descriptions.

I would say anywhere from like 20 to 40. And you're not all at once, but spread out. And if you don't feel comfortable within same thing for step one, you don't feel comfortable looking at 20 to 40 programs. And you're like, I still don't feel like I found the one that's for me. I mean, keep searching, but you, that's probably a good place to stop.

I wouldn't get past 50 programs before I started to look into programs more. And that might be too excessive for some, you know, it might, some might say 30. What would, if you could slap a number on it, how many programs would you say to probably stop at before you start like looking into the ones you've pulled up more in depth?

Yeah, maybe, maybe 30. I think it kind of depends on, you know, how long it's taking you and how, how hard it is for you to find those programs based on your parameters. So just, again, it's kind of a personal thing, but I think 30 is a good ballpark number to kind of think about. And you can always go back later and gather more information if you need to.

So we'll say 30 programs. And so then that moves you to step three. And then what I would say is a sweet spot here is just pulling up the website. And saying, I looked at the website would be too little, too much would be, let's, for example, say a hospital system. I think too much would be pulling up the hospital's strategic plan and saying, I'm going to read their strategic plan so I can see how pharmacy aligns with their strategic plan.

I can ask them really good questions saying like, here, here's what I want to do because it. You know, aligns with the hospital's own strategic plan. So I think that's too much. Maybe investigating beyond the department of pharmacy and looking at all these hot, all the things that are on the hospital website as a whole.

I think that's too much. So the sweet spot would probably be just reviewing everything. Cause it's usually not much. It's probably like a couple of pages or whatnot, just reviewing everything that's listed under the pharmacy department, pharmacy residency program and see what's going on there. I think that's a good sweet spot.

I like that. I think that's a really good explanation of the sweet spot for that step. But this is a lot like the, as we, as I talked through those steps, that's a lot of information. That's a lot like where are you tracking this? How are you tracking this? So I want to segue into that, Taylor, help, help us out.

How would you combine all these thoughts and combine all this work in a place or a method that's going to be effective? Absolutely. I think there's definitely utility in tracking all of these things because you don't want to just go back and not remember anything later. Um, and how you. Saw or how you evaluated the program.

So there's got to be some sort of method that you Use to keep track of those things and write those things down now you can use any sort of Word document if you have a Excel sheet or if you have a notepad that you're jotting these things down That's fine. And I think those are good ways to keep track of that information Um, one of the things that I did want to go ahead and mention that will be available to our exclusive listeners soon is a program evaluation tool that we developed and that allows you to kind of customize based on what's important to you, how you rate those specific programs, and then it'll kind of spit out a score and kind of rank those programs for you so you can kind of get a sense of Based on what is important to you, how do you kind of see this program as a whole?

And so I think these can be a good thing to kind of just narrow your search a little bit and kind of guide your thoughts. So stay tuned for more details about that or email us with questions. If you want to know more about when that. Um, evaluation tool will be available and how you'll be able to gain access to it.

Yes, absolutely. Look for those updates. Look for those releases, check out our show notes below for a link to our website, because there's a, a link to sign up for our mailing list. You get a free copy of our letter of intent checklist. So definitely something to take advantage of now. And to keep you up to date and on track for any future releases that we do.

If you want to continue to hear up to date topics from us and our guests, please like, and subscribe. Remember you can listen to us on all major podcast apps, and don't forget to check out the show notes for this episode in the description below.