Tuesday, January 18, 2011

What's a PhD worth at the finish line? On hiring committees

The blogosphere is roiling over the worth of a PhD.  Should PhD programs take all qualified students? Is the academic "meritocracy" a myth, i.e., are some PhDs more equal than others? Even an outlet as erudite as The Economist suggests doing a PhD might be a waste of time (the always-nameless correspondent calling her ecology PhD "pointless").

In all of these pieces, the worth of a PhD is measured by its convertibility into a tenure-track position. There's lots of ink to be spilled on this topic, and I don't have a grand, philosophical point to add about the worth of the PhD.  However, I'd like to talk about the "finish line" viewed by many PhD students: the hiring committee.

A few years ago, I had a chance to serve on a hiring committee.  If you ever have a chance to do so as a student, I strongly, strongly recommend it -- it was extremely revealing to see what went on in the room, what the committee liked and didn't like, and the mistakes made by candidates, both on their applications and in their interviews.

Bearing in mind that this is a snapshot of one hiring process in one particular discipline, here's my memory of what the committee thought was important, in order from most to least important, in their decision making (I'm deliberately vague so as not to give away identifying details):
  1. Publication record. Unsurprisingly, this was the sine qua non of the hiring process; the list of publications was our first stop in a candidate's CV. We were hiring in a particular sub-discipline, and we wanted to see a strong publication record in that area, both quantity and quality (i.e., top journals, first-name publications).  Here, excess quantity without quality tended to count against an applicant, as did resume-padding (like listing a paper presentation at a conference with "invited talks", without explicitly making the distinction). We also had a few applicants in tangentially related areas, trying to convince us that their work was applicable to the sub-discipline in which we were hiring; these applications were quickly dismissed.
  2. The interview. Our internal rankings changed significantly after the interview -- you might say that your publication record will get you on the short list, but the interview will get you the job. This one surprised me: I thought the interview would be a formality, but the best interviewees presented themselves as both good colleagues and strong communicators, whereas the worst did not. We were very surprised by some of our short-listers, both positively and negatively.
  3. Reference letters. Important, but less important than you might think.  Two reasons for this.  First, the letters are predictable: you ask people who know you best for your letters, so most people had glowing letters from their PhD and post-doc supervisors. Second, hiring committees are broad whereas academic sub-disciplines are narrow, so -- except in a few extreme cases -- if you ask a superstar in your narrow field to write you a letter, few committee members would know the person.  As I recall, letters would only hurt candidates, never help -- in one case, a letter was positive, but embarrassingly short; in another, someone got a superstar to write a letter (the one and only case of a superstar letter where I recognized the name), but it was clear that the person either didn't know or didn't think highly of the candidate.
  4. Job talk. Surprisingly unimportant given the candidate's stress level. Before they walked through the door, we already had a very clear picture of the candidates' research programs.  Further, we started the interview process at 9 AM, and the job talk happened in the afternoon, so by the time the job talk happened we already had an idea of the candidate's communication skills.  And everyone always has a polished, well-prepared talk.  So the job talk was about confirming impressions, rather than about any serious evaluation.
  5. Where you got your PhD. Irrelevant, controlling for all the above.  You got in to Harvard? Good for you. What have you done lately?
So is academic hiring meritocratic? I would say yes, for suitably constrained definitions of a meritocracy.  Even in the sciences, a famous supervisor's name on a paper may ease its way into a top journal, and access to labs and high-quality equipment both depend on the university and the supervisor.  But the most important facts about a candidate -- publication record and interview -- are generally about themselves, not about who they know or where they come from.  In principle, there's no financial or personal barrier to publishing research in a top journal; one need only have a good idea.  And seeing all the applications together, it was pretty obvious which researchers were talented and which were not; there was discussion of course, but ultimately the rank ordering of candidates was not terribly controversial among our committee. 


    Jason Ernst said...

    Nice post, its good to read about what the process is like going from being a student to trying to get hired in academia. If only our university would hire some new profs I could try to see for myself what this process is like, but alas in our department it seems we keep having retirements and deaths but no replacements :S

    Matthew Adam Kocher said...

    This sounds like a fairly reasonable account of hiring at the senior level, where there is already a fairly extensive body of published work. If you're an established academic with a bunch of solid articles at peer-reviewed journals, it isn't going to matter that you got your Ph.D. at UConn 10 years ago instead of Yale. I've seen well-regarded senior people do horrible job talks and still get hired, because giving a nice talk is not really what the committee is after.

    At the junior level, Eckford's take doesn't conform to what I've seen and heard. For brand-new Ph.D.s or ABDs, most of the time there isn't much of a publication record. Since the number of applicants is huge, committees have to find something else to base their decisions on, and typically the institution they come from and the letters of recommendation play a very significant role, at least in getting people into the final rounds of selection. I don't believe that most applicants' work is read with the kind of care that would be required to make a really informed, direct decision about the quality of their work. Most of the time, I believe this happens for the "long-short list," of 10-15 candidates.

    Eckford makes a distinction between "the interview" and "the job talk," but in my experience the job talk is the interview or at least the most important part of it. It is at this stage that many members of a department really start to pay attention, to read and to listen. The job talk may be the only time that people devote enough attention to the candidate to really think about their arguments and evidence and start to put them to the test. That said, my impression is that intangibles like "personality" end up playing a pretty big role at this stage. Also, I've seen a number of really terrible presentations in job talks.

    In regard to his conclusion, that academic hiring is basically meritocratic, I would make a distinction between process and outcome. I haven't seen a ton of evidence that people are thinking some version of "he's one of us," so we'll take him even though he's not as good (though sometimes methodology can look pretty tribal). In other words, I don't see much overt deviation from meritocratic standards in the process. However, evaluating "talent" or "potential" (which is what is mainly happening with junior hires) is devilishly difficult to do. Just think about how many "can't miss" baseball pitchers or NBA players turn out to be busts. There is a ton of uncertainty in the evaluation process and then the outcome (career quality) is endogenous to hiring. That is, on average the person who gets the sweet gig with the low teaching load, lots of leave, excellent colleagues, and tons of resources, typically does better at publishing than the person who gets stuck teaching a 4-4 at Powdunk State.

    Andrew said...

    As I mentioned, this is a snapshot of one hire in one discipline. We were hiring a junior position, but in computer science the typical fresh PhD will have 10+ publications. That person will add more through a couple of years of postdoc between graduation and the tenure-track hiring process.

    For example, I was hired in 2006, so starting here (and below) is the list of publications that were on my successful CV: one journal paper, 18 conference papers, and 2 dissertations (I probably also listed the 3-4 journal papers in press or under review at that time). I understand this might be different in the humanities.

    Your point about "talent" is a good one, but the tenure system more or less excludes late-blooming researchers. The relative merits and drawbacks of tenure are an entirely different discussion.