What's a PhD worth at the finish line? On hiring committees

The blogosphere is roiling over the worth of a PhD.  Should PhD programs take all qualified students? Is the academic "meritocracy" a myth, i.e., are some PhDs more equal than others? Even an outlet as erudite as The Economist suggests doing a PhD might be a waste of time (the always-nameless correspondent calling her ecology PhD "pointless").

In all of these pieces, the worth of a PhD is measured by its convertibility into a tenure-track position. There's lots of ink to be spilled on this topic, and I don't have a grand, philosophical point to add about the worth of the PhD.  However, I'd like to talk about the "finish line" viewed by many PhD students: the hiring committee.



A few years ago, I had a chance to serve on a hiring committee.  If you ever have a chance to do so as a student, I strongly, strongly recommend it -- it was extremely revealing to see what went on in the room, what the committee liked and didn't like, and the mistakes made by candidates, both on their applications and in their interviews.

Bearing in mind that this is a snapshot of one hiring process in one particular discipline, here's my memory of what the committee thought was important, in order from most to least important, in their decision making (I'm deliberately vague so as not to give away identifying details):

1. Publication record. Unsurprisingly, this was the sine qua non of the hiring process; the list of publications was our first stop in a candidate's CV. We were hiring in a particular sub-discipline, and we wanted to see a strong publication record in that area, both quantity and quality (i.e., top journals, first-name publications).  Here, excess quantity without quality tended to count against an applicant, as did resume-padding (like listing a paper presentation at a conference with "invited talks", without explicitly making the distinction). We also had a few applicants in tangentially related areas, trying to convince us that their work was applicable to the sub-discipline in which we were hiring; these applications were quickly dismissed.
2. The interview. Our internal rankings changed significantly after the interview -- you might say that your publication record will get you on the short list, but the interview will get you the job. This one surprised me: I thought the interview would be a formality, but the best interviewees presented themselves as both good colleagues and strong communicators, whereas the worst did not. We were very surprised by some of our short-listers, both positively and negatively.
3. Reference letters. Important, but less important than you might think.  Two reasons for this.  First, the letters are predictable: you ask people who know you best for your letters, so most people had glowing letters from their PhD and post-doc supervisors. Second, hiring committees are broad whereas academic sub-disciplines are narrow, so -- except in a few extreme cases -- if you ask a superstar in your narrow field to write you a letter, few committee members would know the person.  As I recall, letters would only hurt candidates, never help -- in one case, a letter was positive, but embarrassingly short; in another, someone got a superstar to write a letter (the one and only case of a superstar letter where I recognized the name), but it was clear that the person either didn't know or didn't think highly of the candidate.
4. Job talk. Surprisingly unimportant given the candidate's stress level. Before they walked through the door, we already had a very clear picture of the candidates' research programs.  Further, we started the interview process at 9 AM, and the job talk happened in the afternoon, so by the time the job talk happened we already had an idea of the candidate's communication skills.  And everyone always has a polished, well-prepared talk.  So the job talk was about confirming impressions, rather than about any serious evaluation.
5. Where you got your PhD. Irrelevant, controlling for all the above.  You got in to Harvard? Good for you. What have you done lately?

So is academic hiring meritocratic? I would say yes, for suitably constrained definitions of a meritocracy.  Even in the sciences, a famous supervisor's name on a paper may ease its way into a top journal, and access to labs and high-quality equipment both depend on the university and the supervisor.  But the most important facts about a candidate -- publication record and interview -- are generally about themselves, not about who they know or where they come from.  In principle, there's no financial or personal barrier to publishing research in a top journal; one need only have a good idea.  And seeing all the applications together, it was pretty obvious which researchers were talented and which were not; there was discussion of course, but ultimately the rank ordering of candidates was not terribly controversial among our committee.