Wednesday, November 28, 2012

Cumulative Impact-Factor Benchmarking

Speaking of CVs, publications, and impact-factors. Some time ago I got pretty anxious about this whole publications benchmarking story. You know, when some people say that "everyone should publish at least one paper a year", or somebody mentions in passing that "nobody is hired without at least 1 glamours paper", or "second-authors do not count", and so on.

So I decided to do some research myself. I did the following:

  1. Identified some people in my field who did something remotely similar to what I do, and who are or were on the job market within last ~5 years.
  2. For each of them, I downloaded a full list of their publications as undergrads, grad students and postdocs, as that's what they showed (or are showing) on their CVs when looking for a job.*
  3. For every publication I found the impact-factor of the journal it was published in.
  4. I discounted 2nd-author papers and reviews by 75% (so 4 second author papers = 4 reviews = one first author paper). It's obviously a wild guess, and an oversimplification, as the formula would not hold at extremes, but overall it's probably about right.
  5. And finally, I calculated their cumulative impact factor. And then plotted this value vs. years that passed since they got their PhDs.

Here are the results. Black lines represent those who got their TT positions in really cool (glamorous) places. Brown lines indicate successful landing on TT positions in quite decent places (universities, colleges). Blue lines are for those who either got a non-TT positions, or only got some really terrible (unacceptable) offers in some weird places, or did not receive any offers so far.

What do we see here? A bunch of stuff!

  1. To get any kind of a TT position in my subfield you need to reach a threshold of about 60 cumulative IF. That's either 2 publications in Nature, or 15 Plos-ones, or anything in between**.
  2. You need to get it in about 12 years including grad school. A gentle slope means asking for trouble.
  3. Glamorous papers (those sudden jumps in the cIF) do increase your chances, but mostly because they pump up your cIF. Although one can argue that they also improve your image (see that black line among the brown ones, with a distinct CNS jump).

I personally aren't on track yet, but I have some chances to get on the brown tack, if only the papers I'm working on now are published properly (in good journals).

* Practically speaking, I took all papers published before they got their first last-author research paper; plus any non-last-author research papers published in 2 years after that. (This additional complication is necessary, as apparently many people publish their last postdoc paper already after publishing their first PI paper. But I assume they still had it shown in their CVs as "submitted"; thus the adjustment).

** Update: No doubt, the "threshold" will be very different for different fields, and even subfields. My goal was to benchmark myself against those who would have been my peers, had I started my career some 5 years earlier. It would be really great if somebody could make a personalized online benchmarking tool like that, for everybody to use, for my web-programming skills are just not good enough for developing it. If you can do it - please, do it!