Leiter ranking Canadian law schools

Is this a blessing or a curse? Law school rankings have come to Canada. Brian Leiter was hired by Macleans magazine to design measurement criteria, compile data and rank the schools. This year’s ranking – the second that Leiter has done – has just been published at Macleans (here: http://oncampus.macleans.ca/education/2008/09/11/ranking-canada’s-law-schools/). In many ways Leiter’s rankings are a useful contribution. As he has suggested of rankings in the US, they may “unleash academic talent and ambition,” and, as Macleans argues, they might provide prospective students with information about which law school they “will get the most out of.” The measures that he uses also appear generally legitimate. They have the virtue of being not (at least as far as I can imagine) susceptible to the law school gaming that Leiter and William Henderson have been critical of with respect to the US News and World Report analysis. Having said that, I think there are some points that Leiter might not have fully taken into account in assessing Canadian (as opposed to US) law schools. There are also some weaknesses in the data points.

Half of a school’s ranking comes from information about what happens to students after they graduate ,and half of a school’s ranking comes from faculty citations as identified through a search of Hein-on-line. With respect to students, Leiter measures hiring by elite law firms, “national reach” (students hired at firms other than the three which hired the most grads from a school), Supreme Court of Canada clerkships and graduates hired as academics. These are reasonable but incomplete measures. Supreme Court of Canada clerkships are particularly troublesome. Judges at the Supreme Court of Canada are required to represent the regions of Canada – 1 judge from the Maritimes, 3 from Quebec, 3 from Ontario, 1 from the prairies (Manitoba, Alberta and Saskatchewan) and 1 from British Columbia. The Court hears both civil law and common law matters, and sits in both English and French. As a consequence, clerkship hiring is based not only on academic (or law school) merit, but also on region of origin, competence in French and English, and (for one clerk per judge) competence in civil law. Not surprisingly, therefore, McGill and Ottawa – which teach in both English and French, and which teach both civil law and common law – have remarkably successful clerkship hirings. Equally unsurprisingly, the four prairie law schools – Manitoba, Alberta, Calgary and Saskatchewan – have had relatively limited success. Those schools combined have achieved less than half of the total clerkships of the University of Ottawa, even though combined having nearly twice as many students.

In general the output measures on students also do not sufficiently account for public sector hiring of law school graduates. While Supreme Court clerkships and faculty hirings are both public sector, they cover a miniscule proportion of total graduates from the law schools. This is an issue because public sector jobs in Canada (and in the United States?) are as or more desirable than elite law firm jobs, and may be as competed for by law school graduates. In some Canadian cities entry level government jobs may pay as well or better than private law firm jobs (this was historically the case in Winnipeg, for example). Further, in some sectors high profile careers tend to start in government; a recent issue of Lexpert magazine noted that a “disproportionate number” of the corporate tax litigation bar “trace their roots to the other side of the fence as counsel with Justice Canada” (Lexpert Magazine, May 2008 p. 63).

The exclusive reliance on faculty citations in Hein-on-line as a measure of faculty output is also somewhat disappointing. While obviously faculty citations are an excellent measure, they weigh heavily in favor of law schools with older and more established faculty, and create considerable “stickiness” in the rankings given the limited lateral movement in Canadian law schools. The exclusive reliance on faculty citations is unnecessary in Canada. In contrast to the United States, every Canadian law journal is peer reviewed; every paper published in a Canadian law journal (even case comments and book reviews) has likely been reviewed blind by one or two academics expert in the field. This is often the case even for invited papers. As a consequence, the number of publications by an academic in such journals is a real measure of academic quality. Measuring the number of peer reviewed publications by faculty members in the preceding year would contribute usefully to the assessment of faculty accomplishment.

There are other points that can be made on the Leiter rankings with respect to his analysis of the data, the relatively limited stratification in Canadian law schools and the regional nature of many Canadian schools (in part see analysis here: http://utorontolaw.typepad.com/faculty_blog/2008/09/new-macleans-ra.html and here (in the comments): http://oncampus.macleans.ca/education/2008/09/11/ranking-canada’s-law-schools/). My final observation would simply be to ask: what is the point of this exercise? Is it to drive academics to produce more scholarship? Is it to provide students with a better indication of which law school they should attend? Is it to provide elite law firms with information as to which school to hire from? Different groups may perceive the usefulness of these rankings differently. For instance, a junior academic is likely to feel limited motivation from them; a publication this year is not likely to make a material difference to a school’s citation performance for several years, perhaps even many. A prospective applicant to law school could also legitimately question whether knowing the outcome for the top few students – those going to elite law firms, to the Supreme Court or into academia – in any given law school is all that useful. That student may be more interested in knowing the total placement rate for students upon graduation. While my school (Calgary) did relatively well in the elite law firm category (ranking 6th), applicants may be more interested in knowing that last year 99% of those in our graduating class who sought articling positions found them. As a Faculty we are certainly prouder of that statistic than of the elite law firm statistic used by Leiter.

Comments

[michael webster]

Rankings are notoriously difficult to defend or explain on measure theoretic objective grounds.

Nobody cares that most rankings are measure theoretically meaningless, we are too busy pronouncing ourselves to be #1.

This particular ranking arbitrarily assigns weights to to five decision criteria in order to obtain a trade-off.

There is no justification for using these weights, nor is any explanation offered as to why the trade-offs between these criteria can be modeled as if they were probabilistic weights.

Think for a moment on how you would decide between clerking for the Supreme Court and taking a job a firm ranked highly by Lexpert. Would this trade-off make any sense using simply weights?

If not, why would you expect the ranking of law schools using weights to measure the relative importance of these two decision criteria?

Oh, I forgot - to get a number, a meaningless number.

[Brian Leiter]

Thank you for the comments, which raise several good points. It goes without saying that rankings of academic institutions, even when done reasonably well, use measures that are "reasonable but incomplete." The point you make about Supreme Court clerkships is one I was aware of, but the real question is: how to offset the bilingural effect? This is a genuine question. The provincial courts, as I understand it, overwhelmingly hire from schools in the province. Moreover, like the Canadian Supreme Court, they are bizarrely secretive about whom they hire.

I would like to complement the citation measure with a reputation measure. Even with peer-reviewed journals, I'm skeptical of productivity measures for a whole host of reasons. Some have suggeset dropping untenured faculty from the citation measure, which might make sense--I've now tried that in the US context, and it makes a modest difference.

Canadian legal education is different from American, but it is still stratified in various ways, including ways that I have the impression many Canadian academics didn't realize until we started measuring various things. Maclean's was going to rank law schools with or without me. I am fairly confident that with my input, this has been a more informative exercise than it would have been otherwise. But I am sure we can improve it along various dimensions, including some of those noted above.

[Alice Woolley]

Thanks Brian. There is no doubt Macleans was going to rank the law schools, and I am very glad to have our rankings turn on things other than, say, the size of our library!

On your specific comment, the concern I have with the citations, and also with dropping untenured faculty, is that neither the original measure, nor this solution, provides a way of capturing a law school on the rise, with promising junior faculty. I would have thought a better approach might be to have three categories for citations - assistant, associate and full professors (bearing in mind also that people (I think) spend longer at those three ranks here than in the US). More weight could be given to how cited your full professors are, but some weight could be given to your junior faculty performance as well. It's also easier to determine a professor's rank - I don't think whether or not someone is tenured is a matter of public record (and it is not related to rank - you can be either assistant or associate and tenured or untenured).

I also don't see publication in peer reviewed journals as purely a productivity measure, insofar as productivity is quantitative as opposed to qualitative. I do agree that it is a less valuable measure than citations, but to the extent that getting such publications is difficult (which it can be), to the extent that the referee process provides some assurance of quality (which it does), then having a faculty where there is a high rate of peer reviewed publications is an indicator of the quality of that faculty. Not a perfect measure, but none of these measures are perfect.

On the SCC clerkship, and on public hiring generally, I am not surprised that information is hard to come by. I know precious little about this but I wonder whether the Canadian privacy legislation would impede access to data of that type without the person's consent. You are clearly correct on the provincial clerkships, and the Federal Court has the same problems as the SCC only worse. The issue in both cases is not simply bilingualism, but is also regional representation. Judges hire from their region, and underrepresented regions on the Court (like the prairies) suffer as a consequence.