Studying the Accuracy of College Football’s AP Top-Twenty-Why?

The college football world swoons when the first AP Top 25 poll is released in late August.  For starters, its a signal that football is close.  But far too much stock is placed on where teams are ranked (or if they are ranked at all).  Not only by the media and fans, but it also gets used as marketing material for college football programs.

Here’s the problem: the initial AP Top 25 has become less and less meaningful over recent years because its accuracy is so poor.

I studied the last 15 years of AP Top 25 polls (since 2000) and found that whether its the entire poll or just the top of it, the initial poll is less accurate or meaningful as it comes to accurately evaluating the best teams in college football.

I took two approaches at evaluating consistency and correlation of the initial AP Top 25 poll to the final AP Top 25 poll.  First, I only looked at those teams who were ranked in both the initial poll and the final poll.  By ignoring teams who fell out of the AP Top 25 completely after being ranked in it to start the season, we can use regression analysis to examine linear relationships between initial ranking and final ranking.

But that won’t account for teams who were ranked and then fell out, or teams who were not ranked at the start of the season who were ranked at season’s end, to see if the poll is becoming more or less accurate (and thus more or less helpful) over time.  So I took a far more granular approach and simply evaluated top and bottom 5 consistency, as well as looking at the number of total misses we’ve seen (unranked to start the year but ranked by the end).

In either approach, the result was the same:  the initial AP Top 25 is less accurate and less helpful than ever before.

 

Examining Ranked Teams in Initial and Final Poll

There clearly was a trend over time that showed ranked teams in the final AP Top 25 poll were becoming less and less correlated to their initial AP Top 25 ranking.  I broke the 15 year period in half (as close as possible) and analyzed the 2000-2006 period and the 2007-2014 period.

2000-2006

Below, you can see the results when you compare the initial AP Top 25 (x axis) with the final ranking (y axis).  If the voters were perfect, the first, last, and every AP Top 25 in between would be identical.  If the goal of the AP Top 25 is to rank the best 25 teams in college football, in order, the initial poll should be as close as possible to the final poll.  Obviously, “any given Saturday” and there are better teams who lose games to worse teams (aka upsets) but the goal of any poll should be to rank the best 25 teams in order.  Thus, if the pollsters were perfect, the graph below would consist of nothing more than a straight line from the lowest, left-hand quadrant to the upper, right-hand quadrant.  The initial #1 ends up #1.  The initial #10 ends up #10.  The initial #25 ends up #25, and so forth.

Obviously that never happens.  Many teams are ranked initially and fall out entirely.  Many teams are not ranked initially but are ranked by season’s end.  The graphic below depicts those teams ranked both in the initial and final AP Top 25 poll.  The ultimate result is that from 2000-2006, in searching for a linear relationship between initial and final rankings, 22% of where a team is ranked in the final poll is explained by their initial AP Top 25 ranking.  Essentially, a quarter of their final ranking is related to the initial ranking, three-quarters is related to other factors (assuming they don’t drop out altogether).

2000-06 Init vs Final Rank2007-2014

The analysis is the same as the prior period.  Except instead of 22% of the final ranking being explained by the initial poll, that percentage has dropped by almost 2/3rds.  Now, only 8.5% of a team’s final ranking is explained by their initial ranking.

2007-14 Init vs Final RankWhen comparing the charts side by side (as I’ll do below to make it easier), you can clearly see the effect of that big drop down to 8.5%.

2000-06 Init vs Final Rank2007-14 Init vs Final RankOn the left is the 2000-06 graph, on the right is 07-14.

The orange trend line’s slope is far less steep (more horizontal) in the 07-14 period, indicating less of a relationship between initial and final ranking.  And the teams littered at the far reaches of the graph, the top left or the bottom right (where inaccuracy rules), are far more numerous in the 07-14 graph.

To illustrate it more clearly, I added the shading in the lower right of both periods.  Teams in this area were initially ranked much worse in the first AP Top 25 as compared to where they ended the season.  As an example:

  • NO teams ranked bottom 5 of the initial poll & finished top 10 from 00-06.
  • 9 teams ranked in the bottom 5 of the initial poll & finished top 10 from 07-14.

To further demonstrate the progressive weakness of the poll over time, refer to another example using the blue shading on the two charts:

  • 8 teams are ranked inside the same blue shading in the earlier time period
  • 19 teams are ranked inside the blue shading in the later time period.

Just the Top 5

In the early 2000s, the voters were pretty accurate with the top 5 teams.  From 2001-2004, of the 5 top teams in each of those 4 years (20 teams in total), all 20 teams (100%) were ranked somewhere in the first AP Top 25 poll of the year.  No total surprise teams existed, and in fact, on average 3 out of the 5 teams who finished top 5 at the end of the year were also ranked top 5 at the beginning of the year.

But recently, that hasn’t been the case.   Instead of 100% of the top 5 teams being found somewhere in the initial AP Top 25, the last 3 years, 33% of teams who finished top 5 at the end of the year were NOT EVEN RANKED in the first AP Top 25 poll!  Its not that they were ranked 20th and really outperformed their ranking – they were outside the top 25 yet proved to be among the 5 best teams in college football in that season.

Some of these teams included:

  • 2014 TCU – finished #3 – were not even ranked until 9/28 (25th)
  • 2013 Auburn – finished #2 – were not even ranked until 10/12 (24th)
  • 2013 Michigan State – finished #3 – were not even ranked until 10/26 (24th)
  • 2013 Missouri – finished #5 – were not even ranked until 10/25 (25th)

These are massive miscalculations to not even rank these teams inside the top 25.  In the early 2000s, we never witnessed these types of errors from the voters.

 

Just the Bottom 5

Obviously the bottom 5 should be harder to predict.  From 2000-2006, voters had 14 of the 35 teams (40%) who ultimately finished in the bottom 5 somewhere in the top 25 to start the season.  And those 14 teams finished an average of 7.8 seeds different than they were ranked in the first poll.

But since 2007, voters have been worse with the bottom 5.  They had 11 of 40 teams (27.5%) who finished in the bottom 5 somewhere in the initial AP Top 25, so their percentage was well worse.  But additionally, those 11 teams finished ranking 12.6 seeds different than they were ranked to start the season.  So the percentage of even ranking teams has been much worse, and when they did get a team in there, their actual ranking of that team has been much worse.

 

Bottom 15

Of the teams ranked 11-25 in the final AP Top 25 poll, only 49.6% of those teams were ranked in the initial AP Top 25 poll, from 2007-2014.  That number is consistent with the earlier period as well, as 49.5% of teams ranked 11-25 by the end of the year were even ranked in the AP Top 25 at the start of the year.

Its a further example of how voters are missing on teams.  If they we’re seeing consistent examples of unranked teams finishing in the top 5, of course we will see examples of unranked teams finishing in the top 25.  But when less than half of the teams in the bottom 15 at year end were even ranked at the beginning of the year, it begs the question “why should we care about the initial AP Top 25 poll?”

 

AP Top 25 is “Discredited” by the Playoff Selection Committee

College Football implemented a Playoff Selection Committee, and their first rankings of the year don’t emerge until week 10.  In 2015, that will be November 3rd.  It’s really pointless to consider polls prior to that point in time, as polls are now irrelevant when it comes to a committee – they consider far more than just polls.

In fact, the bylaws of the selection committee actually preclude them from even caring what the initial AP Top 25 poll said.  Specifically:  “Committee members will be required to discredit polls wherein initial rankings are established before competition has occurred.”

But more than just the initial poll appears to be precluded from analysis by the Selection Committee.   Because the AP Top 25 poll’s initial rankings clearly are established before any game has occurred, it appears the Selection Committee members should be “required to discredit” the AP Top 25.  Entirely.  All year long, not just initially.

There are a myriad of reasons why its harder to accurately rate the best 25 college football teams in the nation, particularly in mid August.  Many programs which historically were bad or mediocre are becoming better, and in many cases, capable of pulling off upsets over historically stronger programs.  But regardless of the why, the fact is, the initial AP Top 25 is more meaningless now than ever.  And placing much stock in it is assuredly a waste of time.

 

Comments

comments