Follow-up to Discuss Differing Studies Regarding the New England Patriots Fumble Rate Since 2000

By Warren Sharp

I wanted to provide a brief follow up to the “series” of articles I wrote on the New England Patriots Deflate Gate investigation.  [Click here to read the last installment with conclusions]  What started as a simple question:  “why would they intentionally deflate their footballs” turned into uncovering a statistical fumble-aversion anomaly for the Patriots dating back to 2007 which was absent from 2000-2006.  The beauty of the data is, it speaks for itself, especially when its presented visually, and there really is not much I have to add to the picture.

Of course, with any data-driven analysis, there will be a ton of skepticism, criticism and commentary, particularly when that analysis is sports related.  After publishing the summary piece, which laid out all of the key findings quite plainly, a number of others from various walks of life, particularly the statistics field, wrote articles which cast doubt and criticism on my study.  I wanted to briefly comment on some of the prime criticisms, but before I do that, I think its important to understand my main takeaways from the data, to date.

Point #1 –  Something occurred between 2006 and 2007 which allowed the Patriots to fumble the ball at an extremely low rate moving forward when compared to the rate their team fumbled from 2000-06 (Bill Belichick started coaching the team in 2000).

Point #2 –  Whatever occurred caused the Patriots to shift from a team who fumbled the football the league average (in 2000-2006) to a team who was so superior when compared any other team the odds it was a mere coincidence are extremely unlikely.

The data to support both of those findings is described in detail in my last article, from which the main graphical takeaways are:

(click to enlarge)

Understanding those were the primary conclusions of this research, it was fascinating to see each time someone would send over another analysis for my review which was said to “poke holes” in my conclusions.  Obviously, I was anxious to see if someone found other data which would supercede or “trump” what I researched.

Primarily, I was interested to see:

  • “Did they find something that showed the Patriots did NOT change in 2007, and that their 2000-06 fumble rate actually was very comparable to the 2007-14 fumble rate?”
  • “Did they find something that showed the Patriots were not actually statistical outliers in the 2007-14 period, and that the rest of the NFL, particularly the outdoor teams, actually fumbled at the same rate as the Patriots?”

But unfortunately I never saw anything that actually looked at my conclusions, using the same time periods, and determined the fundamentals of my two key points were incorrect.  I’ll start by sharing a common theme in these combative articles and then move to a few specifics.  Most of the pieces were written by statistics professors at universities.  These gentlemen assuredly know more about statistics than I do, considering I do not teach statistics at a university.

Instead of trying to deal with the two periods I clearly laid out, where my conclusions are unmistakable, they all wanted to eliminate periods of time and/or use other means to reduce the sample size.  Whether they broke up the stats by position (and eliminated the QB position entirely from discussion) and then never combined them again to demonstrate any conclusion, or whether they looked at dome teams who played outdoors (who obviously would have about 50% of the sample size that a non-dome team would have), or whether they wanted to focus on shorter periods of time, such as the fumble rate of only the 2014 Vikings, it seemed they were working in the wrong direction.  If I found one year the Patriots were exceptional at their fumble rate (2010) and used it as crux of my argument, I would be accused of poor sample size.  But instead, while I use statistically sound sample sizes over 7 and 8-year periods, the criticism by statistics professors comes by using significantly smaller sample sizes.

The NFL, more than any other sport, is a very tricky one to analyze.  There are only 16 total regular season games each year.  Year to year data can be crazy.  Example:  The Oakland Raiders were the #1 team in offensive red zone scoring percentage in 2014 after being #30 just 2 years ago.  The Raiders!  The larger the sample size, the better, especially in the NFL.

I also found that while most of these pieces try to poke holes in some of the smaller points or comments, most of their own large, big picture takeaways actually appear to align with my own.  Let me explain:

For example, one article did not offer any real conclusions on the overall 2007-14 period.  Instead, that analysis took a fumble rate for RBs and one for receivers, individually, and separated them into two buckets on two different graphs.  If they merely combined the data (which they don’t share), into one graph, I’m fairly certain (from looking at the graphs) that the Patriots would be clearly sitting #1 from 2007-14 as the team who fumbled least frequently.  And if they re-introduced QB fumble rates, I’m sure that position would be even more certain.  But they never bother to combine the data.  Why not?  I can only assume its because it would then agree with my Point #2 from above.

That very same article never looked at the critical 2000-06 period’s data.  If they did, I can only assume they would find they Patriots were right in the middle of the NFL in fumble rate (since their data for the skill position players seems to show something similar as mine, though theirs is segregated by position).   They then would be able to see that something definitely started to happen in 2007 which caused the Patriots to move from a league average to the best team in the NFL by a margin.  Which would then agree with my Point #1 from above.

In essence, at a macro level it “seems” that this highly argumentative piece does not actually conclude anything which refutes either of my two main takeaways.  We can’t tell the “scale” of difference they may have vs. the data I presented, because they never aggregate it to compare, but it appears they are finding that the Patriots are perhaps the most fumble-adverse team in the NFL from 2007-14.  By how much, per their article, we don’t know, nor would we without factoring in quarterback rates into the equation.

In fact, the piece also “encouraged” readers to “check out Brian Burke’s post at Advanced Football Analytics for a more reasoned take on fumble rates“.  Ironically, Brian Burke’s piece was written after he read my initial piece, and he states of (of my findings):

“The charts are convincing, and the implication is that NE benefitted from under-inflated balls is unmistakable. But I wasn’t sure how much stock to put in the numbers for a couple reasons. One is that they were so extraordinary they seemed unlikely to be true.  …”

However, once complete with his own analysis, Burke comes to the same conclusion as I do:

Whoa. In this case NE is at the top of the list, and the next best team is a distant second. Notice how the second team (BLT) through the second to last team (PHI) have rates that are within 1 or 2 plays of each other. NE, however, is better than the next best team by 20 plays per fumble.

You can read Burke’s article for more context, but his comment there is precisely what struck me as well, and which is still the key point during this 2007-14 period.  But for some reason, the highly critical piece praised the Burke article, despite Burke’s article finding the same results in the data as I did.  After reading this critical piece, Burke commented on Twitter:

Why was this piece so viciously attempting to disprove something, when the data they introduce themselves actually appears to (without saying) concur with my primary conclusion?  While at the same time agreeing with another article which is (essentially) providing an identical conclusion to mine?  I cannot speculate as to that reason.

Other pieces I read were more critical of other studies I performed rather than the one with my primary conclusions which is the last one in the series.  The key criticism revolved around the individual player statistics.  And on this point of criticism, it’s entirely valid.  When I ran my numbers on the individual player statistics, I incorporated all fumbles by that player.  The reality is to really assess fumble rates which may have changed, you should attempt to remove punt return fumbles and kickoff return fumbles, because they are using a special “K” ball, not the footballs used by the team when playing offense.

As one article surmised:  “I suspect the cause of the bad data is that Sharp collected his figures from the otherwise excellent Pro-Football-Reference.com, which for some reason includes special teams and postseason fumbles under its “Receiving and Rushing” statistics.”  They are correct.  They re-ran the analysis, removing kicks and punts.  What they found, their overriding conclusion, was the individual Patriots players have a touches per fumble on “offense-only” of 132.9, and when those same players leave the Patriots, it reduces to 107.9.  In other words, they fumble 23% less frequently when playing for New England.  And based on their data, the players who handled the ball the most for New England and then left (Welker, Maroney, Green-Ellis and Woodhead) fumbled 38% less frequently with New England than with their other teams.  This is their study, with proper numbers removing kicks, not my own.

My primary reason for even involving the individual fumble data in the first place (it was clearly the most-requested follow-up investigation from the masses) was as a simple “sanity check”.  I thought the Patriots refusal to fumble from 2007-14 could potentially be explained by them drafting, trading for, or otherwise acquiring players who simply NEVER fumbled.  Clearly, if you suddenly start players who never fumble the ball ever, if its their career “calling card”, you would expect the aggregate team data would reflect that was well.

But where are we now with regard to the individual player data?  Did their analysis show players fumble LESS after leaving New England?  No.  Did it show players fumble the SAME after leaving New England?  No.  It still shows players fumble MORE when leaving New England.

Notice, however, that my 2 key takeaways from the most recent analysis do not even address individual players.  To drill down to that microscopic level was simply a sanity check.  While my initial analysis into the player statistics did not properly remove “K” ball plays as mentioned earlier, after removing the kicks and punts, we still see these Patriots players fumble 23% more on average when playing for other teams immediately after playing for the Patriots.  This allows us to rule out the possibility that stocking their team with “magic non-fumblers” was the reason for why the Patriots stopped fumbling in 2007.

So much like the first critical article I discussed above, this article was quite aggressive, but still (ultimately) concluded the Patriots fumbled more when playing for other teams.  Which was my exact conclusion from my own article, but more than that, it means we need to keep searching for what the answer to this anomaly would be.  Individual player statistics should not be the starting nor ending point for this analysis due to sample size, which gets me to the following:

I really want to get back a key point here:  Drilling down to 1 player or 1 season or one skill position or even a couple of seasons significantly reduces your sample size.  Which significantly increases variance.  Which is unnecessary when the macro data is so compelling and there has yet to be a counter-analysis which disproves the two overriding points that the data appears to show.

I could look at Kevin Faulk, for instance, and tell you that in his initial years in New England, thru 2006, he had 908 offensive touches (eliminating returns) and fumbled 19 times, or one fumble per 48 touches.  But from 2007 until 2011 when he retired, he fumbled just once in 387 touches.  That is an improvement of over 700%!  But that alone proves nothing.  However, when you look at the entire team over the course of many consecutive seasons, it becomes obvious where there are anomalies.

My training is as an engineer.  I’ve been practicing for almost 20 years.  While I never actually looked at a website to define what I do on a daily basis, this description was pretty close:

While not a professional statistician, it doesn’t even take an engineer or any professional trade to uncover the anomaly I uncovered in the Patriots fumble data.  That is the beauty of this entire analysis – any one reading this very article is connected to the internet, and that is all it takes to find and then compile this data.  We can leave it to the statisticians to define the probabilities, etc, from that data (which is what I did when I ran the data by a Data Scientist), but the data itself speaks volumes.

If I have any say in the progression of this story, its: stick with the macro data and either prove or debunk it.

  • To Point #1:  Can you disprove that the Patriots changed in 2007, and prove that their 2000-06 fumble rate actually was very comparable to the 2007-14 fumble rate?
  • To Point #2:  Can you disprove that the Patriots were statistical outliers in the 2007-14 period, and prove that the rest of the NFL, particularly the outdoor teams, actually fumbled at the same rate as the Patriots?

That’s what we need to understand.  My data reflects it.  Brian Burke’s data reflects it.  But I would be very interested to see data which reflects the opposite, should it exist.

If the data that others uncover thru different means and statistics show that 1) the Patriots do change in 2007, and 2) they do move well ahead of the NFL average by some margin from 207-14, but not to the exact extent of the data that I used would indicate, that’s still confirming (not denying) a potential issue, and we have even more data (which is different data) which shows 1) they changed and 2) they are different from the rest of the NFL.  So the first step is, find data that disproves my numbers which are graphed above.

After that, the only question is:  WHAT caused the Patriots (starting in 2007) to move so far away from what they historically were in terms of fumble rate, and so far away from the NFL average?  I’ve heard plenty of ideas, a few of which have already been debunked by the data.  I’ve heard its because the Patriots pass more, but from 2007-14 they pass on 57% of their offensive plays, which is the league average and ranks 18th in the NFL.  I’ve heard its because Brady passes faster, but why did the other skill players become so adept at not fumbling starting in 2007?

FYI: I’ve heard a lot of suggestions to look at 2000-06 home games, because maybe those showed the Patriots were engaging in this tactic early on, when they controlled the footballs at home.  There is absolutely NO evidence of that in the data.  The data shows the Patriots were league average in both HOME and ROAD fumble rate from 2000-06.  They became great in home fumble rate from 2007-14, and outstanding in road fumble rate in that period.  Which is why I continue to state:  SOMETHING happened between 2006 and 2007, and its clear from the data whatever occurred was not occurring prior to 2007.

I’ve made it exceedingly clear I am not 100% pinning it on deflated footballs.  I’ve stuck 100% with the data, and the data cannot prove anything as to WHAT it was.   I’ve never, in radio interviews or my articles, said WHAT it was.  It’s suspicious that the data changes so dramatically in 2007, but we don’t know WHY.  As I’ve said before, the data can only clue us in to WHEN a certain pattern started, and it can sometimes prove what it was NOT (as in non-fumblers on the team, an outstanding head coach – who must have been average while winning Super Bowls thru 2006 but became tremendous starting in 2007, or a secret way to hold the football which was introduced in 2007, etc.).  We won’t find the “WHAT” in the data.  But understanding the data should allow us (and the NFL) to ask smarter questions when attempting to find out the “WHAT”.

______________________________________

Warren Sharp of sharpfootballanalysis.com is an industry pioneer at the forefront of incorporating advanced analytics and metrics into football analysis. A licensed Professional Engineer by trade, Warren applies the same critical thought process and problem solving techniques into his passion, football. After spending years constructing, testing and perfecting computer models written to understand the critical elements to win NFL football games, Warren’s quantitative analytics are used in private consulting work, and elements of which are publicly shared on sharpfootballanalysis.com. To contact Warren, please email [email protected] or send a direct message on Twitter to @SharpFootball.

Comments

comments