I’m a data nerd. I know it. You know it. It’s not like it’s a big secret. My name is Meredith Jones, and I let my geek flag fly.

So it’s no wonder that my nerdy spidey senses tingled late last week and early this week with the release of two new hedge fund studies. The first was eVestment Alliance’s look at small and young funds - version 2.0 of the emerging manager study I first launched at PerTrac in 2006. The second study, authored by hedge fund academic heavy weights Andrew Lo, Peter Lee and Mila Getmansky, looks at the impact of various database biases on aggregate hedge fund performance.

Neither paint a particularly bright picture of the overall hedge fund landscape.

So why aren’t I, Certified Data Nerd and long-time research of hedge funds, rolling around on the floor in piles of printed copies of each study right now? Because, in addition to being a total geek when it comes to a good pile of data, I’m also a big ol’ skeptic, and never moreso than when it comes to hedge fund data.

Here’s the thing, y’all. Hedge fund data is dirty. Actually, maybe even make that filthy. It's "make my momma  want to slap me" dirty. Which is why it is critical to understand exactly what it is you may be looking at before jumping to any portfolio-altering conclusions. Some considerations:

One of the reasons I imagine Lo et al undertook their latest study was to show just how dirty hedge fund data is. They looked at backfill bias and survivor bias primarily, within the Lipper Tass database specifically. Their conclusion? When you adjust for both biases, the annualized mean return of hedge funds goes from 12.6% to 6.3%.

Ouch.

However, let’s consider the following:

No hedge fund database contains the entirety of the hedge fund universe. A 2010, comprehensive study of the hedge fund universe (again, that I completed for PerTrac) showed that 18,450 funds reported performance in 2009. Generally speaking, hedge fund databases cover roughly 7,500 or fewer “live” hedge funds. So, no matter what database you use, there is sample bias from the get go.

And while backfill bias and survivor bias do exist, so does participation bias.

Because a fund’s main motivation to participate in a hedge fund database is marketing, if a fund does particularly poorly (survivor bias) or particularly well (participation bias) it may stop reporting or it may never report. For example, of the top ten funds identified by Barron’s in 2014, three don’t report to Lipper Tass, two are listed as dead, two more aren’t reporting current data and three do report and are current. This could be sample bias or it could be participation bias. Hell, I suppose it could be survivor bias in some way. In any case, it does show that performance gleaned from hedge fund databases could be artificially low, not just artificially high.

As for emerging manager studies – they run into a totally different bias – one I’ll call barbell bias.

Unfortunately, due to wildly unbalanced asset flows over the last five years towards large funds, 85% of all hedge funds now manage less than $250 million. More than 50% of funds manage less than $100 million. Indeed, the hedge fund industry looks a little bit like this:

Some of you may remember my “Fun With Dots” blog from a few months ago. Using that same concept (each dot represents a hedge fund, each block has 100 dots and each line 1,000 dots, for a total of 10,000 dots, or funds) the Emerging vs. Emerged universe looks like this: 

What’s interesting about this is, at least mathematically speaking, every fund in the large and mid sized category could have been outperformed by a smaller fund counterpart, but because of the muting effect that comes from having such a large bucket of small funds, the small fund category could still underperform.

Now, of course, I still found both studies to be wildly interesting and I recommend reading both. Again: Nerd. I also know that people have poked at my studies over the years as well, which, frankly, they should. Part of the joy of being a research nerd is having to defend your methodology. In addition, most people do the best they can with the data they’ve got, but it’s not for nothing that Mark Twain stated there are “Lies, damn lies and statistics."

What I am saying is this: Take all studies with a grain of salt. Yes, even mine. 

In hedge funds - perhaps more than anywhere else, your mileage may vary. You may have small funds that kicked the pants off of every large fund out there. Your large funds may have outperformed your emerging portfolio. You may have gotten closer to 12% than 6% across your hedge fund universe (or vice versa). Part of the performance divergence may come from the fact that it’s hard to even know what the MPG estimates should be in the first place, which is why it’s critical to come up with your own return targets and expectations and measure funds against those indicators and not industry “standards.” 

Sources: Barron's, CNBC, Bloomberg, LipperTass, MJ Alts, PerTrac, eVestment Alliance