This post was last updated on August 23rd, 2019 at 04:06 pm
Note: The purpose of this article is to explain the reasoning and backstory behind our cryptocurrency exchange ratings. If you’d rather just jump to the ratings themselves, you can go to our exchange index page.
On March 19th, Bitwise came out with a report to the U.S. Securities and Exchange Commission (SEC) that made two claims of particular interest to us.
Claims from the Bitwise report of particular interest to us . . .
- Only 10 exchanges had actual volume
- 95% of reported exchange volume is fake
As a data company, the first thing we did was look at these exchanges to see what the commonalities were:
What our examination of the Bitwise report has revealed
Eight of the 10 cryptocurrency exchanges identified by Bitwise as good actors provide historical trade level data (i.e. the most granular and audible form of pricing data available) about exchange activity. That is, the one thing we found in common among the good actors identified by Bitwise, is they were very transparent about trading activity.
In contrast, we found that — of the exchanges explicitly called out by Bitwise as bad actors — every single one of them provides limited trading history and virtually no granularity around trading activity.
Transparency of Trusted Exchanges
| Trusted Exchanges | Historical Trade-Level Data |
|---|---|
| Binance | Yes |
| BitFinex | Yes |
| bitFlyer | Yes |
| Bitstamp | Yes |
| Bittrex | No |
| Coinbase | Yes |
| Gemini | Yes |
| itBit | No |
| Kraken | Yes |
| Poloniex | Yes |
Transparency of Suspect Exchanges
| Suspect Exchanges | Historical Trade-Level Data |
|---|---|
| CoinBene | No |
| RightBTC | No |
| CHAOEX | No |
| IDAX | No |
| LBank | Yes |
| BitForex | No |
| Exrates | No |
| OKEx | No |
| BitMart | No |
| ZBG | No |
| CoinTiger | No |
| SIMEX | No |
| Coinsuper | No |
| Bit-Z | No |
| EXX | No |
| IDCM | Yes |
In other words, the good actors from Bitwise’s report had highly transparent data practices, and the bad actors were largely not transparent.
It makes sense that opacity around exchange data would be correlated with fake volume, toxic activity, and wash trading. Indeed, just like an IRS audit, the more data history and granularity provided by an exchange engaging in nefarious activities, the more likely they are to be caught.
On the other hand, upstanding exchange operators have every incentive to provide high-granularity data (with as much history as possible), as this kind of transparency attracts market makers, generates the broader discovery of an exchange’s markets and trading pairs and engenders trust among institutional traders, investors, and regulators.
We’re excited about the Bitwise report (and the prospect of a Bitwise ETF). The report does a lot to educate the space, is directionally correct, and has forced a lot of people in our industry to do much better.
But, we also believe the Bitwise report left a lot of questions unanswered.
The two things about the Bitwise report that surprised us
- First, we’ve been surprised by the lack of scrutiny around the report, particularly by crypto twitter, which has a reputation for being an all-out war zone.
- Second, we’ve been surprised by the degree to which folks took a document meant to market the approval of Bitwise’s ETF to the SEC, and overgeneralized the findings to apply to all exchanges (even ones not analyzed by Bitwise) and all cryptoassets. Indeed, Bitwise only reported findings on BTC/USD and BTC/USDT pairs.
In short, I think that the crypto community’s response to the report lacked criticality and nuance.
Don’t get me wrong, as a Bitcoin hodler, I want this ETF to be approved. But I believe the claims made by the report are just too large and global to take at face value. And that a more nuanced conversation needs to be had.
When I consider why the Bitcoin world just ran with the claims made by the report, I’m left with two explanations:
- The first is a kind of confirmation bias. We tend to not question things that confirm our assumptions about what is true. And the history of cryptocurrency is riddled with vulnerabilities associated with exchanges, starting with Mt Gox, and continuing until Is Quadriga or DragonEx, or whatever the latest exchange hack is at the time you’re reading this.
- The second reason why I think the Bitwise report received little public scrutiny, is even advanced operators don’t know much about how exchange data really works.
(Note: we do a weekly webinar — which you can find at nomicswebinar.com — about exchange data and how it works. We’ve had everyone from quants, to regulators, to executives and analytics from large crypto companies attend and say that they learned new things. I think the ins and outs of this space are just hard to sink your teeth into, and that’s a contributing factor.)
Our 7 Critiques of The Bitwise Report and The Public’s Response
With all of this established, I’d like to spell out our 7 critiques of the Bitwise report (and the public’s response to it):
- Critique #1 is that the report’s primary purpose is to persuade the SEC to allow a Bitwise ETF. And for this reason, it has an inherent bias. That is, the document is a marketing document first, and a research document second. And as a research document, it was not subject to a formal peer review system that academic research papers are subject to.
- Critique #2 is our most superficial criticism. And it’s that the report seems to have a base 10 bias. The number 10 seems very curious: why not 9 or 11? This appears to embody “base 10 bias.”
- Critique #3 is that their conclusions are not falsifiable, which means that they are not trustless. When the Bitwise team says that only 10 exchanges have actual volume, the determination about which exchanges pass muster vs. not appears to be a qualitative call. That is, someone at Bitwise is looking at the data and giving individual exchanges a thumbs up or thumbs down and there is no stated independent test that stands apart from Bitwise, that a third party can independently apply. You either have to trust their conclusions or not. (Note: Bitwise’s report is a presentation, which is an inherently non-falsifiable form of evidence. It’s my understanding that Bitwise will be sharing a more thorough analysis that will be falsifiable in nature).
- Critique #4 is related to critique #3, and it’s that the cryptosphere has generally responded to the report as if its conclusions are stuck in time; there is far too much over-extrapolation happening on Twitter. Bitwise does not state a formal methodology that can be used to independently rate an exchange so a 3rd party cannot update the list. So today, some of the exchanges identified by Bitwise as being good actors might have flipped and be engaging in wash trading. And exchanges that Bitwise did not indicate as having “actual volume” may have cleaned up their practices.
- Critique #5 is that people have over-extrapolated the report to apply to cryptoassets other than Bitcoin, to markets other than BTC/USD & BTC/USDT, and to exchanges that don’t list Bitcoin. This report, as it stands right not, does not apply to cryptoassets other than BTC, to markets other than BTC/USD & USDT, and exchanges other than the 80 they examined (all of which list Bitcoin). While Bitwise’s analysis was perfect for a Bitcoin ETF, it’s a mistake to apply their findings broadly to all crypto markets.
- Critique #6*
is that there is no stated timeframe for the most important data point in the report. For example, they assert that 95% of the volume on global crypto exchanges is fake. Is this volume on one day of analysis? Is this for 2019 YTD? 2018? The lack of details is important to note. - Critique #7 is that the public’s response to the Bitwise report has been unfair to upstanding exchanges. There are many exchanges like ShapeShift and IDEX that have gone out of their way to comply with regulators, and in ShapeShift’s case, have ostensibly experienced layoffs as a result. Yet too many readers have given the report a cursory glance and concluded that there are only 10 exchanges with true volume: this is an incorrect assumption.
* Thanks to Alex Sunnarborg of Tetras Capital for correcting me on this. Bitwise does report dates on the 95+% fake BTC volume claim on page 61 of their report. We were looking at slide 23, which does not provide dates (while slide 22 & slide 24 do provide dates).
Introducing Nomics’ Crypto Exchange Transparency Scores
Disclosure: Nomics’ customer base includes exchanges. While this might affect our rankings, we should note that these ratings are not subjective: we are simply checking our database and exchange APIs to survey the quality and history of available data. Nomics does not charge for exchange listings.
The purpose of these scores is to rate cryptocurrency exchanges by their willingness to provide auditable history. Again: just like an IRS audit, the more data history and granularity provided by an exchange engaging in nefarious activities, the more likely they are to be caught. And upstanding exchange operators have every incentive to provide high-granularity data (with as much history as possible), as this kind of transparency attracts market makers, generates the broader discovery of an exchange’s markets and trading pairs and engenders trust among institutional traders, investors, and regulators.
Transparency Rating: A+ (Nomics Verified)
A+ Verified Exchanges have completed a “deep data integration” with Nomics. This integration brings current and historical gapless raw trade data to investors via the Nomics.com website and market data API. The A+ distinction also means that Nomics has vetted an exchange’s trade history and verifies that an exchange meets the highest reliability, consistency, and data integrity standards. (To learn more about how an exchange can get A+ verified, please see this article on integrating your crypto exchange with Nomics).
Transparency Rating: A (Full Transparency)
Exchanges with a transparency score of “A” provide high granularity (trade-level) data with full history. These are exchanges that provide every trade on every currency pair, dating back to the inception of these exchange trading pairs.
Transparency Rating: A- (Very Good)
Exchanges with a transparency score of “A-” provide the same high-quality data as exchanges with an “A” rating, but have received a downgraded rating due to qualitative factors such as active SEC investigations, wash trading indicators, etc.
Transparency Rating: B (Good)
Exchanges with a transparency score of “B” provide current trade data (but not full trade history) and recent candles.
In order for an exchange to get this grade, they must provide, at the very least, this much candle history:
- 1-Day Candles: 7 candles with last candle occurring within a rolling 48 hours
- 1-Hour Candles: 24 candles with last candle occurring within a rolling 2 hours
- 1-Minute Candles: 60 candles with last candle occurring within a rolling 10 minutes
Transparency Rating: C (Fair)
Exchanges with a transparency score of “C” provide recent candle data but no trade data.
In order for an exchange to get this grade, they must provide, at the very least, this much candle history:
- 1-Day Candles: 7 candles with last candle occurring within a rolling 48 hours
- 1-Hour Candles: 24 candles with last candle occurring within a rolling 2 hours
- 1-Minute Candles: 60 candles with last candle occurring within a rolling 10 minutes
Transparency Rating: D (Poor)
Exchanges with a transparency score of “D” provide non-historical ticker information.
Transparency Rating: F (Fail)
Exchanges with a transparency score of “F” do not provide a public market data API.
How These Ratings Affect Our API & Website
Ratings On Nomics.com
Our exchange ratings can currently be seen in three places on Nomics.com
- The first place is on our crypto exchange index page
- The second place is on the markets tab of any currency page (for example, see the markets tab for our Ethereum price page)
- The third place is the markets tab on our trading pairs page (for example, see the markets tab for our ETH/BTC trading pair page)
In addition, we’ll allow users to flip a switch to view any page on Nomics.com through the lens of data provided exclusively by A-level exchanges.
Ratings Via Our Market Data API
Very soon, these exchange ratings will be included in our cryptocurrency API. Further, all of our aggregate metrics will be filterable by exchange grade.
Conclusion
We welcome your feedback on this thinking. We’re available via the comments in this blog post, our forums, and this contact form. If you have questions or comments, we are eager to engage.
