Forums  > Trading  > Historical order book reconstruction API for crypto markets  
     
Page 1 of 1
Display using:  

tardis_thad


Total Posts: 8
Joined: Aug 2019
 
Posted: 2019-08-15 16:22
Hi, I'm a founder of tardis.dev which I've built to scratch my own itch as couldn't find anything on the web that would give me historical high resolution data for crypto markets (futures, swaps, options included) with affordable pricing for individuals hence created my own - hope some of you will find it as useful as I do.

It's raw non aggregated historical data API including full order book snapshots & incremental delta updates, tick by tick trades etc. Currently supported are BitMEX, Binance, Deribit, Bitfinex, Bitstamp, Coinbase Pro, Kraken, Crypto Facilities and OKEx.

There is a RunKit playground on the homepage so you can play around with the API and client libs that are available on GitHub (https://github.com/tardis-dev).

I'd really appreciate any feedback you have.

Thanks! Thadeus

nikol


Total Posts: 803
Joined: Jun 2005
 
Posted: 2019-08-15 17:16
It seems to me that:
- Latency for cross-exchanges arb business is important, therefore, for each of this exchanges you should measure and disclose it. And if you collect data from one single location, then its value drops.
- Cryptotraders store their own data (once you have your own feed and internal reconstruction of order book, the rest is just organizational). Once you have it, to start listening and storing new exchange is the matter of 1 day or 2.
- Consistency: for example, by construction, my data are internally consistent with my API from the very beginning.

On the positive side: Web-site design looks good.

I really do not want to discourage. If you believe in it, shoot it.
Good luck!

tardis_thad


Total Posts: 8
Joined: Aug 2019
 
Posted: 2019-08-15 17:35
Thanks, really appreciate the feedback, not discouraged at all, it's a learning experience for me as I'm long time programmer not algo trader. Noted about latency. I measure it already, but it's not yet available via API and not sure if measurements are granular enough, will think about it. I'm aware about single location cons, but right now can't afford to have it colocated per exchange and in some cases 'middle location' (London) can be good enough for certain scenarios (eg: arb between BitMEX - Ireland and Deribit-France) at least it was my line of thinking. I'm aware that more professional firms often invest in own infra that collects the data, but I wouldn't call it 1 or 2 day job - at least it was mine experience. Not sure what you meant by consistency, but data provided by the API is in exchange format so I assume that anyone trading on the exchange will have to know the format anyway and it will be easy enough to convert it to desired format anyway as it's what needs to be done for real-time data anyway.

I wanted to reach people in similar situation to mine - "indie algo traders", but perhaps your're right it's non existing market for this kind of data as most of such traders don't use tick data or store it on their own and larger players have resources to build own infrastructure where they control everything. Anyway this API is something I'm using so at least it's that :)

nikol


Total Posts: 803
Joined: Jun 2005
 
Posted: 2019-08-15 18:42
> API is in exchange format

Yes, of course, you talk about json. Confirmed.

About latency: you can record your PC clock to all data ticks, so everyone can see it while streaming data. If data is collected over the grid of servers, then find the way to sync all those clocks. In principle, most of exchanges have minimal latency of x10 ms (except very few). It is huge comparing to the scale of under-mks, but still...

tardis_thad


Total Posts: 8
Joined: Aug 2019
 
Posted: 2019-08-15 19:07
Indeed, most of the crypto exchanges real-time streams are WebSocket and JSON based, I know some provide FIX but it's not that popular in this space and even it there is FIX it's often built on top of WebSocket (eg Bitfinex).

I see now what you mean about latency, I already do that and each message received has local timestamp stored as well as it's all available via API. I also measure every minute ping and http request/response time for each exchange and assumed it's what you were talking about.
When researching similar services in crypto space I couldn't find historical order book data that was tick level, what I could find was at most 1s snapshots with really high price tag and that pushed me to build something like this - essentially crypto market data replay API - in principle similar to https://www.dxfeed.com/market-replay/ but of course not the same scale.

nikol


Total Posts: 803
Joined: Jun 2005
 
Posted: 2019-08-15 19:11
see this discussion

http://www.nuclearphynance.com/Show%20Post.aspx?PostIDKey=184987

tardis_thad


Total Posts: 8
Joined: Aug 2019
 
Posted: 2019-08-15 19:51
Thanks, I'm already familiar with CoinAPI, it's a fine service and some good points in that thread, although I'm pretty sure they do not provide non aggregated historical order book data and it's only 20 top bids/asks if I understand correctly their docs hence not something I was after as wanted to have a way to reconstruct full market state in any given moment in the past tick by tick.

nikol


Total Posts: 803
Joined: Jun 2005
 
Posted: 2019-08-16 17:23
ICE is your competitor?

"The feed delivers real-time and historical data for the most widely and actively traded cryptocurrencies. Covering 60+ cryptocurrencies from leading venues, markets and exchanges across the globe."

https://www.theice.com/market-data/connectivity-and-feeds/consolidated-feed/coverage-list/cryptocurrencies

tardis_thad


Total Posts: 8
Joined: Aug 2019
 
Posted: 2019-08-16 18:09
Yes, it's in the same market space. It has partnered with blockstream https://blockstream.com/cryptofeed/ to provide such data. There isn't much details what that data feed includes exactly, but I'm pretty sure it's not historical order book data, but real-time only. If anyone has access to it I'd love to know! Overall it feels a lot enterpisey. BTW, I'm not sure why one would like to connect to exchange via 'proxy' that is additional potential point of failure, when can do this directly for free, but perhaps it works fine for certain use cases.

nikol


Total Posts: 803
Joined: Jun 2005
 
Posted: 2019-08-16 20:40
Cannot help, sorry. For me it was a discovery as well, related to news about Bakkt 'cleared to launch Bitcoin futures' via ICE for institutional investors with physical settlement (i.e. with Bitcoins). No idea where institutionals will get BTC if they are officially not yet in the game.

jshore


Total Posts: 1
Joined: Sep 2008
 
Posted: 2019-09-03 10:40
It might be more useful to provide normalized L3 data and/or L2 as opposed to the, sometimes very funky, json formats these exchanges use. There are a few reasons why this might be a good idea:

- many of the exchanges randomly drop trades (so u would need to do a rest call to backfill missing trades)
- your connection will drop at random or periodically, so you need to have multiple connections, ideally from different data centers; gluing this back together into a time-based sequence is easier to do with some sort of record format.
- finally, would save consumers of your data a lot of time in parsing the disparate formats.

Perhaps best to provide both, while knowing that the raw feed will be periodically disrupted and *will* miss trades. I have connected to most of the above and more and we see missing trades in the feed (when this can be detected by a sequential trade ID) multiple times per day.

tardis_thad


Total Posts: 8
Joined: Aug 2019
 
Posted: 2019-09-03 12:18
Agree, in fact it's something I'm working on currently (normalization of L2 and trade data). It will be available as separate API endpoint and via official API clients.

Do you have specific exchanges in mind when it comes to missing trade data via WebSocket connection?

nikol


Total Posts: 803
Joined: Jun 2005
 
Posted: 2019-09-05 19:41
@tardis_thad:

Tip: there is certain interest in historical funding within perpetuals. You could imply it from your data and offer to the market.

tardis_thad


Total Posts: 8
Joined: Aug 2019
 
Posted: 2019-09-06 06:32
Indeed, I already store and provide funding info for perpetuals (from exchanges WS feeds), but as you mentioned that could also be done by recalculating it based on exchange formula and historical market data (which exchange uses to create index)

svisstack


Total Posts: 322
Joined: Feb 2014
 
Posted: 2019-09-20 17:59
>> Thanks, I'm already familiar with CoinAPI, it's a fine service and some good points in that thread, although I'm pretty sure they do not provide non aggregated historical order book data and it's only 20 top bids/asks if I understand correctly their docs hence not something I was after as wanted to have a way to reconstruct full market state in any given moment in the past tick by tick.

CoinAPI here;-) We have the full order book in realtime in API product and historically in the flat files, the order book is tick by tick or as granular as exchange delivers and not limited by the number of levels.

Flat files are available via separate website: https://www.cryptotick.com/
Listing: https://www.cryptotick.com/Shop/Files?path=datatype-exchange-date-symbol/limitbook_full
Sample: https://s3.amazonaws.com/coinapi-samples/limitbook_full/20171229/27606-BITSTAMP_SPOT_BTC_EUR.csv.gz
Docs: https://www.cryptotick.com/Faq


www.coinapi.io && www.cryptotick.com

tardis_thad


Total Posts: 8
Joined: Aug 2019
 
Posted: 2019-09-25 06:54
@svisstack sure, I was talking about CoinAPI which provides historical top 20 bids/asks via REST API if I understand the docs correctly and since it's an API it's more comparable to tardis.dev API. Wasn't trying to confuse anyone, apologies if that was your impression. Your service have historical data that goes further back in history which may be important for some.

radikal


Total Posts: 265
Joined: Dec 2012
 
Posted: 2019-10-07 14:10
I'm pretty impressed. I use CoinAPI for a few pieces of my crypto infra (exchanges we don't connect and record natively) but can definitely see using Tardis to fill gaps, replace missing data, etc. Access to historical funding data is a big win potentially as I don't know have that, for example.

Point being, keep it up! You'll likely have our fund as a customer =)

There are no surprising facts, only models that are surprised by facts
Previous Thread :: Next Thread 
Page 1 of 1