Forums  > Software  > mogoDB + arctic?  
     
Page 1 of 1
Display using:  

Strange


Total Posts: 1528
Joined: Jun 2004
 
Posted: 2018-12-26 17:49
Anyone has any opinions on replacing an SQL DB with MongoDB and using arctic as an overlay for storing time series data and such? What are the pros and cons?

I don't interest myself in 'why?'. I think more often in terms of 'when?'...sometimes 'where?'. And always how much?'

ax


Total Posts: 77
Joined: Jul 2007
 
Posted: 2018-12-26 18:09
i found mongo to be easy to load and difficult to extract. SQL is simple so any database with sql syntax is my first choice. then look into clustering and see if that's an issue; cassandra wasn't easy when clustering was involved

TonyC
Nuclear Energy Trader

Total Posts: 1287
Joined: May 2004
 
Posted: 2018-12-27 05:36
check if your shop has a kdb site license (I seem to have a vague memory that they might)


flaneur/boulevardier/remittance man/energy trader

bullero


Total Posts: 33
Joined: Feb 2018
 
Posted: 2018-12-27 07:30
We gave it a go - I wrote some code on it and it felt okay. If my memory serves me correctly they were able to improve the performance vs. plain vanilla mongodb (?).

Patrik
Founding Member

Total Posts: 1358
Joined: Mar 2004
 
Posted: 2018-12-29 19:46
I've had a decent experience with this: https://www.timescale.com/

depends on use case.

Capital Structure Demolition LLC Radiation

Strange


Total Posts: 1528
Joined: Jun 2004
 
Posted: 2018-12-30 16:38
Well, I am hoping to use a single storage software for
(a) options data (actual snapped option chains)
(b) parametrized vol surfaces
(c) pre-calculated time series of things like variance swaps etc
(d) actual tick data
(e) bar data (1 min and 15 min)
(f) various structured data like event weights
(g) pre-calculated stuff for back-tests
(h) backtest results
(z) etc

I know that I can get away with pretty much any DB engine, SQL or no-sql, but I am trying to figure out if there is any advantage to either. The reason I am looking at mongoDB + arctic is that arctic has some nice things like versioned storage of time series, chunk store etc. Mongo should be able to do everything else on my list and, if I feel like it, I can try to build new storage engines for arctic specific to my data types.

I don't interest myself in 'why?'. I think more often in terms of 'when?'...sometimes 'where?'. And always how much?'

Patrik
Founding Member

Total Posts: 1358
Joined: Mar 2004
 
Posted: 2019-01-01 22:45
If you only need data access from python then arctic may be a reasonable choice. If you aren't 100% sure on "python only" then it may be dicey.

Comparing to timescale (postgres extension) the one thing on the list that isn't obvious is the versioned storage - you'd need to support that through the schema you define yourself one way or another. Pros would be standard postgres so you get client lib access from anything and the operational robustness and tooling of postgres. Mixing time series and non-timeseries data in queries also becomes simple. In general, at least to me, the query power in SQL tends to win out.

Capital Structure Demolition LLC Radiation

anonq


Total Posts: 35
Joined: Aug 2018
 
Posted: 2019-01-02 02:15
I haven't tried using arctic but do use mongodb currently for backtest results, config, and some daily data. Like it a lot for that. I did try storing more granular data but found I would need a decent sized cluster to make it performant. I have some servers in a private datacenter as well as make use of aws so having to host two clusters made it a no go for me.

Strange


Total Posts: 1528
Joined: Jun 2004
 
Posted: 2019-01-04 06:50
Thanks everyone. Upon some consideration, I've decided to continue with a regular SQL database. Whatever pros there are to MongoDB, I decided that it's not worth dealing with yet another product and switching stuff up.

I don't interest myself in 'why?'. I think more often in terms of 'when?'...sometimes 'where?'. And always how much?'

mtsm


Total Posts: 231
Joined: Dec 2010
 
Posted: 2019-01-04 15:57
SQL DBs are fundamentally different from columnar DBs and not really as suitable for time series, although that ultimately depends on what kind of time series you are dealing with.

I believe that some SQL products such as MS SQL have features allowing one to optimize columnar aspects of tables, but no idea how good that is compared to KDB+ for example.

Arctic and Mongo are easy to set up and geared mostly towards python users. I also wonder how feasible that setup is as a production solution.

KDB+ is a whole different world. It's a functional programming environment with a somewhat arcane language, Q. It's awesome, blazingly fast for select vectorized operations, but also has it's limits and needs to be interfaced with other programming environments, unless maybe you deal with pure cash products, but even there I have not seen it being entirely self-contained.

Strange


Total Posts: 1528
Joined: Jun 2004
 
Posted: 2019-01-26 17:20
I think for the amount of data that I have, SQL tables with smart indexing would be good enough. Nothing I do justifies a kdb+ license, that's for sure.

I don't interest myself in 'why?'. I think more often in terms of 'when?'...sometimes 'where?'. And always how much?'
Previous Thread :: Next Thread 
Page 1 of 1