Forums  > Software  > Any advancement on compiled vector oriented languages  
     
Page 2 of 2Goto to page: 1, [2] Prev
Display using:  

jslade


Total Posts: 1182
Joined: Feb 2007
 
Posted: 2018-11-29 00:08
I'd posit you aren't using a repl much because the repl you're using is ass. The only purpose in using a shitbag interpreter of any kind is the repl; otherwise just write everything in golang or C or something.

As for thinking numpy's second class status is a ... nice feature of python coding: that's preposterous. If you really care about execution latency to the extent the overhead involved in calling blas matters, you're are not using python or matlab.

You're making an argument for R if you think 'overwhelming amount of ML research and supporting framework' is the most important thing. R has a better stats stack as well.

As for your beef with my statement on python's numeric syntax; you keep using the word 'syntax' -I don't think it means what you think it means.

"Learning, n. The kind of ignorance distinguishing the studious."

prikolno


Total Posts: 45
Joined: Jul 2018
 
Posted: 2018-11-29 00:38
>I'd posit you aren't using a repl much because the repl you're using is ass.
For context, most of my REPL experience has actually been in MATLAB by far, followed by toplevel (OCaml) then IPython with beaker extensions. So I know where you're coming from, I don't know how else I can assure you that this policy didn't arise from limitations of the REPL environment.

>otherwise just write everything in golang or C or something.
Most of my codebase is actually in C++.

>As for thinking numpy's second class status is a ... nice feature of python coding:
I don't think it's a nice feature, like you I actually hate writing numpy more than MATLAB. My point is that the first class treatment of arrays is a nonfactor.

>If you really care about execution latency to the extent the overhead involved in calling blas matters, you're are not using python or matlab.
That's exactly what I'm doing. Maybe it's easier to explain this with a basic example. In production I estimate standard deviation with SMW because it's O(1) in memory and updates. In contrast, numpy and MATLAB both keep the entire sample in memory so it's O(n) in memory. I don't actually want to use `np.std()` or MATLAB's `std()` because our algorithmic implementations are different. If I write research prototyping code using the third party `std()` then translate it over to production, I could end up with alpha signals that are desynced by 0.01-1 bps, which can throw off when my order is sent off in simulation/prototyping vs production by a few hundred events, which is light and day in PnL. A cleaner way is to put a thin Python/MATLAB wrapper over my own routine.

>You're making an argument for R if you think 'overwhelming amount of ML research and supporting framework' is the most important thing. R has a better stats stack as well.
Yes, mostly true, that's why I say it's a debatable advantage because you might as well take it further and advocate for R instead.

>As for your beef with my statement on python's numeric syntax;
>you keep using the word 'syntax' -I don't think it means what you think it means.
I think you're actually mixing up the definitions of "syntax" and "API". There's no "numeric syntax" in the CPython distribution, you're probably referring to numpy's API. CPython's syntax is completely transparent in a single file (https://github.com/python/cpython/blob/master/Grammar/Grammar) and the parser generator that converts that syntax into C is also completely transparent (https://github.com/python/cpython/blob/master/Parser/pgen.c).

EspressoLover


Total Posts: 381
Joined: Jan 2015
 
Posted: 2018-11-29 03:54

Good questions outrank easy answers. -Paul Samuelson

Jurassic


Total Posts: 253
Joined: Mar 2018
 
Posted: 2018-11-29 10:08
> and I know the cutoff digits for large integers is 70.

What are the issues with using large numbers in calculations in Python? (although this could be a programming language in general question)

rickyvic


Total Posts: 187
Joined: Jul 2013
 
Posted: 2018-11-29 15:48
@jslade what do you specifically do with J?

I just would like to say that most of the things that have been said are true... except matlab is slow. It is not because of the jit, which is fast and keeps being faster and obviously BLAS LAPACK.

My point is: Matlab is not complicated but what sucks is to pay that much money and the language lacks a lot of features, lack of coders... It is insane especially because the routines we use are not that complex, if they are you dont want to use them and you write your code.
So julia was a good idea but undeveloped, all the mlns of dollars in this industry and we cant create a matlab clone tuned more for professional users....
Plus it can be used in research, data manipulation, reporting, even databasing (I use it with success with hdf5 api), but not in real-time, it is just not good for that and the matlab coder is not mature enough.

Question is what can we do now.

One thing I would do with matlab in prod is to do all the params calculations, calibration and historical data manipulation, then feed it to a C real time process using sockets (or other message brokers), the C process reads data in real time and runs the real time algorithms.
Excuse me the down to earth language I learnt programming by doing things.

"amicus Plato sed magis amica Veritas"

prikolno


Total Posts: 45
Joined: Jul 2018
 
Posted: 2018-11-29 16:02
> What are the issues with using large numbers in calculations in Python?

Speed. The runtime scales with ~n^1.585 for two numbers with n-digits. There's asymptotically faster algorithms that improve on the exponent but they're slower for small n. If you're doing this very frequently and with large numbers (most often in cryptography), then it makes a huge difference to use a more exotic algorithm. No impact for finance where numbers can be represented in 64 bits.

svisstack


Total Posts: 320
Joined: Feb 2014
 
Posted: 2018-11-29 16:28
microservices, deploy on k8s and forget? doing job -> yes. dont touch.
you want golang for something ok, other thing for something else it's also ok.

i love matlab but's it's has terrible OO which dont make any sense if you can code in something legit like c++, C#, java. worth to mention you can host the matlab code as rest service or convert to c++ (i never used it actually)

www.coinapi.io && www.cryptotick.com

jslade


Total Posts: 1182
Joined: Feb 2007
 
Posted: 2018-11-29 18:01
@sviss: Matlab OO is completely worthless, and nobody should ever write code this way. Better namespaces would help a lot. Their OO thing fails at being better namespaces, and is a third rate student implementation of OO using eval.

@rickyvik: you do realize that Octave exists right? Most of the low level Matlab packages have been more or less cloned, and the IDE is almost as good. FWIIW matlab had this RtOS target called Xpctarget at one point; I built an inertial navigation gizmo with it. Supposedly you can compile to verilog as well.

I use J for developing new ML and statistical/signal processing algorithms. One of the killer apps; linking to arbitrary C libraries is literally a one-liner. I also use the database for a data tee-off on a feed handler (it's been up for a year and a half now -the EC2 instance is less stable), and a liquidity finding gizmo. It takes a year or two to be useful in it, but it is the highest productivity language I have ever used. I haven't been using it much in the last year and a half, but it's my jackhammer and stradivarius. It's a fairly antisocial programming language, but I'm a misanthrope and prefer not to talk to anyone else anyway.

@prikolno -I see where you are coming from now. I suppose the only use for a repl in your world is running stats and looking at charts -and those are probably fixed enough you'd want them in a static report.

@espresso -I'll let you in on a secret: silicon valley engineers are mostly yeast life. They do things like slap together "cloud" crap in containers not because it's a sound practice, or interesting engineering, but for the same reason they all wear hoodies well into their 30s. They're mostly powerless lemmings, and the main reason "the cloud" exists as a 'go-to' is because accountants find calculating depreciation too difficult.

"Learning, n. The kind of ignorance distinguishing the studious."

rickyvic


Total Posts: 187
Joined: Jul 2013
 
Posted: 2018-11-29 19:30
J is interesting, will never find a developer though....
Octave is not good imo, nothing good about it

"amicus Plato sed magis amica Veritas"

prikolno


Total Posts: 45
Joined: Jul 2018
 
Posted: 2018-11-29 20:00
> I see where you are coming from now. I suppose the only use for a repl in your world is running stats and looking at charts

Yes, glad we're synced. And we do a lot of the modeling and exploration work outside of a REPL.

> linking to arbitrary C libraries is literally a one-liner.

That's actually pretty cool.

Maggette


Total Posts: 1147
Joined: Jun 2007
 
Posted: 2018-12-03 10:06
Everybody is different...but taht is IMHO not the advantage of an REPL.

Working with a REPL is kind of "poor mens test driven development". Lots of code I write is already tested (at least in a trivial way) even before I write a unit test for it.

That has vastly improved my development speed.

Also it is quite great to explore new APIs and language features.

Ich kam hierher und sah dich und deine Leute lächeln, und sagte mir: Maggette, scheiss auf den small talk, lass lieber deine Fäuste sprechen...

rickyvic


Total Posts: 187
Joined: Jul 2013
 
Posted: 2018-12-10 19:04
Basically the two solutions I found acceptable for my workflow and to avoid going insane are:
1)
a)matlab for research offline and hdf5 for db, which so far worked out ok.
b) An api/gateway connection which will push data to a shared memory and keep a book of real time quotes up to date (this is being developed in C++).
c) Then all I need to do is to write the real time algo in matlab and then compile to C or any low level machine code (to be studied and tested).
2)
doing everything in a language like J but not sure how it works in realtime. It means a year from now so will be version 2.
Probably will need to interface to C.

This is for an algorithmic trading system


"amicus Plato sed magis amica Veritas"

bullero


Total Posts: 50
Joined: Feb 2018
 
Posted: 2018-12-10 23:41
@rickyvic - Why don't you just write everything in C/C++? If you really really really want to mix languages you can make them talk via ZeroMQ for example. Now, if you need to solve some math systems quickly you have libs like Blaze, alglib and xTensor for that so no need for matlab or any other matrix language. And finally whats up with the shared (main?) memory, its quite slow.

rickyvic


Total Posts: 187
Joined: Jul 2013
 
Posted: 2018-12-11 12:18
@bullero
thanks for your input. There is no reason to make the research environment in C++, it would take ages and we need rapid prototyping. All we need to do is writing params to a db which will be loaded by the trading algos.
For prod it would be good to have a fast vector oriented language to port the code quickly. Shared memory, I meant a process that reads data and pushes them via zeromq and the algo reads them.
Otherwise the idea would be using a shared memory so all algorithms can read from the same table in parallel.

But I guess that would be slow... so your suggestion is to read the raw data, process them (so rebuild OB and record transactions) in memory and run the algo on the dataset?
Then replicating the same for all strats?

"amicus Plato sed magis amica Veritas"

bullero


Total Posts: 50
Joined: Feb 2018
 
Posted: 2018-12-11 17:00
@rickyvic - Difficult to give any specific suggestions because (in my opinion (whatever its worth ;) )) the implementation is dependent on your trading logic and trading/investment time scale. However, I think its clever to do the research 'backtesting' using the same system you would use for live action. Again, depends on your strategy and time scale. For a traditional 'style investing' approach this is not crucial issue - for a latency critical stuff more so. (I do not mean you should not worry about data granularity)

Now, for intra-day system research I would do the following: I would replay a trading day by sending those market data messages to my system as if it was going live. After consuming the market data I send my actions to (a) hypothetical exchange(s) which will then send me back simulated fills in case such an event happens. The exact simulation of fills is quite complex in practice but you can try to get some sort of statistical estimates. The statistical behavior of this hypothetical exchange may be replicated (to a point) using information distilled from the same market data that I am sending to my box. Btw, there is a large body of literature out there giving you the mathematical framework for building this.

rickyvic


Total Posts: 187
Joined: Jul 2013
 
Posted: 2018-12-11 19:25
This was the whole point of the discussion, if there is a way to code up in a nice language vector oriented the research and then put the same in production or make minimal changes for production.

What came out is that someone uses J effectively and that is fast, Julia has not gone anywhere and matlab is effective offline, unusable in real time, but usable in prod for offline jobs, it can be compiled to C and hardware.

One last thing is that C++ numerical libraries can make the transition easier, bullero says Blaze, alglib and xtensor. Thanks for this, armadillo came out of my search but I dont know how reliable that can be.

"amicus Plato sed magis amica Veritas"

rickyvic


Total Posts: 187
Joined: Jul 2013
 
Posted: 2019-05-09 12:14
Guys someone (not completely a finance quant) told me Julia is gaining traction again. He uses it and since version 1.0 it is much more stable.
There is a conference in Baltimore in July.
Worth a shot maybe.

"amicus Plato sed magis amica Veritas"

Rashomon


Total Posts: 203
Joined: Mar 2011
 
Posted: 2019-06-27 06:21
rickyvic, I can hook you up with a J developer if needed.
Previous Thread :: Next Thread 
Page 2 of 2Goto to page: 1, [2] Prev