
Hello people,
Looking for your opinion on the matter of Kalman filters. Have you successfully used the technique for stock return forecasting? If not, what are the usual applications of Kalman filters within finance? For me to be able to implement it, I will need to invest some serious time to understand it first. This is the reason I am asking whether you guys see any value in it for this purpose.
Regards 




rickyvic


Total Posts: 188 
Joined: Jul 2013 


Should be in basics. Kalman filter is a way to estimate the path of the hidden variables in a state space model. Then the idea is that you need to understand the state space framework and then you apply the kf. Iteratively you can use the kalman filter the compute the conditional expectation of you hidden variable (markovian) the compute the likelihood and so if you have coefficients to estimate you can use ML or similar. You can look at it from the bayesian perspective and you have priors there, that opens up the game to more complex MCMC in case your model is not in closed form.
Stock trading? Why not SS models have applications all over finance and are a good one to keep in your toolbox.
I know there are a lot of applications for trend following where things can get complicated with this stuff.
Keep it simple and robust, bad starting values or priors can cause instability 
"amicus Plato sed magis amica Veritas" 


rusti999


Total Posts: 17 
Joined: Oct 2008 


The Kalman filter itself is the easy part. The hard part is formulating what goes in to your state space model (what hidden states you have, how they evolve over time, and how they give rise to the observed returns). Given you have a good (i.e. closely resembling reality) state space model, then applying the model with the Kalman filter will give you reasonably good forecasts. But again the big if is whether your state space model is good to begin with. 





Yes that is indeed what I have been able to gather. But that also begs the question, if I knew the state space model, or I guess the state space variables, wouldn't that be very similar to knowing the xvariables in a normal linear regression? Wouldn't the only benefit of the Kalman filter in that case be to somewhat improve the accuracy of the model when forecasting?
Or am I missing something? My question is as follows: is there something fundamentally different in trying to determine state space variables compared to determining independent variables in linear regression? Of course this is all in the context of financial time series, I know that these models have been developed for a different purpose from the beginning. 




I'll move my future topics to basics until I've built on my knowledge a bit :) 




rickyvic


Total Posts: 188 
Joined: Jul 2013 


Missing a lot... For reference I would go through this book from durbin and koopman https://www.amazon.com/TimeAnalysisStateSpaceMethods/dp/019964117X
It will also be easy to implement following their algorithms. A lot of the existing packages do not allow easy definitions of models so it is probably easier to follow those instructions to filter and estimate parameters.
The main applications are time varying parameter models, dynamic factor models and autoregressive models. To name a few. 
"amicus Plato sed magis amica Veritas" 



Thanks for the tip. I will look into it and see whether I can handle the maths. 




rod


Total Posts: 391 
Joined: Nov 2006 


I am a bit puzzled by this thread. It makes Kalman filters look trivial.
Trivial? Perhaps in an ideal world in which God grants one a linear state space model and where all noise is additive, white and Gaussian. Who lives in that world?
Even in engineering, where masses and moments of inertia can be measured very accurately, tuning a Kalman filter is a difficult art. Of course, in engineering failure is quite obvious. In econometrics, however...
And the folks at NASA's Jet Propulsion Lab do not use the standard Kalman filters when sending things into space. Not sufficiently robust. They use more robust algorithms one can only find in exotic books that only a few graduate students have even heard of. 



Maggette


Total Posts: 1151 
Joined: Jun 2007 


Agree with rod.
There were exactly two occasions outside of finance (once profesionally with GPS data and once as a state estimation component of a toy project implementing PMC stuff) where I actually used Kalman Filters and new beforehand that it was the way to go.....but still it wasn't easy (or I am dumb, who knows)... and I had to dig deep and came up with some two layer Kalman Filter I stole from a paper.
I now have a very hard time envisioning scenario in finance where I would use Kalman trust the results... 
Ich kam hierher und sah dich und deine Leute lächeln,
und sagte mir: Maggette, scheiss auf den small talk,
lass lieber deine Fäuste sprechen...




Zoho


Total Posts: 28 
Joined: Feb 2018 


"I now have a very hard time envisioning scenario in finance where I would use Kalman trust the results... "
there is example of rather simple scenario in the crosssectional equity approach:
if you have exposures of the stocks to "price" and then you force overall portfolio exposure to be 1 for the current moment and zero portfolio exposure for 1 year ago then when trying to solve this linear system of equations you can apply Kalman filter to get the weights. And the resulting signal will obviously resemble crosssectional momentum. And you can generalize this approach easily. 



rod


Total Posts: 391 
Joined: Nov 2006 


If engineers are intimidated by Kalman filters and quants are not, then there must be a reason. I cannot see what the reason is. What am I missing? 




radikal


Total Posts: 264 
Joined: Dec 2012 


I've used KFs for about a decade in HFT; more often than not they've provided marginal benefit over simpler algorithms like EWMAs and such, but I've appreciated the structure. I've used both EKFs and UKFs in my space days; can't comment on how, but the idea that KFs don't get used in space is crazy  often they are literally the optimal solution.
I'd start here: https://github.com/rlabbe/KalmanandBayesianFiltersinPython
Where things get challenging is that kalman filters lack some very often desired properties (hard bounds, guaranteed numerical stability), there exist some pretty fancy solutions that target this, but in finance, I'd never bother, just have a meta component that ensures things aren't going wonky, and disable/reset in case you end up in unsafe gradient hell. 
There are no surprising facts, only models that are surprised by facts 


rod


Total Posts: 391 
Joined: Nov 2006 


radikal, who mentioned that "KFs don't get used in space"? 




jslade


Total Posts: 1184 
Joined: Feb 2007 


FWIIW my first consulting job was putting a Kalman filter in an early G3 phone with accelerometers. It wasn't so bad, though I had to bandaide it with some tricks to get it to do what I wanted it to (aka ignore gravity).
I know a guy who was successful with Kalmanlike approaches in futures markets. I think he was doing something like what Zoho says. Or, like this:
https://cran.rproject.org/web/packages/MARSS/ https://cran.rproject.org/web/packages/MARSS/vignettes/UserGuide.pdf
I'd be curious about Rod's obscure NASA robust filters, but I think the reason it's trivial is measurement noise is kinda sorta zero, and varma is simpler than shooting rockets into space.

"Learning, n. The kind of ignorance distinguishing the studious." 



EMAs are recoverable basis functions for any linear model. And that extends to any continuous nonlinear model, if you feed them into a universal approximation like a neural network or kernel machine.
Therefore, I'd say your default should just be OLE on a whole bunch of EMAs spanning the full range of potentially relevant time horizons. OLE is still an unbiased linear estimator in the timeseries domain. It's true you need timeseries specific regressions if you want to rely on the pvalues for coefficient signifigance or point estimation. But nowadays you're probably crossvalidating anyways, and the computation cost of regression is basically free.
IMO Kalman filters, or any other statespace model, should probably only be used if you have very strong inductive priors or a small/highvariance dataset. Otherwise just get a shitton of data and throw it at a regression of EMAs. 
Good questions outrank easy answers.
Paul Samuelson 



tbretagn


Total Posts: 277 
Joined: Oct 2004 


Well economists have zero problem fitting kalman filters with 100 points of data so I don't know why you guys are so squeamish about it 
Et meme si ce n'est pas vrai, il faut croire en l'histoire ancienne 


nikol


Total Posts: 784 
Joined: Jun 2005 


Kalman is ok for dynamic smile fitting, except when it jumps. 




rickyvic


Total Posts: 188 
Joined: Jul 2013 


KF is not trivial at all but not even that difficult to implement for basic simple models. Certain is that it is highly unstable if mispecified, which is often the case. There is also a case of sensitivity to starting values, which in the bayesian world would be sensitivity to priors. A sausage machine basically. MCMC works best for state space more complex models, like the ones you can work on in trend following. There too you need to be careful and model instability is often linked to almost non stationary mc samples.
On a more intuitive way you can figure this out easily, you are trying to estimate a hidden state that drives the observable Y, and this state is markov and has some behaviour given by the model of choice. Still the state is not observable, so unless you are denoising, you need to know exactly what to expect from this behaviour or you are just guessing and coming out with a mispecified model.

"amicus Plato sed magis amica Veritas" 


doomanx


Total Posts: 23 
Joined: Jul 2018 


Bit late, but just to weigh in on state space models  I've used them a lot in HFT (for example pressure in related assets). There are some areas where nonlinear dynamics come up and particle filtering is very useful (stochasticvol is the most prominant example I can think of where noise is multiplicative). Some people will do EKF/unscented (I think this is hard to make work in practise). I like Bayesian stats so I use sequential MC for this kind of thing (to make it work in practise you need a resampling step or your importance weights will be crap, but this can be scary with nonstationarity). The hardest part as already mentioned by others are the classic noisy signal processing issues. Choice of prior, GIGO and all that. Filter, filter and then filter again (hint: one can combine the above two approaches via Bayesian anomaly detection). 




Nezoo


Total Posts: 4 
Joined: Nov 2017 


@EspressoLover This is really interesting. I've always done the old regression (OLS, which I presume is the same thing as OLE?) on a bunch fo EMAs, and figured that was a crude hack that I came up with that seemed to get the job done. Didn't know that it was a standard technique or that there was a theoretical basis for it. What do you mean by a recoverable basis function? Quick googling didn't yield any hits. 



rickyvic


Total Posts: 188 
Joined: Jul 2013 


This is what most people do, although I prefer arma models that do pretty much the same thing, you can then make assumptions on the conditional variance of the error (but you can control that effect with regression too). State space too ends up being the same thing, although the biggest advantage is in estimating in one step, which is much better than doing in more steps and with different estimators.

"amicus Plato sed magis amica Veritas" 



Nezoo


Total Posts: 4 
Joined: Nov 2017 


Why is the one step estimation that useful? Also, my problem with ARMA comes when I want to include other possibly nonlinear parameters in the time series model. With the EMA approach, you can pretty easily fit these by just tossing everything (EMA + factors) into a random forest or something. Also the conditional variance estimation approach only really works for normal errors, which don't come up much in my experience, right? 



rickyvic


Total Posts: 188 
Joined: Jul 2013 


Correct, obviously it is easy to overfit that way so it needs care. For conditional variance wrong as garch fits very well daily returns distributions and captures fat tails and more depending on the flavour you use. You can even go multivariate which has obvious implications for portfolio construction. 
"amicus Plato sed magis amica Veritas" 


