Forums  > Pricing & Modelling  > Maintaining production ML Models  
     
Page 1 of 1
Display using:  

Nezoo


Total Posts: 2
Joined: Nov 2017
 
Posted: 2018-07-20 04:05
Suppose you've developed a machine learning model for something. Presumably the underlying parameters should drift slowly and continuously over time. This problem is probably more of an issue in finance than other industries given the nature of things. How do you tend to deal with this? Just retrain on rolling time intervals of data? Train models on longer time frames and shorter time frames and perform some sort of model ensembling?

schmitty


Total Posts: 55
Joined: Jun 2006
 
Posted: 2018-07-22 07:37
Model your ML model parameters directly basis the center of your rolling training window, with relevant features and left-side (of your training window center) residuals as inputs.

finanzmaster


Total Posts: 158
Joined: Feb 2011
 
Posted: 2018-07-22 19:32
You need an AI, which will optimally decide when to re-train you ML :)

Well, there is no universal recipe, often one does a periodical retraining, e.g. bi-weekly. Not because it is mathematically optimal but merely because it is simple.

Another approach may be to have two copies of your model: one in production and another, recalibarated overnight. Then you can estimate how they diverge and if it exceeds some threshold, you investigate what's going on and what to do further.

www.yetanotherquant.com - Knowledge rather than Hope: A Book for Retail Investors and Mathematical Finance Students

finanzmaster


Total Posts: 158
Joined: Feb 2011
 
Posted: 2018-07-22 19:33
You need an AI, which will optimally decide when to re-train you ML :)

Well, there is no universal recipe, often one does a periodical retraining, e.g. bi-weekly. Not because it is mathematically optimal but merely because it is simple.

Another approach may be to have two copies of your model: one in production and another, recalibarated overnight. Then you can estimate how they diverge and if it exceeds some threshold, you investigate what's going on and what to do further.

www.yetanotherquant.com - Knowledge rather than Hope: A Book for Retail Investors and Mathematical Finance Students

ronin


Total Posts: 343
Joined: May 2006
 
Posted: 2018-07-26 09:25
If your model needs training, then it obviously needs periodic re-training. Every few years or so, even the universe may be substantially different. So the parameters will be different as well.

But I wouldn't trust a model that you have to re-train more than every few years or so to accomodate the changing universe. Bi-weekly would be an immediate no, at least from me.

"There is a SIX am?" -- Arthur

Maggette


Total Posts: 1056
Joined: Jun 2007
 
Posted: 2018-07-26 10:51
There are applciations (outside of finance) where the "state" of the system changes regulary. You can use stacking here => a meta-model that decides which state you are in and which model to use. You train that once and not retrain it frequently.

I want to warn you that I only have seen this approach work in areas where the switch of the system is very obvious and often has a physical/logical explaination. Your signal to noise ratio has to be very good. Or else you gonna fit noise.

If you have to retrain your model very frequently you are most probably fitting noise. I mean think about it. What would you think are the effects you are learning there?

Ich kam hierher und sah dich und deine Leute lächeln, und sagte mir: Maggette, scheiss auf den small talk, lass lieber deine Fäuste sprechen...

skAnkh


Total Posts: 1
Joined: Aug 2018
 
Posted: 2018-08-09 14:11
Hi, you might be interested by this PhD thesis: https://ir.uiowa.edu/cgi/viewcontent.cgi?article=5248&context=etd

Especially chapter 4, which defines the notion of "concept drift", how to measure it and how to deal with it. Have a good read
Previous Thread :: Next Thread 
Page 1 of 1