skpatel73 Hi all, I'm a newbie, so please take it easy on me on my first post. I've read a lot about how curve-fitting a system is bad, and I can definitely understand that for a static system whose rules don't change, curve-fitting is a disaster. However, if you design a system that adapts over time by adjusting a few key parameters, in theory you can adjust to changing market conditions and perform a lot better. In essence, you are dynamically curve-fitting, and on paper you can come up with some great looking systems. Is this basically just as bad as curve-fitting in a static system whose parameters don't change? Or is this even curve-fitting at all? Or does it approach curve-fitting when the system can adjust itself too quickly, but it is not so much with a longer time constant? This is the kind of thing that I can test and test, but it seems to go against conventional wisdom. Any advice? Thanks!! Samir
Roscoe Excellent point. In many cases curve fitting is taken to mean something like “making things better after the fact” by such means as repeated testing with varying parameters until no more improvement can be found, while adaptive systems are usually designed to respond to market information (often volatility) as at the current last bar. So what is the difference? To me, curve fitting means altering history with information that could only be gained in the future (ie. the end of the test period) while adaptive systems can only work with the information available at the end of each bar. Does that help?
tobbe IMHO, it’s worse. An adaptive system has more degrees of freedom than a non-adaptive system and is by design more susceptible to curve fitting than a corresponding non-adaptive (static) system. Both types of systems need initial optimization but it is harder or impossible to find the robustness of the set of parameters that are allowed to vary in the adaptive system. Even if you back test to put constraints on how the dynamic parameters are allowed to change, the uncertainty on system performance has increased in comparison. Trying to assess the robustness of a system is essential (to me, at least) and is hard enough without the added complexity of dynamic adaptation. There are other ways of adaptation to market changes that might be more robust though. For example if you have two systems that are negatively correlated it might make sense to try to identify the conditions that can be used to decide when to take trades from either system. And some systems are inherently adaptive (Bollinger etc). There are many good discussions on these topics at this forum. If you haven’t already, try the search function and look for “walk forward” and “adaptive” for starters. cheers, tobbe
James A Here's my take on it, from another newbie. Curve-fitting is where you assume the data should take a particular form, like linear regression assumes the data should fit a straight line. AFAICT adaptive systems may or may not be curve-fitting. For digital signal processing, the adaptive systems seem to be primarily curve-fitting, because the adaptive output is typically compared to an "ideal" output and adjustments are made accordingly. But adaptive systems in trading tend to be more free-form. For example, the Kaufman Adaptive Moving Average doesn't fit the data to a form; it just adjusts the degree of smoothing based on whether or not the data is trending. So I guess the answer is "it depends."
tobbe Out of sample data easily becomes in sample data. As soon as a set of parameters have been discarded as a result of the system not performing well on out of sample data, the out of sample data has become in sample data since it's now part of the optimization process . In such a case testing on the out of sample data doesn't really say much. To me, testing for robustness is trying to assess how sensitive the system is to changes in parameter values as well as how sensitive the system is to minor/major changes in the prerequisites (the environment that the system is thought to work in). This is what I find so hard to do with an adaptive system. Do you use genetic optimization to find new functions (behaviour) on the fly or do you use it to optimize a given set of parameters for some fixed functionality? I think there's a big difference in complexity (as well as degrees of freedom). Anyway, I'm not implying that adaptive systems don't work. It's just they're so hard to test. This is an interesting topic and I hope you'll post updates on your results. cheers, tobbe
我喜欢简单自适应系统(adaptive system), 不喜欢太简单的, 也玩不转复杂的, 比如BP网络. 多年以前我曾经在本论坛谈过一个观点, 世界上只有两种生物是全球性的, 一种是最高级的人类, 另一种是最简单的最低级的细菌. 这么多年过去了,我的观点仍然没变. 没有老公的MM 要考虑我这类的保守党, 基本不会变心, 不过, 不要考虑我了, 我肯定不会变了. 这些年我一直在找办法使用人工智能, 但是, 到现在为止仍然没有找到门在哪儿. 而低级的简单的强鲁棒性的自适应系统却经历了一次牛市两次熊市的生死考验, 我相信它, 不再有任何疑问.