The Probability of Back-Test Overfitting http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2326253 Abstract: Most firms and portfolio managers rely on back-tests (or historical simulations of performance) to select investment strategies and allocate them capital. Standard statistical techniques designed to prevent regression over-fitting, such as hold-out, tend to be unreliable and inaccurate in the context of investment back-tests. We propose a framework that estimates the probability of back-test over-fitting (PBO) specifically in the context of investment simulations, through a numerical method that we call combinatorially symmetric cross-validation (CSCV). We show that CSCV produces accurate estimates of the probability that a particular back-test is over-fit.
其实只要心中不去追求超常绩效，就能避开绝大多数的过拟。 反之，只要存在强烈渴望，想要搞出一个牛B朝天的系统，不管多么小心谨慎最终总是会过拟。 因为做的过程中，无过拟的普通绩效系统被直接忽略了。然后在一堆超常绩效系统中寻找，想要找到一个看起来没有过拟的NB系统，但其实只是还没明白哪里过拟而已。如果找不到就会继续找下去。