Portfolio optimization: choosing weights under a stated risk trade-off

Portfolio optimization is the problem of choosing position weights that achieve the best available trade-off between expected return and risk, subject to constraints on how those weights can be structured. The formulation is old and the mathematics is well understood; the difficulty in practice sits almost entirely in the three inputs the optimizer is asked to trust.

The mean-variance formulation

For a set of candidate positions with expected-return vector \(\alpha\) and covariance matrix \(\Sigma\), the classical Markowitz problem is

\[\underset{w \in \mathcal{W}}{\operatorname{maximize}} \quad \alpha^{\mathsf T}w - \frac{\lambda}{2} w^{\mathsf T}\Sigma w,\]

where \(\lambda\) controls the relative weight placed on risk versus expected return, and \(\mathcal{W}\) encodes constraints such as full investment, position bounds, or sector limits. Smaller \(\lambda\) favors expected return; larger \(\lambda\) moves the solution toward lower variance. This is a convex quadratic program, so a solver finds the exact optimum for any given \(\alpha\), \(\Sigma\), and \(\mathcal{W}\).

The inputs are the hard part

The optimizer treats \(\alpha\) and \(\Sigma\) as known. Both are estimates. In a cross-sectional equity setting, \(\alpha\) is typically derived from a factor's rolling ICIRRolling ICIRThe mean information coefficient divided by its standard deviation over a trailing window, usually annualized. It measures the persistence of ranking skill rather than one period’s IC.Open glossary entry → rather than a raw forecast, since ICIR-based sizing already discounts a signal for how unstable its recent ranking performance has been. \(\Sigma\) is usually a sample covariance matrix estimated over a trailing window, which becomes poorly conditioned when the number of candidate positions is large relative to the estimation window or when candidates are highly correlated.

Because both inputs carry estimation error, the optimizer's exact solution is exact only with respect to inputs that are themselves approximate. Small changes in \(\alpha\) or in weakly identified directions of \(\Sigma\) can move the unconstrained solution by more than the underlying economic difference between candidates would justify.

Why the nominal solution is a starting point

Position bounds and full-investment constraints limit, but do not eliminate, sensitivity to estimation error. Two adjustments address it directly: comparing solutions across adjacent rebalance dates to check whether turnover reflects a stable economic signal rather than estimation noise, and extending the objective to penalize the portfolio's behavior in adverse historical scenarios rather than only its average variance. Both are developed in robust portfolio optimization, which also reports a walk-forward comparison of the nominal solver against worst-case and CVaR-penalized variants on real factor-mimicking portfolios.

Further reading

• Harry Markowitz, “Portfolio Selection”, The Journal of Finance, 1952. The original formulation of return and variance as the two quantities a portfolio should be optimized over.
• Richard O. Michaud, “The Markowitz Optimization Enigma: Is ‘Optimized’ Optimal?”, Financial Analysts Journal, 1989. The classic critique of treating estimated inputs as known quantities, and the starting point for most robust-optimization methods in use today.