Infinity or -4: Gambling With Ridge Regression
bet MORE.
Under the crescent moon, the young statistician quantymacro was poring over his Ridge Regression G10 rates model. Despite its improvements over OLS, he still wasn't satisfied. The residuals nagged at him, and the predictions lacked the sharpness he sought.
As he pondered over the perplexing data, a familiar luminous figure appeared in his study—the Oracle of Variables.
"Still seeking the unattainable, are we?" the Oracle's voice resonated like a whisper carried on the wind.
"Oracle! Perhaps you can help," quantymacro sighed. "Even with Ridge regression, my model isn't performing as well as I'd hoped."
The Oracle gave an understanding nod. "Ridge regression is a commendable step, but what if I told you there's a method that could make your model infinitely better?"
"Infinitely better?" quantymacro raised an eyebrow. "What's the catch?"
"It's a gamble," the Oracle cautioned. "This method could vastly outperform Ridge, but if fortune doesn't favor you, your model could become four times worse."
Quantymacro weighed the proposition, his desire for improvement battling his caution. Finally, his thirst for knowledge won. "I'll take the risk," he declared.
The Oracle's smile deepened. "Very well. The secret formula is..."
Last time we looked at how Ridge Regression is one of the closest things to a free lunch - it will always dominate OLS within a certain range of values. Now we will look at another type of regression, a secret one, where it can be infinitely better than Ridge, and would not be more than fourfold worse in the worst case scenario.