What is Microsoft Research's µ-Parametrization?

µTransfer: A technique for hyperparameter tuning of enormous neural networks

Microsoft Research has done it again! This is pretty exciting for OpenAI. There’s some suggestion that Microsoft and OpenAI may have solved a fundamental AI bottleneck. So let’s get into it.

The research was published this week on March 8th, 2022 a Tuesday on Microsoft Researcher’s blog.

The lead on the study is Edward Hu, a PhD student at Mila, where h…