Microsoft’s new coding LLM is coming after GPT 4

phi-1 is Microsoft’s new language model for coding.

2 min. read

Published onJune 22, 2023

published onJune 22, 2023

Share this article

Read our disclosure page to find out how can you help Windows Report sustain the editorial teamRead more

Key notes

Microsoft is really taking a big step into AI development. Last month, AI was at the forefront of the Microsoft Build conference, and the Redmond-based tech even announced thatCopilot is coming to Windows 11. As a native built-in app, for everything you need,

AI is coming to Microsoft Teams as well,in its Recap Ai tool. And a lot of other tools from Microsoft will use AI, including Microsoft Fabric.

But it seems, Microsoft is also funding AI research. Just recently, the tech giant announcedOrca 13B will be open source. AndLongMem is another good point in AI research: it is the answer for unlimited context length.

And now there is time for another big breakthrough in AI research, coming from Microsoft, of course. The new 1.3B-parameter model coding LLM, called phi-1, is reportedly outperforming GPT 3.5, in only 4 days of training.

What is phi-1 and how does it outperform GPT already?

Phi-1is a new 1.3B-parameter language model for code, with a significantly smaller size than competing models. The language model was trained for 4 days, over 7B tokens (slightly over 50B total tokens seen) followed by finetuning on less than 200M tokens.

Despite being a lot smaller than the competing models, phi-1 attained 50.6% pass@1 accuracy on HumanEval and 55.5% pass@1 accuracy on MBPP (Mostly Basic Python Programs), which are one of the best self-reported numbers using only one LLM generation.

Moreover, despite being trained on much fewer tokens compared to existing models, phi-1 still has a lot of potentials.

The improvements on HumanEval are the phi-1 greatest achievements, as a language model. After tuning, phi-1 managed to execute tasks that were not featured in the finetuning dataset. This means the model adapted and improved the fine-tuning process.

And the most remarkable thing is that phi-1 reorganized and consolidated the knowledge acquired during pretraining, even though the knowledge was not explicitly present there in the first place.

In shorter words, phi-1 not only learns during training but also expands the knowledge on its own. It managed to outperform GPT 3.5 on every level, and it’s just a matter of time until the small model takes on the big ones, like GPT 4.

What do you think about this new AI model? Where do you think AI research is heading? Be sure to let us know your opinions in the comments section below.

With KB5043178 to Release Preview Channel, Microsoft advises Windows 11 users to plug in when the battery is low#

Copilot in Outlook will generate personalized themes for you to customize the app#

Microsoft will raise the price of its 365 Suite to include AI capabilities#

Death Stranding Director’s Cut is now Xbox X|S at a huge discount#

Outlook will let users create custom account icons so they can tell their accounts apart easier#

Microsoft’s new coding LLM is coming after GPT 4#

What is phi-1 and how does it outperform GPT already?#

With KB5043178 to Release Preview Channel, Microsoft advises Windows 11 users to plug in when the battery is low

Copilot in Outlook will generate personalized themes for you to customize the app

Microsoft will raise the price of its 365 Suite to include AI capabilities

Death Stranding Director’s Cut is now Xbox X|S at a huge discount

Outlook will let users create custom account icons so they can tell their accounts apart easier

Microsoft’s new coding LLM is coming after GPT 4

What is phi-1 and how does it outperform GPT already?