Google DeepMind's AlphaCode is a Game Changer for AI Coding

DeepMind invents AI tool to kinda write novel computer code.

Google, which acquired the DeepMind artificial intelligence lab in 2014, might have been on to something. DeepMind and Google Brain have given Alphabet an incredible 1-2 punch in artificial intelligence and deep learning research and R&D.

Today some news came out on DeepMind’s AlphaCode.

Google’s UK-based artificial intelligence unit DeepMind has invented a tool that can write computer code at a competitive level to solve open-ended problems that require critical thinking, logic and understanding of language.

While Microsoft has obviously been after A.I. that could one day code at a human level (GitHub, OpenAI). Now in February 2022 DeepMind says its new AI coding engine is as good as an average human programmer.

Read DeepMind’s blog about AlphaCode here.

• AlphaCode recently achieved an estimated rank within the top 54% of participants in programming competitions by solving new problems that require a combination of critical thinking, logic, algorithms, coding, and natural language understanding.

• DeepMind has created an AI capable of writing code to solve arbitrary problems posed to it, as proven by participating in a coding challenge and placing — well, somewhere in the middle.

I’m not sure I would call it “competitive”, but it’s still impressive.

AlphaCode was tested against challenges curated by Codeforces, a competitive coding platform that shares weekly problems and issues rankings for coders similar to the Elo rating system used in chess.

DeepMind is notorious for liking buzzy headlines to celebrate Google’s research into A.I You can read DeepMind’s pre-print here.

Developing AI systems that can effectively model and understand code can transform these tools and the way we interact with them. Systems that can generate code are not only useful, but also stepping stones that can lead to greater understanding of AI and how it relates to programming. - DeepMind

Code is a fusion of language, logic and problem-solving that is both a natural fit for a computer’s capabilities and a tough one to crack.

• After simulating 10 contests, with more than 5,000 participants, AI system AlphaCode has ranked in the top 54% of competitors.

• There was still work to do to bring it up to the same level as top performing humans, DeepMind said.

These challenges are different from the sort of tasks a coder might face while making, say, a commercial app. They aren’t really real-world stuff at all. I think we could think of them at best as very specialized puzzles that combine logic, maths and coding expertise.

Typical problems include finding ways to place roads and buildings within certain constraints, or creating strategies to win board games. That sounds like the kind of problems AI could be trained to be good at.

What is AlphaCode?

AlphaCode leverages deep learning models to excel at tasks that require “critical thinking, logic, algorithms, coding, and natural language understanding” to excel at competitive programming, something that DeepMind says is “beyond the capabilities of existing AI systems.”

Generating code that solves a specified task requires searching in the huge structured space of possible programs, with a very sparse reward signal. Single character edits can completely change program behaviour even if they don’t cause crashes. Solutions can look dramatically different even for the same problem, and judging if a partial or incorrect program is useful is a difficult challenge.

Microsoft has been teaming up GitHub with OpenAI on similar tasks for quite a while. If you think about it, OpenAI has its own Codex natural-language coding project, and it powers both GitHub Copilot and a test from Microsoft to let GPT-3 finish your lines.

AlphaCode can generate a larger number of possible answers and winnow these down by running the code and checking the output just as a human competitor might. “The whole process is automatic, without human selection of the best samples,” Yujia Li and David Choi, co-leads of the AlphaCode paper, told reporters.

AI Systems Aren’t Producing Valid Code Yet

A lot of progress has been made developing AI coding systems in recent years, but these systems are far from ready to just take over the work of human programmers. The code they produce is often buggy, and because the systems are usually trained on libraries of public code, they sometimes reproduce material that is copyrighted.

I’d go so far as to say OpenAI and DeepMind have a friendly rivalry here. I’ve been writing for a long time about how programmers would be working alongside AI in hybrid AI-human roles soon.

Challenges like these mean that AI coding systems will likely soon be integrated slowly into the work of programmers — starting as assistants whose suggest get too excited about AlphaCode being that solution. Right now it’s enough to demonstrate that the model is capable of parsing and understanding a complex written challenge all at once and producing a coherent, workable response most of the time.

Mike Mirzayanov, the founder of the platform Codeforces, which ran the competitions, called AlphaCode a "promising new competitor". I thought that was pretty funny. DeepMind likes its PR pieces, and sometimes when you dig into the research it’s not that impressive.

If OpenAI and DeepMind are doing this, you can bet somewhere in China it’s also being done. Last year, San Francisco-based research lab OpenAI released Codex, an AI model for translating natural language commands into app code. The model, which powers GitHub’s Copilot feature, was heralded at the time as one of the most powerful examples of machine programming, the category of tools that automates the development and maintenance of software.

Still, any research being done in the AI of coding is super interesting and worthy of praise. Software engineering will eventually intersect with machine learning in more coherent and useful ways.

In DeepMind’s preprint, they detail AlphaCode and how it uses transformer-based language models to generate code at an unprecedented scale, and then smartly filters to a small set of promising programs.

If you enjoyed this article you might enjoy my Newsletter Sublink, where I discuss business breaking news at the intersection of technology.

Finally if you enjoy my content please consider giving me a tip as this is my full-time gig and I cannot continue to write without extensive community support.