Quality is key: GitHub Copilot and code quality

Published via GitHub Executive Insights

We all know the importance of quality code. It makes code more readable and maintainable, and just generally improves the developer experience (DevEx). So, when we introduce a tool like GitHub Copilot into the code creation process, it’s critical to ensure that it maintains our same quality standards. What does the research tell us about the impact GitHub Copilot has on code quality?

Jared Bauer, GitHub staff researcher, commissioned a study to explore this exact topic, and joined host Christopher Harrison to share his learnings. In a nutshell: happy developers create better code, and GitHub Copilot is effective at helping developers write better code.

To test the impact of Copilot on code quality, Jared’s study involved 202 experienced Python developers who were tasked with creating APIs to pass a set of unit tests. The study ultimately demonstrated that those who used Copilot were more likely to create code that passed all the tests put forth.

But passing tests doesn't tell the full story about code quality. It's not just about the code operating correctly — it also needs to be readable and maintainable. Measuring this requires code reviews, which was the next part of the study. All valid submissions were blind reviewed, where reviewers did not know what tools were being used to create the code. THe study found that code created with the help of Copilot was more likely to pass code reviews by over 10%: a statistically significant percentage.

Additional insights were gleaned from the study, including the fact that developers with Copilot both generated, but also deleted, more code than those who didn't use Copilot. The theory behind this behavior is that Copilot allowed developers more time, and gave them confidence to refactor their code to ensure it was of the highest quality possible.

To learn more and continue your journey, explore the resources below:

Insights

Techs

GitHub Copilot

Want to learn more about the strategic role of AI and other innovations at GitHub? Explore Executive Insights for more thought leadership on the future of technology and business.