Investigating AI Productivity: Insights from the Early-2025 Open-Source Developer Study

On Friday, 14 November 2025, Miroslav Blaško held an Open Mic session with the topic "Investigating AI Productivity: Insights from the Early-2025 Open-Source Developer Study". Video and presentation included.

Abstract

The session presented selected findings from a randomized controlled trial conducted by the organization METR (Model Evaluation and Threat Research). The study recruited 16 experienced open-source developers, each with long-term familiarity with their respective repositories, to complete 246 issue tasks. Each task was randomly assigned to either allow or disallow the use of state-of-the-art AI development tools (e.g., Cursor Pro with Claude 3.5/3.7) in early 2025. Contrary to prevailing expectations, the study found that developers using AI tools took, on average, 19% longer to complete tasks than those working without AI. Developers had initially predicted a 24% speed-up, and even after experiencing the slowdown, they continued to believe they had improved by roughly 20%.

Beyond the key findings, the session also examined annotated screen recordings to pinpoint where the slowdown occurred and discussed the most likely contributing factors. In the final section, the session considered how up-to-date the results remain, given the significant improvements in AI tools since the study period. Current tools exhibit higher accuracy and tighter IDE integration, enabling more effective solution searches with reduced interaction and increased support from multiple cooperating agents. The session concluded with a discussion of how these insights could be applied to the use cases of the KBSS (Knowledge-based Systems and Software) research group.

The presentation slides are available at this link.

Further reading:

• Becker, J., Rush, N., Barnes, E., Rein, D. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” METR blog post (10 July 2025).
• Becker, J., Rush, N., Barnes, E., & Rein, D. (2025). Measuring the impact of early-2025 AI on experienced open-source developer productivity. Full paper arXiv preprint arXiv:2507.09089.
• Denicola, D. “My Participation in the METR AI Productivity Study”, (July 15 2025).
• Watanabe, M., Li, H., Kashiwa, Y., Reid, B., Iida, H., & Hassan, A. E. (2025). On the use of agentic coding: An empirical study of pull requests on github. arXiv preprint arXiv:2509.14745.