https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ I have read the full paper and I think their main claim is to be taken with a grain of salt. This study is performed on developers who have been working for at least 5 years on the particular codebase. That said, the biggest take away is they all underestimated their solving performance compared to when using Cursor with Claude 3.7 Even after finishing tasks with AI they still thought to have been faster, while the control group showed the average task performance speed was lower with AI. The main claim of the RCT study is "developers perform 19% slower when using AI". But, the charts at the end have quite broad variances, so this claim would not hold a review I guess. Not significant enough. But definitely AI usage did show any advantage in THIS context.
