editorial
15 posts tagged editorial.
I Shipped a Scoring Bug That Made Improvement Impossible — Here's the Fix
burst_score was computed over a file's entire commit history, which made it a one-way ratchet in the live risk score. A single bursty period years ago could keep a file stuck at CRITICAL forever, even after every other risk factor improved. Fixed in the next patch release.
How phrasing affects LLM compliance with spec security clauses
I tested five ways to phrase a security requirement across four language models. The phrasing most commonly used in practice was near the bottom on all four of them. The phrasing that performed best across all four was the one that said nothing about what not to do.
The Coupling Signal That Kept Inverting Itself
I tried three formulations of co-change coupling for defect risk ranking. Raw co-occurrence inverted on dense codebases. Mutual information made it worse.
Your Transformer Featurizer Only Helps When Your History Is Broken
Transformer embeddings on source code improved defect ranking on exactly the repos where git history had already failed — and added near-zero or negative signal everywhere else. Here's what that conditional looks like in practice.
The Defect Prediction Trap
I fine-tuned a model that scored Spearman ρ=+0.678 on defect risk. Then I zeroed out the git history fields and it dropped to −0.142. The model wasn't reading code. It was relaying commit metadata back as a prediction.
The Metric That Hides Failure
Spearman ρ measures average rank agreement across all files. Precision@K measures whether the files engineers actually review are in the right place. On mature, stable codebases, a ranker can show positive ρ while returning zero bugs in its top 10.
Which LLM Vendor Wins at Defect Risk Prediction?
When evaluating open-weight models for code analysis, the vendor comparison you run in 2024 might give you the wrong answer in 2025.
When Your Labels Lie: File-Level Bug Attribution Inflates Defect Rates
Most defect prediction pipelines share a quiet assumption: if a file was touched in a bug-fix commit, every function in that file was 'buggy' — and that assumption silently inflates defect labels in ways that undermine fine-tuning before it starts.
How to Know If Fine-Tuning Will Help Before You Run It
Before you spend compute on per-repo fine-tuning, one scalar from the codebase's bug-fix history can tell you whether it's worth running. Here's how it works and what it can't predict.
Can a Fine-Tuned LLM Learn Which Code Is Risky?
I spent a few weeks running a controlled experiment: can a small language model, fine-tuned on a specific codebase's history, learn to rank code by defect risk better than a hand-tuned heuristic?
AI Agents Can Pass Tests. They Still Can't Maintain Systems.
AI coding tools have made writing software dramatically easier. A new benchmark shows maintaining it is still the hard part — and the failure modes are predictable.
AI Made Code Cheap. The Bottleneck Is Now Understanding Systems.
AI removed code production as the engineering bottleneck. The new constraint is comprehension - and most developer tooling wasn't built for that problem.
The 20% Rule for Codebases: A Practical Refactoring Playbook
Use this practical worksheet to intersect change and complexity, pick three high‑leverage targets, and add simple guardrails to prevent backslide.
Stop Missing Regressions in PRs: Complexity Policy Checks in CI
Code review misses slow creep. Use Hotspots policies in CI to stop risky complexity changes before they merge — start warn‑only, then flip to blocking.
Introducing Hotspots: Find Risky Functions and Stop Regressions
Meet Hotspots — a lightweight analyzer with explainable risk, shareable reports, and CI policies so you fix what matters and keep it from slipping back.