I tried three formulations of co-change coupling for defect risk ranking. Raw co-occurrence inverted on dense codebases. Mutual information made it worse. A direction-aware formulation conditioned on partner defect history finally worked — but only after a leakage correction and a per-repo label stability screener.
Claim
Co-change coupling — the idea that files which change together are likely risky together — is a positive signal on sparse codebases and an anti-signal on dense ones. Standard normalisation (mutual information) makes this worse, not better. A direction-aware formulation that weights co-change by partner defect history solves the sign flip and ships as a ninth feature in a production ranker.
Reader Problem
You’re adding co-change coupling to a defect risk model. You observe it working on a small Go microservice or a mid-size Python framework. You ship it. On a large, active JavaScript codebase it inverts — the files with the most co-change partners are among the least likely to contain bugs. Your model is now worse on the repos with the most data.
This is a density confound, and it is not obvious until you measure it.
Setup
The starting signal was straightforward: for each file, count the number of distinct files it co-changes with in git history. Call this co_change_partners. The intuition is that a file touched by many other changes has high coupling and is structurally fragile.
I expected this to add lift over activity-only features on top of an existing tabular ranker. Success condition: positive Spearman ρ on a test set of repos ranging from small (gin, flask) to large and dense (svelte, react, django).
First Failure: The Sign Flip
A cross-repo evaluation across 13 repositories revealed that co_change_partners ρ correlates strongly with repo coupling density (Pearson r=−0.679, p=0.011): the higher the mean number of co-change partners per file, the more negative the coupling signal becomes. On dense, high-activity codebases the signal inverted — the two most strongly negative repos had mean coupling densities of 2,655 and 1,020 co-change partners per file respectively.
In the specific repos used for subsequent experiments, this manifested as near-zero or unreliable signal: gin ρ=−0.023, svelte ρ=+0.023 (near noise), react ρ=+0.237 (mildly positive but inconsistent across runs), django ρ=+0.078 — while sklearn ρ=+0.514 was the strongest result, from a moderately sparse repo.
Diagnosis: on a large active codebase, most files co-change with most others because large sweeping commits touch hundreds of files at once. High co_change_partners doesn’t indicate structural fragility — it indicates centrality. Central files (config, build tooling, shared utilities) are touched in the majority of commits and are among the most stable and well-maintained parts of the codebase.
Second Failure: Normalisation Makes It Worse
The standard response to this kind of density confound is normalisation. Mutual information (MI) and its normalised variant (NMI) condition on the marginal activity rate of each file, removing the main effect of base commit frequency.
I computed max NMI per file — the single strongest coupling relationship, after conditioning on marginal activity — and evaluated it on the same cross-repo set.
| Repo | ρ NMI | ρ co-change count | ρ activity baseline |
|---|---|---|---|
| svelte | −0.349 | +0.023 | +0.692 |
| react | −0.160 | +0.237 | +0.253 |
| gin | +0.032 | −0.023 | +0.091 |
| django | −0.550 | +0.078 | +0.680 |
| sklearn | +0.117 | +0.514 | +0.345 |
NMI is not neutral on the sign-flip problem — it amplifies it. On django (ρ=−0.550) and svelte (ρ=−0.349), NMI is more anti-predictive than any other signal I had tested.
The mechanism: max NMI per file selects the partner with the tightest conditional coupling. On the dense repos I tested, that partner was consistently a high-centrality file — a build system entry, a shared utility, an interface definition. The result is that max NMI measures “proximity to the most central file in the repo,” not “structural fragility.” On large codebases, those are negatively correlated.
Normalisation removed the raw count effect and replaced it with a different measure of centrality. The sign flip survived because the root cause was not a counting artefact — it was an aggregation choice.
What Changed: Conditioning on Partner Defect History
The insight from the NMI failure: the problem is not how you count co-change, it’s what you count it with. A file that co-changes with a stable config file is not risky because of that coupling. A file that co-changes with historically defect-prone files is.
Directed coupling weights each co-change relationship by the partner’s defect score:
directed_coupling(i) = Σ_j [ co_change_count(i,j) × defect_score(j) ] / commit_count(i)This eliminates the density confound by design: a file that co-changes with thousands of stable infrastructure files accumulates a score near zero. Only co-changes with genuinely fragile partners propagate signal.
| Repo | ρ directed | ρ co-change count | ρ activity baseline | Δ vs co-change |
|---|---|---|---|---|
| svelte | +0.776 | +0.023 | +0.692 | +0.753 |
| react | +0.260 | +0.237 | +0.253 | +0.024 |
| gin | +0.213 | −0.023 | +0.091 | +0.236 |
| django | +0.452 | +0.109 | +0.660 | +0.343 |
| sklearn | +0.462 | +0.515 | +0.346 | −0.053 |
Sign flip eliminated on each of the five repos in the evaluation set. On svelte — the hardest case, where raw coupling was near-zero and NMI was −0.349 — directed coupling at +0.776 exceeds the activity-weighted risk baseline (+0.692). The signal didn’t just recover; it became the strongest single feature on that repo.
Third Complication: Temporal Holdout Reveals Leakage
The full-history results above compute partner defect scores over the same window as the evaluation labels. That’s leakage: if a file was risky in the second half of history, its partner scores (computed from the full history) already know about it.
Evaluating on a proper temporal holdout — partner scores computed from the training window only, labels from the holdout window — produced a more nuanced picture:
| Repo | ρ DC holdout | ρ DC full | Jaccard | Notes |
|---|---|---|---|---|
| svelte | −0.033 | +0.776 | 0.091 | Leaked — collapses on holdout |
| react | +0.013 | +0.260 | 0.054 | Leaked |
| gin | +0.309 | +0.213 | 0.333 | Genuine signal |
| django | +0.565 | +0.452 | 0.352 | Genuine signal; competitive with activity baseline |
| sklearn | −0.222 | +0.462 | 0.314 | Fails — activity signals anti-predictive on this repo |
Jaccard values from the label stability screener experiment. Note: a later recency-windowed experiment reports react Jaccard=0.063 — a small discrepancy between the two measurement passes. The screener finding is authoritative on this value.
The full-history svelte and react results were substantially leakage. Django and gin show genuine, leakage-free signal. Django holdout ρ=+0.565 is competitive with the activity baseline on the same holdout window — directed coupling is nearly equivalent to the strongest existing signal on this repo.
Why Some Repos Leak and Others Don’t: Label Stability
The key variable turned out to be Jaccard similarity between the set of defect-prone files in the training window versus the holdout window. If the same files are risky across time, training-window partner scores transfer. If the risk surface rotates, they don’t.
| Repo | Jaccard | DC leaks? |
|---|---|---|
| svelte | 0.091 | Yes |
| react | 0.054 | Yes |
| gin | 0.333 | No |
| django | 0.352 | No |
Jaccard < 0.15 predicted leakage on 4 of 5 repos in the evaluation set. Across a broader 45-repo sample, 58% of repos fell below 0.15 — a pattern consistent with architecturally volatile projects where the defect surface rotates between eras, though with 45 repos the distribution is suggestive rather than definitive.
The initial interpretation of this finding was pessimistic: skip directed coupling on low-Jaccard repos. But further experiments showed that wasn’t the right response.
Resolution: Windowed DC Recovers Low-Jaccard Repos
On architecturally volatile repos, full-history coupling mixes multiple architectural eras. Svelte, for example, underwent a major version transition during its history. Full-history co-change mixes pre- and post-transition coupling patterns. A shorter lookback window stays within the current architectural era.
Evaluating 365d and 90d windows against temporal holdouts:
| Repo | ρ dc_full | ρ dc_365d | ρ dc_90d | ρ activity baseline | Jaccard |
|---|---|---|---|---|---|
| svelte | +0.056 | +0.130 | +0.127 | +0.073 | 0.091 |
| react | −0.000 | +0.180 | +0.208 | +0.407 | 0.063† |
| gin | +0.187 | nan | nan | +0.027 | 0.333 |
| django | +0.445 | +0.334 | +0.112 | +0.551 | 0.352 |
| vue | +0.143 | +0.124 | −0.108 | +0.573 | 0.348 |
† The recency-windowed experiment reports react Jaccard=0.063; the label stability screener reports 0.054. Both are below the 0.15 threshold and produce the same routing decision.
On svelte, dc_365d (+0.130) is 2.3× dc_full (+0.056) and beats the activity baseline (+0.073). Low-Jaccard repos don’t need directed coupling to be abandoned — they need a shorter window that tracks the current coupling structure rather than historical ones.
On high-Jaccard repos, windowing hurts: django dc_full (+0.445) > dc_365d (+0.334) because long-stable coupling structure is an asset when it’s actually stable.
The final screener rule. The first condition catches repos like scikit-learn where all activity-based signals are anti-predictive — the ranker’s top-10 predictions contain fewer future bugs than a random sample would. On those repos, no coupling signal built on activity history is salvageable.
if ranker P@10 < base_rate (activity signals anti-predictive on this repo):
skip directed coupling
elif Jaccard < 0.15:
use 365d windowed DC
else:
use full-history DCWhat Shipped
Directed coupling shipped as the ninth feature in the tabular ranker with Jaccard-based screener routing and commit-velocity-informed window selection. The full-history version is used on stable repos; the windowed version on architecturally volatile ones; the feature is skipped on repos where the P@K gate fires (the activity-signal failure mode described in a separate post).
What Went Wrong Or Could Be Wrong
Flask was unlabelable throughout. Flask has near-zero convention coverage — no keyword-based fix labels could be detected. Every coupling evaluation on Flask returned NaN. This is a label pipeline gap, not a signal failure, but it means Flask contributes nothing to the calibration of these results.
Five repos is a small evaluation set for the Jaccard screener. The 0.15 threshold was chosen because it falls in the gap between 0.091 and 0.352 — a bimodal distribution. But this threshold was validated on the same repos used to identify it. The broader 45-repo corpus Jaccard distribution supports the bimodal pattern, but the threshold’s reliability at the boundary needs more data.
Sklearn fails for a different reason entirely. Sklearn’s Jaccard (0.314) is similar to gin and django, but directed coupling still fails on holdout (ρ=−0.222). This is because the signal failure on sklearn is structural: high-activity files are anti-correlated with defect risk (stable core APIs dominate the high-activity bucket). The Jaccard screener cannot catch this — a separate P@K gate is required as a prior check.
Product Implication
A coupling signal needs direction. Raw co-occurrence is symmetric and density-dependent; it cannot distinguish “risky because coupled to fragile things” from “central because touched in every commit.” The product-safe version conditions on whether the partner has a defect history, and adjusts the window to match the architectural volatility of the specific repo.
A screener that routes the signal is not optional. The same metric has genuinely different reliability characteristics on different types of codebases, and pretending otherwise produces the anti-predictive results I observed early in this sequence.
Next Test
Validate the Jaccard threshold on additional repos with known holdout labels. The current calibration is based on five repos. Ten additional repos — especially those near the 0.15 boundary — would either confirm the threshold is robust or reveal that the gap is narrower than it appears.