Deterministic
The effectiveness computation: which comments exist, which were followed by a code change in the PR, and the aggregations by reviewer, team, and period. Running it twice over the same history produces the same number.
// METHODOLOGY
What we measure, how we measure it, and what the metric does not say.
A measurement only deserves trust if it can be questioned. This page defines Releezy Guardian’s central metric, separates deterministic computation from model assistance, and lists the limits we know about.
The metric: reviewer effectiveness
A review comment counts as effective when it is followed by a real code change in the same pull request, between the comment and the merge. It is a comment-to-change conversion measure, computed from Git history, with the same rule for every reviewer: human, AI tool, or Releezy’s own agents.
The name matters: we measure whether the comment was followed by a change, not whether the comment was right. The two correlate, but they are not the same thing. That is why the metric is never read alone.
What is deterministic and what uses a model
Model-assisted
Only the classification of the comment TYPE (nit, logic, security, style). The model never decides whether a comment was effective, and reclassifying types does not change the effectiveness number.
How the baseline is chosen, without circularity
The baseline is not defined by title, seniority, or nomination. It is the reviewers whose comments convert into changes most often, over a minimum volume of activity, recalculated as history grows. The criterion is independent of rank: nobody joins the baseline by authority, and the ruler is not tunable per contributor, which includes Releezy’s own agents.
What the metric does not say
Limits we know about and account for in the reading:
- A correct comment can be answered with an explanation and produce no change. It counts as unconverted, even though it was useful.
- Social authority can inflate conversion: comments from senior people tend to be acted on more. Team-level reading and the longitudinal series help expose that effect; they do not eliminate it.
- Conversion is not causality. A before/after can move because of team composition, seasonality, or PR size. That is why Releezy Concierge tests alternative explanations against the history before recommending, and tracks the same metric after the action.
- Architecture discussions that avoid work without producing a diff do not appear in the metric. It measures the review flow, not the full value of a technical conversation.
Where the numbers on this site come from
The first-party numbers quoted on this site come from customer production environments measured by Releezy Guardian. The base of the main case: 13,784 pull requests, a team of 102 developers, 11 months of history. Every number travels with the base that produced it, and whoever is measured sees their own data.
How a number gets contested
Every metric carries the history that produced it: the comments, the PRs, the time window. A recommendation from Releezy Concierge arrives with that evidence attached, and disagreement is settled by looking at the record. If you find a case where the rule counts wrong, we want to see it: the methodology improves by being contested, not by being defended.
Make continuous improvement part of how engineering works.
Give engineering an ally that turns evidence into action. Improve continuously with trust, connecting every recommendation to facts and every action to its result.