PPA and NPA Calculation in Molecular Validation: A Guide

In molecular diagnostic clinical validation, Positive Percent Agreement (PPA) and Negative Percent Agreement (NPA) quantify the proportion of positive and negative results concordant between a candidate assay and a non-reference comparative method. When a candidate test is evaluated against a predicate device, standard-of-care PCR assay, or routine laboratory method rather than an undisputed reference standard establishing absolute clinical truth, regulatory agencies and payor programs—including the U.S. Food and Drug Administration (FDA) and the MolDx program—prohibit reporting clinical sensitivity and clinical specificity. PPA and NPA must be reported alongside two-sided 95% confidence intervals derived from score-based or exact binomial distributions (such as the Wilson score or Clopper-Pearson method). Standard asymptotic Wald intervals fail near complete agreement. Furthermore, resolving discordant results using a third method cannot alter the primary 2×2 contingency table, and multiplex panels must report agreement on an individualized target-by-target basis.

Key takeaways

• PPA and NPA are mandatory when comparing a candidate molecular assay to a non-reference comparator or predicate device. Clinical sensitivity and specificity are reserved exclusively for comparisons against established, recognized reference standards defining true clinical or pathological status.
• Wald (Gaussian) confidence intervals systematically fail when observed agreement approaches 100% or when positive sample sizes are small, producing impossible intervals such as [1.00, 1.00]. CLSI EP12-Ed3 and the FDA mandate asymmetric intervals, preferentially the Wilson score or Clopper-Pearson exact interval.
• Discordant resolution must never alter the primary 2×2 matrix. Re-testing only discordant specimens with an orthogonal "tie-breaker" introduces conditioning bias. The FDA and MolDx require the primary 2×2 table to remain unadjusted; secondary investigations may only be reported descriptively in an auxiliary appendix.
• Specimen-level metrics obscure target dropouts in multiplex panels. Clinical performance must be reported target by target; pooling targets into a single panel-wide agreement metric masks diagnostic dropouts and leads to regulatory deficiency notices.
• Contrived specimens cannot be commingled with prospective or banked clinical specimens in the primary clinical agreement cohort.

Sensitivity and Specificity vs. PPA and NPA: The Regulatory Divide

A foundational point of friction in clinical validation protocols involves the misapplication of the terms clinical sensitivity and clinical specificity. Under the FDA’s final guidance, Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests, diagnostic accuracy parameters (sensitivity and specificity) can only be measured when the candidate test is compared to a recognized, validated reference method that unequivocally establishes clinical condition (e.g., definitive histological biopsy for tissue pathology or reference quantitative bacterial culture where clinical truth is defined).

In contemporary molecular diagnostics—such as qualitative real-time PCR, digital PCR (dPCR), and targeted next-generation sequencing (NGS)—a candidate assay is rarely evaluated against an absolute reference standard. Instead, it is evaluated against a cleared predicate device, an established laboratory-developed test (LDT), or a standard-of-care testing algorithm. Because non-reference comparators possess their own inherent false-positive and false-negative rates, the true disease or infection status of the tested specimens remains technically unverified.

Reporting clinical sensitivity or specificity under these conditions is scientifically inaccurate and triggers immediate rejection during FDA 510(k) reviews and MolDx Technical Assessments. Laboratories must report:

• Positive Percent Agreement (PPA): The proportion of comparator-positive specimens correctly identified as positive by the candidate assay.
• Negative Percent Agreement (NPA): The proportion of comparator-negative specimens correctly identified as negative by the candidate assay.

Positive Predictive Value (PPV) and Negative Predictive Value (NPV) also cannot be calculated directly from these study cohorts, because study cohorts are intentionally enriched with positive specimens and do not reflect real-world disease prevalence.

Calculating Concordance: The 2×2 Contingency Matrix

Method comparison data are structured using a standard $2 \times 2$ contingency matrix. By convention, the comparative method defines the columns and the candidate assay defines the rows.

Using this matrix, the primary performance metrics are defined as:

$$\text{PPA} = \frac{a}{a + c} \times 100$$

$$\text{NPA} = \frac{d}{b + d} \times 100$$

$$\text{Overall Percent Agreement (OPA)} = \frac{a + d}{a + b + c + d} \times 100$$

Why Overall Percent Agreement (OPA) Can Be Misleading

While OPA reflects the total proportion of concordant calls across the cohort, the FDA advises against relying on OPA as a primary summary metric. In cohorts dominated by negative specimens (common in high-plex infectious disease screening), an assay with high negative agreement ($d$) can show an OPA exceeding 98% even if it fails to detect half of the true positive specimens ($a$). PPA and NPA must always be calculated, reported, and evaluated independently.

Computing Two-Sided 95% Confidence Intervals

Point estimates of PPA and NPA are insufficient on their own; regulatory agencies and clinical accreditation bodies mandate two-sided 95% confidence intervals (CIs). The choice of statistical formula is critical.

The Failure of the Wald Interval

The classical Wald (normal approximation) confidence interval is calculated as:

$$\hat{p} \pm z_{1 - \alpha/2} \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}}$$

While straightforward, the Wald interval relies on the assumption that the sampling distribution of a proportion is normally distributed. This assumption fails near the boundaries ($\hat{p} \to 0$ or $\hat{p} \to 1$) and when sample sizes ($n$) are modest. When a candidate assay achieves 100% agreement ($a = 50, c = 0; \hat{p} = 1.0$), the term $\hat{p}(1-\hat{p})$ evaluates to zero, yielding a standard error of zero and an impossible confidence interval of $[1.00, 1.00]$. For this reason, the Wald interval is rejected by the FDA and the Clinical and Laboratory Standards Institute (CLSI) for diagnostic agreement reporting.

The Wilson Score Interval (Recommended by CLSI EP12-Ed3)

The consensus standard for binary diagnostic examinations, detailed in CLSI EP12-Ed3: Evaluation of Qualitative, Binary Output Examination Performance, is the Wilson score interval. The Wilson score interval inverts the score test and maintains coverage probability near 0% and 100% without producing degenerate intervals.

For observed proportion $\hat{p} = x/n$ and standard normal critical value $z = 1.96$ (for a 95% confidence level), the Wilson score interval is computed as:

$$\text{Center} = \frac{\hat{p} + \frac{z^2}{2n}}{1 + \frac{z^2}{n}}$$

$$\text{Margin} = \frac{z}{1 + \frac{z^2}{n}} \sqrt{\frac{\hat{p}(1 - \hat{p})}{n} + \frac{z^2}{4n^2}}$$

$$\text{Lower Bound} = \text{Center} - \text{Margin}$$

$$\text{Upper Bound} = \text{Center} + \text{Margin}$$

Example Calculation:

Suppose an assay detects 48 positive results out of 50 comparator positives ($\hat{p} = 0.96$, $n = 50$, $z = 1.96$):

1. $z^2 = 3.8416$
2. $\text{Denominator} = 1 + \frac{3.8416}{50} = 1 + 0.076832 = 1.076832$
3. $\text{Center} = \frac{0.96 + \frac{3.8416}{100}}{1.076832} = \frac{0.96 + 0.038416}{1.076832} = \frac{0.998416}{1.076832} \approx 0.92718$
4. Inside the square root: $\frac{0.96(0.04)}{50} + \frac{3.8416}{4(2500)} = \frac{0.0384}{50} + \frac{3.8416}{10000} = 0.000768 + 0.00038416 = 0.00115216$
5. $\sqrt{0.00115216} \approx 0.033943$
6. $\text{Margin} = \frac{1.96}{1.076832} \times 0.033943 \approx 1.82015 \times 0.033943 \approx 0.06178$
7. $\text{Lower Limit} = 0.92718 - 0.06178 = 0.8654 \text{ (86.54%)}$
8. $\text{Upper Limit} = 0.92718 + 0.06178 = 0.9890 \text{ (98.90%)}$

The calculated PPA is 96.00% (95% CI: 86.54% – 98.90%).

Clopper-Pearson (Exact) Interval

The Clopper-Pearson method is also accepted by the FDA. Based directly on the cumulative binomial distribution, it guarantees that coverage probability is never less than $1 - \alpha$. However, because it is strictly conservative, its intervals are wider than Wilson score intervals, requiring larger sample sizes to demonstrate that a lower confidence limit surpasses a specific performance threshold.

Sample Size and the Lower Confidence Bound

Clinical validation protocols must account for the lower confidence bound, not merely the point estimate. Regulatory reviewers and reimbursement auditors evaluate whether the lower bound of performance is clinically acceptable.

If a laboratory evaluates only 10 positive specimens and achieves 10/10 concordant positives ($\hat{p} = 100%$), the Wilson score 95% confidence interval is 72.25% to 100.00%. A lower bound of 72.25% will fail to meet MolDx and FDA technical expectations for high-acuity targets. To secure a Wilson lower bound $\ge 90.0%$ with an observed point estimate of 100%, a minimum of 36 consecutive concordant positive specimens is mathematically required. Designing an enrollment plan that ensures statistical power requires rigorous clinical validation protocol design prior to initiating laboratory runs.

Discrepant Analysis and Discordant Resolution: The Critical Bias Pitfall

A common compliance deficiency in molecular diagnostic validation dossiers occurs when laboratories modify initial 2×2 tables following "discrepant resolution".

INCORRECT (BIASED) WORKFLOW:
Candidate Assay vs. Comparator ---> Identify Discordant Samples ---> Test Discordants with Sanger/dPCR ---> Modify 2x2 Table to "Resolved PPA/NPA" (NON-COMPLIANT)

CORRECT (COMPLIANT) WORKFLOW:
Candidate Assay vs. Comparator ---> Primary 2x2 Table & Initial PPA/NPA Locked ---> Orthogonal Root-Cause Analysis ---> Report Findings in Auxiliary Narrative Table Only

The Statistical Bias of Discrepant Resolution

When candidate and comparator results disagree ($b$ or $c$), laboratories often re-test those specific specimens using an orthogonal method, such as bi-directional Sanger sequencing, digital PCR, or microbial culture. If the orthogonal assay confirms the candidate assay's result, laboratories are often tempted to reclassify the specimen as a "true positive" or "true negative," calculating a "resolved PPA" or "corrected NPA."

Both the FDA (in Section 6 and Appendix 7.3 of its diagnostic statistical guidance) and biostatistical literature demonstrate that this practice introduces severe conditioning bias. Because concordant specimens ($a$ and $d$) are never subjected to the resolver, any errors present among the concordant results remain unexamined, while discordant samples resolved in favor of the candidate assay artificially inflate the final agreement metrics.

Regulatory Reporting Rules

To comply with FDA and CLSI requirements:

1. The primary 2×2 table remains immutable. The unadjusted PPA, NPA, and associated 95% CIs derived from the primary comparison must be reported as the primary clinical performance specification.
2. Post-resolution recalculations cannot be reported as primary performance. Claims of "resolved PPA of 99%" are unacceptable in summary tables or package inserts.
3. Orthogonal testing is presented descriptively. Discordant resolution may be included in an auxiliary, exploratory section to provide root-cause context (e.g., characterizing sequence variants, probe-binding site mutations, or analytical levels below the comparator's limit of detection).

High-Plex Panels: Target-Level vs. Specimen-Level Reporting

Multiplex molecular assays—such as syndromic gastrointestinal panels, 20+ target respiratory pathogen panels, and targeted qualitative NGS oncology panels—present distinct validation challenges.

The Specimen-Level Fallacy

A frequent error in multiplex assay validation is reporting an aggregate "specimen-level percent agreement."

Consider a 22-target multiplex panel tested on 100 patient specimens (2,200 total analytical determinations). If an individual specimen contains Legionella pneumophila that the candidate assay misses, but correctly calls the remaining 21 targets negative, calculating agreement across all calls yields 21/22 (95.5%) agreement for that single specimen. Pooled across the study, the assay could miss 100% of the true Legionella cases while demonstrating an overall specimen concordance exceeding 99%.

CLSI EP12-Ed3 and the FDA mandate that clinical agreement must be calculated and displayed target by target. Each organism, target gene, or sequence variant must have its own dedicated 2×2 contingency table, PPA, NPA, and 95% confidence intervals.

*Note how small sample sizes for low-prevalence targets produce broad, non-informative confidence intervals, underscoring the need for supplemental enriched or banked cohorts.

Managing Low-Prevalence Targets

For rare targets (such as Bordetella pertussis or uncommon antimicrobial resistance markers), prospective clinical collection alone rarely yields sufficient positive cases within practical validation timelines.

• Banked Retrospective Specimens: Validated, frozen specimens with known comparator results may supplement prospective cohorts, provided pre-analytical storage stability has been verified.
• Contrived (Matrix-Spiked) Samples: Spiking quantified reference materials, inactivated organisms, or synthetic constructs into negative clinical matrix is acceptable to demonstrate baseline target recovery at concentrations spanning the assay range (e.g., 1–2× LoD, 3–5× LoD).
• Segregation of Reporting: Contrived samples must never be pooled with native clinical specimens in the primary clinical agreement table. They must be presented in a separate, dedicated section of the validation dossier labeled as contrived specimen performance.

Regulatory and Reimbursement Dossier Strategy

Validation data serve dual purposes: establishing regulatory and accreditation compliance and supporting reimbursement coverage.

CLIA/CAP Validation Binders

Under CLIA regulations (42 CFR § 493.1253) and College of American Pathologists (CAP) checklist requirements, laboratory directors (HCLD/DABMM) must review and approve method comparison data before releasing patient results. Inspectors evaluate whether:

• The comparator method was an appropriate predicate or high-order method.
• Sample sizes were statistically justified before testing.
• Initial and final invalid or indeterminate rates are documented. Samples that yield unresolved invalid calls must be recorded in an auxiliary table and not quietly omitted from the denominator.

MolDx Technical Assessments

For laboratories seeking Medicare reimbursement across jurisdictions administered by Palmetto GBA, Noridian, CGS, and NGS, molecular tests must undergo a rigorous Technical Assessment (TA) to obtain coverage and DEX Z-Code assignment. The MolDx program frequently flags and rejects validation submissions that commit the following structural errors:

• Using resolved 2×2 tables as primary performance data.
• Pooling distinct panel targets into composite specimen-level concordances.
• Submitting confidence intervals calculated using the Wald approximation.
• Failing to demonstrate concordance across all claimed sample matrices.

Navigating these expectations requires an integrated approach that aligns analytical study design directly with MolDx technical assessment requirements before bench testing begins.

Pre-Validation Clinical Agreement Checklist

Before initiating patient sample testing for clinical validation, ensure the protocol addresses the following requirements:

1. Comparator Selection: Confirm that the comparator is an established predicate device, reference method, or validated standard-of-care assay. If non-reference, verify that the protocol specifies PPA and NPA rather than sensitivity and specificity.
2. Statistical Plan: Define the confidence interval method in the validation protocol (Wilson score or Clopper-Pearson exact). Establish target positive and negative sample sizes sufficient to achieve the desired lower confidence limit.
3. Invalid Result Protocol: Define precise criteria for re-testing initial invalid, equivocal, or inhibitory runs, and create a workflow to track both initial and repeat invalid rates.
4. Discordant Workflow: Establish an orthogonal secondary testing plan for investigative root-cause analysis, ensuring the protocol mandates that the primary 2×2 table remains unadjusted.
5. Panel Architecture: For multiplex assays, configure data pipelines to extract and tabulate performance on an individual target-by-target basis.
6. Cohort Segregation: Separate prospective clinical specimens, banked retrospective specimens, and contrived matrix-spiked samples into dedicated, non-commingled validation reporting tables.

How Priestly Consulting can help

Priestly Consulting provides specialized, technical advisory support to clinical laboratories, molecular diagnostic developers, and biotechnology organizations designing and executing clinical validation studies. Our team assists laboratory leaders with biostatistical analysis planning, validation protocol authoring aligned with CLSI EP12-Ed3, and the preparation of comprehensive technical dossiers for CLIA/CAP compliance and MolDx Technical Assessments. To discuss your upcoming validation strategy or review your assay concordance data, schedule a consultation.