Requirements for NAHLN Deployment of New Diagnostic Assays
FINAL DOCUMENT 06.21/2023
Submission Checklist for Feasibility Studies
This document is intended to provide guidance to commercial manufacturers of reagents and testing
platforms regarding the minimum validation requirements for acceptance into the NAHLN. Additional
approvals by the National Veterinary Services Laboratories (NVSL), VS’ Center for Epidemiology and
Animal Health, and the appropriate VS commodity program may be required prior to deployment of
new test methods to NAHLN laboratories.
For the initial stage of approval, the following information is requested. See Appendix A for additional
detail on each of these topics:
1. Intended purpose of the assay including specific use cases
2. Side-by-side comparison of current NAHLN protocol/SOP using an initial test panel obtained
from NVSL [subject to test panel availability*]. Data from this panel should include:
a) Operating range of assay: One or two high positive reference samples that have been
serially diluted to extinction.
b) Assay specificity: for nucleic acid-based assays, an in-silico analysis should be provided
for new primer and/or probe sequences. (Recommended but not required.)
c) Assay Standardization: comparison of NAHLN SOP to the new method.
a. Test panel must contain a minimum of 30 samples and cover high pos, low pos
and negative reference samples.
d) Matrix Inhibition:
b. Data must be provided for each different sample or matrix type that will be used
for the assay. Recommended to include all currently approved sample types.
e) Robustness
c. Data must be provided for at least two lots of reagents, and preferably tested by
two different people.
d. Specific test parameters for nucleic acid-based tests:
Test results provided using at least one and preferably all, commercial
extraction and thermal cycler instruments used by NAHLN laboratories.
e. Specific test parameters for commercial primer/probe reagents:
Test results provided using a minimum of two and preferably three
common commercial master mixes.
f. Specific QC controls for nucleic acid-based PCR tests:
No-template control (NTC)
Analyte-positive control
Inhibition control or extraction control
For RNA assays – reverse transcription control
Requirements for NAHLN Deployment of New Diagnostic Assays
FINAL DOCUMENT 06/21/2023
2
Submission Checklist for Final Validation Study
Once the initial feasibility study data set has been reviewed and approved by the NAHLN MTWG, a
second, larger panel suitable for generating the following data will then be made available through NVSL
[subject to test panel availability*]. A side-by-side comparison of the second panel to the current NAHLN
SOP must be conducted for all the following parameters and provided for final review and approval by
NVSL and NAHLN prior to deployment. See Appendix B for additional details.
1. Analytical Performance:
a. Repeatability: Test three to five currently circulating field samples representing the analyte
activity within the full operating range of the assay.
b. Analytical Sensitivity (limit of detection): a dilution series of the target pathogen diluted in
all the different sample matrices. The dilution series should be to extinction.
c. Analytical Specificity: refers to the ability of an assay to measure a particular organism or
substance, rather than similar organisms, in a sample.
2. Diagnostic Performance: See Appendix B for additional details on each of these requirements.
a. Diagnostic Sensitivity (estimation of false negatives): Ability of the test to identify the
presence of disease correctly.
b. Diagnostic Specificity (estimation of false positive): Measure of diagnostic test accuracy.
c. Diagnostic Reproducibility: Agreement of multiple laboratories using the same diagnostic
procedure.
*If test panels are not available through NVSL, manufacturers may choose to provide their own sample
panels; however, prior approval of panel composition will be required from NVSL and the NAHLN
Program Office before initiating any validation studies.
For questions regarding this process, please contact the NAHLN Program Office at NAHLN@usda.gov
.
For information on test panel availably contact NVSL.info@usda.gov
Requirements for NAHLN Deployment of New Diagnostic Assays
FINAL DOCUMENT 06/21/2023
3
Appendix A: Dossier Requirements for Feasibility Studies
More detail on specific experimental design parameters for each of the required data sets is provided in
this Appendix. For feasibility studies only, sample panels may contain biological samples (live or
inactivated) or spiked samples containing either biological or synthetic analytes.
Intended Purpose of Assay
- Must provide intended purpose for the assay.
o This includes a qualitative and quantitative assessment of the ability of a positive or
negative test result to accurately predict the infection or exposure status of the animal
or population of animals.
o Examples of intended purpose(s):
- Demonstration of freedom from infection in a defined population
(country/zone/herd).
- Confirm freedom from of infection for trade/movement purposes.
- Confirm diagnosis of infected animals.
- Estimate prevalence of infection for risk analysis.
- Determine the immune status of an animal or animal populations post-
vaccination.
o Must identify target animal species, target diseases, and sample types/sample matrices.
- Data for the feasibility testing must be provided comparing the new assay to the equivalent
procedure currently deployed to the NAHLN.
Side-by-side comparison to NAHLN SOP
- Feasibility data must include a side-by-side comparison of the current NAHLN protocol using the
same sample panels. The following data must be provided:
Operating range of the assay
- Use of a high positive reference sample that has been serially diluted to extinction, based on the
assay’s response curve. Sample(s) must be diluted in an analyte-negative matrix of the same
constitution as the sample matrix and be from animals targeted by the assay. The results should
be plotted as a ‘response-curve’, with the response (e.g., optical density, cycle threshold,
counts, etc.) as a function of analyte concentration (amount). Spiked samples are permitted for
this study.
Specific feasibility studies for Ab based assays:
- Test panel must be comprised of at least 30 samples spanning the operating range of the assay.
o Must include one or more reference standards if required for data normalization.
o Must demonstrate adequate separation of results between negative, low positive and
high positive samples.
- The lower OD range should be 0.1 or less for negative controls in an indirect
ELISA,
- OD values should not exceed 2.0 for strong positive controls in
competitive/blocking ELISAs to avoid inaccuracy in plate readers at this range.
- Test panel must contain at least two contemporary field strains and closely related strains to
test for cross-reactivity.
Requirements for NAHLN Deployment of New Diagnostic Assays
FINAL DOCUMENT 06/21/2023
4
Assay Standardization
- Test panel must contain high pos, low pos and negative reference samples.
o If more than one sample matrix will be approved for use with the assay, standardization
data for each sample matrix must be provided.
o In general, it is recommended that at least 30 samples are tested, and include a range
high pos, low pos, and negative sample types.
- Reference samples must represent both known infected and uninfected
animals.
- Reference samples must be well characterized by at least one, and preferably
two, alternate methodologies.
o The alternative of preparing reference samples spiked with cultured
agents or positive sera is inferior, as these samples do not truly
represent the naturally occurring matrix-agent interaction.
o When no other alternative exists, spiking a sample with a known
amount of the analyte or agent derived from culture, or diluting a high
positive serum in negative serum of the same species may be all that is
available.
o The matrix into which the analyte is placed or diluted, must be
identical to or resemble as closely as possible the samples that
ultimately will be tested in the assay.
Specific feasibility studies for nucleic acid-based assays:
- Test results must include a comparison of at least one, and preferably all, commercial extraction
and thermal cycler platforms approved for use in the corresponding NAHLN SOP (e.g., ABI 7500,
QuantStudio5)
- Test results must include a comparison of all commercial master mixes approved for use in the
corresponding NAHLN SOP, and include the following:
o Information on assay components (primer/probe sequences, probe reporter and
quencher dyes, master mix composition and internal controls) should be provided.
o For multiplex real-time PCR assays, reporter dyes should be chosen for compatibility
with common commercial real-time PCR instruments.
o For all real-time PCR assays, threshold determination methods and positive/negative
cutoff values should be provided.
Optimization
Matrix inhibition studies
- Data must be provided for each different matrix type that will be used for the assay.
Robustness
- Data must be provided for at least two lots of reagents, and preferably tested by two different
NAHLN laboratories.
- Quality control data must be provided for at least two lots of reagents used during the feasibility
studies. Reagents used for quality control must be fully described. (Are these commercially
available? Will these be provided to external customers either in the kit or as a separate
reagent?)
o Specific QC controls for nucleic acid-based tests:
Requirements for NAHLN Deployment of New Diagnostic Assays
FINAL DOCUMENT 06/21/2023
5
- No-template control (NTC) – approximately 5% of wells should be NTC and
distributed randomly within the assay or over the plate when 96 or 386 well
plate formats are used.
- Analyte-positive control –must be tested with each plate (at least one well) and
Ct activity must be within the defined operating range of the assay.
- Inhibition control or extraction control – must be tested with each plate.
- For RNA assays – reverse transcription control must be tested with each plate.
Specific feasibility studies for Ag based assays:
- Results must be able to distinguish between samples with and without the analyte.
- Data must show minimal cross-reactivity when non-target antigen is present in the sample or
sample matrix.
- Data must assess repeatability for a range of control samples over several days.
o Recommend running all critical reagents against each other in a checkerboard titration.
Requirements for NAHLN Deployment of New Diagnostic Assays
FINAL DOCUMENT 06/21/2023
6
Appendix B: Dossier Requirements for Final Assay Approval
Once data has been submitted for the feasibility studies and has been approved by the NAHLN, the
following data will then be requested for final review and approval prior to deployment to the NAHLN.
Analytical Performance
- Ideally, the assistance of a statistician and disease expert should be engaged to ensure sample
size and experimental approach are valid.
Repeatability testing
- A minimum of three samples and preferably five samples should be tested, with samples
representing the analyte activity within the full operating range of the assay.
o All replicates must be tested through all steps of the assay, including any extraction
procedures and all working dilutions. It is not acceptable to prepare a final working dilution
of a sample in a single tube from which diluted aliquots are pipetted into reaction vessels, or
to create replicates from one extraction of nucleic acid rather than to extract each replicate
before dilution into the reaction vessels.
- Specific repeatability testing requirements For Ab based assays:
o A reference panel containing no less than 3 (preferably 5) samples covering the operating
range of the assay, and of sufficient quantity for at least 20 runs of the assay over several
days.
- At least one reference sample should be included in an indirect ELISA (pos serum
control) to normalize test samples against.
- CV should not exceed 15% for strong positive samples.
- Specific repeatability testing requirements for nucleic acid-based assays,
o A reference panel containing a minimum of three and preferably five samples covering the
operating range of the assay must be tested using the entire assay procedure, including
nucleic acid extraction.
o Intra-assay data must be provided that has a minimum of five replicates of each sample in
one assay run.
o Inter-assay data must be determined by testing samples over several days, using multiple
operators and at least 20 runs.
Analytical Specificity (ASp)
Evaluation should reflect the test purpose and assay type.
- Specific analytical specificity parameters for antibody detection assays:
o Sera from animals infected with related species should be used to evaluate analytical
specificity.
- Specific analytical specificity parameters for antigen-based assays:
o ASp should be determined by testing well characterized samples from similar or related
pathogens, which produce similar lesions as the target pathogen or are frequently found in
samples containing the target pathogen. For example, to assess the ASp of a FMD antigen
detection ELISA for one particular serotype (e.g., serotype O), it is necessary to assess its
Requirements for NAHLN Deployment of New Diagnostic Assays
FINAL DOCUMENT 06/21/2023
7
reactivity of all sub-strains within this serotype (e.g., O Campos, O Manisa, etc.) to assess
inclusivity. At the same time, it is important to show that the test does not cross-react with
other serotypes such as A, Asia 1, C, SAT 1, 2 and 3. Finally, there is also a need to assess
whether the test cross-reacts with agents from diseases which may produce similar signs,
e.g., vesicular stomatitis, swine vesicular disease and swine vesicular exanthema.
- Specific analytical specificity parameters for nucleic acid-based assays:
o Data demonstrating the discriminatory power of the assay must be provided by testing
pathogens that cause similar clinical syndromes which are likely to be found in the target
sample.
o Example: if the assay is designed to detect avian influenza A virus, pathogens such
as Newcastle disease, infectious bursal disease, etc. should be tested to
demonstrate analytical specificity.
o Data will also be required for pathogens with known variants circulating in animal
populations, such as HPAI and FMD. In these cases, the assay must be evaluated against as
many well-characterized isolates of the virus as available to ensure all strains from a variety
of geographical areas and hosts are detected.
o How the assay will be used and in which geographical regions (example: North
America) will determine whether it is necessary to evaluate some or all available
lineages
Analytical Sensitivity (ASe)
- Limit of Detection (LOD)
This is the estimated amount of analyte in a specified matrix that would produce a positive
result at least a specified percent of the time.
- Specific LOD parameters for Ab based assays:
o Data must be provided demonstrating the LOD using an endpoint dilution in which 10
replicates of each dilution in a log
2
dilution series are run in the assay.
- Specific LOD parameters for Ag based assays:
o Data must be provided demonstrating the LOD using an endpoint dilution in which 10
replicates of each dilution in a log
2
dilution series are run in the assay.
o Screening assays or assays designed to detect sub-clinical infections should have a very high
ASe.
o If available, serial samples from experimentally infected animals could provide temporal
information about the assay’s capacity to detect antigen over the course of the infection.
- Specific LOD parameters for nucleic acid-based assays:
o Data is required showing a dilution series of the target pathogen diluted in all the different
sample matrices. Dilution series should be to extinction – i.e., not detected by the assay in
any of the replicate samples at that dilution.
o Samples must be tested by both the assay and by another standard method of detection.
Requirements for NAHLN Deployment of New Diagnostic Assays
FINAL DOCUMENT 06/21/2023
8
Diagnostic Assay Performance
Diagnostic Sensitivity and Specificity (DSe and DSp)
- Ideally, samples used for determining diagnostic assay performance are derived from testing a
panel of samples from reference animals, of known history and infection status relative to the
disease/infection in question and relevant to the country or region in which the test is to be
used.
- An estimate of the area under the receiver operating characteristic (ROC) curve is a useful
adjunct to DSe and DSp estimates for a quantitative diagnostic test because it assesses its global
accuracy across all possible assay values.
- The designated number of known positive and known negative samples will depend on the likely
values of DSe and DSp of the candidate assay and the desired confidence level for the estimates.
- The table below provides two panels of the theoretical number of samples required, when
either a 5% or 2% error is allowed in the estimates of DSe or DSp.
Theoretical number of samples from animals of known infection status required for establishing diagnostic
sensitivity (DSe) and specificity (DSp) estimates depending on likely value of DSe or DSp and desired error margin
and confidence.
2% error allowed in
estimate of DSe and DSp
5% error allowed in
estimate of DSe and DSp
Confidence
Confidence
Estimated
DSe or DSp
90% 95% 99% 90% 95% 99%
90%
601
864
1493
98
138
239
92%
466
707
1221
75
113
195
94%
382
542
935
61
87
150
95%
372
456
788
60
73
126
96%
260
369
637
42
59
102
97%
197
279
483
32
45
77
98%
133
188
325
21
30
52
99%
67
95
164
11
15
26
- Use of experimentally infected or vaccinated reference animals
o Multiple serially acquired pre- and post-exposure results from individual animals are not
acceptable for establishing estimates of DSe and DSp because the statistical
requirement of independent observations is violated.
o A single time-point sampling of individual experimental animals can be acceptable (e.g.,
one sample randomly chosen from each animal).
o In cases where the near impossibility of obtaining suitable reference samples from
naturally exposed animals necessitates the use of samples from experimental animals,
the resulting DSe and DSp measures should be considered as less than ideal estimates of
the true DSe and DSp.
Diagnostic Reproducibility
This is a measure of the agreement between results obtained in different NAHLN laboratories using the
same protocol, similar equipment, and the same panel of samples.
Requirements for NAHLN Deployment of New Diagnostic Assays
FINAL DOCUMENT 06/21/2023
9
- For all assays:
o A panel consisting of 20-30 samples, including at least three that are present in
quadruplicate, should be tested.
- The panel should consist of samples covering the range of the test, with several
that are very close to the test positive/negative cut-off values.
o The same panel used for repeatability testing can be used for reproducibility, but with
enhanced numbers of replicates.
References:
JACOBSON R.H. (1998). Validation of serological assays for diagnosis of infectious diseases. Rev. sci. tech. Off. int.
Epiz., 17, 469–486.
WOAH– Manual of Diagnostic Tests and Vaccines for Terrestrial Animals 2022. Ch. 1.1.6. Principles and Methods of
Validation of Diagnostic Assays for Infectious diseases. Accessed online 2/1/2023.https://www.woah.org/en/what-we-
do/standards/codes-and-manuals/terrestrial-manual-online-access/
WOAH- Manual of Diagnostic Tests and Vaccines for Terrestrial Animals 2022. Ch. 2.2.1. Development and
optimisation of antibody detection assays. Accessed online 2/1/2023.https://www.woah.org/en/what-we-
do/standards/codes-and-manuals/terrestrial-manual-online-access/
WOAH- Manual of Diagnostic Tests and Vaccines for Terrestrial Animals 2022. Ch. 2.2.2. Development and
optimisation of antigen detection assays. Accessed online 2/1/2023.
https://www.woah.org/en/what-we-
do/standards/codes-and-manuals/terrestrial-manual-online-access/
WOAH- Manual of Diagnostic Tests and Vaccines for Terrestrial Animals 2022. Ch. 2.2.3. Development and
optimisation of nucleic acid detection assays. Accessed online 2/1/2023.https://www.woah.org/en/what-we-
do/standards/codes-and-manuals/terrestrial-manual-online-access/
