Metrology and Standards Needs for Gene Expression Technologies

1
Metrology and Standards Needs for Gene Expression
Technologies

• Krishna Ghosh
• Agilent Technologies
• June 10, 2003

2
Overview of Presentation

• History of Gene Expression Standards Development
• Microarray/Scanner Fluorescent Standards
• Status update
• Next Steps
• Universal RNA Standards
• Status update
• Next Steps
• NIST Feed back-Next Steps

3
Development of Standards for Gene Expression

• Kickoff meeting (NIST, Oct 2002)
• Workshops identified Microarray /Scanner
  Universal RNA
• Collaborative model development and workgroups
• Follow up meeting (NIST, Dec 2002)
• Technical overview of microarray readers and
  their performance
• Monthly teleconferences of working group to
  develop/detail artifact(s)
• Universal RNA Standards Workshop (Stanford, March
  2003)
• Microarray Fluorescence Standards Working Group
  Meeting, (NIST, May 2003)

4
NIST Working Group Microarray/Scanner
Fluorescence Standards

• Working Group Objectives
• Develop an artifact to measure microarray reader
  performance using a standardized method. The
  artifact(s) will be used primarily by
  manufacturers to standardize the measurement of
  specifications.
• Identify the appropriate fluorescent material(s)
  and manufacturing technology to produce a
  NIST-certified artifact
• Provide the required procedures and tools
  required to analyze the artifact
• Accomplishments to Date
• Assembled representatives from microarray reader
  manufacturers (Affymetrix, Agilent, Axon, Perkin
  Elmer, Biorad, Arrayit and others)
• Created a working draft of the artifacts
  features and characteristics
• Reviewed different options for fluorescent
  materials and manufacturing technologies (organic
  and inorganic dyes, metal oxides, nanocrystal
  composites, polymer coating, fluorescent glasses,
  Sol-gel)

5
Artifacts to Measure Scanner Performance Using a
Standardized Method

• Uniformity Artifact Measures scanners
  uniformity and Signal / Noise for bright
  features.
• Detection Limit Artifact Measures scanners
  limit of detection.
• Artifacts will be manufactured by someone other
  than NIST, BUT will be qualified by NIST.
• Artifact(s) intended for use primarily by
  manufacturers to standardize the measurement of
  specifications.

6
Preliminary Scanner Specification Decisions

• Artifacts will be uniformly coated (except for
  fiducials and background regions).
• There will be at least two artifacts per dye one
  in the middle of the dynamic range meant for
  uniformity measurement, the other for measuring
  signal/noise for very dim signals approaching the
  detection limit.
• The dimmer slide should approximate about 0.5
  chromophores per square micron of fluorescent
  material.
• The glass non-flatness should not exceed -10
  microns.
• The parallelism will be lt1 mrad
• The preferred substrate material is glass.
• The artifact will be available as 1.00 mm thick.
• The artifact will be 1x3 inches and can be fitted
  to other users.
• The dye choice should match Cy3/Cy5 as closely as
  possible.
• Photostable and environmentally stable

7
Coating Requirements

• Transparent coating on high purity, low
  background emission slide glass.
• Mimic Cy-3 and Cy-5 dye properties
• Cy-3 Excitation at 532 nm (frequency doubled
  Nd-YAG laser)
• Cy-3 Emission at 562 nm.
• Cy-5 Excitation at 633 nm (Helium-Neon laser)
• Cy-5 Emission at 660 nm
• Long-Term Stability 
• Intensity uniformity over coated area (1 CV)

From presentation of E.Pope
8
Present NIST Artifact Slide Layout
From presentation of J.Corson
9
Summary of Workshop Presentations14th May 2003

• Metal oxide glasses are less prone to
  photobleaching than organic dyes and have
  promising spectral characteristics
• There are a variety of ways to prepare
  fluorescent standards to meet the Artifact Slide
  requirements(Matech)
• Fluorescent microspheres offer photostability and
  compatibility with different attachment
  chemistries(Molecular Probes)
• Nanocrystals (semi-conductor materials) can mimic
  emission wavelengths of fluorescent dyes while
  using common laser and broadband sources (Evident
  Technologies)

10
Microarray/Scanner Fluorescence Standards- Next
Steps -

• Select fluorescent material for the artifact
• Silica doped with metal oxides
• Q dots/Nanocrystals
• Alternate organic photostable fluorescent dye
• Assemble manufacturers capable of making
  specified artifact with uniform coating expertise
• Define recommended use and data analysis
  procedures
• Refine artifact specifications depending on the
  limitations

11
Universal RNA Standards WorkshopStanford
University, March 28-29, 2003
12
Goals of Universal RNA Standards Workshop

• Educational provide participants a forum to
  share various methods and techniques that are
  relevant to defining a standard for Gene
  Expression and RT-PCR technologies
• Awareness determine areas of agreement and
  disagreement on issues of different standards,
  and where there is a need for additional
  information
• Guidance help define how NIST could best help to
  develop the RNA standard(s) and promote its use
• Requirements a reliable, reproducible and
  manufacturable standard to support the use of
  gene expression results for IVD and NDA filings
  (proficiency testing, comparison of submitted
  data with published data, comparison of different
  gene expression platforms), AND can be used by
  manufacturers to certify their products

13
Gene Expression Workflow- Standardization
Requirements
14
Generalized Workflow
How many standards are needed to address the
workflow differences due to Instrumentation,
Reagents, Sample, Operator and Analysis?
15
Session 1Standardization of Biological Component
of RNA Based Molecular Assays

• Focus Review the needs for gene expression
  measurement standards in support of Safety and
  efficacy claims of therapeutic products, and
  Human clinical in vitro diagnostics.
• Session Summary
• Standards can be used for proficiency testing,
  benchmarking, cross-platform comparisons,
  inter-laboratory comparisons
• Significant sources of variation are Array
  Platform, Lab and Array-to-Array etc..
• Standards, analysis tools metrics, and training
  are needed to help assure that platform and
  sample processor are capable of detecting the
  biological truth
• Reference methods and reference materials
  (traceable, assignable) are key to defining
  analytical and clinical performance
  characteristics
• Consideration should be given to
  bioinformational standards with approved gene
  lists and indicator patterns used as training
  sets for decision rules

16
Design for Experiment

• 1.  Tissue extraction liver vs. 5-tissue pool
  from 22 C57 black, adult male mice (NIEHS
  NTP/OHSU)
• 2. Standard RNAs liver, 5-tissue pool (NIEHS)
• Chips
• Standard 18K mouse oligo (Duke)
• Resident cDNA, oligo, Affymetrix, Agilent
• 4. Hybridizations (4) liver vs. liver liver
  vs. pooled with fluor dye flips
• 5. Data quality Arabidopsis10-gene set in
  standard RNAs and on chips (Wang et al, 2002)
• 6. OHSU Data Warehouse Web hosting and data
  sharing, MIAME sheet for experimental details,
  gene annotations, resident analysis and stats
  tools

17
Sources of Variation Preliminary Trends
From presentation of B.Weiss
18
FDA Funded collaborative Project for Evaluation
of Performance Standards for Toxicogenomic
StudiesUsing benchmark genes within mixed tissue
samples

• Identify tissue-selective, low variance rat genes
  from control animal data in large databases
  (populated using a consistent protocol).
• Select tissues with large difference in number of
  tissue-selective genes from control animal data
  
• Model a pilot set of tissue mixtures for the
  standard using database info and test on arrays
• Identify probe sets corresponding to benchmark
  genes on different platforms
• Evaluate the added value of exogenous spike-in
  standards (platform-dependent).

Hsiao et al., Physiol. Genomics 7 97-104, 2001
19
Expected Initial Outcomes

• Identification of probes that can perform
  similarly across platforms
• Determine normal range of false positive/false
  negative rates for MTS
• Determine normal range of lab-to-lab variance for
  MTS
• Determine normal range of cross-platform variance
  for MTS
• Publication of findings.

20
Session 2Metrics for Universal
StandardExpression Arrays

• Focus What standard parameters/metrics are
  needed for existing expression array technologies
  to ensure both intra-laboratory and
  inter-laboratory comparison? What are the key
  functions and features of the standard?
• Session Summary
• RNA sample quality can impact usefulness of
  microarray results (
• Not all RNA is created equal. An RNA quality
  index, taking into account a number of physical
  metrics, can be formulated
• RNA spikes can be used to assess and limit
  variability for this reasonably complex procedure
  
• Transcript pooling offers an organism specific
  approach to assessing sensitivity, specificity
  and reproducibility

21
Session 3Appropriate Standards to Meet Metrics

• Focus What are currently implemented
  intra-laboratory controls and their
  effectiveness? Are these applicable to
  inter-laboratory comparisons and cross-platform
  comparisons?
• Session Summary
• Depending on its purpose, more than one RNA
  standard is probably required, and will evolve
  with time
• Begin with the end in mind what are the design
  specifications to meet the user requirements?
• Complex synthetic sample offers a sample of
  intermediate manufacturing difficulty and
  intermediate customer relevance versus oligo-only
  and complex natural samples
• Universal reference RNAs are a commercially
  available blend of total RNA isolated from
  multiple cell lines designed to maximize gene
  expression profiling

22
Session 4 Metrics for Universal
StandardQuantitative RT-PCR

• Focus What are the key functions and features of
  standards for RT-PCR? What are currently
  implemented intra-laboratory controls and their
  effectiveness?
• Session Summary
• Emerging genetic tests will be evaluated for
  analytical validity, clinical validity, clinical
  utility and ethical, legal and social
  implications
• Internal quantitative standards (e.g., HSK) can
  demonstrate the impact of sample collection and
  processing methods on data quality
• Internal calibrators or standard curves are
  required for accurate absolute quantification
  relative quantification is possible with
  endogenous controls
• Human universal reference RNA can be used for
  QRT-PCR
• QRT-PCR has applied normalization to control
  variation in total RNA mass and amplification
  efficiency, and ratio-ing to verified invariant
  genes

23
Themes from the Universal RNA Standard Workshop

• Multiple sources of data variability
• Different laboratories, platforms, sample types,
  extraction methods, etc.
• Evolving technologies
• Probe/primer design
• Difficulties sharing data
• MIAME is a start
• Annotation problems abound
• Is the same analyte being measured?
• Standard methods and metrics for proficiency
  testing
• Terminology and definitions are needed
• Differences between analytical and clinical
  applications

24
All RNA is Not Created Equal
25
RNA Quality Index

• Sample isolated with minimal degradation
• RNA extracted using the same method
• RNA is proven stable during study
• RNA has a 28S18S ratio above 1.5
• Free of DNA for Real-time validation

The procedures are available but not all tools
are easily used or in simple kit formats.
26
A Good Standard Should

• Allow performance validation of any single
  platform over time.
• Facilitate comparison between various platforms
  used to assay gene expression.
• Be constructed in such a manner as to assure
  consistency over time.
• Include a well defined protocol describing how it
  is made and validated.
• Include two or more samples that allow one to
  make both absolute and relative measurements of
  the abundance of individual transcripts.
• Not be limited to hybridization-based approaches,
  but should be amenable to use with other assays
  such as QRT-PCR

27
Standards to evaluate platform (and laboratory)
performance

• Standard should be relatively invariant and
  regenerable.
• Standard should be formed on probes in common
  between platforms.
• Standard should resemble test samples (e.g., rat
  tissue standard for toxicogenomics) and also
  query a wide range of features on the arrays
• A set of mixed tissue standards that have varying
  ratios of components would contain real and
  quantifiable gene expression changes between
  samples and use endpoints measurable on most
  platforms

28
Microarray Performance Characteristics and
Controls

• Characterization of Array
• Design and fabrication, e.g. platform type,
  surface type, composition and spatial layout,
  number of elements (spot), number of replicates,
  etc.
• Spot elements, e.g. clone, sequence, PCR primer
  pairs, probe length, gene name, etc.
• Built-in controls, e.g. housekeeping genes, etc.
• Microarray Controls
• Internal controls (housekeeping genes, synthetic
  RNA)
• Pooled RNA from cell lines
• Pooled RNA from test samples
• RNA and oligonucleotides from plants and bacteria

29
Session 5Proposed Workshop Recommendations

• Standards are required for several purposes in
  gene expression RNA analysis regardless of
  whether quantitative RT-PCR or microarray gene
  expression technologies are employed.
• Periodic laboratory proficiency testing
• Platform performance validation and baseline
  monitoring
• Cross-platform performance validations
• Inter-laboratory performance validation
• Providing a reference point for regulatory
  agencies evaluating gene expression profiling
  data
• A consistent definition of terminology is needed
• The consensus of the attendees was to develop two
  types of RNA standards, External Synthetic RNA
  Standard Reference Material Internal RNA
  Reference Standard

30
Session 5Proposed Workshop Recommendations

• External Synthetic RNA Standard Reference
  Material
• Moderately complex pool of highly characterized
  synthetic mRNA targets
• Used across all RNA gene expression profiling
  platforms, including both RT-PCR based and array
  based methods.
• Measures the accuracy, dynamic range, sensitivity
  and specificity of each platform under any given
  set of conditions.
• Internal RNA Reference Standard
• Spike-in RNA standards used in conjunction with
  standard probe sequences
• Compatible with all array formats, and available
  to labs making in-house arrays
• Provides an internal measure of the quality of
  any particular array experiment (sensitivity,
  dynamic range, and specificity)
• Array manufacturers could also use this material
  for quality control testing their products
• Reference Method
• Instructions on the correct use of the reference
  materials are required

31
External Synthetic RNA Standard Reference Material

• Modular components
• Individually characterized and pooled
• Extendable and upgradable
• Number of human sequences 96
• Maximum of 2000 bases beginning with 3 end
• Selected component characteristics
• Conserved, minimally polymorphic sequence
• Well characterized tissue specificity, splice
  variants, gene family members, etc.
• Cloned into expression vector, sequenced, high
  purity and stability
• Represents 106 fold range of absolute
  quantitative expression

32
Internal RNA Reference Standard

• Spike-in standard to array probes serves as
  internal positive control of array performance
• One set of 12 sequences
• Target sequence gt 600 nucleotides
• Sequence content acceptable to different array
  platforms
• Cloned into expression vector, sequenced, high
  purity
• Covers dynamic range of the platform
• All array platforms have optimized probes with
  5-10X redundancy across the array
• Instructions on correct use of spike-in standard
  to be provided (standard reference method)

33
Open Questions and Next steps

• NIST Guidance document published by end of June
  2003
• Formation of working group members
• Will NIST take up this project with limited
  resources and budget?
• Will FDA make a formal request to NIST for
  specific standards to facilitate acceptance of
  Tox /Pharmacogenetic data submissions?
• Should the platform manufacturers develop their
  own standards to establish platform performances?