Assessment of a Video Quality Metric presenter Sarnoff Corporation Meeting Name April xx, 2001

1
Results of the ATIS/T1A1.1Ad Hoc Group on
Full-Reference Video Quality Metrics
(FR-VQM)VSF MeetingOctober 3, 2001John
PearsonSarnoff Corporationjpearson_at_sarnoff.com
2
Take Home Messages

• Tariffs can now include Visual Quality Metrics
  (Full Reference)
• The basis for this is a family of 4 Technical
  Reports by ATIS/T1A1
• The T1A1 approach is extensible to additional
  Visual Quality Metrics, and does NOT establish a
  Standard

3
Outline

• Why is measuring Visual Quality important?
• Why is measuring Visual Quality hard?
• International Standards for VQMs
• T1A1 Technical Reports

4
FR-VQM Needs of US Telecom
Q-??
Q-A
Q-B
Company Auses VQM-A
Company Buses VQM-B
Site of Video Origination(e.g., Denver)
Transfer Between Network A B
Site of Video Consumption(e.g., Mexico City)

• Digital video processing can create objectionable
  noise
• End-to-End QoS across the networks of multiple
  companies requires agreement on Quality at
  Transfer Points (Tariffs)
• Tariffs require ANSI sanctioned technical
  documentation

5
Digital Video Creates Patterned Noise ... Human
visual response to patterned noise highly
non-linear ...
Blocky Digital Noise
Random Analog Noise
MSE 27.10
MSE 21.26
Measures like MSE suitable for Analog noise no
longer work for Digital noise
6
Patterned noise in the sky much more
perceptible even though much smaller in terms of
pixel differences
Source Frame
Difference Map
Codec Frame
7
Visual Quality Metrics... correlate well across
scene types, unlike MSE ...
Visual Discrimination Model
Mean-Squared Error
Mean of 80 trials for 20 subjects
Bars show 5 confidence intervals
8
Vital Role of Subjective Database

• Goal of VQMs is to approximate subjective quality
  assessments (SQA)
• The relevance of the SQA depends on
• Test sequences (SRCs)
• Distortion generators (HRCs)
• Viewing conditions and testing protocols
• Producing a relevant SQA is hard

9
Three Kinds of VQMs

• Full Reference (FR)
• a double-ended method and is the subject of this
  Technical Report.
• Reduced Reference (RR)
• only reduced video reference information is
  available. This is also a double-ended method.
• No Reference (NR)
• no reference video signal or information is
  available. This is a single-ended method.
• It is generally believed that the FR method will
  provide the most accurate measurement results
  while the RR and NR methods will be more
  convenient for QoS monitoring.
• The T1A1 Technical Reports concern FR methods

10
Full-Reference VQMs with Normalization
11
International Standards Progress

• VQEG may be several years from recommending a
  FR-VQM standard to ITU
• Its possible that no single FR-VQM will be a
  clear winner
• The FR-VQM field is young, and significant,
  steady improvements are expected over the next
  decade
• Its possible that several different FR-VQMs may
  gain industry acceptance

12
T1A1.1 FR-VQM Strategy an extensible family of
TRs for FR-VQ, enabling Industry to move ahead
without Standards ...

• Provide guidelines for how Industry can
• specify its specific FR-VQM needs
• assess the suitability of existing documented
  FR-VQMs
• drive the development by FR-VQM proponents of
  new/improved FR-VQM algorithms and products
• inter-operate with different FR-VQMs
• Provide guidelines for how FR-VQMs can be
• documented in algorithms, accuracy and
  limitations
• quantitatively cross-calibrated to each another
• Extensible framework enabling addition of FR-VQMs
• Start by specifying two already disclosed FR-VQMs
• Stimulates continued FR-VQM innovation

13
Primary Contributors
14
Family of Technical Reports

• TR A1 Accuracy and Cross-Calibration (Mike
  Brill, Sarnoff)
• defines accuracy (statistical analysis),
  limitations of a FR-VQM
• defines transformation to common scale, for
  cross-calibration with other applicable FR-VQMs
• TR A2 Normalization Methods (David Fibush,
  Tektronix)
• applied to source and processed video before VQM
  calculation
• e.g., spatial/temporal registration, gain/level
  offset calibration, ...
• may utilize special test signals
• TR A3 Peak Signal to Noise Ratio (Steve Wolf,
  NTIA)
• Specify PSNR VQM, following TR A1 and TR A2
  guidelines
• TR A4 Objective Perceptual FR-VQM Using a
  JND-Based Full Reference Technique (David Fibush,
  Mike Brill)
• Specify JND-based FR-VQM, following TR A1 and TR
  A2 guidelines

15
TRA1 Defines Basic Methods

• How to specify VQM accuracy
• with respect to subjective assessments
• based on defined statistical analysis
• How to specify VQM scope/limitations
• type of scene content (signal)
• high/low motion, color/bw, interlaced/progressive
  
• type/severity of artifacts (noise)
• e.g., encoding techniques, bit-rates, blurring,
  blockiness
• subjective testing characteristics
• behavior with viewing distance, resolution,
  gamma,
• expert vs non-expert viewers
• How to cross-calibrate VQMs
• determination of mathematical transformation
  relating one VQMs outputs to anothers

LIMITATIONS
SCOPE
Works well, has been well tested here
16
VQEG Database SRCs
17
VQEG Database HRCs
18
JND/PQR PSNR Limitations no H.263
19
Algorithm Documentation JND/PQR
20
Stripping for JND/PQR Registration
21
Algorithm Documentation PSNR
22
Normalization Requirements
JND/PQR
PSNR
23
VQEG data Logistic-mapped PQR
24
Logistic-mapped PQR for Common Scale provides
approach for cross-calibration...
25
Accuracy -- 3 Methods

• RMSE
• Resolving Power
• Classification of Errors

26
Confidence vs.D-VQM JND/PQR
27
Confidence vs.D-VQM PSNR
28
RMSE

• RMSE root mean square error between subjective
  and objective normalized scores

29
Classification of Errors
30
(No Transcript)
31
Progress

• T1A1.1 Ad Hoc Group created Feb. 2001, co-chairs
  John Grigg, John Pearson
• Mail Ballot Approval August 2001
• Approved by T1A1.1 25 September 2001
• Approved at Plenary meeting of T1A1, 28 September
  2001