Ingen lysbildetittel

1
Network Media Handling

• A common platform and scenario based on Virtual
  worlds, gaming and quality

Andrew Perkis, Peter Svensson, Professors Centre
for Quantifiable Quality of Service in
Communication Systems (Q2S) Department of
Electronics andTelecommuniations Norwegian
University of Science and Technology
2
Q2S

• Centre of Excellence in Quantifiable Quality of
  Service in Communication Systems Q2S
• Funded by Norwegian Research Council
• NOK 30 (EURO 4) million/year
• 45 people (5 Profs., Post.docs, PhDs and
  administration)
• Basic research and laboratory experimentation
• Main Goals
• Network Media Handling
• QoS assessment and monitoring
• QoS mechanisms for dynamic networks.

3
Common architecture - common usage scenario

• Process

• Q2S research plan

• Common usage scenario
• Develop work on Virtual worlds and serious
  gaming
• Align and coordinate research teams
• Team
• Andrew, Peter and Touradj
• Postdocs
• PhDs
• Master students
• Process
• Coordination weekly meetings, MindMap
• Information and communication
• Q2S research plan
• Network Media Handling
• Quality assessment and monitoring

• Networked Media handling
• Technology to present, manipulate and evaluate
  multi media content
• Focus on media representation, error protection,
  perceived quality, functional placement
• QoS mechanisms for dynamic networks
• Mechanisms that affect QoS, emphasis on
  heterogeneous and dynamic environment
• Focus on dependability, security, resource
  allocation
• Quality assessment and monitoring
• Measuring methods and models for QoS
• Measurement of perceived QoS, measurement methods
  and architecture, and traffic performance

4
Network Media Handling

• Focusing on the End-Points
• Functionality in media representation/distribution
  
• Error resilience/protection/concealment
• Media conversion and scalable/layered coding
• Multi-modal perception of audio and video

Interaction between Networks
End-Points Optimized media handling in wired and
wireless networks Congestion and admission
control for multimedia traffic Objective network
measurements for end-to-end QoS Quantifiable
perceptual quality metrics and their use
Application Studies Interactive two-way and
multi-party communication High-quality streaming
media services and applications Virtual acoustics
/ 3D audio systems Digital cinema
5
Multimodal Perceptual Quality Assessment

• Multimodal Perception
• Relationship between audio and video
• Perceptual effects of network impairments
• Relates to end-user expectation and context

Developing Objective Models Study of realistic
network environments Simulation or controlled
laboratory experiments Laboratory for multimodal
subjective experiments Design of quantifiable
metrics for perceived quality
Use of Objective Models Automated monitoring of
end-user perceived quality Relationship to
measurable end-2-end QoS parameters Using
Perceived QoS to adapt and enhance system
performance
6
Architecture
7
Research topics
8
Applications
9
Q2S resources and activities
10
Research results

• Compilations

11
Implementation
End user quality assessment
Congestion control with delivery of erroneous
packets
Classification of data, UEP, FEC
Low delay lossless compression for multichannel
audio
Computational quality measurement for IP-based
audio
12
Creative
Demonstration available during break
Figure 2 Sample stereoscopic video contents use
for measuring the impact of exposure affect on
multimedia quality.
13
Subjective test methods
4 parameter categories visual, audio, gameplay
and network
visual quality metric for HQ images
Novel Evaluation Method DSAQFA
Stereoscopic Quality assessment
14
Models and assesment
Economical and technology aspects of end-to-end
Quality of Services provisioning in the Internet
PEAQ ODG measurement
Focus of Attention (FOA) based Visual Quality
Assessment
15
FP7 Call 4 proposals
16
European proposals
17
EURO NF
35 partners
18
3D Media Internet Services (3DMEDIA) - NoE

• Leading WPs
• European 3D Media Lab integration
• 3D Media Quality of Experience modeling and
  evaluation
• Common software, data sets and repositories
• maintain inventory of participating labs
• enable strong collaboration
• Develop methods for subjective assessments
• Establish virtual 3D MEDIA QoE test facility
• Determine and measure quality features and models
• Contribute to standardization

19
Quality of Experience in Networked Systems and
Technologies (QUEST) - NoE

• Leading WP
• Develop an integrated QoE platform based on the
  datasets, measures and methodologies provided by
  other WPs.
• Explore, design, and implement a certification
  mechanism for quality of experience based on the
  integrated platform.
• Active participation in other WPs
• Subjective and objective image, video and
  audio-visual quality assessment with respect to
  different services, applications and scenarios

20
ENd User quality of experieNCe In multi modAl
neTworked mEdia ENUNCIATE CSA
21
Conclusions

• A good common scenario is under developments
• Substantial involvement
• Cross disciplinary
• Needs to enhance network components
• Has potential for being a platform for many
  integrated projects at Q2S

22
Research results

• Individual

23
High definition assesments

• Digital Cinema
• For very high qualities, PSNR slightly tends to
  be best objective metric so far, compared to
  structural similarity based metrics
• No significant differences in correlation between
  metrics
• Comparison effects of source vs. source
  channel distortion on perceived quality
• At low source distortion levels, PSNR tends to
  overestimate quality with both source channel
  distortion
• With increasing source distortion levels, ?
  between PSNR values of source and source
  channel distortion gets smaller

Ulrich
24
Novel Evaluation Method DSAQFA

• Compare different types of distortion
• Usually distortion expressed in terms of PSNR
• Qualitative characteristics of source and channel
  distortion are different
• PSNR doesnt scale well

Source distortion
Channel distortion
Ulrich
25
Ulrich
26
Results

• PSNR
• Is not a good measure when comparing different
  types of distortion
• Especially for low source distortion ( HQ video)
• Performance increases for higher source
  distortions ( LQ video)
• Methodology
• Well suited for comparing different types of
  artifacts
• Minimum training nesseray
• Fast, reliable, no rating scales

Ulrich
27
Congestion control based on delivery of partly
corrupted packets

• In the presence of bit errors, network throughput
  can be improved and stabilized by allowing
  delivery of corrupted packets
• Also the video quality is improved (in terms of
  PSNR), if appropriate resilience mechanisms are
  employed (example results below)

High bit error rate
Throughput Avg. PSNR Trad.
0.95 Mbit/s 32.62 MAC Lite 1.55 Mbit/s
34.30 Low bit error
rate Throughput Avg. PSNR
Trad. 1.41 Mbit/s 34.72
MAC Lite 1.57 Mbit/s 35.22
Jari
28
Modeling and services

• Computational models for audio and multimedia
  quality assessment
• Developed a priliminary version of a
  computational model for IP-based audio quality
  measurement
• Service and business models
• Developed a simple financial model demonstrating
  effectiveness of cooperation between competing
  providers for end-to-end quality of services
  provisiong
• Based on real Internet topology, investigated
  current probability and potential evolution
  scenarios for end-to-end QoS provisioning in the
  Internet

Eugene
29
PEAQ ODG measurement

• PEAQ ODG measurement results for AAC and Lame
  MP3 codecs with different bitrates and packet
  loss rates

Eugene
30
Focus of Attention (FOA) based Visual Quality
Assessment

• Focus of attention (FOA) has a great influence on
  visual quality assessment. Compared with image
  quality assessment, the influence of FOA on video
  quality assessment is stronger.
• Objective FOA model can be constructed based on
  bottom-up and top-down approaches.
• Our experiments demonstrate that the correlation
  between PSNR and subjective measurements can be
  increased by about 0.33 by integrating FOA into
  quality assessment.
• Based on certain quality features, a visual
  attention based quality metric (VAQM) is proposed
  and it outperforms the existing metrics, such as
  PSNR, SSIM, NTIA model.
• The future work is to find and design more
  suitable quality features and consider the
  temporal attention as well as the audiovisual
  attention analysis in the joint audio-video
  quality assessment.

Junyong
31
High definition quality evaluations
The scatter plots of MOS versus model predictions
Pearsons correlations and their associated 95
confidence intervals for each metric
Fitri
32
Content-creation

• Creation of mixed-reality representations that
  can be used for interactive future 3D-TV
  applications
• Created contents for Stereoscopic projection
• contents are currently created
• Manipulating video sequences to study facial
  motion perception on quality.
• Program and create interactive design and
  contents for augmented reality

Wendy Ann
33
Analysis of the impacts of affective elements

• Examine impact of affective elements in
  multimedia presentation on users perception of
  quality and user experience. The created contents
  will be used to access the impacts of the
  following variables
• Effects of familiarity in the in the perception
  of enjoyment and quality assessment in a
  stereoscopic environment.
• currently planning to conduct an experiment to
  determine the impact.
• Effect of moving Tatcher Illusion technique on
  perception of quality .
• Effects Familiarity over Photometric
  Inconsistencies. This examines how to effectively
  combine natural and synthetic environment.
• Research shows that people are excellent at
  identifying faces familiar to them, even from
  very low quality images, but are bad at
  recognizing, or even matching, faces that are
  unfamiliar (Hancock et al. 2000).
• Write a paper adding emotion-level on the
  users perspective model that affects user
  perception and multimedia quality.

Wendy Ann
34
Quantifying the strategy taken in an ensemble
delayed hand-clappeing
A parameter called compensation factor was
suggested to capture the amount of early clapping
which performers do to compensate the tempo
decrease caused by latency in a musical
interaction.
CF can explain Chafe effect Moderate amounts of
delay are beneficial to improve the collaboration
If CF equals a certain amount, this would be the
condition for which the trial will have a stable
tempo. We call the achieved delay time over
measure duration as critical CF
Compensation Factor increases linearly with delay
Nima
35
Stereoscopic image quality metric

• Incorporating crosstalk, depth and visual
  discomfort
• Stereoscopic techniques provide a very effective
  depth perception sensation because of using the
  binocular cues. The principle behind is of
  displaying two slightly different perspective
  views for left and right eyes.
• Crosstalk is probably one of the most annoying
  distortion in visualization stage by using
  stereoscopic techniques.

• We propose a stereoscopic image quality metric
  which defines explicitly the positive and
  negative depth information in the crosstalk
  artefact psychometric function
• Original psychometric function
  fracM-m1e-( abx)
• Modified psychometric function
  fracM-m1e-( abxy) cy
• Where x, y is the objective measurement of the
  crosstalk and depth stimulus intensity
  separately. Parameter a, b, c control the curve
  location and steepness. m is the minimum on the
  grading scale and M is the maximum.

Liyuan
36
Stereoscopic image quality
Test matrials Original Psychometric Modified Psychometric
Outdoor 0.940 0.977
Ballet 0.729 0.889
Adile 0.703 0.89
Jungle 0.627 0.82
Train data Test data Original Psychometric Modified Psychometric
outdoor ballet, adile, jungle 0.666 0.778
ballet outdoor, adile, jungle 0.703 0.785
adile outdoor, ballet, jungle 0.735 0.801
jungle outdoor, ballet, adile 0.738 0.807
outdoor, ballet adile, jungle 0.653 0.805
adile, jungle outdoor, ballet 0.812 0.843
ballet, jungle outdoor, adile 0.753 0.800
outdoor, adile ballet, jungle 0.676 0.818
Liyuan
37
QoE in Internet television
- Which network parameters can be extracted and
quantized? - How can these parameters be related
to the QoE? - How to effectively monitor the last
mile (ISP access network)? - How can we handle
various user equipment?
Mathias
38
NTNU Recruitment Game

• Application of Serious Gaming
• First Person Game on a Virtual Campus (main
  building)
• Will be used as a recruitment tool for high
  school students in Norway
• Contains presentations of the different studies
  at IME
• The goal is to show students what NTNU looks like
  and what you are able to make after 5 years of
  study

Håvard
39
Game quality parameters
Petter
40
Research plans
41
Computational quality measurement for IP-based
audio

• Computational models for audio and multimedia
  quality assessment for a) IP-based audio, b)
  Videoconferencing, IPTV.
• Quantification of various impairments on audio of
  video quality
• Quantification of parameters for various audio
  and video codecs, packet loss rates, packet loss
  concealment algorithms, etc.
• Subjective studies to investigate (to confirm)
  assumptions about simultaneous effects of
  multiple impairment factors
• Service and business models
• Continue working on economical aspects of
  end-to-end Quality of Services provisioning in
  the Internet (SLAs, inter-provider revenue
  distribution, pricing strategies, etc.)
• Risk assessment methods for IP-networking
• Business models for new services provisioning (3D
  media applications, multimedia, quality effects)

Eugene
42
3D media Internet

• 3D MEDIA related
• Develop methods for AV subjective assessments
• Determine and measure AV quality features and
  models
• Contribute to standardization
• PERCEVAL related
• Experimental design for measuring cross-modal
  effects
• Focus on low-attention and long-term quality
  assessment
• AV quality objective metric design and validation

Ulrich
43
UEP FEC

• Rate control allowing delivery of partly
  corrupted packets
• More simulating results with more realistic video
  sequences and larger network topologies
• Analysis of perceptual significance of different
  data elements (media units)
• Essential in unequal erasure protection schemes
  (UEP FEC, retransmissions)
• Aim is to improve the methods for classifying
  data elements
• Audiovisual quality assessment
• Results to be used for optimizing the trade-off
  between audio and video distortion, as well as
  source and channel distortion
• Subjective metrics to be turned into objective
  metrics

Jari
44
Perceived audio-visual quality assessment for
future media

• Perceived quality assessment for multimodal media
  with complex scenes (HDTV, UHDTV, NHX, 3D, etc.)
• Prepare appropriate media scenarios and dataset,
  design subjective quality methodologies.
• Cognitive model for joint audio-visual quality
  assessment and corresponding objective metrics.
• Spatial and temporal attention analysis, semantic
  multimedia content analysis and the relationship
  with the perceptual quality assessment.
• Integrated platform for quality of experience
  (QoE) certificate
• Define certificate mechanism for QoE methods for
  future media.

Junyong
45
Virtual worlds

• In the new age of virtual reality and mixed
  environment, users become part of the synthetic
  environment to generate new forms of experiences.
  How can we effectively maximize our perception of
  quality on these kinds of applications?
• Synthetic or 3D contents are less expensive than
  natural representations because they are mostly
  pre-defined and can easily be computed in
  real-time. In real-time or live broadcast (e.g.
  news event, weather forecast, live sports) some
  regions are not so salient and therefore possible
  to substitute or combine with synthetic contents.
  How can we effectively combine natural and
  synthetic representations without sacrificing
  quality perception.

Wendy ann
46
Stereoscopic visual quality

• How does the artefact of coding or communication
  on different part of the data (left vs right
  multiview, or color vs depth rendering) affect
  the overall stereoscopic quality
• Same artefect on the different position of left
  and right multiview, or on the color and depth.
  How it affects the people.
• How about the artefacts on the different depth?
  Is there any difference? It seems that people
  will focus more on the foreground(less depth)
  than background(more depth).

Liyuan
47
Multimodal Perceptual Quality Assessment of High
Definition Content

• Codec performance for digital cinema application
• Conducting subjective quality assessment using
  currently standardized methodology in DCI
  specified Digital Cinema environment
• Developing subjective assessment methodology that
  take into account multimodal factor
• Developing objective model that take into account
  multimodal factor

48
Extra slides

• Backups

49
Computational quality measurement for IP-based
audio (1)

• Quality measurement techniques
• Subjective (using human subjects)
• Objective based on comparison with original
  non-degraded signal
• Objective computational models (when no original
  signal is available for comparision)
• Focus on objective computational quality
  assessment for IP-based audio
• Previous work computational quality model for
  narrowband (up to 4 kHz) and wideband (up to 7
  kHz) speech
• Work on a similar model for IP-based audio in
  general

Eugene
50
Computational quality measurement for IP-based
audio (2)

• 100-point scale is used for quality measurement
  and impairment quantification
• Impairments are percepually additive
• Input quantified qualtiy impairments
• Output uni-directional quality estimate should
  be mapped to a subjective quality measurement
  scale
• Impairments codec, bit rate, packet loss rate,
  packet loss concealment, delay (for interactive
  applications), content dependency,noise, etc.

Eugene
51
Computational quality measurement for IP-based
audio (3)

• Current work
• Start with quantification of encoding and packet
  loss impairments Q 100 function (codec,
  bitrate) function (loss rate, packet loss
  concealment)
• Use software implementation of the objective
  accurate PEAQ algorithm, which is based on
  comparison of degraded and original versions of a
  signal
• Qualitify the impairments using experimental PEAQ
  measurement results converting data to the
  perceptually additive 100-point scale.

Eugene
52
Some pics

• Encoding impairment quantification

Loss impairment quantification
Eugene
53
Future work (2)

• We focus more on the talking person and subtitle
  when we are watching TV, but it is difficulty to
  know who is talking by the image analysis or
  speech processing. However, if it is 3DTV
  together with 3D audio, we can use the 3D audio
  to know who is talking now in which position,
  then using this position, we can further know the
  corresponding pixel position in the image, so we
  can differentiate the quality for ROI for the
  better perceived quality

54
Future work (3)

• If people feel the depth, then it will keep on.
  However, if he stopped a while, it need time for
  him to feel the depth again. So how the
  stereoscopic affect the people's eye focus. I
  think it will in certain degree make the people
  more stressed since he can not move the eyes as
  frequently as he wants.
• Besides, if there is depth, maybe people will
  focus on the different things from the pure 2D
  image.