Adaptable bandwidth planning for enhanced QoS support in usercentric broadband architectures Dr' Ilk

1
Adaptable bandwidth planning for enhanced QoS
support in user-centric broadband architectures
Dr. Ilka Milouchewa (FHG) Dirk Hetzer
(T-Systems, MB)
2
Bandwidth on Demand Planning.Topics

• Background
• Bandwidth planning based on reinforcement
  learning
• Operations research for bandwidth scheduling
• Combining reinforcement learning with scheduling
• Q-learning
• Informed learning
• Relational learning
• QORE system for adaptable bandwidth planning
• Integration of bandwidth planning in research
  projects and Telecom networks

3
Bandwidth on Demand Planning.Background (1)

• need for planning of resources in new on-demand
  services (rich-media, IPTV, VoD, gaming) on all
  IP core, replacing ISDN, ATM etc.
• high bandwidth demands, efficiency achievable by
  bandwidth planning and resource reservation
• Scenario Planning based resource reservation in
  advance at aggregation points (DSLAM, BB-PoP)

4
Bandwidth on Demand Planning.Background (2)
Scenario Planning is based on resource
reservation in advance at access
routers considering learning of performance data

• Need for capacity planning based on ADVANCE
  resource reservations for different kinds of QoS
  based applications and best effort traffic
• Total resources (bandwidth) are always
  restricted, therefore advance reservation could
  be used for planning and enhanced utilization
• Learning of performance to predict resource
  requirements
• Optimized resource allocations adapting advance
  resource requests to predicted requests
• Find compromise between different resource
  requests and enhance QoS for all kinds of
  applications (traffic)

5
Convergence Triple Play.Challanging the way of
Service Creation.? Content delivery on demand
Access evolution path
Triple Play evolution path

• h

Telephony (F/M)
Others
Radio Access

• Microwave
• Satellite
• Laser/Optical
• WiMax

DSL-/ 3G-Internet VoIP TV VoD

• GSM
• GPRS
• EDGE
• 3G W-CDMA
• HSDPA
• Broadband wireless access

UNIFIED USER EXPERIENCE
TriplePlay
DSL-Internet VoIP IPTV Smart Home
Fixed Line Access
Cable Internet Communication

• PSTN/ISDN
• FTTP
• Cable
• Broadband wireline access

Broadband Internet
Television
Communications Services evolution path
Rich Media
SMS
MMS
Voice Text Picture Sound
Voice Text Picture Sound Video
Voice
Voice Text
6
User-centric approach for bandwidth
planning.Scenarios and applications Planning for
Triple Play.Example 1 TV-based Telecom Services.

• Integrates telephony service with a users
  television
• Supports the delivery of telephony services in
  conjunction with cable, DSL, and IP-based video
  services
• PSTN, mobile,or VoIP phones

• TV Calling Name
• User directed routing
• Click-to-dial
• Participation TV - Voting - Gaming
  - Shopping
• Messaging
• - Picture Sharing
• - Multimedia Message Display
• - Content Services
• - Voice Mail Screening

Video Distribution Network
Bill Smith 732-699-3232
PSTN /VoIP
7
User-centric approach for bandwidth
planning.Scenarios and applications Planning for
Triple Play.Example 2 TV-Mobile Convergent
Participation Service (Blogging).

• Participation TV as a Mobile Communication/
  Broadcast Communication convergence case.
• Participation TV denotes the integration of user
  feedback/interaction into TV-formats (such as
  game shows).
• Traditionally very limited e.g. by calling into
  the game, sending SMS,

• MMS diaries (blogging) uploads from personal
  pictures, texts via MMS
• Virtual pubs/discos tour (spying on events)
• Virtual classroom
• Contests (e.g. Best amateur news report of the
  day/month)
• MMS/iTV chat
• Other services (personalized weather reports,
  group contests, alerts,etc.)

8
Bandwidth on Demand Planning.Bandwidth planning
based on reinforcement learning (1)
Different learning approaches to optimize and
plan bandwidth
Stochastic automata - a stochastic policy
by associating a probability with each action, so
that actions are chosen at random according to
their probabilities - the policy does not
take into consideration the current state of the
system, when choosing an action.
Reinforcement learning benefits -gt Rewards from
interactions (action) with environment at each
state (dynamical learning) -gt Adaptive control
considering states and actions (change of
bandwidth scheduling dependent on the performance
feedback)
9
Bandwidth on Demand Planning.Bandwidth planning
based on reinforcement learning (2)
Scenarios for usage of reinforcement learning for
bandwidth planning

• Resource usage is derived in
• interaction with environment
• - Using reinforcements, prediction
• is done for the period T

Learning performance (delay, throughput) and
predict resource needs -gt In case of HDTV
Allocate resource in advance for different
traffic classes -gt Multimedia Conference -gt
GRID Transfers
- Optimal sharing of resources in advance for
traffic classes considering reinforcements for
each traffic

• Different strategies for advance
• allocation of resources for
• on-demand traffic considering
• predicted and reinforcements
• of actual resource usage of traffic classes

Adaptation of advance reservations to resource
needs of traffic classes -gt On-demand service
10
Bandwidth on Demand Planning.Usage of QORE for
bandwidth planning (1)

• Reservation in advance parameters depend on
  applications and users

11
Bandwidth on Demand Planning.Usage of QORE for
bandwidth planning (2)

• Problem Reinforcement learning based on rewards
  from environment (delay of best effort traffic)
• --gtfinding optimal bandwidth allocation (optimal
  schedule) which satisfies resource requirements
  in advance of QoS based applications and
  enhancing QoS (delay) of best effort traffic

Reinforcement learning problems for bandwidth
planning -gt User interface for bandwidth
allocation in advance considering traffic
classes -gt Reinforcements periodical performance
measurements for best effort traffic -gt Value
function cumulative rewards evaluating the
schedule for the planning period -gt Interaction
with Bandwidth Broker
12
Combining reinforcement learning methods and
scheduling (1) Obtaining optimal bandwidth
schedules for proactive and reactive
planningusing combination of operation research
reinforcement learning methods
Pattern based scheduling
Conflict-free schedule with minimum duration
Partial displacement scheduling
13
Combining reinforcement learning methods and
scheduling (2).Simple Q-Learning - Approach

• A model-free RL approach combining conflict-free
  scheduling with minimum duration
• Bandwidth schedules are characterized with
  Q-values based on rewards and value function
  (cumulative sum of rewards evaluating exceeded
  end-to-end delay threshold for planning periods)
• Selection of bandwidth schedule for evaluation
• Random selection, e-greedy (the best Q-value)
• Q-value of bandwidth schedule updated every time
  at the end of planning period, for instance a
  day
• Update based on Learning rate (Recency Weighted
  Average..)
• Pure trial and error
• for practical usage not efficient, because
  predictions of resource usage of applications
  are not considered

14
Combining reinforcement learning methods and
scheduling (3) .Simple Q-Learning - Example
15
Combining reinforcement learning methods and
scheduling (9).Relational Q-Learning - Modified
schedule based on patterns

• When pattern is detected, the planned allocation
  is rescheduled
• Using rewards from end-to-end delay measurements
• for daily planning period

16
Bandwidth on Demand Planning.Usage of QORE for
bandwidth planning (1)

• Integration of patterns in scheduling algorithms
  based on reinforcement learning

Patterns - abstractions for structures
describing behaviour of performance parameters
for network connections - extracted from
monitoring data bases
Bandwidth planning focus





End

Routers

systems





Network connections

• Different kinds of patterns considered for
  bandwidth planning
• Outliers
• Threshold overload patterns
• Patterns describing traffic, QoS parameter
  behavior, routing events and anomalies of network
  connections

QoS

Traffic

Immediate
Routing
Failure


par
ameter


volum
e

resource
event
s

events








request








Patterns describing normaland abnormal
behavior



of multivariate time series data

17
Bandwidth on Demand Planning.Usage of QORE for
bandwidth planning (2)

• QORE components and interaction with monitoring
  and planning database

QORE automated tool for adaptable QoS-oriented
proactive and reactive bandwidth planning

• Components using common knowledge database for
  monitoring planning
• - User interface for advance reservation      
• - QoS parameter monitoring
• Scheduling algorithms
• Resource constraints simulator
• - Effective bandwidth estimation
• - Pattern analyser
• - Visual data mining for bandwidth planning

18
Scenario for bandwidth planning in converged
fixed / mobile environment Selection of optimal
access network for content delivery
19
Summary.Automated adaptable bandwidth planning
in Internet based on reinforcement learning,
QoS parameter patterns and scheduling heuristics

• Novel management strategies for an automated
  proactive and reactive bandwidth planning in
  Internet using QoS monitoring data (QORE system)
• Integration of data mining technologies using
  patterns for bandwidth planning
• Bridging a gap in operations research techniques
  for bandwidth planning integrating dynamic
  learning of QoS parameters (pattern detection)
• Integrated architecture based on reinforcement
  learning, operations research and data mining
  for bandwidth planning

20
Bandwidth on Demand Planning.Outlook ongoing
work in EU projects

• NETQOS
• Application specific bandwidth planning concepts
• Policy based bandwidth management and planning
• DAIDALOS
• Enhanced QoS brokerage architectures based on
  resource reservation in advance
• Advance resource reservation for mobile QoS based
  services
• Practical integration of the adaptable bandwidth
  planning of QORE in network management systems