Measuring the Digital Divide with PingER

1
Measuring the Digital Divide with PingER

• Prepared by Les Cottrell, SLAC, for the
• Round Table Developing Countries Access to
  Scientific Knowledge,
• October 23-24, 2003, ICTP Trieste, Italy
• www.slac.stanford.edu/grp/scs/net/talk03/ictp-oct0
  3.ppt

Partially funded by DOE/MICS Field Work Proposal
on Internet End-to-end Performance Monitoring
(IEPM), also supported by IUPAP
2
Outline

• How PingER makes the measurements
• Typical usage benefits
• Set the stage for what values mean
• Application to Digital Divide
• Hows the world doing today, where is it going?
• Comparisons with development indices
• Summary

3
Methodology

• Use ubiquitous ping
• Each 30 minutes from monitoring site to target
• 1 ping to prime caches
• by default send10x100Byte pkts
• 10x1000Byte pkts
• Record loss RTT, ( reorders, duplicates)
• Derive throughput, jitter, unreachability

4
Architecture
WWW
HTTP
Ping
SLAC
Reports Data
FNAL
Archive
Archive
Monitoring
Monitoring
Monitoring
Cache
Monitoring
Remote
1 monitor host remote host pair
Remote
Remote
Remote

• Hierarchical vs. full mesh

5
Countries Monitored
6
Recent additions

• Added remote hosts in Albania, Macedonia,
  Serbia/Montenegro, Belarus, Turkey, Armenia,
  Mexico, Cuba, Azerbaijan, Tajikistan,
  Turkmenistan, Kyrgyzstan, Philippines Namibia
• Contacts
• Working with contacts in Vietnam, and Tunisia
• Working with Iran site to set up monitor host
• Increased hosts monitored from CERN to give
  better European view
• Now monitoring 60 countries from CERN
• 80 from SLAC

7
Countries Monitored
Used to monitor Only 1 host
Need gt 1 host to reduce anomalies

• 80 countries
• 480 sites
• 800 hosts
• 3600 pairs

8
PingER Benefits

• Aimed at end-user (net-admin sophisticated
  user), planners
• Measures analyzes reports round-trip times,
  losses, availability, throughput ...
• Uses ubiquitous ping, no special host, or
  software to install/configure at remote sites
• Low impact on network lt 100bits/s, important for
  many DD sites
• Covers 80 countries (99 of Internet connected
  population)

• Provides quantitative historical (gt 8yrs) and
  near real-time information
• Aggregate by regions, affiliations etc.
• How bad is performance to various regions, rank
  countries?
• Trends who is catching up, falling behind, is
  progress being made?
• Compare vs. economic, development indicators etc.
• Use for trouble shooting setting expectations,
  identify needed upgrades, choosing a provider,
  presenting to policy makers, funding bodies

Monitoring site vs. Remote sites screen shot
9
What do values mean?

• Loss most important single metric
• lt 1 good, (throughput 1/sqrt(loss))
• gt 1 voice over IP slightly annoying
• gt4-6
• non-native language speakers unable to
  communicate,
• video conferencing becomes irritating,
• hard to edit files remotely (characters on screen
  lag keyboarding)
• Significant degradation of TCP performance
• gt10-12 TCP sessions time out, FTP fails, mail
  still works (keeps retrying)
• RTT (Round Trip Time)
• Remote instrument control needs low RTT
• gt500ms significant interactive voice problems,
  typing/echo problems

10
Throughput - meaning

• Derived throughput 1460 Bytes / (RTT
  sqrt(loss))
• 56 - 64 kbps home modem limit, Internet backbone
  1980
• 128 kbps ISDN
• Home DSL / cable modems low limits 200-500kbps
• Internet backbone late 80s 1.54 Mbps
• Internet backbone early 90s 45Mbps
• Internet backbone today 10Gbits, big sites
  155Mbps 1Gbps
• Testbeds/cutting edge end to end TCP throughput
  (Land Speed Record, Guinness Records Book)
• 2001 1Gbps
• 2003 2.4Gbps/s (February) gt 5.6Gbps (October)

• Example of the increases in capacity of a well
  connected site in US

11
Usage Examples
To North America
Ten-155 became operational on December 11.
Smurf Filters installed on NORDUnets US
connection.
Upgrades ping filtering
To Western Europe
Peering problems
12
Usage Examples

• Identify need to upgrade and effects
• BW increase by factor 300
• Multiple sites track
• Xmas summer holiday

• Selecting ISPs for DSL/Cable services for home
  users
• Monitor accessibility of routers etc. from site
• Long term and changes
• Trouble shooting
• Identifying problem reported is probably network
  related
• Identify when it started and if still happening
  or fixed
• Look for patterns
• Step functions
• Periodic behavior, e.g. due to congestion
• Multiple sites with simultaneous problems, e.g.
  common problem link/router
• Provide quantitative information to ISPs

13
Rate Limiting
At any given time, about 5 of monitored hosts
are doing this, most in developing countries.
Recently (August 2003) seen an increase in ping
rate limiting
RTT
Loss
RTT
Loss
boromir.nask.waw.pl
boromir.nask.waw.pl
2 hosts at same site see sudden step-like
increase in loss from lt 1 to 20-30 at similar
time
gollum.nask.waw.pl
gollum.nask.pl
Loss
Loss
RTT
RTT
Another host in Poland sees no problems, i.e.
helps to have another nearby host
www.pol34.pl
www.pol34.pl
Similar effects for Greek (uoa.gr), Bulgarian
(acad.bg), Kazakhstan (president.kz), Moldovan
(asm.md) and Turkish (metud.edu.tr) sites If no
step function or nearby host may not notice, so
also compare synack vs ping Can ping routers
along path to see where onset occurs
14
Digital Divide Regions

• Design regions
• to match well known world regions and
• to have similar connectivity within region
• Then order by derived throughput
• Derived throughput MSS 1460Bytes /
  (RTTsqrt(loss))
• Want to show general behavior variability
  (outliers)
• Developed
• U. S.Canada, JapanTaiwanSingaporeKorea,
  AustraliaNZ, Europe (excl. SE Europe, Russia)
• Developing (Digital Divide)
• Africa, S. America, C. America, C. Asia, China,
  S. Asia, Caucasus, M. East, SE Europe, Russia

Israel has much better connectivity than
neighbors in Mid East so distorts Mid East
results, move to Europe?! Greece is part of
Europe, should it be part of S. E. Europe, choice
varies with time
15
Region Map

• Also have affinity groups, e.g. AMPATH, Silk
  Road, CMS, XIWT and can select multiple groups

16
Current State Aug 03 (throughput)

• Within region performance better
• E.g. CaEDUGOV-NA, Hu-SE Eu, Eu-Eu, Jp-E Asia,
  Au-Au, Ru-RuBaltics
• Africa, Caucasus, Central S. Asia all bad

Acceptable gt 500kbits/s, lt 1000kbits/s
Bad lt 200kbits/s lt DSL Poor gt 200 lt 500kbits/s
Good gt 1000kbits/s
17
(No Transcript)
18
Trends
S.E. Europe, Russia catching up Latin Am., Mid
East, China keeping up India, Africa falling
behind
Derived throughputMSS/(RTTsqrt(loss))
Africa shown for only Uganda seen from
SLAC, since adding new countries with very
different throughputs distorts result
19
Russia

• Russian losses improved by factor 5 in last 2
  years, due to multiple upgrades

• E.g. Upgrade to KEK-BINP link from 128kbps to
  512kbps, May 02 improved from few loss to
  0.1 loss

20
Loss Comparisons with Development (UNDP)
Positive correlation with Human Development or GDP
21
Europe
Derived throughputMSS 1460Bytes /
(RTTsqrt(loss))
Netherlands
Belgium
Turkey
22
Network Readiness IndexHow Ready to Use Modern
ICTs ?
Market
(US)
Environment
Political/Regulatory
(SG)
Infrastructure
(IC)
(US)
Individual Readiness
(FI)
NetworkReadinessIndex
Readiness
Business Readiness
(US)
Govt Readiness
(SG)
(SG)
(FI)
Individual Usage
(KR)
Usage
Business Usage
(DE)
( ) Which Country is First
(FI)
Govt Usage
(FI)
Slide prepared by Harvey Newman, Caltech for
ICFA
From the 2002-2003 Global Information Technology
Report. See http//www.weforum.org
23
Network Readiness

• NRI from Center for International Development,
  Harvard U. http//www.cid.harvard.edu/cr/pdf/gitrr
  2002_ch02.pdf

NRI Top 14 Finland 5.92 US 5.79 Singapore
5.74 Sweden 5.58 Iceland 5.51 Canada
5.44 UK 5.35 Denmark 5.33 Taiwan 5.31 German
y 5.29 Netherlands 5.28 Israel
5.22 Switzerland 5.18 Korea 5.10
AR focus
Internet for all focus

• Using derived throughput MSS1460B / (RTT
  sqrt(loss))
• Fit to exponential is better

24
Challenges

• Effort
• Usually negligible for remote hosts
• Monitoring host lt 1 day to install and
  configure, occasional updates to remote host
  tables and problem response
• Archive host 20 FTE, code stable, could do with
  upgrade, contact monitoring sites whose data is
  inaccessible
• Analysis your decision, usually for long term
  details download use Excel
• Trouble-shooting
• usually re-active, user reports, then look at
  PingER data
• have played with automating alerts, data will/is
  available via web services
• Ping blocking
• Complete block easy to ID, then contact site to
  try and by-pass, can be frustrating
• Partial blocks trickier, compare with synack or
  TCP/ping
• Derived throughputs poor for well connected sites
  (lt0.1 loss)
• Funding to sustain efforts

25
Collaborations Funding

• 35 monitoring sites in 15 countries
• Plan to add ICTP Trieste if funded
• Other projects used toolkit, e.g. XIWT, PPCNG/EDG
  
• SLAC with help from FNAL
• Digital Divide collaboration (MOU) with ICTP,
  Trieste
• eJDS
• looking for a EU grant for eJDS and PingER
• Need funding for coming year
• Working with DoE, NSF, Pew Charitable Foundation
  
• Tasks
• (0.5 FTE) ongoing maintain data collection,
  explain needs, reopen connections, open firewall
  blocks, find replacement hosts, make limited
  special analyses, prepare make presentations,
  respond to questions
• ( 0.5 FTE) extend the code for new environment
  (more countries, more data collections), fix
  known non-critical bugs, improve visualization,
  automate some of reports generated by hand
  today, find new country site contacts, add route
  histories and visualization, automate alarms,
  detect rate limiting earlier, update web site for
  better navigation, add more DD monitoring
  sites/countries, improve code portability,
  understand regions better
• Also looking for small grants for helpers in
  developing countries
• ICFA show importance to policy makers, funding
  agencies, identify sympathetic contacts at
  agencies, get support

26
Futures

• More work on understanding regions
• Better/quicker detection of rate limiting
• Extend deployment, in particular Africa
• Demonstration in coordination with WSIS in Geneva
  Dec 2003

27
Summary

• Valuable light-weight tool for end-to-end
  performance
• Good for trouble-shooting, planning, setting
  expectations
• World wide coverage
• Performance from U.S. is improving all over
• Performance to developed countries are orders of
  magnitude better than to developing countries
• Poorer regions 5-10 years behind
• Poorest regions Africa, Caucasus, Central S.
  Asia
• Some regions are
• catching up (SE Europe, Russia),
• keeping up (Latin America, Mid East, China),
• falling further behind (e.g. India, Africa)

28
More Information

• PingER
• www-iepm.slac.stanford.edu/pinger/
• MonaLisa
• monalisa.cacr.caltech.edu/
• GGF/NMWG
• www-didc.lbl.gov/NMWG/
• ICFA/SCIC Network Monitoring report, Jan03
• www.slac.stanford.edu/xorg/icfa/icfa-net-paper-dec
  02
• Monitoring the Digital Divide, CHEP03 paper
• arxiv.org/ftp/physics/papers/0305/0305016.pdf
• Human Development Index
• www.undp.org/hdr2003/pdf/hdr03_backmatter_2.pdf
• Network Readiness Index
• www.weforum.org/site/homepublic.nsf/Content/Initia
  tivessubhome

29
Africa Getting Started

• Recommendation on Monitoring from Open Round
  Table, Trieste 2002
• Devote resources to monitor in real time the
  connectivity of research educational
  institutions in developing countries
• MOU signed between SLAC eJDS/ICTP
• Use extend SLAC PingER/project for monitoring
  eJDS network of participants
• Find contacts at potential sites
• Dec 02 send emails to eJDS participants,
  ICFA/SCIC representatives
• Explain value to participants, needs for hosts to
  monitor, contacts
• Contact contacts, explain again
• When get host, check it is pingable
• Could be multi-month process
• 75 contacts provided successful hosts
• 25 pings blocked and unable to resolve, emails
  exchange petered out
• Still need contacts for many countries

30
Africa Oct 03

• Hosts in Ife-Ife/Nigeria, Accra/Ghana,
  Kampala/Uganda, Windhoek/Namibia, UCT/ZA,
  Johannesburg/ZA, Mosselbay/ZA
• Carriers
• GH uses UUNET/Satworks, NA uses UUNET/xantic, NG
  uses TELIANET/NewSkies, UG uses
  Level(3)/globalconnex
• ZA varies from site to site UUNET/ALTERNET, CW
  Telecom S. Africa, CAIS telcom S. Africa
• UG, NA, NG, GH use satellites (gt 600ms)
• ZA uses landlines

31
Africa RTT

• Monitored from N. America Europe
• Depends on remote site (not monitoring site)
• Satellite for all except S. Africa
• Ghana problems

32
West Africa Loss

• Ghana very poor performance
• Sudden increase in losses on August 18th
• Not rate limiting according to synack
• Sometimes get down to a few
• Route ESnet-UUNET/ALTER.NET

• Losses appear on last 2 hops in Ghana
• Nigeria better
• Route via TELIANET/newskies

33
Africa Derived Throughput

• S. Africa (UCT) best, followed by Uganda, Nigeria
  and Ghana
• Throughput to Nigeria site home DSL/cable
• Throughput to Ghana site modem dialup
• Uganda site SLAC late 1980s

Derived throughput MSS1460Bytes RTT
sqrt(loss)
34
Summary 1/2

• Beware limited number of countries monitored and
  even then limited number of sites in each
  country. Need more contacts
• Countries with no sites e.g. Algeria, Cameroon,
  Kenya, Malawi, Mozambique, Senegal, Tanzania,
  Tunisia, Zambia
• Countries with only one (possibly anomalous)
  site Uganda, Ghana, Nigeria, Namibia
• Africa poorest performance region
• Factor 30 behind Europe in performance today
• ZA and Uganda better performers
• Best Uganda site, 8 years behind Europe
• Ghana bad, Nigeria poor, similar to connectivity
  to homes in Developed nations
• Overall Africa not catching up
• Ghana, Nigeria falling behind

35
Summary 2/2

• Little uniformity in routes, many carriers
• Unlike S. America Caucasus where AMPATH
  Virtual Silk Highway have improved performance
• Hopefully Africa ONE project will help
• Undersea fiber to link countries of Africa
  together and to one another

36
Extra Slides
37
Visualization

• Keep it simple, enable user to do their own by
  making data available
• Tables
• Time series (www-iepm.slac.stanford.edu/cgi-wrap/p
  ingtable.pl)
• select metric (loss, RTT etc.), time ticks,
  packet size, aggregations from/to, etc.
• Color code numbers, provide sort, drill down to
  graphs, download data (TSV), statistical
  summaries
• Monitoring site vs. Remote sites
  (www-iepm.slac.stanford.edu/cgi-wrap/table.pl)
• Select metric, region aggregations
• Drill down to time series, download data
• Graphs
• Select source(s)/destination(s), metric, time
  window, SQL selects, graph type

38
Publish information

• www.slac.stanford.edu/cgi-wrap/pingtable.pl gt
  tabular reports, also download data
• Data accessible from MonaLisa
• Implementing web services access prototype
• Includes PingER, IEPM-BE, RIPE-tt, I2 E2Epi
  OWAMP
• Use GGF/NMWG schema/profile, e.g.
• path.delay.roundTrip

!/usr/bin/perl use SOAPLite my
characteristic SOAPLite -gt
service(http//www-iepm.slac.stanford.edu/tools/s
oap/wsdl/profile_06.wsdl') -gt
pathDelayOneWay("tt81.ripe.nettt28.ripe.net)
print characteristic-gtNetworkTestTool-gttoolNa
me,"\n" print characteristic-gtNetworkPathDela
yStatistics-gtvalue,"\n"
39
Rate Limiting Moldova
RTT
Loss
lises.asm.md
cni.md
Bulgaria
Moldova
40
(No Transcript)
41
Loss Comparisons with Development (UNDP)
Positive correlation with Human Development or GDP
Weaker with education literacy
42
Europe
NREN Core Network Size (Mbps-km)
2000
Leading
10M
Belgium
2001
Advanced
1M
Netherlands
In transition
100K
Lagging
10K
1K
Turkey
100
Source From slide prepared by Harvey Newman,
presented by David Williams ICFA/SCIC talk on
Serenate report. Data from the TERENA Compendium
Derived throughputMSS/(RTTsqrt(loss))