A long-lived home-grown Linux-based low-cost leading mass-market service Pierre AUBERT

1
A long-lived home-grown Linux-based low-cost
leading mass-market servicePierre AUBERT Eric
OLIVERI FranceTelecom/Home Communication
Services/Broadband Internet
Wanadoo Portals
2
We talk about mass market net computer
engineering

• the 1 in France
• a simple web page
• the Big Picture
• pure Open source Linux PC hardware
• some thoughts about the past and the future
• works well, costs little ? delivers more from
  less

3
The 1 portals in France

• wanadoo.fr voila.fr MMGs IM/IRC
• 65 reach 13 M unique visitors
• 80 M pages to 4.2 M customers daily
• Every days peaks
• 150 000 simultaneous users
• 1 Gb/s Internet content
• 5 000 new users/mn
• 100 000 web pages/mn
• transport conn/s, forks/s, SELECTs/s

4
We talk about mass market net computer
engineering

• the 1 in France
• a simple web page
• the Big Picture
• pure Open source Linux PC hardware
• some thoughts about the past and the future
• works well, costs little ? delivers more from
  less

5
A simple web page
6
, but
7
We talk about mass market net computer
engineering

• the 1 in France
• a simple web page
• the Big Picture
• pure Open source Linux PC hardware
• some thoughts about the past and the future
• works well, costs little ? delivers more from
  less

well, mostly
8
Not only a web server

• Front office
• Network Load Balancers Firewalls DNS
• Web servers Apache/Mathopd/TUX, mostly PHP
• Databases mySQL, postgreSQL
• Back office
• Publishing Content management
• Audience collection
• Databases
• Supervision, backups, etc.
• sometimes open source/Linux/PC hardware
• most often non-open source/non-Linux/non-PC
  hardware

9
content management
audiencecollection
10
3 ways to deliver Web service

• Apache for dynamic content
• Mostly PHP, some C, some SSI
• 20 M pages for 2.5 M users daily
• 12 servers Bi-CPU Intel, type 1 60 Mb/s
• Mathopd for static content
• Maximize browser cache effect
• 2000 HTTP/second
• 6 servers 1-CPU Intel, type 0 25 Mb/s
• TUX for redirection service
• Built-in Linux kernel
• 2500 HTTP/second 2 Mb/s
• 3 servers (1 is enough) 1-CPU Intel, type -1
  )

11
Some other services

• Cookies (identity, tracking)
• 1250 HTTP/sec
• 4 servers 1-CPU Intel type 0 10 Mb/s
• Databases authentication
• 1000 reads/mn 150 simultaneous connections
• 1 server 1-CPU 1G RAM, type 1
• Databases search engine crawler
• 500 M docs 4 Tb 100k writes/mn
• 12 servers 2-CPU 5x72Gb each, type 1
• WAN-LAN
• DNS 4 servers 1-CPU Intel type 0 Zebra
  (BGP/OSPF)
• Firewall load-balance appliance 1-CPU Intel,
  type -1
• Tricks lingerd, nscd, multicast
• tuning 20 000 sockets per IRC server

12
Supervision etc.

• 60 parameters per server on each of 1000
  servers
• Net-SNMP, big brother/big sister, Perl
• ad hoc monitoring ( 30) 1 h to develop and
  deploy
• 20 servers 1-CPU Intel, type 1
• At-the-fly graph generation 50 000 parameters
• 300 samples/s 3 servers Bi-CPU Intel, type 1
• RRD-Tool SNMP managers (Perl) Apache
• Env Network from HWCPU to application
  internals
• Backups, storage
• Amandas self-planning, no routine restore ?
  1/10th FTE
• 2 servers P2-400MHz 2,4To (2/GO/y) cheaper
  than tape (6/GO/y)
• backs up daily 300 servers

13
We talk about mass market net computer
engineering

• the 1 in France
• a simple web page
• the Big Picture
• pure Open source Linux PC hardware
• some thoughts about the past and the future
• works well, costs little ? delivers more from
  less
• well, mostly

14
Lessons from the past

• Choose what fits best
• 1996 from FreeBSD to Solaris
• 2003 from Solaris to Linux
• voila.fr launched 1998 Apache/PHP/Linux
• Service first
• system administrators are front-line
• diagnose and fix fast (or alternate
  service/systems)
• end-user satisfaction systems health
• accounting transparency (CAPEX OPEX per
  server/day)

15
Lessons from the past

• PC hardware is a key factor
• Failures are facts not problems
•  Small is beautiful  cost of redundancy is
  1/N
• Competitive mass market sourcing
• PC operating system is hw-independent
• Reuse assemble components
• Open source OS is a key factor
• Best use of hardware resources
• Flexible kernels Excellent network support
• Application level
• Open source platforms Internet applications
• Interoperability is in DNA

16
Lessons from the past

• Design skills
• Not a sequential  product  process
• Expert staff allow short long term savings
• Efficiency
• Low-range (90) 3 to 5 k/year
• High-end PC servers cost 5 times more
• Mid-range Unix servers cost 20 times more
• Numerous distributed small systems
• scale at front office
• keep back office infra stable
• distribute bandwidth, IO buffers, axis

17
We talk about mass market net computer
engineering

• the 1 in France
• a simple web page
• the Big Picture
• pure Open source Linux PC hardware
• some thoughts about the past and the future
• works well, costs little ? delivers more from
  less
• well, mostly

18
Costs little

• Some figures
• 1 system administrator for 40 servers
• 1 low-range server 3 to 5 k/year OPEX
• Per unique visitor 1 /year
• Per 1k web pages 0.5
• margin cost 2 c OPEX, 3 c CAPEX
• Cost breakdown OPEX
• 15 shell core
• 15 hardware
• 15 bandwidth
• 5 maintenance/support
• 50 man power

19
Costs little  works  well

• Home page Solaris
• 12 Sun 4-CPU 6 dynamic 6 static content
• 400 k/year OPEX (amort. included)
• Additional CAPEX 10 per 1k daily pages
• Home page Linux
• 18 PC 12-CPU 12 dynamic 6 static content
• 100 k/year OPEX (amort. included)
• Additional CAPEX 3 per 1k daily pages? works
  well

20
Works well

• Voila.fr ranked 1, Wanadoo.fr ranked 3
• () Googles home page always 1 but thats not
  a real portal home page
• Both fastest real portal home pages
• Unavailability mostly due to a sub-service
• freeware expensive fees maintce poor
  support
• Open source proprietary code cant fix
• This sub-service delivers less from more
• )

21
Next Steps

• Convince top management that Open-Source
  solutions can replace proprietary one!
• Most agree on web front office (1-tier)
• Most disagree on 2nd tier Jonas,Jboss /
  Websphere, 9iAS
• All disagree on DB MySQL, PostgreSQL / Oracle,
  DB2, SQLs
• Carefully select an appropriated database for
  your needs
• Deploy Oracle on PC Linux attached disks
  (Instead of Sun Solaris SAN or NAS )

22
Thank you
Some Open Source Oriented Question ?
Pierre.Aubert_at_francetelecom.com Eric.Oliveri_at_franc
etelecom.com