UC San Diego Infrastructure

1
UC San Diego Infrastructure

• CalREN-XD/HPR Workshop 2008
• September 16, 2008

Elazar C. Harel eharel_at_ucsd.edu
2
Agenda

• Funding model
• Production network
• Cyberinfrastructure needs
• Research network
• Green Cyberinfrastructure
• Issues Challenges

3
Funding Model

• Flat fee per communications user
• Unlimited data and video, including domestic
  calls
• Wired and wireless
• Built-in equipment wiring replacement cycles
• Scope does not include bleeding edge
• High reliability (fiber diversity, backup
  generators)
• Improved security, email, directory, helpdesk,
  VPN, VoIP, etc.

4
Fiber Network
Red backbone Blue distribution Green
off-campus
5
Production Network

• 70,000 data ports
• 10gb backbone and 1/10gb connection to every
  building
• 2gb links to the commodity Internet (via CENIC)
• 10gb link to global research networks (via CENIC)
• 230 on-campus, 42 off-campus, and 105 student
  residential buildings
• 22 dark fiber connections to off-campus and
  non-UCSD locations
• 10,467 inter-building fiber strands (2,710 miles)
• 1,100 Routers and Switches
• 2,400 wireless access points, all migrating to
  secure 802.11n
• 3,500 concurrent wireless users, with peaks to
  5,000
• 30,000 unique wireless devices in the last 30
  days
• 67,000 unique IPs over last 30 days

6
Network Growth
7
Cyberinfrastructure Needs

• High-speed network directly to user/instrument
• High-speed peered network to service providers
  (campus and cloud)
• Co-location and/or condo compute clusters
• High-performance, reliable, and redundant massive
  storage
• Knowledgeable support staff
• Funding model that encourages use

8
Cyberinfrastructure Needs Example

• Needed a High performance biological data
  storage, analysis, and dissemination
  cyberinfrastructure that connects
• Genomic and metagenomic sequences
• MicroArrays
• Proteomics
• Cellular pathways
• Federated repositories of multi-scale images
• With Interactive Remote Control of Scientific
  Instruments
• Multi-level Storage and Scalable Computing
• Scalable Laboratory Visualization and Analysis
  Facilities
• High Definition Collaboration Facilities

9
Alternative Endings
More of the Same? CI is an Expense
Change? CI is an Investment
10
More of the Same

• CENIC files for chapter 11 - purchased by Lehman
  Brothers
• Larry Smarr sells OptIPuter parts on eBay
• UC San Diego ranks 437 in U.S. News World
  Report
• Worlds helpdesks staffed by americans
• Worlds toys are Made in the U.S.A
• Climate is warming up no clouds

11
Change

• Jim Dolgonas Larry Smarr win Nobel prize in
  networking
• California ranks 1 in contributions to research
• Unprecedented growth in NSF, NIH, and DOE grants
  to UC
• Minimum of 10Gb connections whenever needed
• High-speed peering with cloud providers
• Funding models that encourage use

12
Research Cyberinfrastructure

• Network
• Storage
• Computing
• Staff support
• Funding model

13
Research Network

• Uncongested high-speed access to researchers
• High-speed access to central storage and
  computing
• Accommodate experimental protocols
• Buildings labs connected as research needs
  arise
• Infrastructure is ready to flex as needs expand
• Reasonable security controls
• Joint funding by interested parties

14
Planned Optical Network Biomedicine
15
Single Lab Example

• Phase 1a, June 2008 completed
• Core equipment _at_ SDSC
• Labs in Leichtag, CMM/W
• Blackboxes

• Phase 1b, Aug/Sept. 2008
• CalIT2
• CMM/E
• Bioengineering
• Salk Institute
• LJ Inst. Allergy Immunology

16
Research Network Costs

• Core Infrastructure The campus funds the cost of
  the core components, including items such as
  backbone WDM gear, fiber links, and the Research
  distribution switch. This cost is between
  500,000 and 700,000.
• Building Each building router and a 10Gb link
  upgrade costs about 80,000. Funding sources
  need to be identified and agreed upon.
• Lab Costs for each lab switch can vary from
  5,000 to 70,000 depending on the labs needs.
  The following switch vendors are currently
  recommended Cisco, Dell, or Force10. These costs
  will be paid by the lab/dept/dean/VC
• Multiple 10Gb research links cost is highly
  variable and will be determined on a case-by-case
  basis when required.
• All equipment needs to be replaced every 3-5
  years.

17
Cyberinfrastructure Model
UCSD Storage
HPC System
Cluster Condo
PetaScale Data Analysis Facility
UC Grid Pilot
OptiPortal
Research Cluster
Digital Collections Manager
Research Instrument
N x 10Gb
18
(No Transcript)
19
Green Cyberinfrastructure

• GreenLight 2M NSF-Funded Project
• Two Sun modular data centers (a.k.a. Blackboxes)
• Sun has shown up to 40 reduction in energy
• Active management of disks, CPUs, etc.
• Measures temperature at 5 spots in 8 racks
• Power utilization in each of the 8 racks

20
Issues Challenges

• Funding model for research CI network
• Changing culture toward more centralization
• Physical space, power, cooling, emission
• Cloud services role in research computing
• Requests for new ports exceed estimates
• Hardware costs not decreasing as in prior years
• Staff recruitment and retention
• Network Security

21
QA