MCNC-RDI%20ppt%20template%20white%20background

1
Grid Computing in the Industry
North Carolina Statewide Grid
Duke University, Nov 2, 2004
Wolfgang Gentzsch Managing
Director MCNC Grid Computing and
Networking Services This Lecture aims at
complementing your background in distributed
computing technologies and developments,
providing an actual overview of Grid Computing,
and its applications and benefits for Education,
Industry and Economy.
2
My current Background MCNC The Grid
Infrastructure for North Carolina
NC Statewide Grid Initiative
MCNC Enterprise Grid

• MCNC
• Microelectronics Center of North Carolina
• Founded in 1980
• Independent, private, non-profit organization
• Operates NCREN since 1984
• Past Supercomputer Center for universities
• Now Grid Service Provider, offering Video,
• Network, Grid, and Datacenter Services
• 50 employees

NC BioGrid
NCREN North Carolina Research Education Network
3
The Industrys View of a Grid An IT
Utility on a . . . Grid Middleware (the
glue) Managing . . . Networked
Distributed Resources
4
Why Should We Care about Grids ? Its
the next big thing !
Grid technologies advance Science and Education
in that we can do things which havent been
possible before. Grid infrastructure attracts
and enables new businesses and creates new jobs,
especially in today's rural areas. Grids make
us more competitive by better utilizing
resources, bringing results to market faster,
and delivering with higher quality.
5
Grid Benefits Department,
Enterprise, and Global Grids

• Access transparent, remote, secure, wireless
• Sharing enable collaboration over the network
• Failover migrate/restart applications
  automatically
• On Demand get resources, when you need them
• Productivity more work done in shorter time
• Virtualization access compute services, not
  servers
• Heterogeneity platforms, OSs, devices,
  software
• Resource Utilization increase from 20 to 80
• Virtual Organizations build dismantle on the
  fly

6
However, There is Still a Long Way to Go !

• Grids are over-hyped currently, they promise
  much more
• than they can really offer.
• Grid technology is far from mature and complete.
• Grid standards are (mostly) still missing.
• Grids are very complex IT infrastructures.
• Grids bring new challenges sharing resources,
  loosing
• direct control, security, intellectual
  property, legal, social,
• political issues . . .

7
Grid Architecture, Technology, Standards
8
Architectural Requirements

• Availability
• Downtime impact
• Individual jobs
• Maintenance windows
• Scalability
• Growth 1-3-5 years
• Scaling strategy and
• Response to peak loads
• Technology refresh, evolution
• Manageability
• Skill set / workload of SA
• Expected stability
• Code management, software distribution
• Security
• User authentication
• Internet access
• Data Security requirements

• Data Distribution
• Location, volume, refresh
• Security of data
• Usability
• Administrative Skill set and
• Client environment
• Psychological factors important
• Operations Management
• 100s CPUs / SA
• Resources added in large blocks
• Change control is critical

9
The Globus Toolkit and the Open Grid Services
Architecture (OGSA)

• Integration of Grid technologies and Web Services
• OGSA defines a Grid Service
• In terms of WSDL interfaces, defines mechanisms

• creating distributed systems
• lifetime management
• change management

• credential management
• notification

• Notification
• Authorization
• Service Creation
• Service Registry
• Manageability
• Concurrency

10
Global Grid Forum (GGF)

• Community-driven set of working groups that are
  developing standards and best practices for
  distributed computing ("Grids" and
  "Metacomputing") efforts
• Formed in 2001 (merger of US, APAC, Euro efforts)
• Open Grid Services Architecture (OGSA)

11
Enterprise Grid Alliance (EGA)

• Industry-driven consortium to implement standards
  in industry products and make them interoperable
• Founding members Oracle, Sun, Fujitsu,
• Enforce complementary position in pushing GGF
  into the academic corner

12
The Changing Grid Landscape
Developer View Write here Run anywhere
Service Composition
Utility Computing
HPTC Grid
OnDemand
Web Svcs
Compute Power from the wall socket
Standards
Public SOA
Outsourcing/Hosting
Software as a Svc
. . .
HP
Technologies
IBM
IBM
Enabled Business Models
Enterprise IT Mgmt Products
Sun

Market making / Productization
Details see backup slides
13
Cluster Grid Implementation, Example

14
Department Grids
Browser to CGM (Remote Server Setup
Configuration)
Compute Grid Manager
Auto OS Deployment Grid Installation/Mgmt Centrali
zed Server Mgmt
Auto Download of Modules
Solaris Servers
Linux Servers
Workstations (Linux or Solaris)
15
Enterprise Grid Reference Architecture
Browser Access via GEP
SunRay Access
Workstation Access
Optional Control Network (Gbit-E)
Myrinet
Myrinet
Servers, Blades, VIZ
Myrinet
Linux Racks
Grid Manager
Workstations
Sun Fire Link
Data Network (Gbit-E)
NAS/NFS
HA NFS
Simple NFS
Scalable QFS/NFS
16
Grid Enterprise Policy Management
Clear definition and documentation of
priorities and policies is key to Grid resource
management
17
Globus Avaki over multiple local Grids

1. Grid Engine cluster mgmnt within an admin domain
   file system area
2. Globus/Avaki knits together resources, handles
   files, binary management, and high level resource
   selection

Organization B
Organization C
Organization A
Avaki Data Grid
18
Applications for The Grid

• Single-CPU Jobs jobmix, many users, many serial
  applications, suitable for grid (e.g in
  universities and research centers)
• Array Jobs 100s/1000s of jobs, one user, one
  serial application, varying input parameters,
  suitable for grid (e.g. parameter studies in
  Optimization, CAE, Genomics, Finance)
• Massively Parallel Jobs one job, one user, one
  parallel application, no/low communication,
  scalable, fine-tune for grid (time-explicit
  algorithms, film rendering, pattern recognition)
• Parallel Jobs one job, one user, one parallel
  application, high interprocs communication, not
  suitable for distribution over the grid, but for
  parallel system in the grid (time-implicit
  algorithms, direct solvers, large linear algebra
  equation systems)

19
Grid Computing Today in Industry
20
Computing as a Utility gtgt Enhancing The
Grid with a Business Model
What's a Utility?

• On Demand Get a service at your finger tip
• From the Wall Socket Don't care about the
  infrastructure
• Metering Billing Pay as you go, for what you
  use
  
  

Like electricity, water, gas, heat, telephony
21
Customer Challenges
22
Critical Customer Requirements
23
Who Uses Department and Enterprise Grids?

• Life Sciences
• Startup and cost efficient
• Custom research or limited use applications
• Multi-day application runs (BLAST)
• Exponential Combinations
• Limited administrative staff
• Complementary techniques

• Electronic Design
• Time to Market
• Fastest platforms, largest Grids
• License Management
• Well established application suite
• Large legacy investment
• Platform Ownership issues

• Financial Services
• Market simulations
• Time IS Money
• Proprietary applications
• Multiple Platforms
• Multiple scenario execution
• Need instant results analysis tools

• High Performance Computing
• Parallel Reservoir Simulations
• Geophysical Ray Tracing
• Custom in-house codes
• Large scale, multi-platform execution

24
How Real is all this Stuff?
direction of technology adaptation
vendorinterest
researchactivity
department enterprise
global
25
Company Example Sun Microsystems
26
Sun Microsystems Grid Computing

• Slogan The network is the computer
• Strategy N1, manage N computers as 1
• Acquisitions
• Gridware gt Sun Grid Engine, Distributed
  Resource Manager
• Pirus gt Dynamic Storage Provisioning
• Terraspring gt Automatic System Provisioning
• Centerrun gt Automatic Application Provisioning

27
Managing Grids What we did for the big
SMPs......we are now doing for the
Network

• Resource Virtualization
• Domains, Zones
• Interdomain resource mgmt
• Routing
• Soft cabling within the box

Solaris
N1
28
Virtualization of Resources
Service1
Vertical
Horizontal
StorageNetwork
Service2
Service3
Compute
Storage
Compute Data Grid
29
Services sharing resources
Services
Services
Services
2
3
1
VirtualNetwork
VirtualStore
VirtualCompute
N1 managing services, not servers
30
N1 Provisions Grid Computing
Web Server
App Svr
App Svr
DB
Web Server
Web Server
App Svr
Web Server
DB
31
The N1 Effect on EfficiencyRadical improvement
in costs uptime
80
N1

• Sys Utilization

6-15
500

• Server/Admin

15-30
Costs
100TB

• Terabytes/DBA

1TB
500

• Ports/Admin

50-100
99.999

• Availability

99.9
Uptime
Hours

• Time to Deploy

Weeks
32
Grid Computing Use Cases,
Deployments, Best Practices
33
Example White Rose Grid in England

• Leeds, York Sheffield Universities
• Deliver stable, well-managed HPC resources
  supporting multi-disiplinary research
• Deliver a Metropolitan Grid across the
  Universities

34
WRG Hardware
Maxima
Snowdon
Pascali
Titania
Maxima
35
WRG Architecture Overview
portal
GT2.0
White Rose Grid
GT2.0
GT2.0
GT2.0
GT2.0
GEEE
GEEE
GEEE
GEEE
Solaris
Solaris
Solaris
Linux
Titania
Maxima
Snowdon
Pascali
36
WRG Key Components

• Globus Toolkit 2.4
• Provides a secure means for inter-campus actions
• Transferring jobs
• Moving data
• Gathering information about resources
• Grid Engine Enterprise Edition
• Manages the campus grid compute resources
• Delivers a single interface for a heterogeneous
  grid
• Guarantees a share of campus resource for grid
  and local users

37
WRG Key Components

• Grid Portal Development Kit
• Provides a portal interface into Globus Toolkit
• Transferring jobs
• Moving data
• Gathering information about resources
• MyProxy
• MyProxy provides a server with client-side
  utilities to store and retrieve delegated X.509
  credentials via the Grid Security Infrastructure
  (GSI).

38
Grid Engine Enterprise Edition, Share Policies
GT2.0
GT2.0
GEEE
GEEE
Solaris
Solaris
GT2.0
GT2.0
GEEE
GEEE
Solaris
Linux
White Rose Grid
39
Example Grid Service Provider MCNC The Grid
Infrastructure for North Carolina
NC Statewide Grid Initiative
MCNC Enterprise Grid

• MCNC
• Microelectronics Center of North Carolina
• Founded in 1980
• Independent, private, non-profit organization
• Operates NCREN since 1984
• Past Supercomputer Center for universities
• Now Grid Service Provider, offering Video,
• Network, Grid, and Datacenter Services
• 50 employees

NC BioGrid
NCREN North Carolina Research Education Network
40
GCNS Grid Computing and Networking Services
Mission
Advance education, innovation and economic
development throughout North Carolina
by delivering next generation information
technology that enables the academic,
research, government and business
communities to discover, create, share and
apply knowledge.
41
The GridThe next IT Infrastructure of North
Carolina

• Proving ground for Grid
• Successful prototype apps
• Catalyst for collaboration
• International recognition

NC BioGrid
2002 2003 2004 2005
42
The Grid The next IT Infrastructure of North
Carolina

• Cluster and SMP resources
• Research platform for GTEC
• Core component in NCGrid
• Revenue generation

MCNC Enterprise Grid
NC BioGrid
2002 2003 2004 2005
43
The Grid The next IT Infrastructure of North
Carolina

• State-wide partnership
• Leverage lessons learned
• Grid education training resource
• Enable first mover applications

NC Grid Initiative
MCNC Enterprise Grid
NC BioGrid
2002 2003 2004 2005
44
MCNC 3-Year GCNS Grid Roadmap

• Grid Service Provider
• Network, Computing, Data, Video
• Partner with SC Sites
• Build GTEC Service Portfolio
• Start Grid Consulting
• Annual GSP Workshop

Easy Access Training, Web Courses
Access Grid Node Grid Appliance
QuadA
Awareness Creation Conferences, Workshops,
PR MCNC Enterprise Grid NC BioGrid NC Statewide
Grid
CY03
CY04
CY05
CY06
45
North Carolinas Foundation for Grid NCREN
4-7 MCNC-owned Clusters distributed throughout
the state Locations still under evaluation
Cisco
EPA
Existing Blend of owned and leased fiber and
circuits moving toward resilient rings powered by
Cisco routers Planned Strong focus on owned and
leased fiber, Lambda, and few circuits, in
resilient rings powered by Cisco routers and Wave
Division Multiplexers
46
GCNS Enterprise Grid
Avaki Data Grid
Global Grid Resource DB (GIIS)
Users
Campus Grids
Grid Gatekeeper / Interactive Nodes
Data Grid Access Servers (8 total, i.e. 1 per 8
nodes)
LSF Master Job Scheduler
32-CPU SGI Altix Linux SMP Server
128-CPU IBM Linux Cluster (64 nodes)
8-TB Storage
47
Network, Grid and Data Center Services
GTEC, NLR, ANR and other Innovation Initiatives
DEPLOYMENT
State-wide Grid Services
Enterprise Grid Services
Value-add Information Systems Services
Self-serve Data Center Services
Information Security Services
Data Archival Services
Hosting Infrastructure
Grid Computing
Information Assurance
DATA CENTER
NCREN
48
The NC State-Wide Grid Roadmap

• 06/04 Do-Grid-Yourself Workshop
• 07/04 Phase 1, Awareness Creation
• 10/04 06/05 Deliver/Connect Grid Appliance
• Clusters to University Partners
  in NC
• 10/04 Develop Do-Grid-Yourself Training
  Course
• 10/04 Start QuadA Project Access, Accounting,
  
• Authentication, Authorization
• 12/04 Start Deliver Grid Training to Partner
  Univs
• 01/05 06/05 Work with Grid Users to Port Apps
• 03/05 Build Access Grid Node
• 06/05 1st NC Statewide Grid Workshop

49
Grid Projects Driving Grid Adoption

MCNC Grid SP

Grid ColLab

Kids Grids

Gaming Grid

School Grids

Grid-Info Grid

Download-The-Grid
AAAA
Project
Grid Training Courses

Startup Grid
Grid Appliance
Grid
Portal
GGF EGA NC
Statewide Grid MCNC
Enterprise Grid NC BioGrid MCNC
Supercomputing
2000 20004
2008
50
Grid Vision
51
Our Vision The Three Waves of Grid Computing
The Research Wave The Industry Wave
The Consumer Wave Technology, Prototypes
Grid-Enabled Products
Commodity
Virtual Organizations
Enterprise Solutions
IT Utility Standards

Interoperability
Integration GGF, IETF, OASIS
GGF, EGA, IETF, OASIS
Legal, Ethical, Political
Orgs GCNS Awareness Creation
GCNS Easy Access
GCNS Grid Service Provider
52

• Grids Today - Grids in 3 - 5 Years

• Focus on Research - Focus on RD and business
• Compute-oriented - Petaflops linked w/
  Petabytes
• Proprietary interfaces - Standards GGF, OGSA,
  DRMAA
• Mental Firewall - Security, policies,
  identity
• Difficult to build - Standards, services,
  solutions
• Difficult to manage - Sun N1, IBM Autonomic, HP
  DC
• Difficult to use - Grid Portals transparent,
  remote, secure
• Many technologies - Globus Toolkit 3.x

53
Security
Scalability
Affordability
Manageability
NetworkComputer
Interoperability
Availability
Performance
MobilityE2E
Services/Solutions/Partners
54
The Next 10 YearsThe Grid,the Operating
System of the next Internet

• High Capacity rich in resources
• High Capability rich in options
• Persistent stable infrastructure
  knowledgeable workforce
• Evolutionary able to adapt to new technologies
  users
• Usable accessible, robust, and easy-to-use
• Scalable growth must be part of the design
• Flexible able to support new applications
• Fault-tolerant resilient to changes and errors

Grid Computing Making the Global Infrastructure
a RealityBerman, Fox, Hey, 2003
55
FinallyAnyone, anywhere, anytime, any device,
any data,connected to The Grid

• Integration of new devices, data and information
  sources
• Cell phones, PDAs, smart sensors, sensor arrays,
  health
• monitors
• Devices embedded in cars, engines, roads,
  bridges, clothes,...
• Huge amount of data for real-time analysis
• Policies, grid economy, to maintain stability
  and efficiency
• Organizational and societal structures, to
  bridge political
• and social boundaries

56
Vision Grid, the
Time Machine of 21st Century

• 19th Century Time Machine Steam Engine
• Enabled mass production, reduced time to market
• 20th Century Time Machine Combustion Engine
• Much faster from Munich to Dortmund
• 21st Century Time Machine Grid Engine
• Safer, better, new, more products services
  research
• in much shorter time than today !

Grid will have great impact on Science, Business,
and Society ( See examples in the
following presentations )
57
Time Machines
The Innovation Engine
Thank You !

wgentzsch_at_mcnc.org http//www.mcnc.org