Large Scale Distributed Computing

1
Large Scale Distributed Computing

• B. Ramamurthy

2
Evolution of Internet Computing
scale
Parallel HPC
Semantic discovery
??????
Automate (discovery)
Discover (intelligence)
Transact
Integrate
Interact
Inform
Publish
time
3
Computing Models

• High performance computing Ex grid computing
• Parallel computing
• Cloud computing
• Internet scale computing

4
Beginnings of The Grid

• Beginnings of the grid in Search for Extra
  Terrestrial Intelligence (seti_at_home project)
• http//planetary.org/html/UPDATES/seti/index.html
• The Wow signal http//www.planetary.org/explore/t
  opics/seti/seti_history_10.html

5
IT Issues

• Modernizing legacy systems
• Web-enabling business services
• Automating business processes
• Integration within organization
• Data and application integration
• Collaboration with outside organizations
• Security, accountability, availability,
  reliability,

6
Grid Organizations

• Global Grid Forum (GGF)
• www.globalgridforum.org
• a community-initiated forum of thousands of
  individuals from industry and research leading
  the global standardization effort for grid
  computing. 
• The Globus Alliance
• www.globus.org
• conducts research and development to create
  fundamental technologies behind the "Grid," which
  lets people share computing power, databases, and
  other on-line tools securely across corporate,
  institutional, and geographic boundaries without
  sacrificing local autonomy. 
• Enterprise Grid Alliance (EGA)
• http//www.gridalliance.org/en/index.asp
• EGA working groups have been proposed to address
  (1) Reference model (2) Component provisioning
  (3) Data provisioning (4) Utility accounting (5)
  Grid security.
• GGF EGA ? OGF
• Open Science Grid (OSG) ?

7
Grid Technology

• Enabling technology.
• Natural evolution of distributed systems and the
  Internet.
• Middleware supporting network of systems to
  facilitate sharing, standardization and openness.
• Infrastructure and application model dealing with
  sharing of compute cycles, data, storage and
  other resources.
• Promoted by NSF through its Network Middleware
  Initiative (NMI).
• Publicized by prominent industries as on-demand
  computing, utility computing, etc.
• Move towards delivering computing to masses
  similar to other utilities (electricity and voice
  communication).

8
Types of Grid

• Batch-oriented
• High performance applications
• High Throughput applications
• Example Condor

• Service-Oriented
• View all the resources and functions as services.
• Build application models around services.
• Anatomy of the grid
• Physiology of the grid
• It is this genre of grid that will move the grid
  technology towards business applications.
• Example Globus

9
Service-oriented Standards

• Open Grid Services Architecture (OGSA)
• Open Grid Services Infrastructure (OGSI)
• Globus Toolkit (GT3) is a reference
  implementation
• In GT4, OGSI has been replaced by WSRF (Web
  services resource framework)

10
OGSA, OGSI and WS
From tutorial Satomayors GT3 Tutorial
http//www.casa-sotomayor.net/gt3-tutorial-working
/
11
Features of Grid Service
Logger object Levels of logging Info, .. Warn,
Error, Fatal Filtering and redirecting to file,
console
Provides Standard security
Provides notification of events

Stores service properties and States for
discovery, monitoring, negotiations, etc.
Services with ability to migrate
Permanent services such as naming service
that get activated and terminated with the
container
12
Current Status
Internet
Information/ Application Servers
Clients/Consumers
Internet
Application
Application
13
Where are we heading?
Web-enabling information Web-enabling
applications/forms HTML
Business to Consumers (B2C)
Business to Business (B2B) Application to
application
Web Services, XML Standards for specifying
operation in SOAP (Simple Object Access Protocol)
14
Beyond Search Engines Enabling Information
Technology and Scientific Applications
Simple Search (stateless)
Financial Build Portfolio
Medicine Find Cure
Environment Plan Forestation
Travel Plan a Trip
Complex multi-business applications
15
Web Services Standard

• A common operation on the Internet is search, the
  results of which is consumed by humans.
• We want to develop complex multi-business
  applications that are beyond the current
  search-type applications.
• Webservices (WS) is a standard that has been
  introduced by W3 consortium to address this
  important transition.
• Grid takes the web services to the next level a
  grid service (GS) is a web service.
• GS WS state standard features for security,
  reliability, integration,
• Grid specifies a standard architecture,
  infrastructure, protocols and application program
  interface (API) for an open enterprise system.

16
Technology Pipeline
Web 2.0
Web 3.0
17
Virtualization

• Encapsulating service operations behind a common
  message-oriented service interface is called
  service virtualization.
• Isolates users from details of service
  implementation and location.
• Assumes support of a standard architecture.
• Webservices (WS) can do this, however life cycle
  management, fault handling and other features we
  have seen in the Grid are not available with WS.
• Grid specification addresses these issues using a
  core set of standard services.

18
On to data-intensive applications

• Applicable to a certain class of applications
• MapReduce GFS from Google
• MapReduce HDFS
• Amazons suite EC2, S3, mechanical turk, etc.

19
MapReduce