Grid Computing: Concepts and Perspectives

1
Grid Computing Concepts and Perspectives

• José C. Cunha, CITI/DI-FCT/UNL

2
(No Transcript)
3
Transparency and Virtualisation

• To provide adequate functionalities to the
  end-user with the minimal required knowledge
  about the internals details of computer
  operations
• The concepts of transparency have been changing
  as time and technology evolve
• Raw hardware, Assembly, High-Level Languages,
  etc....Operating Systems,...., Text editors and
  processing tools...., Specific application
  packages...
• Grid Computing is related to one of the current
  attempts of increasing the virtualisation levels

4
Distributed Computing

• Physically distributed computations and data
• Goals
• Adapt to geographical application distribution
  and mobility of users and devices (mobile phones,
  PDA)
• Enable communication / access to remote users
  (eg, e-mail), applications and information (eg,
  databases)
• Distribution (Local/global networks)
• Users / Access / Processing / Archiving Sites
• Availability and Reliability

5
Parallel Computing

• Relies on multiple processors cooperating for
  the coordinated and simultaneous resolution of
  parts of a given problem
• Supported by different kinds of Computer Systems
  and Architectures
• Shared / Distributed memory multiprocessors
• Local computer networks and Clusters of PCs
• Large-scale distributed computational Grids

6
Parallel Computing

• Application Problems
• -- which run too slow in sequential computers
• -- which could be solved by a supercomputer but
  this is too expensive
• -- which could not be solved even by
  supercomputer in the required time for the result
  to be useful
• Parallel Computing solutions
• -- goal is to reduce execution time, compared to
  sequential execution

7

• Where is the potential for parallelism?
  Applications?
• Science and Engineering
• Fluid dynamics
• Particle systems
• Weather forecast and Climate modelling
• ? Complex models - simulations with large amounts
  of data
• Economy and Finance
• Financial models
• Simulation of VLSI circuits
• Test generation, fault diagnosis
• Databases
• Parallel search
• Parallelism across transactions (multiple
  simultaneous users)
• ...
• Search of solution / state spaces
• Pattern recognition / image processing
• Natural language processing (parallel text
  mining)
• -etc.

8
Problem-solving perspective

• Parallel Computing requires
• Decomposing the application into parts
• Launching tasks in parallel processes
• Planning the cooperation between tasks
• in general, this is very difficult... and
  requires expertise both in Computer Science and
  in the Application Domain...
• giving motivation to develop
• Integrated environments for solving classes of
  related problems in each application domain

9
Problem-Solving Environments

• -- specific methods for each problem domain are
  encapsulated into software components (libraries,
  packages, and saved in catalogs and repositories)
  
• -- development support tools are also made
  available to the end-user.
• Application components and computational tools
  are integrated into a single unified environment
  (PSE)
• Easy-to-use by the end-user

10
Plug and Play Components (courtesy Prof. David
Walker)

• Can link the output of one component to the input
  of another.
• Store components in a repository.

11
(No Transcript)
12
Impact of PSEs in many areas (1990-1999-2000...)

• Fully developed PSEs in the Industry, e.g.
  Automotive, Aerospace
• Many applications in Science and Engineering
• Design optimisation
• Application behavior studies (parameter sweeping)
• Rapid prototyping
• Decision support
• Process control
• Emerging areas Education, Environment, Health,
  Finance
• A new profile of end-user, beyond the scientist
  and engineer

13
Collaboration

• End-users (scientists,engineers,etc.)
• Solve a particular problem in a specific
  application domain
• Perform experiments
• PSE Developers
• Develop new algorithms and techniques
• Integrate them into components and place them in
  component repositories
• Develop tools to support problem specification
  and application composition
• Develop tools to help the user choose the best
  solutions and to locate the resources

14
Evolution ofApplication Characteristics

• Complex models simulations
• Large volumes of input / generated data
• Difficult interpretation and classification
• High degree of User interaction
• Offline / online data processing / visualisation
• Distinct user interfaces
• Computational steering (eg change parameters)
• Multidisciplinary
• Heterogeneous models / components
• Interactions among multiple users / collaboration

15
Modern applications demanding more ambitious
goals

• Enable heavy applications in science and
  engineering
• Complex simulations with visualisation and
  steering
• Access and analysis of large remote datasets
• Access to remote data sources and special
  instruments (satellite data, particle
  accelerators)
• distributed in wide-area networks, and
• accessed through collaborative and
  multi-disciplinary PSE, via Web Portals.

16
Application Grand Challenges

• Climate modelling to understand the Earth's
  climate and predict future changes
• Computational fluid dynamics to design aerospace
  vehicles and cars
• Numerical turbulence to develop realistic fluid
  and particle simulations of plasma turbulence
• Rational drug design to discover / design new
  drugs with simulations of molecular structure

17
Enabling factors for Grid Computing

• Faster processors / High-Performance Computing
  using standard / open OS
• Advances in distributed and parallel computing,
  in software engineering, and problem-solving
  environments
• The Internet
• World Wide Web infrastructure and services
• Broadband communications (eg optical-based)

18
(No Transcript)
19
The Grid

• Treat CPU cycles and software like commodities.
• Enable the coordinated use of geographically
  distributed resources in the absence of central
  control and existing trust relationships.
• Computing power is produced much like utilities
  such as power and water are produced for
  consumers.
• Users will have access to power on demand
• When the Network is as fast as the computers
  internal links, the machine disintegrates across
  the Net into a set of special purpose appliances
• Gilder Technology Report June 2000

This slide is courtesy of Professor Jack Dongarra
20
Concept of a Grid

• Gathers a diversity of resources, distributed at
  large-scale
• supercomputers and parallel machines, and
  clusters
• massive storage systems
• databases and data sources
• special devices
• Provides globally unified access to virtual
  resources
• Transient to support experiments
• (computation, data, scientific
  instruments)
• Persistent
• (databases, catalogues, archives)
• Collaboration spaces

21

• An application job splits into multiple
  components which are spread on the distributed
  grid computing environment
• ? need to locate one another
• ? to establish communication connections
• ? to send data
• This requires a high degree of virtualisation of
  resources and high-level user interfaces.

22

• Instead of
• Manually subdivide algorithms
• Manage their execution on separate machines
• Need to have a separate user login account in
  each machine
• Manually merge and integrate the results
• Exploit Grid tools to the same, more or less
  automatically, in a virtualised environment
• Single login access point
• Access to logical resources

23
(No Transcript)
24
Virtualisation

• Generic approach to
• Allow logical access to types of remote,
  heterogeneous, and distributed resources
• As if they were a single larger homogeneous
  resource, locally available
• Applies to computation, storage, and network
  resources and to any other LOGICAL RESOURCE
• Resources are aggregated into pools
• Dynamically adjust resource mappings to match
  application demands

25
Grids Towards large-scale computing environments

• Analogy to the Electrical Power Grid
• Simple local interface...
• Transparency...
• Pervasive access...
• Secure...
• Dependable...
• Efficient...
• Inexpensive...
• The Computational, Data, and Interaction Grids
• Not really true (yet!?)

26
Question Is this just an academic exercise? No!

• Real applications needs
• Solve new or larger problems by aggregating
  available resources at large-scale
• for bigger, longer experiments, and more accurate
  models
• Easier access to remote resources
• a large diversity of computation, data and
  information services
• Increased levels of interaction for increased
  productivity and capability to analyse and react
• enable coordinated resource sharing and
  collaboration across virtual organisations

27
Applications and User Profiles

• Computational Grids
• provide a single point of access to a
  high-performance computing service
• Scientific Data Grids
• Access large datasets with optimized data
  transfers and interactions for data processing
• Virtual Organisations and Interactions
• Access to virtual environments for resource
  sharing, user interaction and collaboration
• Real-time interactions for decision support
• Information and Knowledge services
• Access large geographically distributed data
  repositories, e.g. for data mining applications

28
Grid is an evolving field

• Multiple views, perspectives
• Concepts, models and architectures still being
  defined and tested
• Applications still emerging
• Wide variety of interests

29
Applications example

• Virtual access to distributed supercomputing
• For complex computations
• Migrate CPU-bound operations to more powerful
  remote computing resources supported by large
  virtual supercomputers, assembled to solve
  problems too large to fit on a single computer
  system

30
NetSolve The Big Picture (David Walker)
Client
Schedule Database
AGENT(s)
Matlab Mathematica C, Fortran Java, Excel
S3
S4
S1
S2
C
A
31
Applications example

• Virtual access to special instruments
• electron microscopes, particle accelerators, wind
  tunnels,
• coupled with remote supercomputers, DBs,
• to enable
• interactive use,
• online scenario comparisons,
• and collaborative data analysis

32
View Scientific Data Grids

• EU DataGrid projects
• Large-scale environment for accessing and
  analysing large amounts of data
• High energy physics, Biology, Earth observation
• Petabytes of data (1 000 000 Giga)
• Thousands of researchers
• Scalable storage of datasets replicated,
  catalogued, distributed in distinct sites

33
View - Virtual Organisations

• Resource sharing and collaboration between
  dynamically changing collections of individuals
  and organisations
• e.g. Consortium of companies collaborating in a
  design of a new product
• Sharing design data, Collaborative simulations,
  etc
• e.g. Scientists collaborating in common
  experiments via a distributed virtual laboratory

34
Example Collaborative Immersive Visualisation

• Scientific simulations, experiments, and
  observations generate vast amounts of data.
• Observer in the same virtual space as the
  visualised data and can navigate within
• Multiple observers can co-exist in the same
  visualisation space - ideal for remote
  collaboration.
• CAVE a fully immersive environment. Systems with
  stereoscopic projections onto 3 walls and the
  floor.

35
CAVE
36
Applications example

• Parameter studies
• Rapid, large-scale parametric studies
• A single program is run many times
• To explore a multidimensional parameter space

37
Grid - summary of ideas

• Grid
• ---gt cooperation / computation
• ---gt resource-sharing for specific application
  goals

38
Grid Application Characteristics

• Large volumes of data, requiring
• Efficient management and search
• Parallel and distributed processing
• Integration of distributed, heterogeneous
  components in highly dynamic and interactive
  environments
• Dynamic, distributed, and mobile application
  entities, requiring appropriate management of
• Structure, interaction, and coordination
• Dynamic organisation of small, medium, or large
  scale collections of distributed entities

39
Distributed and Grid Computing Systems

• Increasing levels of interaction among components
• New forms of dynamic behavior
• Due to mobility
• Due to more frequent changes in system and
  application configurations
• Due to changes in interaction and behavior
• Increasing scale in terms of system and
  application components

40
More Complex Applications and Environments

• Large number of components
• Complex interactions
• Dynamic configuration

41
Software Engineering Challenges

• Suitable levels of flexibility in all stages of
  the software lifecycle
• Application specification and design
• Program transformation and refinement
• Simulation
• Code generation
• Configuration and deployment
• Coordination and control of the execution

42
Component Based Development /Software
Architecture
Repositories (Skeletons/Templates/Patterns)
Abstract Description Language
specify, design, compose
For structure, behaviour, computation, and
interaction
Mappings
verify, analyse, evaluate, predict
Programming Levels (Models)
Resource Description and Discovery
Deploy and Configure
Grid Execution Environments
control, coordinate execute, reconfigure
Methodology
43
Global conceptual layers

• Software architectures
• Coordination models
• Resource management
• Execution, monitoring and control
• Support infrastructures

44
(No Transcript)
45
Very complex systems

• Aim at providing unifying abstractions to the
  end-user
• Large-scale universe of distributed,
  heterogeneous, and dynamic resources
• Critical aspects
• Distributed
• Large-scale
• Multiple administrative domains
• Security and access control
• Heterogeneity
• Dynamic

46
Grid types

• Space scale Local, metropolitan, regional,
  national, global
• Time scale logically aggregate resources for
  long or short periods of time
• Crossing borders Resources can span a single or
  multiple organisations, or a service provider
  space

47
Layers of a Grid Architecture

• User Interfaces, Applications, PSEs
• Programming Models, Development Tools and
  Environments
• Grid middleware Services and Resource
  Management
• Heterogeneous Resources and Infrastructure

48
Grids and Distributed Systems?

• What are the differences?
• The distinctive aspects
• Higher levels of the transparency for the
  end-user
• Higher levels of integration of services
• Virtualisation of resources

49
The main questions

• Grid benefits, challenges
• Grid architectures
• Standardisation efforts
• Architecture (Open Grid Service Arch/Infrast)
• Execution Models Workflows, Events, Transactions
• System services Security, Monitoring, Billing
  and Accounting, Implementation
• Grid deployment
• Local, national, and global grids
• Grid Application Development
• Economics

50
Semantic Grid

• Will give information a well-defined meaning to
  better enable computers to understand the content
  of documents, and thereby allow people, agents,
  and services to work together.
• Agents for user recommender systems and
  application support, resource monitoring and
  discovery, and for building intelligence in PSEs