Reinventing Explicit Parallel Programming for Improved Engineering of High Performance Computing Software

1
Reinventing Explicit Parallel Programming for
Improved Engineering of High Performance
Computing Software

• Anthony Skjellum, Purushotham Bangalore, Jeff
  Gray, Fei Cao, Barrett Bryant
• University of Alabama at Birmingham
• Department of Computer and Information Sciences
• http//www.cis.uab.edu
• ICSE 2004
• W3S Software Engineering for HPCS Applications

2
Premises of Position Paper

• Explicit parallel programming to be reinvented
• MPI, BSP, and relatives too low level, too hard
  to develop/maintain
• Accidental Complexity results from their use
• Improve vs. replace explicit parallel programming
• Refactor middleware by capturing
  orthogonalizing features, raising level of
  abstraction, exploring patterns
• Middleware only subtracts and never adds
  scalability/performance (Brightwells law), so
  lets add less if possible

3
Premises of Position Paper

• Several software engineering areas show promise
  to help OO, generic programming, generative
  programming, AOSD, MDA, MIC, component
  architectures, design patterns
• New notations should map as well or better to
  existing and emerging hardware/architecture if
  possible
• Scalability and performance of resulting
  applications remain key requirements
• Refocus certain academics from reimplementing
  MPI-1.2 systems for lowest latency, high
  bandwidth

4
Latest Motivator for Change CRA Workshop Report

• Replacing an object-based MPI (object-based
  standard) with another would not be a big step
  forward
• MPI programs now looked on as legacy code
  problem
• Huge investment in MPI codes (e.g., US DOE)
• More MPI codes being developed all the time
• Computing Research Association, Report of
  Workshop on The Roadmap for the Revitalization of
  High-End Computing, organized by Computing
  Research Association, Edited by Daniel A. Reed,
  Washington D.C., June 16-18, 2003.

5
Some MPI Realities

• After 10 years, MPI training still an issue
  (though widely taught at basic level!)
• MPI is not necessarily used well
• Myths exist on how to use MPI correctly
• Unsafe MPI programs abound
• Ties to particular MPI implementation quirks
• Some use bad MPI features (e.g., ALLTOALL,
  IPROBE)
• Failure to use advanced MPI features where useful
  (e.g., ISEND/IRECV vs. SEND/RECV)
• Posting receives early vs. late (fallacy)

6
Some Application Realities

• Parallel codes are often behind sequential codes,
  more rigid
• Conflation of data distribution and correctness
  of parallel application
• Lack of evolvability of parallel code
• MPI architecture within code set very early in
  parallelization, with specific machine/topology/MP
  I implementation/level of knowledge of
  designers/implementers frozen in
• Almost no fault tolerance in parallel codes

7
Outline for new approach

• Called Refactoring MPI
• Demonstrating validity for clusters
  (multicomputers) and grids
• Designing new applications based on Refactored
  MPI
• Studying means to move legacy software forward,
  where appropriate (beyond scope of current paper)

8
Refactoring MPI

• Capture semantics/features of
• MPI-1.2
• MPI-2.1
• DRI (data reorganization interface)
• MPI/RT
• BSP
• Offer a simpler, consistent model for achieving
  these features in a unified/component friendly
  environment
• Extract design patterns for communication and
  computation from these specifications

9
Build new applications differently

• Move away from middleware library implementations
  header files to component libraries
  constraint specifications application builders
• Look for ways to let application builders apply
  correctness semantics more optimistically than
  monolithic middleware currently allows
• Look for ways to build only required underlying
  communication fabric and complexity needed for a
  given application

10
How can software engineering techniques help?

• MDA/MIC capture the features of MPI relatives
  in a meta-model, define complete, orthogonal,
  minimal features for apps needing middleware
  (minimize dead code)
• Principles from generative programming (cf,
  Czarneki et al) for code generation from (GME)
  models may be useful to create actual code
• Ideally, design alternatives can be compared in
  terms of expression of data transfer, including
  concerns of overlapping transfers with
  computation

11
How can software engineering techniques help?

• AOSD helpful with global concerns
• Now, global concerns either implemented by high
  level programmer, and/or libraries make
  conservative assumptions on resources
• AOSD could help spread fault characteristics
  across application and automatically generated
  middleware
• AOSD could also be helpful in instrumentation for
  performance understanding (either source or
  binary)
• Specifically co-aspecting Refactored MPI
  components and application would allow high-level
  optimizations (e.g., less conservative
  assumptions about queues, message ordering)
• Aspecting models used to generate parallel codes
  (two level aspecting) also of interest

12
Summary

• Positions in this paper represent new/early
  effort
• Need to improve explicit parallel programming
• New/recent software engineering methodologies
  help address this goal
• Goal Achieve equal or greater scalability,
  performance, predictability, productivity than
  MPI-1 / MPI-2 programs
• use AOSD, MDA, patterns, and componentized
  middleware design to support data parallel and
  irregular parallel computations
• Reflect on features of MPI relatives (why
  designed, what verticals addressed, why/how)
• Address dynamic resource issues (cf, grid,
  adaptive applications requirements)
• Achieve other benefits (flexibility, ability to
  factor in fault aspects, other separation of
  concerns)