The Open Grid Computing Environments Project

1
The Open Grid Computing Environments Project

• Marlon Pierce
• Community Grids Laboratory
• Indiana University

2
Acknowledgements

• Funding from NSF NMI (2003-2007) and OCI SDCI
  (2007-2010).
• Current participants
• Indiana University (Pierce, Gannon)
• RENCI (Kandaswamy)
• RIT(von Laszewski)
• SDSC (Wilkins-Diehr)
• SDSU (Thomas, Edwards)
• TACC (Dahan)

3
Outline

• Web Portals and Science Gateways
• OGCE efforts
• OGCE Portal Software
• Portal tools
• Java COG, GTLAB
• OGCE Gateway Services
• GFAC, GPIR
• Software Engineering Issues
• What is next?

4
OGCE Goals

• To provide easily installable, well-tested
  software for building Web client and service
  components that constitute a Grid Computing
  Environment.
• Science Web Portal --gt GCE --gt Science Gateway
• To support developing groups through training,
  outreach, and divine intervention.
• Gateways have many needs that cant be solved by
  downloadable software alone.

5
What Is a Web Portal?

• Aggregate content from multiple sources into a
  single display.
• Typically consume RSS/Atom news feeds.
• More powerful versions these days support Flickr,
  calendars, games, etc.
• Gadgets, widgets
• Examples iGoogle, Netvibes, My Yahoo!

6
Science Portals and Gateways

• Science portals resemble standard portals, but
  must also
• Support access to computing and storage
  resources.
• Allow users remote, Unix-like access to these
  resources.
• Provide access to science applications and data
  sets.
• So security is crucial.
• And we must provide value added services as well
  as user interfaces.

7
A Comprehensive Gateway Architecture
8
Components for Science Portals

• OGCE is founded on the principal that portals
  should be built out of reusable parts.
• Key standard in our first phase the JSR 168
  portlet specification.
• Portlets can run in multiple containers
• uPortal, Sakai, GridSphere, LifeRay, etc.
• Allows us to build Grid specific components and
  deploy along side other goodies Sakai
  collaboration tools, contributed portlets, etc.

9
OGCE Portal Software
10
OGCE GPIR portlet can interoperate with TeraGrid
and your own GPIR services.
11
Manage TeraGrid MyProxy credentials with the OGCE
ProxyManager portlets.
12
OGCE file management client portlets interact
with TeraGrid GridFTP servers.
13
General purpose batch and interactive job
submission to GRAM, WS-GRAM is supported.
14
Dashboard Portlet
The dashboard portlet allows users to track jobs
on the selected resource. The user can view
either his own set of jobs or get information on
all submitted jobs.
14
15
(No Transcript)
16
Queue forecasting portlets work with the NWS
QBETS to predict wait times and deadlines.
17
PURSe portlets manage user requests for portal
accounts and Grid credentials.
18
Condor and Condor-G
19
OGCE IFrame Portlet can be used to integrate
external sites.
20
Building Your Own Grid Portlets
21
Coding Portlets

• Portlets are just servlet-like Java classes.
• Basic API key methods
• doView(), processAction().
• These are coupled to JSP pages (typically)
  through tag libraries and request dispatchers.
• OGCE supports Velocity portlets
• So we must provide the coding logic for
  processAction().
• COG abstraction layers provide this.

22
CoG Abstraction Layers
Development Support
Nano materials
Bio- Informatics
Disaster Management
Portals
Applications
GT2
GT3 (X)
GT4 WS-RF
Condor
Unicore
SSH
Others
23
Task
Task Handler
Task Specification
The class diagram is the same for all grid tasks
(running jobs, modifying files, moving data).
Service
Security Context
Service Contact
Classes also abstract toolkit provider
differences. You set these as parameters GT2,
GT4, etc.
24
Task and Specification

• Task tasknew TaskImpl(mytask,
• Task.JOB_SUBMISSION)
• task.setProvider(GT2)
• JobSpecification spec
• new JobSpecificationImpl()
• spec.setExecutable(rm)
• spec.setBatchJob(true)
• spec.setArguments(-r)
• task.setSpecification(spec)

25
Service and Security Context

• Service servicenew
• ServiceImpl(Service.JOB_SUBMISSION)
• service.setProvider(GT2)
• SecurityContext securityContext
• CoreFactory.newSecurityContext(GT2)
• //Use cred object from ProxyManager
• securityContext.setCredentials(cred)
• service.setSecurityContext(
• (SecurityContext)securityContext)

26
Service Contact and Submit

• ServiceContact serviceContact
• new ServiceContact(myhost.myorg.org)
• service.setServiceContact(serviceContact)
• task.setService(
• Service.JOB_SUBMISSION_SERVICE,
• service)
• TaskHandler handlernew GenericTaskHandler()
• handler.submit(task)

27
Coupling CoG Tasks

• The COG abstractions also simplify creating
  coupled tasks.
• Tasks can be assembled into task graphs with
  dependencies.
• Do Task B after successful Task A
• Graphs can be nested.

28
Problems with Portlet Development

• Grid portlets typically wrap each single Grid
  capability in a separate portlet
• Problem is that Grid portlets need to combine
  these operations
• Portlets are entire web applications, so we need
  a component model for portlets reusable portlet
  parts
• Even with the COG Abstraction Layer, we must
  still do a lot of coding to biuld new
  applications.
• To address these problems we have adopted Java
  Server Faces
• Provides several nice Model-View-Controller
  features
• JSF provides an extensible framework (tag
  libraries) for making reusable components.
• Apache JSF portlet bridge allows you to convert
  standalone JSF applications (development phase)
  into portlets (deployment phase).

29
Grid Tag Libraries and Beans (GTLAB)

• GTLAB provides common components for building
  portlets using tags and reusable parts.
• The goal of GTLAB to simplify Grid portlet
  development
• Enable rapid development
• GTLAB capabilities include Grid operations with
  XML based tags within Java Server Faces (JSF)
  framework.
• Grid tag libraries are built using JSF custom
  component development techniques
• Grid tags are interfaces to backing Grid beans
• End users pass values to Grid beans by using tag
  attributes.
• We build on Java CoG 4s abstraction layer.
• Each backing Grid bean has equal capability with
  a portlet application in case of Grid portlet
  approach.

29
30
GTLAB Example

• Grid tags are associated with Grid services via
  Grid beans
• Grid Beans wrap the Java COG Kit (version 4)
• We show an example JSF page section below.
• This allows you to develop new Grid portlets
  with no additional Java code.

• lthtmlgt
• ltbodygt
• ltfformgt
• ltosubmit idtest actionnext_page /gt
• ltomyproxy idpr hostnamegf1.ucs.indiana.edu
  port7512 lifetime2 usernamemnacar
  password /gt
• ltojobsubmit idtask hostnamecobalt.ncsa.tera
  grid.org providerGT4 executable/bin/ls
  stdouttmp/result stderrtmp/error /gt
• lt/osubmitgt
• lt/fformgt
• lt/bodygt
• lt/htmlgt

30
31
Grid Tags Associated Grid Beans Features
ltsubmit/gt ComponentBuilderBean Creating components, job handlers, submitting jobs
lthandler/gt MonitorBean Handling monitoring page actions
ltmultitask/gt MultitaskBean Constructing simple workflow
ltdependency/gt MultitaskBean Defining dependencies among sub jobs
ltmyproxy/gt MyproxyBean Retrieving myproxy credential
ltfileoperation/gt FileOprationBean Providing Gridftp operations
ltjobsubmission/gt JobSubmitBean Providing GRAM job submissions
ltfiletransfer/gt FileTransferBean Providing Gridftp file transfer
ResourceBean Describes common properties among all tags and beans. Passing values given by standard visual JSF components.
32
How to prepare application pages

• Developers embed Grid tags snippet into JSF page
• These components are non-visual and are not
  displayed in HTML.
• Resource bean provides bridging with form inputs
  and GTLAB framework.
• lthoutputText value"Taskname "/gt   lthinputText
  value"resource.taskname" /gt   ltomultitask
  id"multi" persistent"true" task name"resource
  .taskname" /gt
• Dynamic values to Grid tag attributes are
  provided by Resource bean.
• Only visual component is ltosubmit/gt tag that is
  associated with action method of GTLAB.

32
33
GTLAB Dashboard PortletExample

• ltosubmit idtrack actionlist_page /gt
  
• ltomultitask iddashboard tasknametrack
  persistenttrue gt
• ltomyproxy idproxy hostnamegf1.ucs.indian
  a.edu lifetime2
• usernameresource.usern
  ame passwordresource.password /gt
• ltojobsubmit idjobA
  hostnamecobalt.ncsa.teragrid.org
• providerGT4
  executable/bin/whoami
• stdouttmp/result
• stderrtmp/error /gt
•  
• ltojobsubmit idjobB
  hostnamecobalt.ncsa.teragrid.org
• providerGT4
  executable/bin/showq
• stdintmp/result
  stdouttmp/list
• stderrtmp/error /gt
•  
• ltodependency iddepend taskjobB
  dependsOnjobA /gt
• lt/omultitaskgt
• lt/osubmitgt

33
34
Tracking and Managing Jobs

• GTLAB manages lifecycles of jobs and monitor
  their status.
• Grid operations are usually batch processes
• We provide callback mechanism to follow up the
  jobs
• GTLAB creates handlers for jobs and persistently
  stores them.
• GTLAB handlers manages the job events such as
  stop, cancel or resuming the running jobs.
• GTLAB provides archive for job metadata and
  allows managing the archive
• Handler tag helps to organize users job
  repository
• ltohandler iddelete action"monitor.delete"
  gt
• ltfparam id"task" name"taskname
  value"task"/gt
• lt/ohandlergt

34
35
OGCE Gateway Services
36
Web Services in Scientific Communities (G.
Kandaswamy)

• Web services are used to wrap scientific
  applications to
• Describe, publish, discover and consume
  scientific applications in a standard way
• Compose complex workflows from scientific
  applications
• Run and monitor complex workflows on distributed
  resources
• Such web services that wrap scientific
  applications are called application services

36
37
A Simple Application Service
Command-line Arguments
SOAP Request
Output Results
SOAP Response
Host1
Host2
37
38
Things Are Usually More Complicated
38
39
The Problem

• Application services may not be available during
  a workflow execution
• Unreliable resources (software, computers,
  networks)
• Heavy load on service
• Does not meet QoS or security requirements of
  client
• Workflows cannot complete unless all services are
  available

39
40
GFAC Solution

• A Generic Application Factory
• A persistent web service that knows how to create
  instances of any application service
• Use a Generic Application Factory to create
  instances of application services on-demand from
  workflows

40
41
Implementation

• The Generic Application Factory (GFac)
• The Generic Service Toolkit A toolkit that
  wraps any command-line application as an
  application service
• Without writing any web service code
• Without modifying the application in any
  significant way

41
42
Creating an Application Service (1/2)

• Write ServiceMap document to describe your
  service
• Write Application Deployment Description
  document to describe a deployment of your
  application
• Upload the above two documents to a Registry
  service

42
43
Creating an Application Service (2/2)
Portal
Service Provider
Host1
1. Create service request
2. Get ServiceMap Host Description
3. Create service
4. Configure service
5. Register WSDL
5. Register capabilities
Host2
43
44
Invoking an Application Service
Portal
5. Get Application Deployment Description and
Host Description
Host2
User
2. Access service
3. Return user interface
4. Invoke Service
7. Return results
4. Run application
6. Send notifications
Host3
44
45
Software Engineering Issues
46
OGCE Code Repository

• We use SourceForge, SVN
• http//sourceforge.net/projects/ogce
• Other SourceForge tools are useful.
• Replaced old OGCE bugzilla with SF bugzilla
  recently after we were attacked by robots.

47
Portal Build System

• The portal download gives you everything you need
  to get started except Java.
• Includes Tomcat, GridSphere, Ant, and Maven.
• Assume you have a Grid somewhere.
• Build system (recently revised) is designed to
  build everything in one command.
• mvn clean install
• Also designed to support extensibility (I.e.
  replace GridSphere with Sakai) and simple updates
  of portlets.
• We use Maven 2 exclusively.
• Nice for managing third party jar dependencies.
• It can call Ant as necessary
• Testing portals is another matter
• Normal unit test systems like Junit are not
  really appropriate.

48
JMeter Test Suite
File Transfer portlet unit tested in JMeter UI
check for valid HTML response
Create lots of unit tests, run, and see results
in a dashboard
49
Nightly Builds and Tests on NMI Testbed
50
Whats Next?
51
Some Future Issues

• Better support for science tools, not just bare
  grids.
• Experiment builder, Xbaya workflow manager,
  metadata repository services and clients.
• Better support for TeraGrid Science Gateways
• Logging, auditing, integration with GridShib
• JavaScript Grid abstraction layers and agent
  services to support non-portlet clients.
• More projects obviously we are interested in
  working with the OSG

52
What About Web 2.0?

• This is another talk entirely.
• http//grids.ucs.indiana.edu/ptliupages/presentati
  ons/Web20Tutorial_CTS.ppt
• http//grids.ucs.indiana.edu/ptliupages/publicatio
  ns/Web20ChapterFinal.pdf
• See also recent OGF 19 and 21 Workshops.
• Join us at SC07 for the GCE07 Science Gateway
  Workshop
• 20 peer-reviewed or invited talks, with focus on
  Web 2.0.

53
More Information

• OGCE Web Site
• www.collab-ogce.org
• Announcements Atom Feed
• http//collab-ogce.blogspot.com/atom.xml
• Contact me mpierce_at_cs.indiana.edu

54
Some Example Portals
55
LEAD Gateway Portal
NSF Large ITR and Teragrid Gateway - Adaptive
Response to Mesoscale weather events -
Supports Data exploration,Grid Workflow
56
TeraGrid User Portal
57
User Portal Sharable Portlets

• Account Management
• view projects and allocation usage
• view system account usernames
• view DNs registered for account
• add users to projects
• supports gt3500 users
• Resource
• view comprehensive list of TG resources and their
  attributes
• view job queues, load, status of resources

• Documentation
• current User Info documentation
• contextual help for all interfaces
• Consulting
• TG help desk information
• portal feedback channel
• Allocation
• Info about how to apply for/renew allocations

58
North Carolina Bioportal

• Principal collaborators John McGee and Lavanya
  Ramakrishnan
• Features
• access to common bioinformatics tools
• extensible toolkit and infrastructure
• OGCE and National Middleware Initiative (NMI)
• leverages emerging international standards
• remotely accessible or locally deployable
• packaged and distributed with documentation
• National reach and community
• TeraGrid deployment
• Portals hosted at RENCI and NCSA
• Education and training

59
UNC-Charlotte Visual Grid Portal
Project Lead Prof. Barry Wilkinson Portal
Developer Jeremy Villalobos
60
(No Transcript)
61
(No Transcript)