Wrapping Scientific Applications As Web Services Using The Opal Toolkit Sriram Krishnan, Ph.D. [e-mail]

1
Wrapping Scientific Applications As Web Services
Using The Opal ToolkitSriram Krishnan,
Ph.D.sriram_at_sdsc.edu
2
Motivation

• Enable access to scientific applications on Grid
  resources
• Seamlessly via a number of user interfaces
• Easily from the perspective of a scientific user
• Enable the creation of scientific workflows
• Possibly with the use of commodity workflow
  toolkits

3
Some Problems

• Access to Grid resources is still very
  complicated
• User account creation
• Management of credentials
• Installation and deployment of scientific
  software
• Interaction with Grid schedulers
• Data management

4
Towards Services Oriented Architectures (SOA)

• Scientific applications wrapped as Web services
• Provision of a SOAP API for programmatic access
• Clients interact with application Web services,
  instead of Grid resources
• Used in practice in NBCR, CAMERA, GLEON, among
  others

5
Big Picture
Gemstone
PMV/Vision
Kepler
State Mgmt
Application Services
Security Services (GAMA)
Globus
Globus
Globus
Condor pool
SGE Cluster
PBS Cluster
6
Scientific SOA Benefits

• Applications are installed once, and used by all
  authorized users
• No need to create accounts for all Grid users
• Use of standards-based Grid security mechanisms
• Users are shielded from the complexities of Grid
  schedulers
• Data management for multiple concurrent job runs
  performed automatically by the Web service
• State management and persistence for long running
  jobs
• Accessibility via a multitude of clients

7
Possible Approaches

• Write application services by hand
• Pros More flexible implementations, stronger
  data typing via custom XML schemas
• Cons Not generic, need to write one wrapper per
  application
• Use a Web services wrapper toolkit, such as Opal
• Pros Generic, rapid deployment of new services
• Cons Less flexible implementation, weak data
  typing due to use of generic XML schemas

8
The Opal Toolkit Overview

• Enables rapid deployment of scientific
  applications as Web services (lt 2 hours)
• Steps
• Application writers create configuration file(s)
  for a scientific application
• Deploy the application as a Web service using
  Opals simple deployment mechanism (via Apache
  Ant)
• Users can now access this application as a Web
  service via a unique URL

9
Opal Architecture
10
Implementation Details

• Service implemented as a single Java class using
  Apache Axis
• Application behavior specified by a configuration
  file
• Configuration passed as a parameter inside the
  deployment descriptor (WSDD)
• Possible to have multiple instances of the same
  class for different applications
• Distinguished by a unique URL for every
  application
• No need to generate sources or WSDL prior to
  deployment

11
Sample Container Properties
the base URL for the tomcat installation
this is required since Java can't figure out the
IP address if there are multiple network
interfaces tomcat.urlhttp//ws.nbcr.net8080
database information database.usefalse database.u
rljdbcpostgresql//localhost/app_db database.use
rltapp_usergt database.passwdltapp_passwdgt
globus information globus.usetrue globus.gatekeep
erws.nbcr.net2119/jobmanager-sge globus.service_
cert/home/apbs_user/certs/apbs_service.cert.pem g
lobus.service_privkey/home/apbs_user/certs/apbs_s
ervice.privkey parallel parameters num.procs16
mpi.run/opt/mpich/gnu/bin/mpirun
12
Sample Application Configuration
ltappConfig xmlns"http//nbcr.sdsc.edu/opal/types"
xmlnsxsd"http//www.w3.org/2001/XMLS
chema"gt ltmetadatagt ltusagegtlt!CDATApsize.py
opts ltfilenamegtgtlt/usagegt ltinfo
xsdtype"xsdstring"gt lt!CDATA
--help Display this text
--CFACltvaluegt Factor by which to expand mol
dims to get coarse
grid dims default
1.7 ... gt lt/infogt
lt/metadatagt ltbinaryLocationgt/homes/apbs_user/bin
/psize.pylt/binaryLocationgt ltdefaultArgsgt--GMEMCE
IL1000lt/defaultArgsgt ltparallelgtfalselt/parallelgt
lt/appConfiggt
13
Application Deployment Undeployment

• To deploy onto a local Tomcat container

ant -f build-opal.xml deploy -DserviceNameltservic
eNamegt -DappConfigltappConfig.xmlgt

• To undeploy a service

ant -f build-opal.xml undeploy -DserviceNameltserv
iceNamegt
14
Service Operations

• Get application metadata Returns metadata
  specified inside the application configuration
• Launch job Accepts list of arguments and input
  files (Base64 encoded), launches the job, and
  returns a jobID
• Query job status Returns status of running job
  using the jobID
• Get job outputs Returns the locations of job
  outputs using the jobID
• Get output as Base64 Returns an output file in
  Base64 encoded form
• Destroy job Uses the jobID to destroy a running
  job

15
MEMEMAST Workflow using Kepler
16
Gemstone Access to Molecular Science
17
Future Work

• Opal 2.0
• Currently under way - in design phase
• Use of Axis2 for better performance
• 6-8x performance improvement over Axis1.2
• Pluggable Resource Provider Model
• Easier to integrate access to resources via GRAM,
  DRMAA, CSF4, etc
• Alternate mechanisms for I/O staging - GridFTP,
  RFT, etc
• Type checking for application inputs and
  arguments
• State persistence via Hibernate

18
Summary

• Opal enables rapidly exposing legacy applications
  as Web services
• Provides features like Job management,
  Scheduling, Security, and Persistence
• More information, downloads, documentation
• http//nbcr.net/services/

19
Demos

• Workflows using Kepler
• Exploratory science using Gemstone
• Continuity Web services

20
Tutorial Using Opal to deploy a real scientific
application as a Web service

• Sriram Krishnan, Ph.D.
• sriram_at_sdsc.edu

21
Goals

• Start off with a clean slate
• Install software prerequisites
• Install the Opal toolkit
• Deploy scientific application as Web service
• Access service via command-line client

22
Before we get started

• Should have Java (1.5.x) and Apache Ant (1.7.0)
  installed
• Install Apache Ant from http//ant.apache.org/
• Set environment variables
• JAVA_HOME to point to the JDK installation
• ANT_HOME to point to the Ant installation
• Make sure that these can be accessed from the
  command prompt
• Add JAVA_HOME/bin and ANT_HOME/bin to PATH
• Run the commands java and ant to ensure that
  all variables are set correctly

23
Download Software Prerequisites

• Might want to create a new folder called
  Tutorial on your Desktop
• Download and Extract Axis
• http//nbcr.net/services/downloads/tutorial/axis-1
  _2_1.zip
• Extract inside Tutorial directory
• Download and Extract Tomcat
• http//nbcr.net/services/downloads/tutorial/jakart
  a-tomcat-5_0_30.zip
• Extract inside Tutorial directory
• Set the environment variable CATALINA_HOME
• Control Panel -gt System -gt Advanced

24
Download Opal

• Download and Extract Opal
• http//nbcr.net/services/downloads/tutorial/opal-w
  s-1.0RC1.zip
• Extract inside Tutorial directory
• Compile the sources
• On the command prompt, cd to the above location
• Type ant -f build-opal.xml compile
• Watch for error messages, if any

25
Deploy Axis inside Tomcat

• Copy axis directory inside axis-1_2_1\webapps
  into CATALINA_HOME\webapps
• Copy activation-1_0_2.jar and
  mailapi-1_3_1.jar from opal-ws-1.0RC1\lib
  into CATALINA_HOME\common\lib
• Start Tomcat from the command prompt
• Type cd CATALINA_HOME\bin
• Type startup.bat
• Test the Axis deployment
• http//localhost8080/axis/happyaxis.jsp
• Watch for error messages, warnings are OK

26
Edit Opal properties

• Using Wordpad, open opal-ws-1.0RC1\etc\opal.prope
  rties
• Look over the various properties being set
• Modify the property tomcat.url by replacing
  localhost with the the IP address of your
  machine
• Type ipconfig /all to get your IP address

27
Opal Properties
parallel parameters num.procs1
mpi.run/Users/sriramkrishnan/Misc/mpich-1.2.7/bin
/mpirun the base URL for the tomcat
installation this is required since Java
can't figure out the IP address if there
are multiple network interfaces
tomcat.urlhttp//localhost8080 database
information database.usefalse
database.urljdbcpostgresql//localhost/app_db
database.userapp_user database.passwdapp_p
asswd globus information
globus.usefalse globus.gatekeeperlocalhost2
119/jobmanager-sge globus.service_cert/Users/
sriramkrishnan/certs/app_service.cert.pem
globus.service_privkey/Users/sriramkrishnan/certs
/app_service.privkey
28
Download Scientific Application

• We will use the application OpenBabel
• Designed to support molecular modeling,
  chemistry, and many related areas, including
  conversion of file formats and data
• http//openbabel.sourceforge.net/
• Download http//nbcr.net/services/downloads/tutor
  ial/openbabel-2.0.0awins.zip
• Extract inside Tutorial\Babel directory
• Type babel from the above directory to ensure
  that it installed fine

29
Edit Babel Config

• Using Wordpad, open opal-ws-1.0RC1\etc\babel_conf
  ig.xml
• Look over the various elements
• Change the value of the binaryLocation to point
  to your Babel executable

30
Babel Config
ltappConfig xmlns"http//nbcr.sdsc.edu/opal/types"
xmlnsxsd"http//www.w3.org/2001/XMLS
chema"gt ltmetadatagt ltusagegtlt!CDATA./babel
-iltinput-typegt ltnamegt -oltoutput-typegt
ltnamegtgtlt/usagegt ltinfo xsdtype"xsdstring"gt
lt!CDATA ... Currently supported
input types alc -- Alchemy file prep -- Amber
PREP file ... Currently supported output
types ... caccrt -- Cacao Cartesian
file cacint -- Cacao Internal file ...
gt lt/infogt lt/metadatagt
ltbinaryLocationgt/Users/sriramkrishnan/bin/babellt/b
inaryLocationgt ltdefaultArgsgtlt/defaultArgsgt
ltparallelgtfalselt/parallelgt lt/appConfiggt
31
Deploy application using Ant

• On the command prompt, cd to your Opal
  installation
• To deploy application, type the following
• ant -f build-opal.xml deploy -DserviceNameBabelSe
  rvicePort -DappConfigetc\babel_co
  nfig.xml
• Restart Tomcat
• Type cd CATALINA_HOME\bin
• Type shutdown.bat
• Type startup.bat
• Check list of services
• http//localhost8080/axis/services

32
Run command line client

• On the command prompt, cd to your Opal
  installation
• Set classpath by typing classpath.bat
• To run the client, type the following
• java edu.sdsc.nbcr.opal.GenericServiceClient -l
  http//localhost8080/axis/services/BabelServicePo
  rt -a "-ipdb sample.pdb -h -opdb output.pdb" -f
  etc\sample.pdb
• Replace localhost with your neighbors IP address
  to use their Babel service

33
Whats Next?

• To add another service, you just need to add
  another application config
• And deploy using Ant - thats all!
• You can write custom clients for your Web
  services
• Gridsphere, Gemstone, Kepler, PMV/Vision,
  Continuity

34
More Information

• http//nbcr.net/services
• Downloads, Papers, Presentations
• More documentation
• Advanced configuration like Database, Globus, and
  Security setup
• Feel free to drop us a note - contacts available
  from our web page