Using the Pylons Web Framework for Science Gateways - PowerPoint PPT Presentation


PPT – Using the Pylons Web Framework for Science Gateways PowerPoint presentation | free to view - id: 1ac7bd-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Using the Pylons Web Framework for Science Gateways


GCOM solves a 3-D, time-dependent, curvilinear Navier-Stokes flow equations. ... adaptable curvilinear grid; fine grain structure. CI Environment: Portal: ... – PowerPoint PPT presentation

Number of Views:502
Avg rating:3.0/5.0
Slides: 41
Provided by: maryt159


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Using the Pylons Web Framework for Science Gateways

Using the Pylons Web Framework for Science
  • Mary Thomas
  • Department of Computer Science
  • San Diego State University
  • Presented at the GCE08 Workshop
  • Supercomputing, November, 2008, Austin, Texas

Todays Presentation
  • Introduction, background, motivation
  • Description of the Pylons Web Application
  • Demo Portal Architecture
  • Science Application General Coastal Ocean Model
  • Future Work

UPDATE!!! Challenge Proving Access to TeraGrid
Resources to Communities
  • NSF TeraGrid
  • Resource Providers 9 partners
  • 40 GigE backbone
  • 40 TFlops
  • Multi PByte storage

Science Gateways
  • Science gateways provide common, shared, simple
    interfaces to TeraGrid resources for large
  • Community models/services access resources and
  • Users avoid difficult and complex steps needed to
    get accounts, learn computer systems, etc.
  • Gateways are built using a variety of tools
    ranging from simple, customized tools to
    comprehensive frameworks.
  • Age old problem capability vs. complexity
  • greater capabilities -- authentication, account
    management, interfaces, logging, etc.
  • more complexity - resulting in long learning
  • We need to simplify development cycle

Open Grid Computing Environments (OGCE)
  • Funded by the National Science Foundation (NSF)
    to facilitate science gateway development
  • includes services, toolkits
  • Simple Java Portlet framework targeted for
    TeraGrid and cyberinfrastructure
  • Maven based easy install.
  • TeraGrid gateways using OGCE, including
  • Linked Environments for Atmospheric Discovery
  • TG Visualization Gateway,
  • Open Life Sciences Gateway(OLSG),
  • Virtual Laboratory for Earth and Planetary
    Materials (VLab)
  • Note other languages/frameworks used on TG.
  • OGCE lowers barrier for access to distributed

Project Motivation
  • There are other languages/frameworks available in
  • GridSphere, WebSphere, Sakai
  • none provide the services of the Java based OGCE
  • portlet dev environment non-trivial.
  • Other language choices include Python, PHP, Ruby,
  • all keeping pace with Web 2.0 technologies.
  • Early grid portals were written using scripting
  • GridPort (Perl5/CGI), Clarens - simple method for
    building portals/gateways
  • note Perl 6 promises to be object oriented but
    it has been under development since 2000, and is
    not yet ready for public use.
  • rapid development cycles are still popular and
  • Can we provide a gateway framework in another
    language that compliments OGCE?
  • Easy install, GSI integration
  • Technologies including Web 2.0/SOA/Component, WS
    client, etc.

TeraGrid Python-based Projects
  • UCSF Chimera - A visualization system 24,
    VisPort TeraGrid Visualization Web service.
  • National Virtual Observatory (NVO) - astronomy.
  • The Computational Infrastructure for Geodynamics
    (GIG) Science Gateway for the Geodynamics
  • Grid Analysis Environment (GAE).
  • Python Toolkits
  • Chimera is available as an extensible download,
    but is only for chemical modeling.
  • Clarens is Django-based extensible toolkit
  • Two of the projects listed above are built using
    Clarens (GIG, GAE).

A Word About Frameworks
  • Frameworks are collections of components that are
    integrated and form the foundation for an
    application and provide the support needed by a
    Web application to do its job.
  • application runs inside the context of the
  • required for modern Web apps because of the
    number and complexity of components, services,
    data, interoperability, etc.
  • framework includes clients and services,
    databases, templating/view, etc.
  • The list of projects that are labeled Web
    application frameworks is long - from Wikipedia
  • 17 server-side languages
  • over 160 frameworks, including 36 in Java, 8 in
    Perl, 54 in PHP, and 14 in Python.
  • the capabilities provided by these frameworks are
    diverse, and choosing the right framework for a
    gateway project easily becomes a complex task.

Advantages of Python
  • A scripting language
  • dynamically compiles on first execution/changes
  • unlike other languages (e.g. Perl) that reload
    every call.
  • Natively object-oriented, with a large number of
    modules available.
  • It well known as a scientific computing language
  • has many numerical scientific computing tools
    including SciPy, NumPy,
  • Visualization tools NCAR (National Center for
    Atmospheric Research) PyNGL and PyNIO plotting
  • Several portal web application frameworks to

Another Advantage Python Web Services Gateway
Interface (WSGI)
  • A W3C specification for web and application
    servers to communicate and interoperate with
    other web applications and servers.
  • WSGI functions in the same way as the servlet
    architecture used by Java web applications, GCI
    for Perl, etc.
  • Interoperable similar to JSR 168 for portlets
  • allows external components to be integrated, so
    can customize the framework
  • E.g. adding templates or AJAX libraries
  • Applications written with one framework
    can/should be able to run on any server
    supporting the common APIs and libraries.
  • This narrows our choices frameworks python

Simple Choice Python has GSI Modules
  • In addition to being a good programming
    environment, GSI connectivity is already solved.
  • GT4 includes several Globus Python Core release
  • pyGlobus provides an interface to pre-web
    service versions of Globus
  • pyGridWare Python implementation of the Web
    Services Resource Framework (WSRF) and WS-N
    protocol families
  • pyGSI provides Python support for Grid security
    including Grid style proxy certificates through
    command line tools and APIs.
  • But, can we identify a framework that integrates
    GSI tools?
  • Not so easy

Candidate Python Frameworks
  • Evaluated several frameworks
  • Plone, Zope, Django, TurboGears, Pylons
  • Both in the research lab and in teaching courses
    at SDSU
  • All have typical portal services accounts,
    database, session history, etc.
  • Most designed to be content management systemS
  • CMSs come with complex data bases which are not
    needed for grid portals
  • difficult to remove or work around these
  • cumbersome when adapting to Grid services
  • Most are old by Web standards,
  • have absorbed Web 2.0 technologies at late stages
  • re-factored to support WSGI (Web Server Gateway
  • Motivations for a choice
  • can we integrate existing python GSI tools or
    must we write our own?
  • since many TG researchers are faculty, can
    students figure out how to use/modify the

Pylons Web Application Framework
  • Pylons designed from ground up to use WSGI
  • The Pylons model-view-controller (MVC)
    request-response architecture
  • Model contains the data that the application
    works with.
  • models do not contain any information on the
    meaning of this data.
  • often the model refers to database tables.
  • View reads the data from the model and displays
    it to the user.
  • for Pylons this is the template - part of the
    software system used to build Web pages.
  • Controller manages the logic of the application.
  • activates views to display data to the user or
    gets information from the user and stores them to
    the models.
  • Any number of components and libraries developed
    by other projects can be (and are) incorporated
    into the system
  • each of these components is kept separate
  • you can just use the component interface in your
    Python script.

More Pylons Advantages
  • Natively WSGI
  • Designed to use Web2.0/WSGIfrom the beginning
  • It has a very light basic installation because it
    is not inherently designed to be a content
    management system (CMS).
  • Uses Python eggs which are like Maven
  • Directory structure is simple and flexible and
    reflects MVC architecture clearly (next slide)
  • For each new function, you write/modify a script
    for M, V, and C
  • Compiles on the flymost changes show up
  • Other systems are moving to revised architectures
    reflecting the success of the approach of Pylons
    as well.
  • TurboGears 2.0 is now being built on Pylons.
  • Django considering.

Pylons Architecture
  • The basic operation of the Pylon framework
  • (1) the user selects a new page, (e.g. job
  • (2) the Pylons router script is then read to map
    the request to an action (controller) script
  • (3) the controller script selects the correct
    module, and executes the block of code which may
    include local or external calls to other
    applications, connections to the grid, etc.
  • (4) at end of block, the controller calls a
    script to render the HTML(template) based on data
    passed to the template script
  • (5)when done, this HTML is returned back along
    the script chain to the router which serves the
    HTML data to the right URL.

Key Features Security/Authentication
  • Security/Authentication optional to use
  • Uses AuthKit module to log the user into the
  • complete authentication and authorization
    framework for WSGI applications
  • written specifically for Pylons to manage
    authentication tasks
  • but, can be used with any WSGI application
  • Maintains session history.
  • Once a user is validated, authentication
    information is visible to all modules and
    components being used by the portal.
  • access can be restricted based on roles (user,
    group, etc.).
  • Pylons supports a number of SQL databases
  • Optional use portal/WS can run without a
  • Key Question can we integrate GSI?
  • Yes, or this talk would be about another language.

Key Features Routing
  • Routing/Beautiful URLs
  • Typically, a URL for a Web portal has an ugly
  • E.g., the URL used to see weather in Pierrevert,
    France is
  • http//
  • Routes is the Python implementation of the Rails
    routes system for mapping URL's to
    Controllers/Actions and generating URL's
  • Routes makes it easy to create pretty and
    concise RESTful (REpresentational State
    Transfer.) URLs.
  • This means that the ugly URL above could look
  • http//
  • In the Pylons M-V-C architecture, controller maps
    pretty URL to an action and script or code.
  • Will show example later in talk.

Key Features SQLAlchemy
  • Pylons integrates SQLAlchemy Database
  • SQLAlchemy is a Python SQL toolkit and Object
    Relational Mapper (ORM)
  • ORMs map the database structure to objects.
  • For example, the SQL command
  • SELECT,, users.fullname
  • FROM users
  • Is replaced by the single SQLAlchemy command
  • s select(users)
  • The ability to access the entire database using
    object access methods simplifies the programming
  • Although not necessarily the database design
  • Note Pylons supports many databases, you dont
    have to use SQLAlchemy.

Key Features Simple Deployment
  • Simple Deployment
  • New Pylons projects are built using Paste.
  • Web development and application installation tool
    similar to Ant or Make
  • Paste creates python middleware needed for
  • router, config, and mapper template files
  • python webserver that can process WSGI requests
  • directory structure needed for a full site (to be
    populated by the developer) - reflects M-V-C
  • basic files, and data to support a new portal.
  • A single command installs the application
  • paster create -templatepylons pyGateSite
  • The portal server is started using the following
  • paster serve reload development.ini

Key Features Dynamic Update Interactive
  • Dynamic Update most important feature for dev
  • Pylons reload feature supports dynamic class
    loading for all MVC layers
  • modifications to the code cause compiler to
    recompile the binary and re-load it into the
    server while keeping the server live
  • substantially reduces development cycles.
  • Note Portlet frameworks do not support this
  • Interactive Debugger
  • all Pylons projects inherit the ErrorHandler
  • catch any exception, present information on a Web
  • also has log files.

Key Features Web 2.0 Interfaces
  • Pylons creates RESTful web services using the
    paster script command restcontroller option. This
    instantiates a RestController class that is
    automatically populated with the requisite
    skeletal actions.
  • Google Code
  • Appenginemonkey interfaces to the Google
    Application Engine
  • PyWebKitGtk provides an API for developers to
    program WebKit/Gtk
  • Pyjamas for Google Web Toolkit
  • iGoogle style Web apps easy to host (example
    shown later)
  • Desktop Applications
  • pyjamasd converts any Pylons project to desktop
  • JavaScript support for AJAX/Template libraries
    such as the Yahoo! UI Library (YUI).

Developing Pylons Gateway Portal
  • Once we decided on the Python Web app framework,
    it was time to build a portal.
  • Initial efforts focused on exploring what is
    needed to connect to GSI services/TeraGrid.
  • Set up some experiments to develop a basic suite
    of portal components
  • Accounts
  • GSI authentication - MyProxy
  • Job submission GSI
  • Web Service clients (e.g. GPIR)
  • Web 2.0 Could we also develop a portal that
    could be accessed as a Google Gadget/Facebook
    app, etc.?
  • The tools were out there, but could we use GSI?
  • Real World Applications - two projects right now
  • The General Coastal Ocean Model (GCOM) lt-- main
  • The CyberCHEQS Chemical Equilibrium Model

Test Application GCOM
  • GCOM solves a 3-D, time-dependent, curvilinear
    Navier-Stokes flow equations.
  • simulate stratified flows over uneven terrains
  • adaptable curvilinear grid fine grain structure
  • CI Environment
  • Portal Community Model
  • Simulations run by educators, coastal managers
    and government agencies
  • integrate into climate simulators.
  • Web Services
  • Client to vast numbers of ocean data
  • Host services for analyzed data, simulations,
    other models to use during computation
  • Execute
  • clouds, clusters, HPC systems, workstations.

GCOM CI Architecture
How Pylons GSI works
  • Snippets of code used for myproxy-login and job
    submission are given for
  • the router
  • the controller, and the template
  • the first box, the routing mappings are shown
    (lines 4 5).
  • Note also that there are two URLs for signing
    into the portal.
  • This facilitates virtual URLs with the web app.
  • The web in Fig. 3 shows authentication the
    TeraGrid MyProxy
  • service and the resultant credential obtained.
  • The code below is from the code mapping
    authentication requests to controller modules

Developing portal components Examples
  • Code is shown for
  • router - shows how the URLs and/or URIs map to
    the controller.
  • controller - maps the request to an action, which
    can be
  • a template script
  • A remote service (or a script to call a remote
  • database calls.
  • design point controller script should not do
    much other than hand off requests from the router
    and return responses back to the router.
  • template - script that gets called by the
    controller to build the HTML view.
  • Examples are for
  • myproxy credential retrieval
  • simple job submission

Example MyProxy Authentication
  • WebApp host must have gsissh working
  • Portal executes a local Python script that
    connects to the MyProxy service.
  • This provides an added layer of security as all
    scripts live outside the hosted Web file space.
  • Uses the Expect module, which operates similar to
    the PerlExpect module.
  • Note no need to use a Web Service (avoid
    latencies when possible).
  • Once the certificate is retrieved
  • it is stored locally (either in database or on
  • location information is stored as a session
  • can be used by any other modules.
  • All authorized components have access to the GSI
  • Tested myproxy-logon, grid-cert-info, and

MyProxy is used for all actions, including web
pages, URL redirects, etc.
  • map.connect(map1','/auth',controller'auth',
    action'index' )
  • map.connect('map2','/auth/signin',controller'auth
    ',action'signin' )
  • map.connect('map3','/auth/login',
    controller'auth', action'signin' )
  • map.connect('map4','/auth/myproxy_logon',controlle
  • action'myproxy_logon' )
  • map.connect('map5','/auth/myproxy_logon_action',co
  • action'myproxy_logon_action
  • Example above shows how maps
    authentication requests to controller modules
  • URL would look like http//acel.dnsalias.

MyProxy controller -
  • def index(self)
  • ..
  • def myproxy_logon(self)
  • return render('/auth/myproxy_logon.mako')
  • def myproxy_logon_action(self)
  • import pexpect
  • c.request request.params
  • c.request_params request.params
  • username request.paramsR-'myproxy_username'
  • assign other form parameters
  • child pexpect.spawn( myproxy_cmd )
  • try
  • ichild.expect(pexpect.TIMEOUT,err10,
  • except Exception, e
  • return render('/auth/myproxy_logon.mako'
  • if i success
  • child.sendline(password)
  • try
  • jchild.expect(R-str2, str3, errstr1,
    errstr8, errstr9
  • Depending on design- can be multiple controllers
  • the pexpect commands used to connect to remote
    myproxy server.
  • In the first part of the code, form variables
    obtained from WSGI used to build command shown
  • /usr/local/globus-4.0.6/bin/myproxy-logon -T - -p 7512 -l username
  • Last line after command is executed, render
    calls the template code to build HTML page.

MyProxy template - myproxy_logon.mako
ltinherit file"/layout.mako"/gt lth3gtGCOM Grid
Certificate Proxy lt/h3gt ltform action"/auth/myprox
y_logon_action" method"post"gt ltpgtTeraGrid
Portal Username ltinput type"text"
name"myproxy_username"gt ltpgtCertificate
Passphraselt/tdgt ltinput type"password"
name"myproxy_password" gt ltpgtMyProxy Server
ltinput type"text" name"myproxy_hostname"
value""gt ltinput
type"text" name"myproxy_port"
value"7512"gt ltinput type"submit"
name"authform" /gt lt/formgt
  • Web page built using Mako template system
  • invoked in the last line of the controller.
  • Mako is also object oriented, and template
    scripts can inherit from other scripts.
  • accomplished in line (1), where the template
    inherits the overall page style, header, and
    footer from layout.mako.

GSI enabled Job Execution
GSI Job Submission
controller'exec', action' testscript '
) map.connect('mapping304', '/exec/testscript_acti
on', controller'exec', action'
testscript ' )
  • Just need to add actions to the code
  • Maps actions to the controller script
  • Note that two URLs connect to the same

GSI Job controller -
def scriptst(self) return render('/exec/scrip
tst.mako') def scriptst_action(self) username
request.params'username' hostname
request.params'hostname' script
request.params'script' cmd script must
be first cmd cmd ' hostname'hostname
cmd cmd ' username'username fi,foe
os.popen4( cmd, mode't' ) results
foe.readlines() c.command cmd c.results
results return render('/exec/scriptst.mako')
  • Snippets of the controller code in
  • In the action method, the form elements are
    collected and built into a command argument list
  • then passed to local script
  • design allows the developer to test with any type
    of language or script (Unix, Python, Java, etc.).
  • Again, we dont use WSs unless needed

GSI Job template -
  • nnnn

Basic pyGate Portal
Google Application Example
  • To host a service, we used paster contoller
    newcontroler command
  • create a controller
  • for this example, we used http//my_url/pySGScript
  • the request must be added to the file
  • a controller must be configured.
  • Pylons WSGI framework will return the correct XML
    document for Google.

Pylons Portal as a Google Application
  • Code snippet used to host portal as a Google
  • lt?xml version"1.0" encoding"UTF-8" ?gt
  • ltModulegt
  • ltModulePrefs title"pyGate Demo Portal"
  • scrolling"true height"500"/gt
  • ltContent type"url" href"https//myurl5000/"gt
  • lt/Contentgt
  • lt/Modulegt
  • Put this XML on some URL.
  • Hosted website at my machine
  • Works Well
  • All GSI functions worked well.
  • Need to have software sense client.

Results are for a GCOM Job running on TeraGrid
NCSA host.
  • Portal and Web Services based on Pylons Web
  • WSGI (WS-Gateway Interface)
  • AJAX (asynchronous JavaScript and XML)
  • Use any Python tools/libraries
  • GSI MyProxy, GT4, Globus Core Python Toolkit,
    pyWSRF pyGlobus, gsissh, etc.
  • Features
  • dynamic hosting - code-base changes sensed and
  • URL routing to hide hosting details, pretty
  • Paste deployment (like Mako) so easy to package
    and distribute
  • Uses templates for layout of HTML
  • Hosting of portals and web services in same

Future Plans
  • SDSU pyGate Framework
  • Bundle a demo portal
  • OGCE incubator project
  • Emulate additional OGCE capabilities
  • Software release package scheduled to be
    available for friendly users in December
  • Applications
  • GCOM
  • CyberCheqs (NSF CI-TEAM
  • Others
  • New components
  • file browser
  • job monitoring
  • simple data visualization
  • Web 2.0
  • pyjamas/Google Web toolkit for desktop
  • creating deployable eggs
  • Deploy pylons services on cloud computing
  • investigate hosting GSI enabled secure web

Contact Information
  • Mary Thomas
  • Advanced Computing Environments Lab
  • Pylons Web App Framework
  • Google Code pages
  • Teragrid Gateways
  • http//