Using the Pylons Web Framework for Science Gateways - PowerPoint PPT Presentation

Loading...

PPT – Using the Pylons Web Framework for Science Gateways PowerPoint presentation | free to view - id: 1ac7bd-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Using the Pylons Web Framework for Science Gateways

Description:

GCOM solves a 3-D, time-dependent, curvilinear Navier-Stokes flow equations. ... adaptable curvilinear grid; fine grain structure. CI Environment: Portal: ... – PowerPoint PPT presentation

Number of Views:502
Avg rating:3.0/5.0
Slides: 41
Provided by: maryt159
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Using the Pylons Web Framework for Science Gateways


1
Using the Pylons Web Framework for Science
Gateways
  • Mary Thomas
  • Department of Computer Science
  • San Diego State University
  • Presented at the GCE08 Workshop
  • Supercomputing, November, 2008, Austin, Texas

2
Todays Presentation
  • Introduction, background, motivation
  • Description of the Pylons Web Application
    Framework
  • Demo Portal Architecture
  • Science Application General Coastal Ocean Model
  • Future Work

3
UPDATE!!! Challenge Proving Access to TeraGrid
Resources to Communities
  • NSF TeraGrid
  • Resource Providers 9 partners
  • 40 GigE backbone
  • 40 TFlops
  • Multi PByte storage

4
Science Gateways
  • Science gateways provide common, shared, simple
    interfaces to TeraGrid resources for large
    communities.
  • Community models/services access resources and
    services
  • Users avoid difficult and complex steps needed to
    get accounts, learn computer systems, etc.
  • Gateways are built using a variety of tools
    ranging from simple, customized tools to
    comprehensive frameworks.
  • Age old problem capability vs. complexity
  • greater capabilities -- authentication, account
    management, interfaces, logging, etc.
  • more complexity - resulting in long learning
    curves.
  • We need to simplify development cycle

5
Open Grid Computing Environments (OGCE)
  • Funded by the National Science Foundation (NSF)
    to facilitate science gateway development
  • includes services, toolkits
  • Simple Java Portlet framework targeted for
    TeraGrid and cyberinfrastructure
  • Maven based easy install.
  • TeraGrid gateways using OGCE, including
  • Linked Environments for Atmospheric Discovery
    (LEAD),
  • TG Visualization Gateway,
  • Open Life Sciences Gateway(OLSG),
  • Virtual Laboratory for Earth and Planetary
    Materials (VLab)
  • Note other languages/frameworks used on TG.
  • OGCE lowers barrier for access to distributed
    environs.

6
Project Motivation
  • There are other languages/frameworks available in
    Java
  • GridSphere, WebSphere, Sakai
  • none provide the services of the Java based OGCE
  • portlet dev environment non-trivial.
  • Other language choices include Python, PHP, Ruby,
    etc.
  • all keeping pace with Web 2.0 technologies.
  • Early grid portals were written using scripting
    languages
  • GridPort (Perl5/CGI), Clarens - simple method for
    building portals/gateways
  • note Perl 6 promises to be object oriented but
    it has been under development since 2000, and is
    not yet ready for public use.
  • rapid development cycles are still popular and
    needed.
  • Can we provide a gateway framework in another
    language that compliments OGCE?
  • Easy install, GSI integration
  • Technologies including Web 2.0/SOA/Component, WS
    client, etc.

7
TeraGrid Python-based Projects
  • UCSF Chimera - A visualization system 24,
    VisPort TeraGrid Visualization Web service.
  • National Virtual Observatory (NVO) - astronomy.
  • The Computational Infrastructure for Geodynamics
    (GIG) Science Gateway for the Geodynamics
    Community
  • Grid Analysis Environment (GAE).
  • Python Toolkits
  • Chimera is available as an extensible download,
    but is only for chemical modeling.
  • Clarens is Django-based extensible toolkit
  • Two of the projects listed above are built using
    Clarens (GIG, GAE).

8
A Word About Frameworks
  • Frameworks are collections of components that are
    integrated and form the foundation for an
    application and provide the support needed by a
    Web application to do its job.
  • application runs inside the context of the
    framework
  • required for modern Web apps because of the
    number and complexity of components, services,
    data, interoperability, etc.
  • framework includes clients and services,
    databases, templating/view, etc.
  • The list of projects that are labeled Web
    application frameworks is long - from Wikipedia
    article
  • 17 server-side languages
  • over 160 frameworks, including 36 in Java, 8 in
    Perl, 54 in PHP, and 14 in Python.
  • the capabilities provided by these frameworks are
    diverse, and choosing the right framework for a
    gateway project easily becomes a complex task.

9
Advantages of Python
  • A scripting language
  • dynamically compiles on first execution/changes
  • unlike other languages (e.g. Perl) that reload
    every call.
  • Natively object-oriented, with a large number of
    modules available.
  • It well known as a scientific computing language
  • has many numerical scientific computing tools
    including SciPy, NumPy,
  • Visualization tools NCAR (National Center for
    Atmospheric Research) PyNGL and PyNIO plotting
    routines.
  • Several portal web application frameworks to
    consider.

10
Another Advantage Python Web Services Gateway
Interface (WSGI)
  • A W3C specification for web and application
    servers to communicate and interoperate with
    other web applications and servers.
  • WSGI functions in the same way as the servlet
    architecture used by Java web applications, GCI
    for Perl, etc.
  • Interoperable similar to JSR 168 for portlets
  • allows external components to be integrated, so
    can customize the framework
  • E.g. adding templates or AJAX libraries
  • Applications written with one framework
    can/should be able to run on any server
    supporting the common APIs and libraries.
  • This narrows our choices frameworks python
    WSGI

11
Simple Choice Python has GSI Modules
  • In addition to being a good programming
    environment, GSI connectivity is already solved.
  • GT4 includes several Globus Python Core release
    modules
  • pyGlobus provides an interface to pre-web
    service versions of Globus
  • pyGridWare Python implementation of the Web
    Services Resource Framework (WSRF) and WS-N
    protocol families
  • pyGSI provides Python support for Grid security
    including Grid style proxy certificates through
    command line tools and APIs.
  • But, can we identify a framework that integrates
    GSI tools?
  • Not so easy

12
Candidate Python Frameworks
  • Evaluated several frameworks
  • Plone, Zope, Django, TurboGears, Pylons
  • Both in the research lab and in teaching courses
    at SDSU
  • All have typical portal services accounts,
    database, session history, etc.
  • Most designed to be content management systemS
  • CMSs come with complex data bases which are not
    needed for grid portals
  • difficult to remove or work around these
    features
  • cumbersome when adapting to Grid services
  • Most are old by Web standards,
  • have absorbed Web 2.0 technologies at late stages
  • re-factored to support WSGI (Web Server Gateway
    Interface)
  • Motivations for a choice
  • can we integrate existing python GSI tools or
    must we write our own?
  • since many TG researchers are faculty, can
    students figure out how to use/modify the
    framework?

13
Pylons Web Application Framework
  • Pylons designed from ground up to use WSGI
  • The Pylons model-view-controller (MVC)
    request-response architecture
  • Model contains the data that the application
    works with.
  • models do not contain any information on the
    meaning of this data.
  • often the model refers to database tables.
  • View reads the data from the model and displays
    it to the user.
  • for Pylons this is the template - part of the
    software system used to build Web pages.
  • Controller manages the logic of the application.
  • activates views to display data to the user or
    gets information from the user and stores them to
    the models.
  • Any number of components and libraries developed
    by other projects can be (and are) incorporated
    into the system
  • each of these components is kept separate
  • you can just use the component interface in your
    Python script.

14
More Pylons Advantages
  • Natively WSGI
  • Designed to use Web2.0/WSGIfrom the beginning
  • It has a very light basic installation because it
    is not inherently designed to be a content
    management system (CMS).
  • Uses Python eggs which are like Maven
    installations.
  • Directory structure is simple and flexible and
    reflects MVC architecture clearly (next slide)
  • For each new function, you write/modify a script
    for M, V, and C
  • Compiles on the flymost changes show up
    immediately
  • Other systems are moving to revised architectures
    reflecting the success of the approach of Pylons
    as well.
  • TurboGears 2.0 is now being built on Pylons.
  • Django considering.

15
Pylons Architecture
  • The basic operation of the Pylon framework
  • (1) the user selects a new page, (e.g. job
    execution)
  • (2) the Pylons router script is then read to map
    the request to an action (controller) script
  • (3) the controller script selects the correct
    module, and executes the block of code which may
    include local or external calls to other
    applications, connections to the grid, etc.
  • (4) at end of block, the controller calls a
    script to render the HTML(template) based on data
    passed to the template script
  • (5)when done, this HTML is returned back along
    the script chain to the router which serves the
    HTML data to the right URL.

16
Key Features Security/Authentication
  • Security/Authentication optional to use
  • Uses AuthKit module to log the user into the
    portal
  • complete authentication and authorization
    framework for WSGI applications
  • written specifically for Pylons to manage
    authentication tasks
  • but, can be used with any WSGI application
  • Maintains session history.
  • Once a user is validated, authentication
    information is visible to all modules and
    components being used by the portal.
  • access can be restricted based on roles (user,
    group, etc.).
  • Pylons supports a number of SQL databases
  • Optional use portal/WS can run without a
    database
  • Key Question can we integrate GSI?
  • Yes, or this talk would be about another language.

17
Key Features Routing
  • Routing/Beautiful URLs
  • Typically, a URL for a Web portal has an ugly
    construct
  • E.g., the URL used to see weather in Pierrevert,
    France is
  • http//weather.msn.com/local.aspx?wealocationswc
    2014
  • Routes is the Python implementation of the Rails
    routes system for mapping URL's to
    Controllers/Actions and generating URL's
  • Routes makes it easy to create pretty and
    concise RESTful (REpresentational State
    Transfer.) URLs.
  • This means that the ugly URL above could look
    like
  • http//weather.msn.com/France/PierreVert
  • In the Pylons M-V-C architecture, controller maps
    pretty URL to an action and script or code.
  • Will show example later in talk.

18
Key Features SQLAlchemy
  • Pylons integrates SQLAlchemy Database
  • SQLAlchemy is a Python SQL toolkit and Object
    Relational Mapper (ORM)
  • ORMs map the database structure to objects.
  • For example, the SQL command
  • SELECT users.id, users.name, users.fullname
  • FROM users
  • Is replaced by the single SQLAlchemy command
  • s select(users)
  • The ability to access the entire database using
    object access methods simplifies the programming
    complexity
  • Although not necessarily the database design
  • Note Pylons supports many databases, you dont
    have to use SQLAlchemy.

19
Key Features Simple Deployment
  • Simple Deployment
  • New Pylons projects are built using Paste.
  • Web development and application installation tool
    similar to Ant or Make
  • Paste creates python middleware needed for
    Pylons
  • router, config, and mapper template files
  • python webserver that can process WSGI requests
  • directory structure needed for a full site (to be
    populated by the developer) - reflects M-V-C
  • basic files, and data to support a new portal.
  • A single command installs the application
    template
  • paster create -templatepylons pyGateSite
  • The portal server is started using the following
    command
  • paster serve reload development.ini

20
Key Features Dynamic Update Interactive
Debugger
  • Dynamic Update most important feature for dev
  • Pylons reload feature supports dynamic class
    loading for all MVC layers
  • modifications to the code cause compiler to
    recompile the binary and re-load it into the
    server while keeping the server live
  • substantially reduces development cycles.
  • Note Portlet frameworks do not support this
    capability.
  • Interactive Debugger
  • all Pylons projects inherit the ErrorHandler
    middleware
  • catch any exception, present information on a Web
    page
  • also has log files.

21
Key Features Web 2.0 Interfaces
  • Pylons creates RESTful web services using the
    paster script command restcontroller option. This
    instantiates a RestController class that is
    automatically populated with the requisite
    skeletal actions.
  • Google Code
  • Appenginemonkey interfaces to the Google
    Application Engine
  • PyWebKitGtk provides an API for developers to
    program WebKit/Gtk
  • Pyjamas for Google Web Toolkit
  • iGoogle style Web apps easy to host (example
    shown later)
  • Desktop Applications
  • pyjamasd converts any Pylons project to desktop
    applications
  • JavaScript support for AJAX/Template libraries
    such as the Yahoo! UI Library (YUI).

22
Developing Pylons Gateway Portal
  • Once we decided on the Python Web app framework,
    it was time to build a portal.
  • Initial efforts focused on exploring what is
    needed to connect to GSI services/TeraGrid.
  • Set up some experiments to develop a basic suite
    of portal components
  • Accounts
  • GSI authentication - MyProxy
  • Job submission GSI
  • Web Service clients (e.g. GPIR)
  • Web 2.0 Could we also develop a portal that
    could be accessed as a Google Gadget/Facebook
    app, etc.?
  • The tools were out there, but could we use GSI?
  • Real World Applications - two projects right now
  • The General Coastal Ocean Model (GCOM) lt-- main
    focus
  • The CyberCHEQS Chemical Equilibrium Model
    (CI-TEAM)

23
Test Application GCOM
  • GCOM solves a 3-D, time-dependent, curvilinear
    Navier-Stokes flow equations.
  • simulate stratified flows over uneven terrains
  • adaptable curvilinear grid fine grain structure
  • CI Environment
  • Portal Community Model
  • Simulations run by educators, coastal managers
    and government agencies
  • integrate into climate simulators.
  • Web Services
  • Client to vast numbers of ocean data
  • Host services for analyzed data, simulations,
    other models to use during computation
  • Execute
  • clouds, clusters, HPC systems, workstations.

24
GCOM CI Architecture
25
How Pylons GSI works
  • Snippets of code used for myproxy-login and job
    submission are given for
  • the router
  • the controller, and the template
  • the first box, the routing mappings are shown
    (lines 4 5).
  • Note also that there are two URLs for signing
    into the portal.
  • This facilitates virtual URLs with the web app.
  • The web in Fig. 3 shows authentication the
    TeraGrid MyProxy
  • service and the resultant credential obtained.
  • The code below is from the router.py code mapping
    authentication requests to controller modules

26
Developing portal components Examples
  • Code is shown for
  • router - shows how the URLs and/or URIs map to
    the controller.
  • controller - maps the request to an action, which
    can be
  • a template script
  • A remote service (or a script to call a remote
    service)
  • database calls.
  • design point controller script should not do
    much other than hand off requests from the router
    and return responses back to the router.
  • template - script that gets called by the
    controller to build the HTML view.
  • Examples are for
  • myproxy credential retrieval
  • simple job submission

27
Example MyProxy Authentication
  • WebApp host must have gsissh working
  • Portal executes a local Python script that
    connects to the MyProxy service.
  • This provides an added layer of security as all
    scripts live outside the hosted Web file space.
  • Uses the Expect module, which operates similar to
    the PerlExpect module.
  • Note no need to use a Web Service (avoid
    latencies when possible).
  • Once the certificate is retrieved
  • it is stored locally (either in database or on
    disk)
  • location information is stored as a session
    object
  • can be used by any other modules.
  • All authorized components have access to the GSI
    credential.
  • Tested myproxy-logon, grid-cert-info, and
    myproxy-init.

28
MyProxy router.py
Router.py is used for all actions, including web
pages, URL redirects, etc.
  • map.connect(map1','/auth',controller'auth',
    action'index' )
  • map.connect('map2','/auth/signin',controller'auth
    ',action'signin' )
  • map.connect('map3','/auth/login',
    controller'auth', action'signin' )
  • map.connect('map4','/auth/myproxy_logon',controlle
    r'auth',
  • action'myproxy_logon' )
  • map.connect('map5','/auth/myproxy_logon_action',co
    ntroller'auth',
  • action'myproxy_logon_action
    ')
  • Example above shows how router.py maps
    authentication requests to controller modules
  • URL would look like http//acel.dnsalias.
    net/gcom/auth/login

29
MyProxy controller - signon.py
  • def index(self)
  • ..
  • def myproxy_logon(self)
  • return render('/auth/myproxy_logon.mako')
  • def myproxy_logon_action(self)
  • import pexpect
  • c.request request.params
  • c.request_params request.params
  • username request.paramsR-'myproxy_username'
  • assign other form parameters
  • child pexpect.spawn( myproxy_cmd )
  • try
  • ichild.expect(pexpect.TIMEOUT,err10,
    success)
  • except Exception, e
  • return render('/auth/myproxy_logon.mako'
    )
  • if i success
  • child.sendline(password)
  • try
  • jchild.expect(R-str2, str3, errstr1,
    errstr8, errstr9
  • Depending on design- can be multiple controllers
    -
  • the pexpect commands used to connect to remote
    myproxy server.
  • In the first part of the code, form variables
    obtained from WSGI used to build command shown
    below
  • /usr/local/globus-4.0.6/bin/myproxy-logon -T -
    myproxy.teragrid.org -p 7512 -l username
  • Last line after command is executed, render
    calls the template code to build HTML page.

30
MyProxy template - myproxy_logon.mako
ltinherit file"/layout.mako"/gt lth3gtGCOM Grid
Certificate Proxy lt/h3gt ltform action"/auth/myprox
y_logon_action" method"post"gt ltpgtTeraGrid
Portal Username ltinput type"text"
name"myproxy_username"gt ltpgtCertificate
Passphraselt/tdgt ltinput type"password"
name"myproxy_password" gt ltpgtMyProxy Server
ltinput type"text" name"myproxy_hostname"
value"myproxy.teragrid.org"gt ltinput
type"text" name"myproxy_port"
value"7512"gt ltinput type"submit"
name"authform" /gt lt/formgt
  • Web page built using Mako template system
  • invoked in the last line of the controller.
  • Mako is also object oriented, and template
    scripts can inherit from other scripts.
  • accomplished in line (1), where the template
    inherits the overall page style, header, and
    footer from layout.mako.

31
GSI enabled Job Execution
32
GSI Job Submission router.py
map.connect('mapping303','/exec/testscript',
controller'exec', action' testscript '
) map.connect('mapping304', '/exec/testscript_acti
on', controller'exec', action'
testscript ' )
  • Just need to add actions to the router.py code
  • Maps actions to the controller script
  • exec.py
  • Note that two URLs connect to the same
    controller

33
GSI Job controller - exec.py
def scriptst(self) return render('/exec/scrip
tst.mako') def scriptst_action(self) username
request.params'username' hostname
request.params'hostname' script
request.params'script' cmd script must
be first cmd cmd ' hostname'hostname
cmd cmd ' username'username fi,foe
os.popen4( cmd, mode't' ) results
foe.readlines() c.command cmd c.results
results return render('/exec/scriptst.mako')
  • Snippets of the controller code in exec.py
  • In the action method, the form elements are
    collected and built into a command argument list
  • then passed to local script
  • design allows the developer to test with any type
    of language or script (Unix, Python, Java, etc.).
  • Again, we dont use WSs unless needed

34
GSI Job template -
nnnn
  • nnnn

35
Basic pyGate Portal
36
Google Application Example
  • To host a service, we used paster contoller
    newcontroler command
  • create a controller
  • for this example, we used http//my_url/pySGScript
  • the request must be added to the routing.py file
    and
  • a controller must be configured.
  • Pylons WSGI framework will return the correct XML
    document for Google.

37
Pylons Portal as a Google Application
  • Code snippet used to host portal as a Google
    Gadget
  • lt?xml version"1.0" encoding"UTF-8" ?gt
  • ltModulegt
  • ltModulePrefs title"pyGate Demo Portal"
  • scrolling"true height"500"/gt
  • ltContent type"url" href"https//myurl5000/"gt
  • lt/Contentgt
  • lt/Modulegt
  • Put this XML on some URL.
  • Hosted website at my machine
  • Works Well
  • All GSI functions worked well.
  • Need to have software sense client.

Results are for a GCOM Job running on TeraGrid
NCSA host.
38
Summary
  • Portal and Web Services based on Pylons Web
    Framework
  • WSGI (WS-Gateway Interface)
  • AJAX (asynchronous JavaScript and XML)
  • Use any Python tools/libraries
  • GSI MyProxy, GT4, Globus Core Python Toolkit,
    pyWSRF pyGlobus, gsissh, etc.
  • Features
  • dynamic hosting - code-base changes sensed and
    reloaded
  • URL routing to hide hosting details, pretty
    URLS
  • Paste deployment (like Mako) so easy to package
    and distribute
  • Uses templates for layout of HTML
  • Hosting of portals and web services in same
    system

39
Future Plans
  • SDSU pyGate Framework
  • Bundle a demo portal
  • OGCE incubator project
  • Emulate additional OGCE capabilities
  • Software release package scheduled to be
    available for friendly users in December
  • Applications
  • GCOM
  • CyberCheqs (NSF CI-TEAM
  • Others
  • New components
  • file browser
  • job monitoring
  • simple data visualization
  • Web 2.0
  • pyjamas/Google Web toolkit for desktop
    application
  • creating deployable eggs
  • Deploy pylons services on cloud computing
    resources
  • investigate hosting GSI enabled secure web
    services.

40
Contact Information
  • Mary Thomas mthomas_at_sciences.sdsu.edu
  • Advanced Computing Environments Lab
    http//acel.sdsu.edu
  • Pylons Web App Framework
  • Google Code pages
  • Teragrid Gateways
  • http//www.teragrid.org/gateways
About PowerShow.com