Condor Web Services

1
Condor Web Services BirdbathSOAP Functionality
for Condor

• Clovis Chapman c.chapman_at_cs.ucl.ac.uk
• Charaka Goonatilake c.goonatilake_at_cs.ucl.ac.uk

2
Condor Overview
Jobs (command-line)
Submission Machine (Schedules Jobs)
Central manager
Execution machines
3
Scenario Federating Condor pools
Remote client
Condor pool A
Local Scheduler
Q
Central manager
4
Scenario Federating Condor pools
Remote client
Condor pool A
Condor pool B
Local Scheduler
Local Scheduler
Q
Q
Central manager
Central Manager
5
Motivations

• Facilitate the development of third-party tools
  and applications capable of interacting with
  Condor remotely.
• E.g. Build higher-level application specific
  scheduler that submits jobs to one or more Condor
  pools according to application semantics
• Seamless integration of Condor resources in Web
  Grid Service applications
• Build Web Service support within the Condor
  architecture itself
• i.e. Rather than rely on external wrappers
  (e.g. Globus Toolkit)
• Expose a wide range of high-throughput computing
  services based on Condor capabilities (e.g.
  workload management, scheduling, checkpointing,
  job migration, etc.)

5
6
Why SOAP? Alternatives

• Building wrappers around command line tools
• Globus Toolkit 3
• Good when interacting with different types of
  resource managers. Otherwise Birdbath has a
  richer, more scalable and streamlined interface
  which is more suited to Condor.
• Condors GAHP protocol ASCII stream-based
  protocol.
• Built within Condor itself. Simple Condor
  specific protocol for remote submission
• No Type Safety / interface declaration / session
  management.
• DRMAA API built-in support
• API for the submission and control of jobs to one
  or more Distributed Resource Management Systems
  defined by the GGF.
• Allows access to Condor through Local procedure
  calls
• C binding only / Weak fault tolerance

7
Condor Architecture overview
Collector
Startd
7
8
Condor Architecture overview
Collector
Negotiator
Startd
Schedd
8
9
Condor SOAP extensions
Obtain resource ClassAds Obtain all daemon
ClassAds
Start transaction Submit Job Obtain Job ClassAds
Get/Send file Cancel/hold/release job
Execution machine
Execution machine
Submission machine(s)
Central manager
Schedd
Collector
Negotiator
Execution machine
Execution machine
9
10
Clients for the Condor Web Services
Custom Client holdJob()
Submission machine(s)
SOAP library
Schedd

• All libraries can be generated (automatically)
  based on interface description provided by us
  (WSDL)
• The WSDL interface defines which operations can
  be called and their parameters (e.g.
  queryAds(Constraint))

11
Quick application example Collector Query

• Using Axis API.
• Generate stubs from WSDL file using WSDL2Java
• e.g. Obtain information about all resources that
  have over 512Mb RAM
• locator new CondorLocator()
• collector locator.getCollector(new
  URL(http//machineport))
• ads collector.QueryStartdAds(Memorygt512)

11
12
Quick Application example Job Submission

• Transaction based process
• New Transaction()
• New Job ID ()
• Send Files (input files, binaries)
• Send Job Description(ClassAd)
• Commit Transaction()
• Can only commit if everything else has succeeded
• File transfer is in segments (specified by users)
  using simple protocol
• Can also submit DAG-Man jobs

13
Collaboration with OMII GridSAM plugin

• Web Service based job submission service
• Uses JSDL (Job Submission Description Language)
  for standardized job submission to a wide range
  of resource managers

Middle-tier
GridSAM
SGE Plug-in
JSDL request
Submission machine(s)
Condor ClassAd
Condor WS plug-in
Schedd
GridSAM Service can now be on a different host
14
Still to be done

• Security (SSL for now)
• Certificate based security
• Using mappings to local accounts
• Exposing more components (e.g. shadow for remote
  checkpointing and retrieval of partial output
  files)
• Booking resources (pre-allocation) / Flocking
  through redirection and recommendation / etc

14
15
Download

• http//www.cs.wisc.edu/condor/birdbath

15