The%20Bologna%20Batch%20System:%20Flexible%20Policy%20with%20Condor

1
The Bologna Batch System Flexible Policy with
Condor
2
The Bologna Batch System

• Custom batch scheduling system for local users at
  INFN in Bologna, Italy.
• Istituto Nazionale di Fisica Nucleare
• Dr. Paolo Mazzanti initiated the idea.
• Implement on a small subset of machines within
  the larger nationwide INFN Condor pool
• INFN Condor Pool 300 CPUs
• INFN-Bologna Condor 100 CPUs
• Bologna Batch System 50 CPUs

3
The Bologna Batch System

• Paolo wanted two things
• Resources to remain in the larger INFN Condor
  pool
• But local jobs prioritized and managed specially
  differently than elsewhere at INFN.
• Local control over policies of local resources --
  without abandoning commitment to a shared
  resource pool.

4
Where We Started

• Basic Condor Policy
• Opportunistic resources
• Jobs only run when machines are otherwise idle
• Jobs can be preempted for machine owners or
  higher-priority users
• Fair-share across INFN pool
• Highest priority user in the pool gets first
  crack at a given resource
• The more you use, the worse your priority becomes
• Some problems
• Long-running vanilla jobs (with no checkpointing)
  were frequently preempted before running to
  completion
• Users dislike waiting for a resource if they only
  want to run a short job
• High-priority users from other INFN sites running
  on local resources while lower-priority local
  users wait.

5
BBS Policy Requirements

• Prioritize local work
• Share resources, but run outside jobs as backfill
• Treat local servers as dedicated resources for
  local jobs, but opportunistic resources for
  other jobs.
• Run outside Condor jobs only if the server is
  idle.
• Run local batch jobs regardless of other system
  load or console activity.
• Preempt outside Condor jobs to allow local batch
  jobs to run, but dont preempt local jobs for
  outside work.

6
BBS Policy Requirements

• Ensure resource availability for both short and
  long-running jobs
• Prioritize short batch jobs so that they are
  never kept waiting by long batch jobs.
• Prevent long batch jobs from being preempted or
  starved by short jobs.
• Never waste resources
• No idle CPUs when jobs are waiting to run!
• No preemption of vanilla jobs!
• Preemption ideal if you can checkpoint, but here
  we cant

7
A Contradiction!

• No way to guarantee resource availability for
  short or long jobs without reserving some CPUs
  for each
• ...But no way to avoid idle CPUs without allowing
  them to start any kind of job
• If CPUs reserved for short jobs are used for long
  jobs, they become unavailable to run short jobs.
• If CPUs reserved for short jobs are not used for
  long jobs, theyre being wasted when there are no
  short jobs to run.
• What to do, what to do

8
A Solution!

• Allow resources to be temporarily overcommitted
• We treat one CPU as two
• On a two-CPU machine, define four Condor VMs
  (virtual machines) two for short jobs and two
  for long jobs.
• Allow jobs to be suspended rather than preempted
• Think of as checkpointing to swap
• OR allow jobs to be de-prioritized temporarily
• If memory is adequate, allow suspended long
  jobs to continue running at a poor OS priority
  and steal cycles whenever active short jobs are
  busy doing I/O.

9
Everybody wins!

• Short jobs start right away on dedicated short
  VMs
• Long jobs arent preempted by short jobs, but
  rather suspend temporarily or run at a lower
  priority.
• Outside jobs run only when no Bologna jobs
  waiting.
• All CPUs available to all types of jobs.
• No idle CPUs when jobs are waiting.

10
Okay, how?

• Flipside of flexibility is complexity!
• Its pretty cool that Condor allows you to
  combine dedicated and opportunistic scheduling in
  one system, but it takes a bit of work to get it
  all set up
• It took the world-experts in writing Condor
  policy expressions (thats us) many weeks of
  concerted effort to get this all working
  smoothly.
• Luckily for yall, weve already done the hard
  part, and now you can copy it. ?

11
Copy it from where?

• Bologna Batch System document
• http//www.cs.wisc.edu/pfc/bbs.doc
• A detailed walk-through of the specific policies
  and the necessary Condor configuration to make
  each one work.
• Line by line examples of how we implemented each.
• A work in progress.
• Just this February, we added material.
• Open to contributions what clever things have
  you done?
• Whats in it? Lets take a look

12
Advanced Policies In the BBS Document

• How to mark certain jobs as being of one type or
  another.
• (And how to verify jobs are what they say they
  are.)
• How to mark certain machines as being of one type
  or another.
• How to configure multiple Condor VMs on a single
  CPU.
• How to implement different policies for different
  VMs on the same machine.
• New how to make one VM aware of whats running
  on another VM

13
Simple for Users

• Although policy is complicated, the interface for
  users is kept simple
• Users call bbs_submit_long or bbs_submit_short,
  just as they would condor_submit
• Short jobs start quickly, but those that run for
  gt1 hour are killed.
• Long jobs will run to completion...
• bbs_submit_ scripts automatically add the
  appropriate classad attributes to the job to take
  advantage of the long or short running VMs on
  Bologna Batch Servers.

14
What About COD?

• Condors Computing On Demand
• Designed to solve a similar problem
• short-lived interactive computations that need to
  run immediately, without preempting longer batch
  jobs.
• Some differences
• COD focused on truly interactive work, not batch
  computation
• BBS focused on local control of a shared
  resource, and balancing the needs of short and
  long-running batch jobs
• COD tasks not represented as jobs in a queue,
  accounted for resource usage, etc. they just
  run.
• COD is more of a instant Condor rsh
• BBS preserves normal Condor job management
  structure jobs are submitted like always,
  accounted for in the same way, etc.

15
Any Questions?

• Ask me now...
• Feel free to email me at pfc_at_cs.wisc.edu.
• Check the Bologna Batch System document at
  http//www.cs.wisc.edu/pfc/bbs.doc
• Thanks!