Birds of a Feather: Enterprise Condor Use Cases

1
Birds of a FeatherEnterprise Condor Use Cases

• Jason Stowe

2
Condor In Enterprises
3
Experience in Production Environments
4
What is an Enterprise Environment?
5
Companies Only?
6
Any Organization Using Condor with
7
Demanding Users
8
Organization gtGroups of Demanding Users
9
Purchased Computer Capacity
10
Need
11
Computation
12
Done.
13
Easily.
14
On a Deadline.
15
In-House, Third Party Applications without
modification
16
in order to do their jobs
17
Not just companies
18
Includes Research/Govt/Academic environments
19
A valid definition ofEnterprise Condor Users?
20
I started off as a Demanding Condor User
21
Condor Week 2005
22
Condor for Movie Pipelines
23
Condor Week 2006
24
Web-based Management Tools, Consulting, and 24/7
Support
25
A Conversation with Miron
26
Add new (Enterprise) Features to Condor
27
Condor Week 2007
28
Negotiation Features for Demanding Groups of Users
29
So weve defined Enterprise, but
30
Why Condor?
31
Summarized by Bob Nordlund
32
In 2007http//www.cs.wisc.edu/condor/CondorWeek2
007/presentations/nordlund_Condor_In_Flight.ppt
33
"Condor was selected... -Multi-OS
Support -Simple, Scalable, and Flexible -Active
Community -Free"
34
Multi-OS Support
35
Windows
36
Many Varieties of Linux
37
Solaris
38
And if youve got it, Mac OSX
39
Condor works with most anything Keeps users
options open
40
Easy to Download and Use
41
Run any executable as a job
42
Makes parameter sweeps, monte-carlo simulations
easy
43
Scalable
44
Many Facilities with X000-X0000 condor VMs/slots
45
Flexible
46
Not just OS/Platform flexible
47
Start (NonCondorLoadAvg lt 0.5)
48
Control when jobs run where
49
Powerful, Command Lines, Developer APIs
(DRMAA/SOAP)
50
Condor Community
51
(No Transcript)
52
community
53
A group of people living under the same
government.
54
A group of people having common interests the
scientific community
55
Sharing, participation, and fellowship.
56
(No Transcript)
57
Support Sharing, participation, and
fellowship.
58
So thats why Condor in an Enterprise, but
59
How are they using it? Commercial Use Cases
60
Batch Monte-Carlo Computation with
Windows/Linux/Solaris
61
Embarrassingly Parallel
62
From Bob Nordlund of HartfordLife_at_CondorWeek
2007
63
http//www.cs.wisc.edu/condor/CondorWeek2007/prese
ntations/nordlund_Condor_In_Flight.ppt
64
Faced a Challenge -Prepare comprehensive risk
analysis report for all products for investment
analysts. -Run three orders of magnitude more
projections through the system. -Perform
stochastic on stochastic on stochastic
analysis. -2.4 billion individual simulations.
65
IT Responded Infrastructure -500 dual-core,
dual-CPU AMD servers -20 Racks -Dedicated GigE
switches -Condor pool with HAD -4 filers with
8TB of storage -Purchased, installed and
configured in 6 weeks Application -Java-based
liability model -75x performance
gain -Meta-scheduling application -In-memory
market scenario generation"
66
Specifically, it mentions Condor In production
since 2004 ... Dedicated and non-dedicated
execute nodes 1000 Two-socket, dual-core HP x86
servers 1000 desktops, notebooks Linux central
managers with HAD Linux and Windows
schedulers Windows execute nodes
67
Using the same Condor installation.
68
Condor scales
69
Without changing software,end-user training, etc.
70
Just added more hardware.
71
Workstation Pools Finance, Geospatial Imaging
72
Workstations are only utilized 1/3 of the day,
intermittently
73
Considerable Latent Compute Capacity
74
Perfect for Overnight Batch and Master/Worker
jobs
75
Effective Policies for Idle Workstations
76
As mentioned in WatersOnline
77
In 2007http//www.watersonline.com/public/showPa
ge.html?page468641
78
Some finance institutions have saved considerable

79
Running Pools of X000s Workstations
80
3rd Party Master Worker Jobs
81
MatLab, MoSes, TAS, etc.
82
Condor Can Manage Resources Worker Counts
83
So on a 100 machine pool, with applications A B
84
(No Transcript)
85
Lastly
86
Movies
87
CORE Feature Animation on Disney Picture, "The
Wild"
88
Condor for Movies75 Million Jobs 1000 CPUs
(Linux/OSX)70 TB storage
89
From my slide in 2005 Condor Week1000 2.8
GHz. Processors, Linux 4GB RAM70-100 Terabytes
Several Filers50 Million Renders so
far(Vanilla Universe)
90
Batch, with many, many
91
Groups of Demanding Users
92
But Thats just a few use cases
93
Other CondorWeek presenters Altera, Micron,
Oracle, etc.
94
You may have new ones
95
Talk with Condor Team.Talk with other users.
Talk with me.
96
Help the community continue to work well for
everyone.
97
Thank you NeSC. Questions?http//www.cyclecomp
uting.comjstowe _at_ cyclecomputing.com