The Agile Heartbeat - PowerPoint PPT Presentation

Loading...

PPT – The Agile Heartbeat PowerPoint presentation | free to view - id: 2065-MDI4Y



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

The Agile Heartbeat

Description:

Reduced build time from 3 hours to 30 minutes. Complete build and ... Last working build of the day becomes the nightly 'Broken builds are a thing of the past' ... – PowerPoint PPT presentation

Number of Views:74
Avg rating:3.0/5.0
Slides: 76
Provided by: CMPMed
Category:
Tags: agile | build | heartbeat

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: The Agile Heartbeat


1
The Agile Heartbeat
  • John Graham-Cumming
  • Co-founder
  • Electric Cloud, Inc.
  • Mountain View, CA
  • jgc_at_electric-cloud.com

2
Agenda
  • The death of the Nightly Build
  • Continuous Integration as seen by the build
    manager
  • The Agile Heartbeat
  • Stepping Stones to Continuous Integration
  • Tools that can help

3
The Death of the Nightly Build
4
Goodbye Nightly Build
  • Build is too long to run overnight
  • There are too many variants
  • There is no night round the clock
    development
  • The nightly build is always broken

5
Aside to Enterprise IT
  • Automated daily builds are one of the most
    widely disregarded best practices of software
    development. Most software vendors perform
    frequent, and often daily, builds, but for
    enterprise IT organizations, this is the
    exception, rather than the rule.
  • Forrester Research report Reining in Runaway
    Builds, February 2006.

6
Aside to Enterprise IT
  • Skip the Nightly Build phase altogether
  • Go straight to continuous integration
  • But if not, nightly builds are still a best
    practice

7
Long builds?
8
While waiting?
9
Real-world Build Trouble
  • 2007 survey by Osterman Research
  • 352 software companies
  • Only 9 said they dont have build problems

10
The Top 3
  • Managing Dependencies 50
  • Debugging broken builds 40
  • Cross-platform builds 29
  • Not going to talk about those
  • but some pointers

11
Managing Dependencies
  • Useful articles
  • Dependency ManagementDr Dobbs, April
    2006http//www.ddj.com/dept/architect/184406479
  • Tips and tricks of the automatic dependency
    generation masterCM Crossroads, October
    2006http//www.cmcrossroads.com/option,com_smf/It
    emid,180/topic,69672/

12
Debugging Broken Builds
  • Useful article
  • Debugging MakefilesDr Dobbs, March
    2007http//www.ddj.com/dept/debug/197003338

13
Cross-platform Builds
  • Useful article
  • Cross-platform BuildsDr Dobbs, January
    2005http//www.ddj.com/articles/2005/0501/

14
Infrastructure Problems
  • Builds take too long 31
  • Maintenance of scripts 30
  • Dispersed build systems 28
  • Need better reporting 26
  • Large volume of builds 24
  • Unclear build process 23
  • Developers cant do full builds 17

15
Continuous Integration
16
Continuous Integration
  • a software development practice where engineers
    integrate frequently, leading to multiple
    integrations per day. Each integration is
    verified by an automated build and test to detect
    integration errors as quickly as possible.
  • http//www.martinfowler.com/articles/continuousInt
    egration.html

17
Client Story Intuit
  • Intuit implemented Continuous Integration for the
    QuickBooks code base
  • Reduced build time from 3 hours to 30 minutes
  • Complete build and smoke test of latest sources
    every 30 minutes all day
  • Broken builds result in email to developers
  • Last working build of the day becomes the
    nightly
  • Broken builds are a thing of the past

18
The Five Elements
  • Continuous Integration requires
  • Automated builds
  • Self-testing builds
  • Fast builds
  • Every engineer commits every day
  • Every commit starts a build

19
Automate the Build
  • anyone should be able to bring a virgin
    machine, check the sources out of the repository,
    issue a single command, and have a running system
    on their machine

20
Automate the Build
  • Implications of the build command
  • Build location must be specifiable
  • Source location, where to put the binaries
  • Build variants must be specifiable
  • Debug or release, target platform/processor
  • Must set a suitable exit code
  • Must know when to halt because of errors
  • Needs to build the installer
  • Needs to be able to clean up
  • Needs to be able to install or verify the tool
    chain

21
IHTM Getting Acquired
  • I worked for a start-up that was acquired by A
    Big Co.
  • ABC didnt have automated builds, they had a team
    of harassed build starters
  • A week after we merged many of my engineers still
    couldnt build ABCs code on their machines
  • It Happened To Me

22
Automated Builds
  • Some important benefits
  • A new employee can be up and running in the time
    it takes to run build.
  • If build is stored in the SCM old software
    versions can be quickly rebuilt.
  • Anyone can be the build manager.

23
IHTM The Old Version
  • Working at Some British Co. had to patch a
    two-year old version of a driver for an
    Australian Bank.
  • No way to build the old code.
  • Ended up a delivering a patch containing an
    entirely new version.
  • Crossed our fingers that AB didnt figure it out.

24
Usmans Law
  • make clean doesnt
  • Achilles Heel of many automated builds systems
    is clean up doesnt work properly.
  • Essential that can return to virgin state
  • Must remove all generated files
  • Objects, code, temporary
  • Uninstall DLL, OCX, SO

25
Make Your Build Self-testing
  • any body of code must have a set of automatic
    tests that can check a large part of the code
    base for bugs
  • Could be a combination of
  • A smoke test
  • Developer written unit tests
  • Subset of full test suite selected by QA
  • Doesnt have to be the full test suite.

26
Everyone Commits Every DayEvery Commit Builds
  • Imagine 20 engineers working an 8 hour day
  • Each commits once, so 20 builds
  • One build and test every 24 minutes
  • More engineers means more builds
  • Ultimate goal is a build per commit
  • No commits unless last build is green

27
Per Commit Builds
  • Advantages of One Build Per Commit
  • If the build breaks the culprit is immediately
    known
  • The culprit still has the state of the code in
    their mind
  • Build breakages a fixed rapidly eliminating build
    crunches
  • A common goal is 10 minute builds

28
Espresso Builds
  • The ultimate Build Per Commit is the Espresso
    Build
  • Engineer commits code
  • Engineer types build
  • Engineers gets and drinks a coffee
  • Build and test results are ready

29
Keep the Build Fast
  • Extreme Programming has a goal of 10 minute
    builds
  • For most people thats unrealistic
  • But implementing CI means having fast builds, and
    the build speed will throttle the integration
    speed

30
Client Story LSI Logic
  • Software builds were taking 3 to 4 days (without
    testing)
  • Decided to be agile
  • Needed to build and test 50 unique products every
    day

31
Client Story LSI Logic
  • Reduced smoke test time to 1 hour through
    reengineering
  • Implemented parallel building to reduce a single
    build from 150m to 20m
  • Replaced home grown build system with commercial
    build automation tool
  • Can now build everything every day

32
The Agile Heartbeat
  • Status of the per-commit build indicates the
    health of the software project
  • With many commits per day and many builds per day
    its the heartbeat of the software team
  • The Agile Heartbeat is a useful single measure of
    progress

33
Stepping Stones to CI
34
Stepping Stones
  • Fully Automate
  • Fast Builds
  • Lava Lamps
  • Build and Test
  • Pre-Flight Builds

35
1. Fully Automate
  • Create the build command
  • Can probably be created without acquiring new
    tools
  • Use shell scripts, Perl, or other languages to
    wrap up existing scripts
  • Make sure it works from a clean machine all the
    way through installation with no intervention

36
1. Fully Automate
  • Build when SCM changes
  • A common practice is to monitor the SCM system
    for changes
  • Then wait for a quiet period (no changes) of 15
    minutes
  • Then launch a build using build

37
1. Fully Automate
  • A good opportunity to try out a Build Automation
    tool is to use it to just monitor SCM and kick
    off a build
  • Lets you try out the tool without implementing
    all of CI first

38
2. Fast Builds
  • Set a realistic goal
  • 10 minute build and test is unrealistic for most
    teams
  • Perhaps build and test in 1 hour

39
2. Fast Builds
  • Possible approaches
  • Upgrade old hardware (particularly slow disks)
  • Buy SMP hardware
  • Manually restructure build for parallel building
  • Use free parallel build tools (e.g. make -j,
    distcc)
  • Buy commercial parallel build tool (e.g.
    ElectricAccelerator)
  • Parallel builds will give the best speedups

40
Parallel Build Gotchas
  • make -j, distcc and other tools frequently break
    or produce bad builds
  • Thats not their fault!
  • Many build systems are not parallel safe
  • They created assuming a serial build

41
Parallel Build Gotchas
  • Missing dependencies
  • Assuming this happens before that without
    specifying that time dependency
  • Reuse of generated files
  • Example creating the same intermediate generated
    code file in different rules
  • Writing to a shared file
  • Forces the build to run serially

42
IHTM Poor Speedup
  • For most organizations Ive dealt with the
    maximum speedup they get from free tools is 2-4x
  • All of them break down as they push for greater
    and greater speedup when they hit a non parallel
    safe part of the build
  • At East Coast Co. GNU Make -j achieved a 2.5x
    speedup on a 32-processor machine, but a
    commercial tool produced 5.2x

43
Aside Amdahls Law
  • Maximum possible speedup in a parallel build
    is 1 / ( F ( 1 - F ) / P )
  • P is the number of processors available
  • F is the fraction of the build that has to run
    serially

44
Amdahls Law Example
  • A build lasts for 4 hours
  • The longest portion that must run on a single
    processor is creating the installer which takes
    10 minutes
  • The maximum possible speedup with 10 processors
    is 7.3x

45
Amdahls Law Graph
46
3. Lava Lamps
  • Create a build monitoring tool visible to
    everyone
  • e.g. Red and Green Lava Lamps indicating build
    status
  • http//www.artima.com/weblogs/viewpost.jsp?thread
    67492

47
3. Lava Lamps
  • Step 1s automatic build sets the build monitor
    status automatically.
  • Then make the build run as many times as
    possible.
  • Lava lamps become indication of software health.

48
3. Lava Lamps
  • Lava lamps are only practical for small teams
    with one build variant
  • Real world alternatives
  • Internal web page showing build status (make it
    each developers home page)
  • Flat panel display showing build status mounted
    high up
  • Email alerts when build is broken

49
Real World Status
50
Culprit Identification
  • When the build fails email people who last made
    changes
  • Before build starts interrogate SCM for names of
    users whove submitted since last build
  • With frequent, short builds the culprit list will
    be short

51
4. Build and Test
  • Integrate smoke test and/or unit tests with build
    script
  • Now lava lamps show build and test status
  • QA can also propose tests that should be included
    automatically

52
IHTM You Broke It, You Own It!
  • Working as a partner to Near Seattle, Co. was
    forced to own the build because I broke it.
  • The build manager floated from person to person.
    If you broke the build you owned it until the
    next person who broke it.
  • The current build breaker was indicated by a
    skull and crossbones flag on their cube.

53
5. Pre-Flight Builds
  • Put full builds in the hands of developers
  • Shared server resource for doing builds
  • Internal web page for launching builds
  • Branch or shared directory to build from
  • What to build and test
  • Who to email the results to
  • Queue of builds on FIFO basis

54
5. Pre-flight Builds
  • Problems to contend with
  • Long build queue
  • Disk space management
  • A new engineering rule
  • No commits until a pre-flight build has passed on
    the proposed changes

55
Client Story Qualcomm
  • 100s of different build configurations
  • Individual builds taking 2 hours
  • Implemented parallel builds to reduce build time
    to 15 minutes
  • Now we make incremental changes we might not
    have tried before simply because of the build
    time

56
Tools
57
Tools that can help
  • Three major categories
  • Build acceleration
  • Build automation
  • Build visualization
  • Wikipedia has a long page on CI
    toolshttp//en.wikipedia.org/wiki/Continuous_int
    egration

58
Build Accleration
  • Free accelerators
  • GNU Make -j and friends
  • distcc
  • ccache
  • Best known commercial tool is ElectricAccelerator

59
Build Acceleration
  • Some tools claim to speed up builds by running
    them regularly
  • Or by managing a group of machines that run many
    builds

60
Build Automation
  • Very crowded space
  • Wikipedia has extensive list of tools
  • Best known free tool is CruiseControl
  • Well known commercial tools are IBM Rational
    BuildForge and ElectricCommander

61
Build Automation
62
Build Automation
63
Build Automation
64
Build Visualization
  • Typical builds contain hundreds or thousands of
    steps
  • Dependencies are poorly understood
  • Parallel building adds a layer of complexity
  • Agile development means multiple builds per day

65
Build Visualization
66
Build Visualization
67
Build Visualization
68
Questions to ask
69
Questions to ask your vendor
  • Does the tool support scheduled builds?
  • What are the reporting and anaytical tools?
  • Does it support pre-flight and per-commit
    builds?
  • Does the tool support Identity Management?
  • Does the tool support multiple teams?

70
Does the tool scale?
  • Agile development means lots of fast builds
  • How does the tool scale as
  • the build load increases
  • additional servers, platforms or targets are
    added
  • the organization grows

71
Is the tool easy to adopt?
  • Rewriting Makefiles, build scripts, tests, etc.
    should not be necessary
  • Avoid any tool that doesnt leverage your
    existing build infrastructure
  • Should be able to kick the tires without a
    major investment

72
Wrap up
73
Best Practice Review
  • Fully automated build and test
  • Build when SCM changes
  • Visible, automated build status
  • Culprit identification
  • Pre-flight builds for developers

74
Ultimate Goal 1, 1, 1
  • One change
  • by one developer
  • results on one full build and test

75
Further Help
  • CM Crossroads Ask Mr Make
  • Not just about Make!
  • http//www.cmcrossroads.com/articles/ask-mr.-make.
    html
  • Agile Journal
  • http//www.agilejournal.com/
  • www.jgc.org
  • All my writing and speaking about GNU Make and
    builds
About PowerShow.com