Software project managemnt for building high frequency trading systems - PowerPoint PPT Presentation


PPT – Software project managemnt for building high frequency trading systems PowerPoint presentation | free to download - id: 3beb71-MTBmZ


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Software project managemnt for building high frequency trading systems


Software project managemnt for building high frequency trading systems Andrew Kumiega Infinium Capital Management * * * * * * * * * * * * * * * * * * * * * * * K/V ... – PowerPoint PPT presentation

Number of Views:83
Avg rating:3.0/5.0
Slides: 44
Provided by: infotechNo3


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Software project managemnt for building high frequency trading systems

Software project managemnt for building high
frequency trading systems
  • Andrew Kumiega
  • Infinium Capital Management

Finance Industry
  • Financial market industry will hit almost 90
    billion by 2015, driven by strong growth in
  • Over 75 of daily volume on exchanges are
    computer generated orders which require real time
  • Algorithm release cycles are critical to a firms

Chicago Trading/Finance Industry
  • Not including Banks or Insurance firms.
  • Approximately 100 firms at any one time. Firms
    are well hidden to avoid PR.
  • Approximately, 12000 employees in Chicago
    employed by these firms.
  • Approximately, 40 50 of employees are in
  • Source Roy Talman.

Industrial RevolutionComputer RevolutionFinancia
l Technology Revolution
  • Production machines replace hand labor of skilled
    individual craftsmen.
  • Machinist's are rewarded for high quality parts.
    Parts are regularly reworked to obtain the
    required quality standard.
  • Computer numerical control (CNC). Computer
    controlled machines replace human machine
  • Machine operators are rewarded for
  • Cycle time reduction by better computer programs.
  • Cycle time reduction by reducing step up time.
  • Monitoring of the machine for out of control
    situations using Statistical Process Control
  • Algorithm-controlled finance (ACF). Computer
    algorithms and computers are replacing human
    traders and investment managers.
  • Large percentage of trades on exchanges are
    generated by computer algorithms. Stoll (2006)
  • Fund managers are continuously benchmarked to
    industry standards using static methods that can
    be gamed.

Automated Trading Systems
  • An automated trading system consists of
  • Interacting trade execution algorithms
  • Quantified trade selection rules
  • Business logic necessary to enter into and exit
    from positions in the financial markets
  • Technology
  • High speed information networks
  • High speed clustered servers to execute
  • Data warehouses for trade capture and risk
  • Redundant systems for 24/7 operations worldwide

Automated Trading Systems
  • Types of systems
  • Trigger trades
  • Filter trades
  • Multi-factor trades
  • Automated trading systems
  • Computer makes both the buy and sell decision.
  • Computer automatically executes the
    stock/option/fixed income trade.
  • Computer automatically hedges the position
  • Computer performs risk management of the
    positions to insure the portfolio construction is
    to specification.
  • Humans are used only to monitor the machines.

The good, the bad and the ugly
  • Some Good, Some bad and some ugly
  • Build software
  • Design and manage network
  • Hire Quants to build elegant algorithms
  • Build proprietary risk tools
  • Build custom servers using custom boardst capital
  • Start the system
  • Trading systems have multiple trade offs
  • Purchase software
  • Purchase network
  • Purchase algorithm
  • Use vendor supplied risk tools
  • Use Prime supplied Co-Lo computers
  • Start the system

The good, the bad and the ugly
  • Mad House Racing Method THC method
  • Hire two tech people
  • Buy a system
  • Buy a price feed
  • Create a trading algorithm based on other
    peoples ideas.
  • Back test 1 -2 products for 6 months
  • Hire several junior traders to shadow trade for 4
  • Produce a trading plan and get capital
  • Start the system
  • Remember the algorithmic business is a
  • competitive business similar to car racing.

The good, the bad and the ugly
  • Formula One method
  • Create a new mathematical algorithm. Get
    published in a tier one journal
  • Hire three PhD C programmers and 1 tech people
  • Back test for 10 -200 products for 5 years
  • Buy or create a price feed
  • Buy or create a trading system
  • Analyze Network routes
  • Analyze Co-Locations
  • Design and build custom computer for speed
  • Perform standard risk to reward management
  • Hire several junior traders to shadow trade for 6
  • Produce a trading plan and get capital
  • Start the system

Construction (Nascar) method of building a High
frequency trading system
History of K/V methodology
  • The method was jointly developed by Andrew
    Kumiega and Ben VanVliet.
  • Methodology was developed in and applied at
    multiple trading firms between 1995 and current
  • Most firms consider their development methodology
    to be proprietary which is the leading reason for
    lack of industry information
  • Methodology has been applied successfully at
  • Market making firms
  • Hedge funds
  • Fund service providers
  • Investment managers
  • Methodology has been refined over various
  • Excel/VBA original methodology
  • Excel/VBA to C methodology
  • Quality Money Management methodology
  • Based on industrial engineering includes
  • Rapid development
  • Stage Gates with capital controls
  • Statistical Process control

Risks addressed by methodology
  • Operational Risk the probability that
    technology-related problems, either internal or
    external, will interrupt a firms business.
  • Lack of understanding of automated trading system
  • Lack of well defined interfaces
  • Lack of risk tools
  • Lack of flexibility of system for changes in
    trading environment
  • Project Risk the probability of loss due to
    events that adversely affect the success of a
  • Quality of the end product
  • Increased costs
  • Delayed completion
  • Project failure
  • Lack of future product awards due to a time over

Key ideas in K/V software methodologyConstruction
  • Utilize a team of Financial Engineers, Senior
    Programmers, Traders and Partners
  • Build prototypes in Excel or Matlab for speed to
  • Replace excel calculations with VBA/C code
  • Replace excel interface with C or C
  • Utilize a spiral design
  • Foundation first in Excel
  • Calculations infrastructure second in code
  • Implementation third in C

Need for speed to Market in finance
  • Success in finance has become the ability to
    launch and manage new products quickly.
  • Finance has become a mature technology industry
    with time constraints for new product development
    and high quality expectations from customers.
  • High frequency execution speeds are measured at
    the sub second level for large order.
  • Operational algorithm risk from lack of testing
  • Software quality risk from lack of testing
  • Run away algorithm risk

Why use Excel/Matlab/etc as the prototyping
language for trading system development
  • The pre-built, standardized functions in Excel
    that can easily handle difficult and/or time
    consuming financial calculations, such as
  • The embrace of Excel by most colleges and
    universities as the dominant tool for teaching
    finance. Multiple textbooks in finance include
    examples in both Excel and VBA code.
  • A large number of commercial market data feeds
    that integrate into Excel. Among these are firms
    are Bloomberg, Spryware and Reuters.
  • A large number of commercial calculations from
    third party vendors such as FinCad, Numerix, and
  • The ease of use that allows desk traders, who
    generally lack formalized education or training
    in programming and software engineering, to
    develop simple trading systems.
  • The ability to add higher level languages such as
    Solver, Matlab, SPlus and etc into excel through
    the use of DLL Add-ins
  • Very easy to build a quick application that needs
    a Hash table structure linked to live time data

Overview of the Kumiega-Van Vliet Trading System
Development Methodology ( K V )
  • Hybrid solution that combines multiple software
    frame works that has been tuned for trading
    system development using a modeling language and
    an object orientated language
  • Waterfall Methodology
  • Spiral Methodology
  • Stage-Gate? Methodology
  • Go. Go on to the next stage of the waterfall.
  • Hold. Hold development at the current stage for
    reconsideration at a future date or continuing
    usage as a proto-type
  • Return. Return to a previous stage for
    additional research or testing.
  • Kill. Kill the project entirely.
  • Stage-Gate is a registered trademark of R.G.
    Cooper Associates Consultants, Inc., a member
    company of the Product Development Institute.

Deficits in standard software development
  • Water fall
  • Waterfall methodology tends to put too much
    emphasis on planning all details must be defined
    up front before design and implementation can
  • The waterfall methodology does not include a gate
    processa management decision as to whether to or
    how to continue development based upon the system
    s potential
  • Spiral/Agile
  • Loops can grow without end and there are no
    constraints or deadlines to terminate iteration
  • Agile has no criteria for transition from one
    tool set to another, say from Excel to C.
  • Team Programming framework(Visual Studio Team)
  • Lacks a frame work that allows for multiple
    programming tools (Excel, Matlab, Mathematica,
    C) to be used in a different stages
  • Lacks the concept of gates tied to resources
    (Money, Hardware, Programmers)

Advantages of K/V (hybrid) development
  • Rapidly develop prototype working system
  • Easy for multiple groups to review calculations,
    functionality, and GUI
  • Provided clear specifications for programmers to
    build a final application in C
  • Provided test cases for the programmers
    implementing the system in C
  • Provide a potential revenue stream utilizing
    Excel that can be used to fund the final
  • Solves many of the issues listed as key sources
    of spreadsheet errors as described in the Eusprig

K/V overview
  • Borrows from waterfall and spiral models, and
    Coopers Stage-Gate new product development
    methodology, with real optionality.
  • Consists of four stages Research, Back testing,
    Implementation, Management, with gates.
  • Each stage is a Plan-Benchmark-Do-Check spiral.
  • Each stage should use different software
    development tools. The initial tools are based
    on rapid development tools such as Excel, Matlab,
    Mathematica, Resolver. The late stage
    development tools are based on C, C, F, .Net,
    Java utilizing based calculations
  • Fully explained in Quality Money Management and
    Journal Articles

Agile component of methodology
  • Plan
  • Determine the problem to be solved, gather
    information, and then plan and document a course
    of action to solve it (i.e., what do we need to
  • Benchmark.
  • Research and compare alternative solutions to
    arrive at the best practice. Over the four
    stages of our methodology we benchmark
    quantitative methods, data cleaning and
    optimization algorithms, technology, and
    portfolio and risk management processes.
  • Do
  • Carry out the best practice course of action
    (i.e., we do it)
  • Check
  • Check to see if the desired results were
    achieved along with what, if anything, went
    wrong, and document what was learned (i.e., how
    did we do?).
  • If results are not satisfactory, repeat the
    cycle using knowledge gained.

Quality Money Management
  • An integrated business, software and risk
  • An iterative design methodology for algorithmic
    software development
  • Quality centric
  • Allows for stage project funding
  • A method that is focused on machine control
  • Allows for continuous process improvement

Quality Money Management Methodology
Water fall component
Research and Document Design
Back Test
Example Pairs Trading System
  • We selected companies that are fundamentally
    similar and should therefore have similar price
  • Statistical arbitrage is a bet that the
    relationship between the prices of two stocks
    will revert to a long-term mean.
  • Trigger trade is entered when the spread is
    above/below a threshold value (e.g. 2 standard
  • Trading strategy was created for this example.
  • Full paper available upon request
  • BVanVliet_at_IIT.Edu

K/V development software stage
  • Stage I. Research and Document Design
  • Describe idea
  • Fully articulate the business logic and
    quantitative methods
  • Over the iterative research process, planning
    each loop will require team members to re-define
    goals and set boundaries for that research.
  • Research quantifiable methods
  • Derive proprietary algorithms
  • Apply publicly available research from journals,
    books, the Internet, or white papers
  • Prototype
  • Quickly build several generations in order to
    evaluate whether a particular idea warrants
    further investigation and to promote risk-based
    iterative development
  • Check performance
  • Develop a clear plan on how to measure
    performance and define success and the
    variability of the trading system
  • Gate 1 Review mathematics and methodology
  • This gate will prevent development of the
    trading/investment system from moving to the back
    testing stage until the required activities and
    deliverables have been completed in
  • a quality manner.

A key output of stage 1 is to identify the
outputs that can be controlled
  • Mean/Median Profit and Loss
  • Average, Range and Standard Deviation of Returns
  • Sharpe and Sortino Ratios
  • Average Time in Trade
  • Number of Winning and Losing Trades, Winning and
    Losing Days
  • Drawdowns
  • Maximum Negative Excursions
  • Information Coefficient / Spearman Correlation

Standard Application of SPC to Trading Systems
  • The sample mean returns of a trading system
    change, within UCL/LCL on an X-bar chart.
  • The dispersion of returns widens and narrows,
    within limits on an R-chart or Sigma chart.
  • For an equity trading system, the system may
    over/under-weight a sector, reflected on a P
    chart, used for percentages.
  • For a high frequency trading system, the average
    time in trade changes, shown again on an X-bar
  • SPC can be used in multiple stages of the
    algorithm development process.

Software methodology continued
  • Stage II. Back Test
  • Gather data
  • Build a customized database of historical data
    and purchase or build a software tool that allows
    for proper back testing of the system Required
    data may either not exist at all or is
    prohibitively expensive based upon the
    prospective returns of the trading/investment
  • Develop data cleaning methods
  • Development of a Data Transformation Management
    System (DTMS) that will scan data for errors and
    irregularities is essential
  • Perform algorithm benchmarking with sample data
  • 1. Profitable both in sample and out of sample.
  • 2. Profitable in sample, but not out of sample.
  • 3. Unprofitable both in sample and out of
  • 4. Use statistical process control to
    check for ambiguity in trading
  • algorithm output.
  • Check Performance and Probationary Trade
  • Shadow trade the algorithm
  • Not indicative of performance due to lack of SPC
    control algorithms
  • Mainly used to help design interfaces, risk
    controls and SPC controls
  • Gate 2 Review research results and formalize
    development plan

Standard in sample trading performance reporting
  • System was run over 4 years 1/1/2002 to
  • In-sample results for first 2 years.
  • Results show the system generates attractive
    performance metrics for the first 2 years.

SPC Indication of System out of control or in an
ambiguous state
  • Any single measurement above or below the upper
    or lower control limits.
  • 5 consecutive measurements moving in the same
    direction up or down.
  • 2 measurements greater than 2 standard deviations
    from the mean.
  • 3 measurements more than 1.5 standard deviations
    from the mean.
  • 7 consecutive measurements on one side of the

SPC for Investment Decisions KV Stage 2
Two year out of sample results
  • Statistical Process control would not allow this
    algorithm to proceed forward to Stage 3. The
    stopping of algorithm prevented spending
    resources on a losing system or forced further

Software methodology continued
  • Stage III. Implement
  • Plan and Document Technology Specifications
  • Fully define the functionalities and performance
    requirements of the trading/investment system.
    The specification documents allow the team of
    hardware engineers and programmers to quickly
    build the system with the correct functionalities
    and to the proper specifications.
  • Design System Architecture
  • Architecture documents are blueprints of the
    hardware and software that will form the
    architecture of the trading/investment system,
    including the financial calculations, realtime
    data and user interfaces, order routing
    connections, reporting functionalities, and any
    other necessary processes.
  • Build and Document the System
  • The process of construction will be a
    step-by-step march through the architecture
  • Architecture documents themselves will be
    evolving as the team spirals through
    implementation using an agile approach utilizing
    the four stages of plan, do, benchmark and check
  • Check Performance and Probationary Trade
  • Purpose of probationary trading is to allow the
    trader or money manager time to use the trading
    tools and determine what additional reporting
    tools need to be built to properly manage the
    embedded risks
  • Gate 3 Review development of system

K/V development software stage
  • Stage IV. Monitor
  • Plan Performance and Risk Processes
  • Reports will present the portfolio statistics and
    performance metrics, risk calculations, and
    provide documentation of the attribution of gains
    and losses
  • Define Performance Controls
  • These risk control techniques help the team
    understand whether or not the system is working
    within specifications and in conformance with the
    back test and/or a benchmark
  • Markets are stochastic and trading/investment
    system performance will be stochastic as well.
  • Perform SPC analysis
  • Are the underlying market distributions the same
    as the ones that were used to generate returns
    for the back test?
  • Is the system in conformance with the back test
    and the benchmark index?
  • Are the inputs or outputs of the process
  • Determine causes of variation in performance
  • Systems will need to be continuously tweaked
    utilizing a Six sigma approach
  • Repeat the entire waterfall process for
    continuous improvement.

K/V Monitoring using SPC for a daily FX trading
algorithm presented for publication
K/V Monitoring using SPC for a FX algorithm
presented for publication
K/V Monitoring using SPC for a FX algorithm
presented for publication
K/V Monitoring using SPC for a daily FX trading
algorithm presented for publication
System identified as out of control
  • Stage IV. Monitor
  • Determine causes of variation in performance
  • Once a system is deemed to be out of control the
    system needs to fixed.
  • Applying Ford 8d to identify the root cause.
  • Apply design of experiments to identify
    interactions between factors that has caused the
    process shift.
  • Perform Kaizan analysis
  • Check for data latency
  • Check for new competitor with better algorithm
  • Release newer algorithm
  • Redesign algorithm to outperform new competitor
  • Repeat the entire waterfall process for
    continuous improvement.

The good, the bad and the ugly
  • K/V Methodology
  • Is it better than the other two methods?
  • All methods have issues.
  • Does it produce consistent results?
  • Does it reduce the risk of a run away algorithm?
  • Does it produce better results?
  • Which race track do you plan to race your
    algorithm on?

Academic approach to building high frequency
trading system development?
  • A quantitative trading system is a complex
    machine and should be built and managed
    according to process engineering theory.
  • A hybrid development cycle utilizing different
    tools for different development stages is optimal
    for quantitative finance
  • Stage-Gates for quantitative software development
    are required to due to complexity of the systems
    being dev
  • Quantitative finance requires a specialized
    process due to the large amount of cutting edge
    RD, time pressures and cross functional teams
    with specialized skills
  • Next life cycle stage in Quantitative Finance is
    the quality stage. This is due to the fact as
    all industries mature they eventually have to
    focus on process quality to produce goods and
    services that win wars.