Taming Development and Data Chaos in Military Logistics: The ICIS Experience With Lightweight Languages

1
Taming Development and Data Chaos in Military
LogisticsThe ICIS Experience With Lightweight
Languages

• Gary Kratkiewicz BBN Technologies
• Geoffrey Knauth
• BAE Systems

2
ICIS Background

• Web-based software system
• Analyzes future combat and humanitarian
  operations
• Measures ability of Defense Logistics Agency to
  satisfy wartime requirements for following items
• Fuel
• Food
• Replacement items (repair parts, clothing,
  medical items, etc.)
• Identifies potential problems
• When will I run out of stock?
• What should I buy today for tomorrows needs?
• Do I have adequate sources transportation?

3
Accomplishments

• Early Web-based DOD system
• Saved over 100 million
• By computing reduced inventory needs for fuel and
  food
• Adopted as part of DOD doctrine
• Months of calculations reduced to hours
• What-if capability leads to better decisions
• Short analyses expanded role to crisis action
• Used in military operations to
• Assess readiness
• Compute operational requirements

4
Development Notes

• Started in 1994
• Grew from small prototype to large system in 4
  phases
• User interface moved to Web in stages from 1996
  to 1999
• Approximately a third of code is in Lightweight
  Languages
• Significant data issues

5
Data, Data, Data

• Large amounts of data
• 800-1000 input files 200 different formats
• Gigabytes of input and output per analysis
• Hundreds of analyses active
• Data management issues
• Received from many different agencies
• Regular and irregular updates
• Meandering formats
• Duplication between analyses
• Computations
• Intermediate output files

6
1. Prototype ICIS 1994/1995

• Original prototypes completely coded in Tcl/Tk
• Vague, rapidly changing, and conflicting
  requirements
• Less than 1 MB of data
• Ease of developing/tailoring lightweight
  prototypes greatly contributed to initial success

7
2. Production ICIS 1995/1996

• Lots of data approaching 1 GB
• Addition of (mostly) well-specified computational
  modules
• Real use with classified data in mid-1996
• Mix of languages
• Tcl/Tk for GUI and glue for module execution
• Perl for data preprocessing
• C/C for computational modules

8
3. Web ICIS 1996-1999

• Address remote access issues
• Moved to Web in stages
• On server
• Tcl/Tk glue for module execution
• Perl data preprocessing
• C/C and Java computational modules
• Perl and C dynamic generation of HTML and
  JavaScript
• In browser
• HTML static GUI elements
• JavaScript form-based GUI elements
• Java AWT applets interactive GUI elements

9
4. Redesigned Web ICIS 2001/2002

• Improve GUI and better manage data
• Data approaching 1 TB
• On server
• JScheme managing data module execution
  (sophisticated glue)
• Perl data preprocessing
• C/C and Java computational modules
• In browser
• HTML static GUI elements
• JavaScript (much less) simple GUI elements
• Java AWT applets more complex GUI elements
• Java Swing applets most complex GUI elements
• JScheme for understanding scenario

10
Source Code Evolution
11
Summary of Lightweight Language Use in ICIS

• Rapid prototyping
• LL replaced by HL completely for computations,
  partially for GUI
• Highly iterative, innovative design development
• LL used in Data Set Manager, patent pending
• Processing of meandering data
• Gluing together heavyweight modules
• LL replaced with more sophisticated LL
• Building little languages
• Test code

12
JScheme Application Flexibility

• Interpreter for debugging
• Scripts for component functionality
• Can be a server for execution management of
  multiple tasks
• Example implementation
• BBNs Data Set Manager

13
User Asks ICIS Questions
Example How much Fuel do I need?

• User interaction with ICIS is web-based
• User defines composite scenario
• User specifies results of interest
• ICIS runs the assessment
• Numerous modules might be run
• It might take a while
• Results appear in browser

14
An Example Small Assessment
15
Problems

• Obvious approach of using shell scripts and files
  doesnt cut it here
• Too much data
• Many computations recomputed needlessly

16
A Full ICIS AssessmentDoes Many Computations
The readable version of this graph takes up a
large wall.
17
Data Set Manager

• Data is described by features, not file names
• A module spec describes
• Inputs and outputs of a module
• Process relationships with other modules
• Modules can be implemented in any language
• A module spec is applied to a composite scenario
  to produce an execution script
• The script is executed
• All computations are memoized

18
Execution Management Needs Dynamic Capabilities

• Data Set Manager takes full advantage of Scheme

19
Implementing the Data Set Manager in JScheme was
fairly easy because

• Quasiquote was used to go from module spec to
  execution script
• A full language was required to write module
  specs
• Executing the script was just walking a tree
  structure

20
Murphy Strikes
Start
Goal
Mistrial!
21
MVC JScheme MC, Java V
Draw graph from knowledge of assessment state
Generate User Options
Show Details
Try Again
22
Data Set Manager Conclusion

• Core implementation is 3804 lines of JScheme code
• Supported by 287 lines of Perl, 102 lines of SQL
• ICIS Module specs add another 1853 lines of
  JScheme
• Current APIs for other languages are
• 688 lines of C
• 496 lines of Perl
• 356 lines of Java (1 class)
• 120 lines of JScheme
• BBN has applied for two patents