The Software Matrix Simplifying Software Salvage

1
The Software MatrixSimplifying Software Salvage

• Riddhiman Ghosh
• Advisor Dr. James Fawcett

2
Introduction

• Approximately 100 billion lines of source code at
  work in the world today.
• Large fractions of code in systems are
  functionally equivalent.
• However systems are often constructed without
  leveraging existing code bases less than 15 of
  new code serves an original purpose.
• Reinvention of the wheel in the software
  industry.

3
Introduction

• Software construction is needlessly error-prone
  and expensive.
• Were maintaining multiple copies of essentially
  the same software.
• Time and cost of developing, testing and
  documenting a piece of software is multiplied by
  the number of equivalent copies in existence.

4
Software Reuse

• Study of reuse recycling of software assets
  has been an important branch in the software
  engineering discipline.
• Effective reuse promises
• Reduced development and maintenance costs
• Gains in development schedule and quicker
  time-to-market
• Increased robustness and quality
• Not a new idea.

5
Software Reuse

• Systematic Reuse an institutionalized
  organizational approach to product development in
  which software assets are intentionally created
  or acquired to be reusable
• However, few organizations practice systematic
  reuse, in spite of a recognition of its potential
  benefits.
• Reuse is hard!

6
Gap between theory and practice in Reuse

Not all the theoretical oriented and
sophisticated solutions presented by
researchers have ...thrilled the
practitioners... Zand, M. ...the reuse
community has worked on complex technologies and
methods with high ceremony, yet most of the
software community seems to be looking for
simpler solutions... High ceremony methods
require an organization with high process
maturity to achieve success. Griss, M.

7
Software Salvage

• What we seen in the industry is not software
  reuse, but rather software salvage
• Salvage lifting of a significant block of
  existing systems and inserting them into a newly
  developed system.
• Radar Systems Department, General Electric
  Company (Syracuse, NY) routinely attempted
  salvage in the building of a new radar.

8
Reuse vs. Salvage

• We make a distinction between the terms software
  reuse and software salvage.
• Reuse connotes immutability
• the individual pieces meant for reuse cannot be
  modified theyre designed and implemented to be
  adaptable but with no intent to change even a
  single character of source code.
• Salvage makes no guarantee of immutability
• very often source code of individual pieces being
  salvaged is modified.

9
Problem

• Effective salvage is difficult to accomplish
• The large pieces we want to recycle often have
  many dependencies on parts we dont want.
• These dependencies usually require expensive
  changes to the part being salvaged.
• Pulling out pieces from different systems, and
  getting these extricated pieces to work together
  is challenging.
• Think of salvage as analogous to transplanting
  the heart from one living organism to another!
  Similar problems, when we pull out a part from a
  software system due to the connectedness inherent
  in typical software.

10
Problem

• It is difficult to salvage existing parts
  systems and build new systems from them, with
  ease.

11
Goal

• Enable leveraging of major parts of systems
• Eliminate or reduce changes required by salvage
  so that salvage moves closer to the reuse model
  (immutability), without requiring high-process
  organization.
• Simplify software salvage to make it a useful
  paradigm.

12
Prior Approaches

• Two major (and fairly recent) approaches towards
  encouraging reuse have been
• object-oriented reuse
• component-oriented reuse.

13
Object-Oriented Reuse

• Reuse has been one of the classic motivations of
  object-orientation
• Object oriented theory
• mandated discipline in writing code through
  best-practice guidelines such as separating
  interface from implementation
• Encouraged reuse through inheritance,
  composition, parameterization

14
Object-Oriented Reuse

• However OO technologies didnt quite engender the
  reuse revolution that was hoped for.
• Perhaps the only OO libraries widely used
  user-interface frameworks (such as MFC) and
  libraries for data structures.
• While OO techniques have made compiler libraries
  an effective means of reuse, business
  organizations have had a harder time getting
  leverage from their pre-existing software assets
  only through the creation and consumption of OO
  libraries.

15
Object-Oriented Reuse

• The definition of an object is purely technical
• Defined as an encapsulation of state and behavior
• No direct mapping to the physical unit that is
  actually deployed, versioned and potentially
  reused.
• Objects dont fit the salvage model well
• In terms of granularity, they are of an
  inappropriate size to be mixed and matched.
• They are the size of grains of sand, while what
  salvagers are typically looking for in building
  systems, are bricks.

16
Component-Oriented Systems

• The logical next step was the notion of
  components
• a coherent package of software implementation
  that can be independently deployed and composed
  with other components.
• of coarser granularity than objects
• independent and deployable implies executable or
  loadable code
• The defining feature of component technology is
  that a system, which is composed of components,
  can be repaired or made better by updating
  components, without rebuilding the whole system.

17
Component-Oriented Systems

• There are several component technology offerings
• COM, CORBA, JavaBeans/EJB, .NET Components
• Adopting a component-oriented approach (rather
  than only concentrating on programming language
  as in OO theory) is beneficial.
• But the existence of these component technologies
  has not solved the problems that plague software
  salvage, discussed earlier
• Packaging techniques alone are not enough

18
Our Approach

• We are of the view that components are useful
  only if there is a framework that actively
  supports and promotes their reuse.
• .NET and J2EE have framework support, however it
  is focused on generic industry problems (network
  communication, web publishing, application
  security, etc.)
• We seek to support vertical, perhaps proprietary,
  applications, where building a huge framework is
  impractical from a return on investment point of
  view.

19
Our Approach

• Make salvage easier by
• viewing applications as compositions of different
  pieces
• having a framework of collaborating pieces from
  which applications can be composed dynamically

• Try to achieve benefits of the Software-IC
  model a plug-and-play approach to software
  construction.

20
Our Approach

• We are limiting the hard problem of general reuse
  to a smaller domainof salvage within an
  organization, to be used in the construction of
  modest-sized systems (about 200,000 lines of
  code).
• To address this domain would be to provide
  solutions of value to the small and medium-sized
  software shops that have been resistant to adopt
  systematic reuse.

21
Software Matrix

• The Software Matrix is a framework that actively
  supports and promotes the salvage of components.
• We focus on the reuse of major blocks of code
  rather than low-level components
• By employing message-passing and mediator
  structures and by supporting the discovery of
  needed types, weve built a pluggable
  architecture that can gracefully adapt to salvage
  operations.

22
Software Matrix

• In particular, the Matrix is a runtime
  infrastructure that acts as a substrate into
  which individual pieces of an application
  different blocks of code can plug.

23
Software Matrix
From these plugged-in individual pieces the
Matrix dynamically composes applications.
24
Software Matrix

• This infrastructure was named the Software Matrix
  in order to connote a very structured pattern of
  building software.
• System elements are embedded in this substrate
  a matrix /collection through which applications
  are dynamically composed.
• The image of endless banks of incubators of
  humans in a matrix, as from The Matrix motion
  picture, isnt entirely unintentional, if
  somewhat flippant.

25
Cells

• The individual blocks of code that plug-in to the
  Matrix are called Cells and are the building
  blocks out of which applications are built.

26
Cells

• Cells represent the unit of composition and reuse
  in our system.

27
Cells

• An application is built through the collaboration
  of Cells, and all Cells communicate with each
  other strictly through messages (we use XML
  encoded messages)

28
Cells

• One of the problems of salvaging extracting
  parts of monolithic (or very tightly-coupled)
  applications.
• By enforcing a separation of concern between
  Cells by inserting a message bus between them we
  are insisting on loose-coupling between different
  pieces of an application, which makes salvaging
  these pieces for use in future applications much
  easier.

29
A closer look at Cells

• Every Cell contains
• a message queue
• holds request and response messages during
  collaboration with other cells.

30
A closer look at Cells

• Every Cell contains
• a capability list
• used to advertise the capabilities of a cell to
  other cells, via the Matrix. (E.g.
  SU.Math.Convolution)
• The capability list is used by the Matrix in
  order to discover the right cells for system
  construction.

31
A closer look at Cells

• Every Cell contains
• a globally unique identifier (GUID)
• Cells can be uniquely identified using a GUID.
  This is used by the Matrix for several operations
  such as discovery and registration.

32
A closer look at Cells

• Every Cell contains
• functionality
• Cells also contain the functionality that allows
  them to be considered as software assets with
  potential for reuse.

33
A closer look at Cells

• All cells subscribe to a common protocol (ICell)
  that specifies how to
• register and un-register with the Matrix
• advertise capabilities
• send and accept messages
• collaborate with other cells (could be
  synchronous, asynchronous or one-way)

ICell
MyCell
34
A closer look at Cells

• Every cell also has an entry-point (start), and
  is given a chance to execute once it is
  plugged-in.
• This entry-point (empty/non-empty) decides
  whether a cell will be only a passive server,
  or itself actively seek collaboration from other
  cells.

35
Example

• Sample application needs to read data samples
  from an input file, perform a signal processing
  operation on the a data (e.g., filtering), plot
  the results of the operations on the system
  display, and log the results to a file.
• The major pieces of the application would be
  responsible for
• file operations
• signal processing
• graphical plotting

36
Example

• These would be written as cells and plugged-in to
  the Matrix.
• The Matrix would then assume the responsibility
  of constructing the application from these
  individual cells.
• In order to perform its task a cell may need
  services of another cell. But cells do not
  explicitly bind to other cells they only
  specify what message type they need handled, and
  the Matrix discovers cells capable of handling
  that message.
• If no suitable cell found, a not supported
  message is generated.

37
Example
38
Example
Sequence of events in the construction of the
sample application
39
Example

• So what we see here is dynamic composition we
  are building a system from pieces that exist on
  the Matrix at runtime. The compositional aspects
  (as opposed to only computational aspects) of
  software are being taken care of by the Matrix.
• It automatically connects the right pieces at
  run-time without having to bind to anything
  explicitly at compile-time.
• This method of software construction, with its
  dynamic nature and loosely coupled building
  blocks, is amenable to salvage operations. The
  very same Cells could be used to build other
  applications.

40
How do cells plug-in?

• Cells are implemented as plug-in modules, and
  are realized using .NET components.
• The Matrix uses the reflection and
  late-binding (Fusion) features of the .NET
  framework to discover and register plug-ins.
• Given an .NET assembly, the Matrix will reflect
  over the contained types to determine which of
  them implement the ICell interface in order to
  recognize valid plug-ins.

41
How do cells plug-in?

• Throughout its lifetime, the Matrix runtime
  infrastructure monitors a known location the
  plug-in directory, in order to discover if any
  new cells are present.
• On detecting a valid cell, the Matrix registers
  the cell with itself. This newly registered cell
  (along with previously registered cells) is then
  available for system construction.

42
How do cells plug-in?
At periodic intervals the Software Matrix
interrogates a known location in order to
discover if any new Cells are present, and if so
registers them.
43
Steps to take advantage of the Matrix

• In order to enable salvage, pieces of an
  application (at the time of writing it, or
  existing pieces) are wrapped in a Cell.
• Create a wrapper that inherits from ICell (e.g.
  we wish to create FileManager Cell responsible
  for common file operations )
• FileManager ICell
• ...

• ?

44
Steps to take advantage of the Matrix

• The Capability List of this FileManager Cell is
  then populated to indicate the types of messages
  it is capable of handling.

• ?

CapabilityList.Add(SU.FileManager.Files.Read) C
apabilityList.Add(SU.FileManager.Files.Write) C
apabilityList.Add(SU.FileManager.Files.Search)
CapabilityList.Add(SU.FileManager.Files.Compressi
on)
45
Steps to take advantage of the Matrix

• Override the process method to add appropriate
  message processing
• Basically you specify what is to be done in
  response to a particular message type delegate
  calls to your implementation.

• ?

46
Steps to take advantage of the Matrix

• Compile, and copy the resulting binary into the
  plug-in directory of the Software Matrix. The
  Matrix will automatically detect and register the
  cell, and it will be available for composition.

• ?

47
Steps to take advantage of the Matrix

• If a cell wishes to use other cells (or is a
  program executive for instance), it will probably
  say
• ...
• Result syncSend(
• "SU.FileManager.Files.Search",
• Params)
• ...
• Here it is trying to locate a cell that can
  handle the named message type.

48
Features

• Fine-grained message-passing
• Many problems associated with software salvage
  arise from extracting one of several
  tightly-coupled pieces of a system.
• We are using message passing at a much finer
  level of granularity than is normally seen
• The Matrix requires messages be the only mode of
  collaboration between different parts of an
  application, thereby decreasing the degree of
  coupling.
• This loose coupling is critical to the success of
  salvage operations.

49
Features

• Dynamic Composition
• The Matrix takes care of the compositional
  aspects of software by automatically discovering
  and connecting the right pieces and building an
  application at runtime
• By adding a few more cells into the Matrix if
  needed, we can build new applications by reusing
  existing cells.

50
Features

• Support for System Evolution
• Since the appropriate cell to serve a particular
  message type is selected at runtime, the Matrix
  supports evolution of software systems.
• Evolving requirements can be accommodated easily
  by unplugging or modifying only those cells that
  represent the affected part of the system.
• Easy to field-replace cells all one has to copy
  the new cell over the old cell in the plug-in
  directory.
• Effective way to support program maintainability,
  bug fixes/upgrades.

51
Features

• Simplicity
• The Matrix is a supporting infrastructure that is
  lightweight (the bare infrastructure is roughly
  only 2000 lines of code)
• Simple to use. The infrastructure comes with
  documented full source (if needed) and sample
  applications.
• The underlying technology used is the .NET
  component model which is considered superior and
  less complex as compared to other models such as
  COM.

52
Assessment

• Built applications in the regular style and
  using the Software Matrix.
• Working on an assessment model with quantitative
  and qualitative criteria for effective
  comparison. How easy is salvage?
• SLOC
• Complexity
• Changes necessitated, cognitive distance
• Performance

53
Comparison with prior and related work

• There exists a combination of middleware and
  component technologies such as COM and CORBA that
  have reusability as one of their goals.
• COM, CORBA are considered highly complex,
  over-specified and require heavy-weight
  supporting infrastructures.
• The Matrix was meant to address the specific
  problem of salvage and is lightweight (approx.
  2000 SLOC) and simple.

54
Comparison with prior and related work

• Most middleware technologies use the
  procedure-call model of collaboration. Having
  components bind to exact function signatures is
  not flexible enough leads to the
  tightly-coupled systems that make salvage
  operations difficult.
• The Matrix uses message-passing as the mode of
  collaboration between components. This leads to
  looser coupling of system elements resulting in
  potentially easier salvage in the future.

55
Comparison with prior and related work

• Message Oriented Middleware (MOM) and to a
  certain extent Web Services also use
  message-passing (most web services use XML-RPC,
  though they support a message-passing interface)
• The Matrix differs from them in the granularity
  and scale of message-passing, since we are
  focused on local compositions of cells
  (components) to build applications, rather than
  accessing remote functionality over the network.

56
Conclusion

• The Software Matrix is a runtime substrate with a
  plug-in architecture that enables simpler
  software salvage.
• Enforces loose coupling of system elements
• Helps in the discovery and collaboration of
  components.
• Easy to dynamically compose new applications from
  existing cells.
• Provides support for product evolution and
  maintenance important in large systems.
• Simplifying software salvage is a worthy goal and
  will have a positive impact on productivity in
  the software industry.

57
End of Presentation