Metrics

1
Metrics Efficiency

• Explain why metrics are used
• Outline the Goal, Question, Metric approach
• Justify Asymptotic Complexity
• Assess the complexity of simple algorithms

2
Metrics What and Why?

• Measure of an interesting attribute
• An attribute belongs to an entity
• A metric attaches a value to an attribute
• Examples

3
Why do we want metrics?

• To compare features of different items
• Is x faster or more reliable or cheaper than y
• To assess the value of an individual item
• How fast is z?
• Other features quality, reliability, cost
• To make predictions
• If w has a complexity score of c, how long will
  it take to develop?
• Is the design for module m too complex

4
What can we measure?

• We can measure attributes of
• Products
• E.g. a program efficiency, quality
• Processes
• The methods used cost effectiveness
• Resources
• Inputs to a process
• E.g. people productivity
• E.g. software tools - usability

5
Desirable features of a metric

• Valid
• actually measures the attribute
• Does counting if statements measure complexity?
• Reliable
• consistent and repeatable
• Is a single users rating of usability reliable?
• Can be manipulated
• Combine measures, take averages,
• This is affected by scale type of metric

6
Metric Scale Type

• Nominal (classify search, sort, calculation)
• Value is a name
• Ordinal (first-1, second-2, third-3 )
• Difference between values no meaning
• Interval (A date)
• Differences between values are meaningful
• Ratio (Execution time)
• There is a fixed 0 point
• Absolute
• A count

7
Scale Type Affects Manipulation

• Nominal (classify search, sort, calculation)
• Cant calculate average or compare values
• Ordinal (first-1, second-2, third-3 )
• Can put into order, but cant calculate mean
• Interval (A date)
• Can average compare
• Cant say one is twice another
• Ratio of differences is meaningful
• Ratio (Execution time) Absolute
• Can average say one is n-times another

8
Deciding what to measure
Would be similar for network performance

• Goal, Question, Metric
• What is the purpose/goal of the investigation?
• E.g. to select the the best algorithm
• What questions are relevant?
• E.g. how fast? How much memory used?
• What measures can answer the questions?
• Experimentally measured speed
• Algorithmic complexity (see later)
• Other issues
• Reliability, ease of measurement
• Asking the right questions

Another Example Goal Effectiveness of web
server Question How fast How long do
users wait Metric Throughput (average?)
Waiting time (average, worst)
9
What Is Complexity?

• Complexity has two meanings
• Algorithmic complexity
• Performance efficiency
• Measures of time space demands
• Standard approach asymptotic complexity
• Module complexity
• Measure of difficulty / understandability
• No standard approach

10
Measuring Algorithm Efficiency

• How long will the algorithm take?
• Why not just measure the programs time?
• What factors affect Efficiency/Complexity?
• Desirable features of a complexity metric
• Independent of programming language
• Independent of hardware implementation
• Allows comparison of algorithms
• Robust
• Allows determination of time on any PC
• Asymptotic complexity doesnt do this

11
Asymptotic Complexity

• Identify the key operation
• Usually depends on the problem domain
• Sorting comparison
• Numerical analysis multiplication/division
• Virtual memory memory access
• The expensive operation
• Count operations for given input size
• Results in a formula
• Investigate formula for large input size
• Essentially keep only the big terms
• Asymptotic means as a value gets large

12
Example

• Joining all the strings in an array
• Key operation
• Joining a string
• Analysis
• Let n the number of strings
• There will be n-1 joins
• The complexity is about n for large arrays
• Can ignore the 1

13
Big-O Notation

• A way of comparing function size
• Classification Examples
• Can you give a rule for identifying O(f(n))

14
The effect of the main term
Complexity
n2
n Log n
n
Log n
Size
15
Why Asymptotic Complexity?

• Advantages
• A measure of the algorithm
• Independent of
• implementation details
• Hardware
• Robust
• Straightforward to calculate
• Disadvantages
• May not be true for small amount of input
• Ignores slight differences in algorithms

16
Example 1 Sorting an Array

• Bubblesort
• for (int i0 i lt data.length-1 i)
• for (int ji j lt data.length j)
• if (dataj gt dataj1)
• swap(dataj, dataj1)
• Complexity
• Approximately n times round outer loop
• Inner loop approximately n comparisons
• Overall complexity O(n2)
• Rough guide nested loop ? n2

17
Sorting an array Quicksort

• Quicksort
• quicksort(start, end, data)
• int mid partition(start, end, data)
• quicksort(start,mid)
• quicksort(mid,end)
• Complexity
• Complexity of partition is O(n)
• quicksort will be called O(log n) times, doing a
  partition each time
• Overall quicksort is O(n log n)

18
Example 2 Accessing a Tree
Are 0, 13 in the tree?

• Ordering
• Values in a left (right) descendent of a node are
  less (greater) than the value in a node
• Depth of a tree
• A 3-layer tree can hold 7 items (formula?)
• A tree with log2 N layers can hold N items
• How long would it take to check for an item?

19
Trees that arent compact

• Trees are often built as values arrive
• If values arrive at random, the tree is usually
  reasonably compact lots of branching
• If we are unlucky, a linear tree can result
• How many comparisons are needed in this worst
  case?

Yes values to the right are bigger than the
value at a node. All left branches are empty.
0
3
7
8
Is this a tree?
11
Are 0, 13 in the tree?
19
70
20
Summary

• A metric is a measurement of an attribute
• Goals, Questions, Metrics
• Systematic way of identifying metrics
• Asymptotic complexity
• Comparing algorithmic efficiency
• Identify key operation
• Formula counting operations for input size
• Only retain most significant terms
• Robust implementation independent