Ra

1
Benchmarking Methodology
Raúl García-Castro Asunción Gómez-Pérez ltrgarcia,
asun_at_fi.upm.esgt
September 27th, 2004
2
What has been done? in D 2.1.1 Survey of
Scalability Techniques for Reasoning with
Ontologies
Ontology Technology/Methods
Experimentation
Evaluation
Benchmarking
Measurement
Desired attributes Weaknesses Comparative
analysis ...
Continuous improvement Best practices
Recommendations

• Overview of benchmarking, experimentation, and
  measurement
• State of the Art of Ontology-based Technology
  Evaluation

3
What are we doing? T 2.1.4 Benchmarking
methodology, criteria, and test suites
GOAL Provide a framework for benchmarking
activities in WP 2.1

• Benchmarking results
• Comparative analysis
• Compliance with norms
• Weaknesses
• Recommendations on tools
• Recommendations on practices

• Ontology tools
• Ontology building tools
• Annotation tools
• Querying and reasoning services
• Semantic Web Services technology

Benchmarking Methodology

• General evaluation criteria
• Interoperability
• Scalability
• Robustness

• Benchmark suites for
• Interoperability
• Scalability
• Robustness

• Benchmarking supporting tools
• Workload generators
• Test generators
• Monitoring tools
• Statistical packages
• ...

4
What will be done? T 2.1.6 Benchmarking of
ontology building tools

• Interoperability
• Do the tools import/export from/to RDF(S)/OWL?
• Are the imported/exported ontologies the same?
• Is there any knowledge loss during
  import/export?
• ...

Tools/Partners

• Experiment results
• test 1
• test 2
• test 3
• ...

• Experiment results
• test 1
• test 2
• test 3
• ...

UPM

• Experiment results
• test 1
• test 2
• test 3
• ...

• Benchmarking results
• Comparative analysis
• Compliance with norms
• Weaknesses
• Recommendations on tools
• Recommendations on practices

Benchmarking ontology building tools
...
NO OK OK
...
...
...

• Benchmark suites
• RDF(S) Import capability
• OWL Import capability
• RDF(S) Export capability
• OWL Export capability
• ...

• Benchmarking supporting tools
• Workload generators
• Test generators
• Monitoring tools
• Statistical packages
• ...

5
Benchmarking methodology
Methodology
Processes
Processes
Processes

• Benchmarking process is
• Planned
• Collaborative

Inputs
Outputs
...
Task 1
Task n
More Semantic Web oriented More KW oriented
6
Benchmarking methodology
Plan
1.- Benchmarking goals identification

• Goals depend on the organisations vision,
  objectives, and strategies.

2.- Benchmarking subject identification

• Analyse the current tools in the organisation.
• Select, understand, and document the tool whose
  improvement would significantly benefit the
  organisation, according to end user needs or
  expectations, organisational goals, etc.

3.- Management involvement

• Inform the organisation's management about the
  benefits of the benchmarking study and its costs.
  
• Management support is needed to proceed and when
  implementing changes based on the benchmarking.

4.- Participant identification

• Identify and contact the members of the
  organisation that are involved with the selected
  tool.
• Select and train the members of the benchmarking
  team.

5.- Benchmarking planning and resource allocation

• The planning must consider time and resources.
• The planning must be integrated into the
  organisation's planning.

6.- Benchmarking partner selection

• Identify, collect, and analyze information about
  the tools that are considered the best.
• Select the tools to benchmark with and make
  contact with someone in their organisations.
• The partner organisations may not belong to KW.
  Not all best in class tools are developed by KW
  partners.

7
Benchmarking methodology
Experiment
7.- Experiment definition

• Determine the experimentation plan and method.
• Define the experiment that will be performed.
  The experiment must collect not just the data on
  the performance of the tools but the reasons of
  this performance.
• Communicate the partners the experimentation
  plan and method and agree on it.

8.- Experiment execution

• Perform the experiment according to the
  experimentation plan and method.
• The collected data must be documented and
  prepared for analysis.

9.- Experiment results analysis

• Compare the results obtained from the
  experiments and the practices that lead to these
  results.
• Document findings in a report, including the
  best practices found (if any).

8
Benchmarking methodology
Improve
10.- Benchmarking report writing

• The benchmarking report must provide an
  understandable summary of the benchmarking study
  with
• An explanation of the benchmarking process
  followed.
• The results and conclusions of the experiments.
• The recommendations on improving the tools.

11.- Benchmarking findings communication

• Findings must be communicated to all the
  organisation (including identified participants)
  and to the benchmarking partners.
• Collect and analyze any feedback received.

12.- Benchmarking findings implementation

• Define a planning for the implementation of the
  benchmarking findings.
• Implement the necessary changes in order to
  achieve the desired results.
• Periodically monitor the benchmarked tool.

13.- Recalibration

• Recalibrate the benchmarking process using the
  lessons learnt.
• The benchmarking process should be repeated
  forever in order to obtain a continuous
  improvement.

9
Benchmarking Methodology
Raúl García-Castro Asunción Gómez-Pérez ltrgarcia,
asun_at_fi.upm.esgt
September 27th, 2004