Research Profiles: A Mental Model for Managing and Measuring Different Types of RTD Programs

1
Research Profiles A Mental Model for Managing
and Measuring Different Types of RTD Programs

• Performance Assessment of Public Research,
  Technology, and Development Programs
• June 17-18, 2004
• Brussels
• Gretchen B. Jordan, Ph.D.
• Sandia National Laboratories
• gbjorda_at_sandia.gov, 202-314-3040

Work presented here was completed for the U.S.
DOE Office of Science by Sandia National
Laboratories, Albuquerque, New Mexico, USA under
Contract DE-AC04-94AL8500. Sandia is operated by
Sandia Corporation, a subsidiary of Lockheed
Martin Corporation. Opinions expressed are
solely those of the author.
2
Measurement perturbs the system. We must measure
appropriately.
Innovation is a multi-stage process and each
stage requires different management strategies
(E.B. Roberts, 1988).

• The existing set of metrics suffers in part
  because
• They miss the temporal dimension between
  scientific outputs and technology
  accomplishments, and
• Motivations for measurement differ. The
  short-term criteria for control are often in
  conflict with the long-term criteria for
  evaluating value and strategy realization (E.
  Geisler (2000).

3
Motivation for DOE Research Profiles Research

• Observe that research organizations differ
• Differences are magnifying as RTD
• becomes more specialized and complex and
• interdisciplinary and collaborative
• Increasing pressure to describe value of research
  and manage for results
• No theory links diverse strategies to management
• Organizational innovation studies address links
  but primarily involve product development
• Stokes and others suggesting new RTD models or
  taxonomies look at differences in task and
  purpose, but dont link that to organization and
  management practices

4
DOE Project Overview

• Research by Sandia labs in collaboration with Dr.
  Jerald Hage and the Center for Innovation,
  University of Maryland
• Sponsored by U. S. Department of Energy (DOE)
  Office of Basic Energy Sciences to define
  innovative measures of operational and scientific
  performance.
• Since 1997 concentration has been on
  understanding and developing tools to assess key
  factors in the research environment that foster
  excellence in order to improve.
• Hypothesis RTD performance will be highest when
  organizational structure and management practices
  fit the strategy and primary task (profile) of
  the research.

5
The Competing Values Framework shows trade offs
on two dimensions among four competing views on
organizational effectiveness.
Flexible Structure
Creativity, Cutting edge output, Growth
external support
Morale cohesion, Commitment, HR development
Quality Outcome
Internal Focus
External Focus
External positioning, Productivity, Goal
achievement
Timeliness, Stability, Efficiency
Controlled Structure
(covers ALL types of organizations) Cameron and
Quinn 1999
6
The desired balance across competing values
depends on the situation, internal and external
Flexibility
Flexibility
Start up Firm
Build Expertise
External Focus
Internal Focus
50 year Old firm
Internal Focus
Production Mode
External Focus
Control
Control
Flexibility
Flexibility
Dynamic World
Ambidextrous Organization Or Wholonics
Internal Focus
Stable World
Internal Focus
External Focus
External Focus
Control
Control
7
Average of over 1000 organizations
University of Michigan data shows differences for
RTD organizations
Hi-Tech Manufacturer
Flexibility
Flexibility
Clan (HR, Morale)
Adhocracy (Innovation, Growth)
40
40
Clan
Adhocracy
30
30
20
20
10
10
Internal Focus
External Focus
Internal Focus
External Focus
10
10
20
20
30
30
Hierarchy (Efficiency, Stability)
40
Market (Productivity, Sales)
Market
Hierarchy
40
Control
Control
Source Cameron and Quinn, 1999
8
DOE Study is defining dimensions to recognize
diversity in research organizations and tasks.
Dynamic, Emerging Field
Narrow scope, Small
Stable, Mature Field
Broad Scope, Big
What dimensions distinguish research
characteristics?
Most Research Laboratories
Extend Existing
Radically New
9
Proposed DOE theory of RTD diversity Four
profiles depending on scope and objectives
Narrow Scope, Small A component or niche market

Be Sustainable Master
Be New Create
Incremental Change Specialized Task
Radical Change Complex Task

Be First Produce
Be Better Improve
Broad Scope, Large A systemic or global problem
Names of profiles borrowed from Cameron, Quinn,
et al. University of Michigan
10
Examples for both science and technology
Narrow Scope, Small A component or niche market
Be Sustainable Master
Be New Create
Physics of nanoscale Quantum computing

Academic research Biotech capabilities
Incremental Change Specialized Task
Radical Change Complex Task
CERN Fuel Cell Vehicles
Genomics research 2004 Nisson global car

Be Better Improve
Be First Produce
Broad Scope, Large A systemic or global change
11
Hypothesis Organizational Design and Management
Differ Depending on RTD Strategy and Task
RTD Strategy/Work Profiles
Narrow Focus Small RTD
Master
Create
Incremental Specialized Task
Radical Complex Task
Task and Desired Outcomes
Produce
Improve
RTD Environment Profiles
Broad Focus Large RTD
Time to Explore Intellectual Integration Encourage
Change
Values Individuals Teamwork HR Development
Work Environment
Rich in Resources Technical Management Low burden
Systems
Vision and Strategies Plan Execute Strategic
Relationships
Source Research Profiles (Jordan extending
Cameron and Quinn)
Controlled Structure
12
Research Profiles Theory Fits Existing
Organizational Theory

• Larger organizations, big science, require more
  coordination, support systems, hierarchy and
  control than small
• Radical innovation requires more flexible, fluid,
  responsive organization
• Tolerance for risk, uncertain timing of outputs
• Openness to external events and frequent
  adjustment/ redefinition of task
• Complex Teams/ Inter-Functional integration
• Rewards for entrepreneurial activity
• High lateral communication
• Strong feedback loops

13
Preliminary Data Shows Significant Differences
Across Project Characteristics

• For basic science vs. technological tasks
• Science projects were lower on stable funding,
  reputation, competencies, salaries, quality of
  staff, critical thinking, external collaborations
  
• Technological were lower on vision, relationship
  with sponsor, investment in future, lab and
  project metrics, cross fertilization, integrated
  portfolio
• For small (lt 1M) vs. large projects
• Large projects were lower on investment in
  future, lab and project metrics, rewards for
  merit, allocation on internal funds, career
  development, identifying new opportunities,
  integrated portfolio
• Small projects lower on external collaborations
• Projects with complexity of labor (6
  departments) rate 14 lower
• Stable funding, vision, project planning, lab and
  project metrics, identification of new
  opportunities, integrated portfolio, career
  advancement, professional development, allocation
  of internal research funds, lab systems and
  services, research competencies, competitiveness
  due to overhead

14
Potential Applications of Research Profiles

• Better data on health of the research environment
  and how to manage RTD
• Portfolio analysis, balancing
• Performance measures appropriate for strategy and
  structure/task
• Better descriptions of RTD programs and their
  value (Program Logic)

15
DOE Study Defined 36 Key Attributes, Now in a
Research Environment Survey
Flexible Structure
External Focus
Internal Focus
Controlled Structure
Attributes are in four sets, shown here with
survey question number. Graphic displays survey
response (mean time true)
16
Survey collects data on profile of work
17
Organizations profiles can be complementary
Or an organization can cover all profiles an
example of RTD for a public good
18
Validating the link between Profile, management
and outcomes requires better measures of
progress.
Small RTD Projects, Narrow Scope
Create- Be New
Master - Be Sustainable
Radically fundamentally new ideas, techniques
Incremental advancement in existing ideas,
capabilities
Value individuals Build teamwork Committed to
human resources growth
Time to explore, take risks Intellectual
Integration Encourage change, critical thinking
Technical Progress
Complex Radical
Specialized Incremental
Management Practices
Rich in resources Good technical
management Efficient management systems
Clear goals strategies Plan execute
well Build strategic relationships
Incremental improvement in existing products,
processes
Radically or fundamentally new
products, processes
Produce - Be First
Improve - Be Better
Large RTD Projects, Broad Scope
19
Two approaches to get better measures

• Differentiate existing or new measures by
  Research Profile
• Develop technical progress measures (hands on
  with research projects) defining measures that
• Come earlier than publications,
• Can be tracked over time, and
• Fit positively with the daily routine of doing
  RTD

20
Four Profiles Different Outputs, Outcomes
(Logic)Different Performance Indicators
21
Logic of RTD Investment to Build/Maintain
Capabilities -- Be Sustainable
Master
Resources
Activities
Outputs
Outcomes
Ultimate Outcome
Build and maintain capabilities
Staff hired Learning opportunities Forums
held Partnerships identified Tools
acquired Research projects underway Technical
reports

• Use of this extension of capabilities to
• Build or sustain an educated workforce
• Know what others are doing
• Modify a tool for your own use
• Be agile, ready to enter a new area if need be
• Increase use of an idea, product or technology

Incremental new idea, technique, tool Knowledge
exchanged, written down (codified) Great
contributors participating Communities of
Practice established Equipment, tools
functioning at peak Excitement increased
External Influences Social/cultural,
Technical, Economic, Political/regulatory
22
Logic of RTD Investment to Discover New Ideas --
Be New
Create
Improve
Resources
Activities
Outputs
Outcomes
Ultimate Outcome
Radically new idea, bench scale prototype,
material Proof of concept Robust
model Analytical design for technology, product,
process Publications Portfolio of highly unusual
projects Emerging fields

• Use the new knowledge or prototype
• in other/ further research
• in technology, product, or process development
• to inform policy or regulatory decisions
• for competitive or public advantage

Research plans reports Data from experiments,
simulations Designs Design tests
Conduct strategic basic applied research
External Influences Social/cultural,
Technical, Economic, Political/regulatory
23
Logic of RTD Investment for Developing a New
Technology -- Be First
Produce
Improve
Resources
Activities
Outputs
Outcomes
Ultimate Outcome
Identified applications Converge on project
aimed at technical need Tests of lab-scale, then
intermediate scale prototypes
Radically new knowledge, technology, product, or
process at system or commercial scale Technical
progress on functionality, feasibility Scale-up
prototypes Strategic coalitions

• Use the new knowledge or product to
• capture a new market
• have first mover advantage
• have a mission advantage (e.g. security)
• have high yield, expected high yield

Conduct new technology development
External Influences Social/cultural,
Technical, Economic, Political/regulatory
24
Logic of RTD Investment to Improve a Technology
-- Be Better
Improve
Resources
Activities
Outputs
Outcomes
Ultimate Outcome
Tests Market demonstrations Performance cost
data analyzed Technical problem diagnosed

• Use the improved knowledge or product/process
• to differentiate for larger market acceptance
  or mission use
• to have consistent quality
• to enforce standards

Incrementally improved knowledge, technology,
product, process Standards for technology,
product and/or process Steady technical progress
Manufacturing process stable, optimized
Conduct engineering, manufacturing, or quality RTD
External Influences Social/cultural,
Technical, Economic, Political/regulatory
25
Application of this mental model Research
Profiles

• Continue to build a database to test
  relationships and validate Research Profiles
  theory and instruments
• Profile data and research environment assessment
• Technical progress data
• Encourage others to distinguish among Research
  Profiles when managing and assessing RTD programs
  and when studying the determinants of innovation

One size does not fit all profiles.