Ernesto Fernandez Polcuch UNESCO Institute for Statistics International Conference on S

1
Three approaches to STI measurement in developing
countries

• Ernesto Fernandez PolcuchUNESCO Institute for
  StatisticsInternational Conference on ST Policy
  Research and Statistical Indicators08-10
  November 2006 - Colombo, Sri Lanka

2
Structure of the presentation

• UIS strategy in the field of ST stats.
• Three approaches
• Careers of Doctorate Holders (CDH)
• Innovation
• RD
• Developing countries specificities and way
  forward

3
UIS ST Strategy
4
UNESCO International Review of ST Statistics and
Indicators

• Carried out jointly with UNESCO Division of
  Science Analysis and Policy (SC/AP) and Regional
  Office for ST in Latin America and the Caribbean
  (ROSTLAC) in 2002/2003.
• Objectives
• To review priority science policy information
  needs.
• To examine existing ST statistical and indicator
  systems.
• To identify key areas for future development of
  ST statistics.
• To define the future role and strategy of the UIS.

5
General ST Policy Issues

Extremely Important
Resources
Not or Somewhat Important
Impact on Society
VeryImportant
Dissemination of knowledge tech. transfer
ST output
Institutions Mechanisms
0
20
40
60
80
100
6
Immediate term priorities (Human Financial
Resources)

• RD personnel
• Expenditure in RD
• Human resources devoted to ST
• Science education
• Higher education
• International mobility
• Gender

7
Medium term priorities (Innovation)

• Measurement of innovation in agriculture and
  other non-manufacturing sectors
• Promoting the use of indicators reflecting
  sub-national (regional) innovation systems
• Measuring minor or incremental innovation
• Measuring innovative applications of existing
  products or processes (surveys of use of
  technologies)
• Output

8
Longer term priorities (Output
Impact)

• Output
• Bibliometric tools adapted for the analysis of
  scientific output in developing countries
• Technology output indicators
• Impact indicators
• Measuring social impact of ST,
• Impact of ST on agriculture,
• Public perception of ST

9
Ongoing activities

• ST Survey operation and data guardianship
• Training in ST statistics Workshops
• Standard setting/Methodological developments
• Analysis / Publications
• Collaborations / Partnerships

10
ST Survey operation and data guardianship

• Conducting global Survey on Statistics of Science
  Technology Biennially currently second round
• Maintaining database on ST statistics
• Data publishing on the UIS website
• Contributions to external agencies

11
Survey on Statistics of Science Technology

• The 2004 ST statistics survey was launched in
  May, 2004. This was the first UNESCO statistical
  questionnaire to be available for completion
  on-line via the internet. The latest resulting
  data were released on the UIS website May 2006.
• The 2006 ST statistics survey was launched in
  June/July 2006.
• OECD and EUROSTAT provide data for their Member
  States. RICYT provides data for Latin America.

12
Survey on Statistics of Science Technology
(continued)

• Questionnaire on Statistics of Science and
  Technology Instruction Manual for completing
  questionnaire.
• Data are collected from each country from either
  the institution responsible for ST policy or
  statistics (e.g. Ministry of Science and
  Technology, Ministry of Research and Higher
  Education, National ST Council or similar
  organization) or the National Statistical Office
  through a single questionnaire which cover data
  on all institutions carrying out RD activities
  in the particular country.

• Data collected
• RD personnel by occupation gender
• RD personnel by sector of employment
  occupation
• RD personnel by sector of employment gender
• Researchers by formal qualification sector of
  employment
• Researchers by formal qualification gender
• Researchers by fields of science sector of
  employment
• Researchers by fields of science gender
• Total expenditure in RD by sector of performance
• Total expenditure in RD by source of funds

13
2004 Survey on Statistics of Science Technology
Results of the 2004 survey are now published on
the UIS website Data are available for 114
countries and territories. Here is an overview of
the sources
http//www.uis.unesco.org
14
Evolution of 2006 survey
15
http//www.uis.unesco.org

• Data and Indicators published
• Number of Total RD personnel (Headcount-HC and
  Full-time equivalent-FTE) by sex
• Total RD personnel (FTE) by sector of employment
• Researchers (HC and FTE) by sex
• Researchers (FTE) by sector of employment
• Technicians and equivalent staff (HC and FTE) by
  sex
• Other supporting staff (HC and FTE) by sex
• Researchers per million inhabitants (HC and FTE)
• Technicians per million inhabitants (HC and FTE)
• Women as a percentage of RD personnel (HC and
  FTE)
• Women as a percentage of Researchers (HC and FTE)
• Total Gross Domestic Expenditure on RD (GERD) in
  local currency and international Purchasing Power
  Parity (PPP) dollars
• GERD by sector of performance ()
• GERD by source of funds ()
• GERD as a percentage of Gross Domestic Product
  (GDP)
• GERD per capita

16
UIS is UN lead agency for ST Statistics

• Contributes data for
• UNESCO HQs UNESCO Science Report 2005, UNESCO
  World Report, International Report on Science
  Technology and Gender 2006.
• UN Statistical Division UN Statistical Year Book
• UNDP Human Development Report
• World Bank World Development Indicators
• Specific Requests

17
ST Statistics Workshops

• Diagnosis
• Production of ST statistics in many developing
  regions (particularly in Africa and Asia) remains
  low. Capacity building is needed.
• UIS Response Regional workshops
• To increase the number of countries regularly
  producing quality ST indicators.
• To create local capacities for the production of
  such indicators, with the final aim of
  establishing sustainable local ST statistics
  systems
• To promote the use of ST indicators, seeking
  comprehension for evidence-based ST policy
  making.
• To share experiences with other developing
  countries in the field of ST indicators, and to
  address the problems that countries may have
  encountered in collecting ST statistics.
• To gain knowledge about the particular
  characteristics of ST statistics data collection
  and use in the context of countries in the same
  region.
• To generate initiatives that could be used as a
  demonstration for good practices in other
  countries of the region.

18
ST Statistics workshops 2005/06

• Southern and Eastern Africa Regional Workshop
  held in Uganda, Sept. 2005
• South Asian Regional Workshop held in India,
  Nov. 2005
• South-East Asian Workshop held in Indonesia,
  March 2006
• Francophone Sub-Saharan Africa held in Senegal,
  Oct. 2006
• Upcoming Workshops
• Workshop in Central Asia will be held in
  Kazakhstan, Nov. 2006
• Workshop in Arab States, will be held in Jan.
  2006
• Workshop in South-East Europe will be held in
  Croatia, Mar. 2006

19
Standard setting/Methodological developments

• Measuring Innovation in Developing countries
  Annex to the Oslo Manual (2005)
• Oslo Manual (Guidelines for collecting and
  interpreting innovation data)
• central reference document for the statistical
  definition of innovation and forms the basis for
  surveys of innovation throughout the world.

20
Careers of Doctoral Holders CDH

• Objectives
• to track the careers of doctoral holders, with
  the aim of obtaining information on the loss of
  highly qualified specialists, the so-called
  brain drain.
• to design an international survey tool which
  would help track the careers of doctorate holders
  which would better inform policy makers
  worldwide, both at global and local levels.

21
Analysis / Publications

• Data on the web site.
• UIS Publications (ST Bulletin)
  1st
  Investment in RD
  2nd
  Bibliometric Indicators
  3rd Women in
  Science
  (can be downloaded from
  the UIS website)
• UNESCO Science Report 2005
• International Report on Science, Technology and
  Gender 2006
• UNESCO World Report
• History of Science Statistics at UNESCO

22
Women in Science

• International Report on Science, Technology and
  Gender 2006
• to be published by
• UNESCO Natural Science Sector
• UIS Contribution the chapter on Statistics on
  Science, Technology and Gender (STG) with an
  Annex on statistical overview.
• Collaboration with European Commission (Research
  Directorate-General) and a group of worldwide
  experts.
• UIS Bulletin on Women in Science available
  on-line in Sept 2006.

23
Collaborations / Partnerships

• UNESCO HQs
• OECD
• Eurostat
• RICYT (Latin America)
• NEPAD / AU / ATPS (Africa)
• ASEAN
• IDRC
• INRS (Quebec)
• ISESCO
• Inter-Academy Council
• ISDB

24
Quality of data

• Efficient use of resources

Validity and reliability
Consistency over time and space
Relevance to policy
Accessibility and affordability
Potential for disaggregation
Comparability through standards
Currency and punctuality
Coherence across sources
Clarity and transparency
25
Careers of Doctorate Holders (CDH)
26
UIS approach to Human Resources statistics CDH
project

• Methodology developed from the scratch together
  with OECD Eurostat.
• Aimed both at developed and developing countries
• With participation from experts from both
  developed and developing countries
• Now being piloted
• Promoting the methodology by encouraging
  developing countries to conduct such surveys and
  produce cross-nationally comparable statistics on
  careers of doctorate holders
• What about the Sri Lankan Tracer Study of
  Graduates and Postgraduates? Does it fit into
  this project?

27
Relevance of the CDH project

• World economy is increasingly based on knowledge
  and information.
• Knowledge is now recognised as the driver of
  productivity and economic growth.
• As a result, there is a new focus on the crucial
  role of highly qualified individuals who
  represent a key to the production, application
  and transmission of knowledge.
• Statistics on the global trends in human
  resources for Science and Technology (HRST) are
  very week.
• The quality and comparability of international
  data on migration is particularly weak.
• Diversity of data collection methods hinders
  international comparability, and does not provide
  information on career paths and mobility patterns.

28
Background and Process

• Background
• Brain drain or loss of skilled personnel
  identified as priority by UIS and UNESCO Paris
• OECD and EU are interested in
• skills loss from developing countries
• Moves from university to industry
• US NSF also interested in changing US patterns of
  academic recruitment
• Process
• To meet the users demands UIS, OECD Eurostat
  launched a project on Statistics on the Careers
  of Doctorate Holders.
• Each organization took responsibility for issues
  appropriate to their mandate. Expertise and
  resources are shared across the three partners.
• Expert group Argentina, Belgium, Canada, China,
  Denmark, France, Germany, India, Italy, Japan,
  Malaysia, Portugal, Russian Federation, Spain,
  Switzerland, Ukraine, Uganda, the United States.

29
Objectives and Task forces

• Objectives
• To design an internationally comparable tool for
  tracking the careers of doctorates holders and
  highly qualified people in different countries.
• To pilot tests of this instrument in volunteer
  countries.
• to collect and exchange information on the career
  paths of holders of doctorates from existing data
  sources and the new survey tool.
• Task forces
• Production of output tabulations, and development
  of common definitions (led by OECD and Canada)
• Drafting of methodological guidelines (led by
  Eurostat and Portugal).
• Development of a model survey with a supporting
  methodology (led by UIS).

30
Output tabulation program

• Conceived to be used by countries when delivering
  the requested data to the international
  organizations
• Consists of several tables organized in seven
  broad groups personal characteristics,
  educational history, work history, scientific
  output, perception of current work situation,
  international mobility retrospective and future
  plans.

31
Definitions

• Doctorate
• Citizenship and residential status
• Recent doctorate recipients
• Employed persons
• Unemployed and inactive persons
• Researcher
• Temporary/permanent employment, full-time and
  part-time employment, etc.
• Postdoc

32
Methodological guidelines

• Define and structure the target population
• Describe the sources for building sampling frames
• Present some aspects of data collection, data
  processing and results estimation.
• Should be seen as an orientation for countries
  that plan to launch a CDH survey as well as a
  tool to improve and align their existing national
  surveys to an international standard.

33
Target population

• Consists of individuals fulfilling the
  following criteria
• having an education at ISCED 6 level (doctorates)
  obtained anywhere in the world, and
• being resident (permanent or non-permanent)
  within the national borders of the surveying
  country.

34
Target population structure

• Target population, residents in country X (the
  surveying country) that are
• Citizens of country X with doctorate awarded
  within country X
• Citizens of foreign countries with doctorate
  awarded in country X
• Citizens of country X with doctorate awarded
  abroad
• Citizens of foreign countries with doctorate
  awarded abroad
• Non-target population, residents outside country
  X that are
• Citizens of country X, with doctorate awarded
  within country X that have left country X
• Citizens of foreign countries, with doctorate
  awarded in country X that have left country X
• What to do with doctorates abroad
• - Groups E and F are usually taken into
  consideration in the reporting countries where
  these doctorate holders are resident.
• - If countries are able to construct a sample
  frame, and reach group E, the survey could also
  be extended to this group, even if countries are
  not requested to report information to the
  international organizations on them.

35
Sources for constructing sampling frames

• National education registers
• Universities
• International and national foundations
• National libraries
• Population censuses
• Research institutes
• Professional organizations
• Central registers of foreigners
• Alumni organizations
• Previously conducted surveys on doctorate holders

36
Survey types

• Cohorts of recent graduates
• administered on graduation and repeated every few
  years
• simple and straightforward, but
• limited coverage of the target population
• Snapshot surveys (cross-sectional retrospective
  sample of all doctorates)
• one time, administered on an occasional basis
• more appropriate tool for collection of
  information related to the entire target
  population
• covers all cohorts at the same time
• quickly provides information on the mobility and
  career development of doctorate holders
• Should not be seen as mutually excluding but
  rather complementary

37
Instruments (developed by UIS)

• Core model questionnaire containing core
  questions which would provide data to fill in
  the output tabulations. The questionnaire will be
  accompanied with a manual comprising the
  definitions and other guidelines.
• CDH questions data base comprises the set of
  questions among which the suitable questions can
  be chosen and applied by countries when designing
  CDH surveys. CDH data base will be web-based and
  openly accessible.
• The core model questionnaire is mainly foreseen
  for countries which do not have CDH type surveys
  yet.

38
Construction of Core model Questionnaire

• The following existing CDH questionnaires were
  used
• USA (SED and SDR),
• Canada (SED),
• Switzerland (Survey of Tertiary Graduates plus
  Doctorates surveyed the year after they received
  their degree),
• Portugal (Survey of former PhD Scholarships
  receivers concerning their professional
  situation),
• India (Pilot study on the Career Profile and
  Professional Achievements of Doctorates in
  science from the Selected Central
  Universities/Institutions),
• Ukraine (Structure of database of Doctorate
  Holders in Ukraine)

39
Modules

• Doctoral Education (EDU)
• Recent graduates (REC)
• POSTDOCS (POS)
• Employment situation (EMP)
• Career-related experience and scientific
  productivity (CAR)
• International mobility (MOB)
• Personal characteristics (PER)

40
Doctoral Education

• Information on educational history of
  doctorates holders, such as
• institution
• field of science
• duration
• sources of financial support
• country of previous degrees

41
Recent graduates

• Data on the holders of doctorate who received
  their degree in the last 2 years.
• Provides a complete educational history,
  including names and locations of secondary and
  each postsecondary institution, dates of
  attendance, field of study, and date of degrees.
• Enquires about the time elapsed before obtaining
  first CAREER PATH JOB .

42
POSTDOCS

• Would your principal job be considered a POSTDOC
  position in your country of residence? (PROBLEM
  WITH DEFINITION)
• Title, Field of ST, total length of POSTDOC
• Reasons for taking this POSTDOC
• Activities involved in POSTDOC (Research,
  teaching, other)
• Main source of financial support

43
Employment situation

• Detailed information on employment status,
  working hours, principal employer, sector and
  type of employment (postdoc, temporary or
  permanent employment, part-time or full-time job
  ), annual earnings, job-satisfaction.
• Relationship of job doctorate degree.
• Retrospective career history within ten past
  years (including occupation, dates, years of
  experience, and reasons for leaving position).
• multi-job pattern is foreseen.
• Section for unemployed and inactive.

44
International mobility

• Information on mobility patterns, including
  inflows (group D) and outflows (group C).
• Enables to distinguish temporary mobility from
  permanent mobility as well as to identify the
  reasons for departure and return.
• Provides data on intentions to move out of the
  country within the next year including the
  destination planned.
• Enquires about linkages with country of origin.

45
Career-related experience and scientific
productivity

• Data on scientific output and experience, as well
  as teaching.
• Section for researchers, including reasons for
  taking up research careers.
• Enquires about international cooperation
  activities.

46
Personal characteristics

• Marital status
• Number of dependents
• Place of birth, date of birth, citizenship status
  (in order to better deal with dual nationality as
  well) and residential status.
• Contact information

47
Way forward

• Instruments are being tested.
• Joint publication with OECD Eurostat
• Involve more countries to create an international
  database

48
Innovation
49
UIS approach to innovation statistics

• Analyse experiences in innovation measurement in
  developing countries.
• Propose amendments to Oslo Manual.
• Development of an Annex to the Oslo Manual
  Measuring Innovation in Developing Countries,
  with the participation of experts across the
  developing world.
• Dissemination of Oslo Manual Annex. Discussion
  of its applicability in other developing
  countries.

50
Measuring Innovation in developing countries

• After the publication of the 2nd Oslo Manual,
  also developing countries started conducting
  innovation surveys.
• The design of the surveys was intended to comply
  with Oslo Manual standards, with adaptations for
  capturing the particular characteristics of
  innovation processes. Adaptations were prepared
  by each country separately and with different
  approaches.
• Bogotá Manual published by RICYT (Ibero American
  Network on ST Indicators) first effort to
  compile particularities and guide the design of
  cross-nationally comparable innovation surveys.

51
Annex to OM (continued)

• UIS circulated a base document prepared by RICYT
  to a vast network of experts in the developing
  world covering China, Thailand, Singapore,
  Malaysia, Hungary, India, Lebanon, South Africa,
  Tanzania, and Thailand.
• UIS drafted the final annex based on this input.

52
Characteristics of innovation in developing
countries

• Size and structure of markets and firms
• Instability
• Informality
• Particular economic and innovation environments
• Reduced innovation decision-making powers
• Weak innovation systems
• Characteristics of innovation

53
Innovation measurement in developing countries

• The definition of innovation needs to remain
  unchanged, as well as those concerning its
  subtypes.
• The concept of potentially innovative firm is
  incorporated.
• Measurement priorities
• Innovation capabilities (Human resources,
  Linkages, Quality assurance systems, ICTs)
• Expenditure on innovation activities
• Organizational innovation

54
Principal adaptations

• ICTs in innovation surveys
• strategic use of new technologies (Front office
  vs Back office)
• Linkages
• linkage agents and types of linkage
• geographical location of linkages
• Innovation Activities
• Hardware purchase, and Software purchase
• Industrial design, and Engineering activities
• Lease or rental of machinery, equipment and
  other capital goods
• In-house software system development
• Reverse engineering
• Human resources training
• Quality and environmental management

55
Methodological issues for developing country
contexts

• Information systems specificities weakness of
  statistical systems
• Application of the survey
• Questionnaire design
• Frequency
• Publication
• Difficulties

56
Thinking ahead

• The role of entrepreneurs and their attitudes
  towards innovation.
• The intention to capture innovations driven by
  factors other than market forces, and in
  particular innovations conducted by the public
  sector.
• The adaptation of methodology to measure
  innovation in the primary sector (particularly in
  agriculture).
• The need for better measuring minor or
  incremental changes, including innovative
  applications of existing products or processes,
  and the so-called 'backwards integration' of
  technological capability.
• The development of indicators reflecting
  sub-national (regional) innovation systems.

57
Issues arising in the follow-up to the Annex

• Innovation in informal sector?
• Innovation from traditional knowledge?
• Surveying innovation, rather than RD, in
  business (and informal) sector?

58
Innovation data collection

• In 2007, innovation data and metadata will be
  collected through a pilot project.
• Only countries with existing innovation surveys
  will be approached.

59
Research and Experimental Development (RD)
60
UIS approach to RD statistics (currently)

• Applying Frascati Manual (FM) as much as
  possible.
• Disseminate the FM methodology through workshops
• Listen to countries (in workshops) and be
  attentive to issues where the FM is not suitable
  for developing countries and look for solutions
  together with the country professionals.

61
Breakdowns required by UIS

• RD personnel by gender
• RD personnel by occupation and gender
• RD personnel by sector of employment and
  occupation
• RD personnel by sector of employment and gender
• Researchers by formal qualification and sector of
  employment
• Researchers by formal qualification and gender
• Researchers by fields of science and sector of
  employment
• Researchers by fields of science and gender
• Total expenditure in RD by sector of performance
• Total expenditure in RD by source of funds

62
HOW do we collect data?

• RD Surveys. Innovation surveys. Combined
  RD-innovation surveys. -gt Good quality
  questionnaires are needed!
• Administrative data (budget, personnel list)
• ST management information systems
• Time-use surveys
• Estimations
• Different strategies for different sectors one
  size does not fit all!

63
Conclusions
64
Conclusions How do we ORGANIZE a sustainable ST
statistics system?

• Convince policy-makers
• Involve multiple actors
• User/producer consultation
• Create national ST statistics groups
• Shared ownership of data
• Quality is key capacity building of producers
• Step-by-step approach
• Combine statistics with ST information systems
• Network with colleagues from other (similar)
  countries
• Prioritize areas of work RD innovation CDH?
• Develop sector specific approaches and tools
  GOV-HE-BUS-PNP-FOR

65
Conclusions

• Three different approaches to development of
  methodologies for developing countries
• RD, focus on comparability.
• Innovation, focus on adaptation.
• CDH, joint development from the scratch.
• The Annex to the Oslo Manual as an inspiration
  for further work on other areas, particularly
  RD.
• Developing countries are not homogeneous, UIS is
  trying to get closer, but new developments will
  need significant participation of national
  experts.

66
Thank you!

• http//www.uis.unesco.org
• UNESCO Institute for Statistics C.P. 6128
  Succursale Centre-ville, Montreal, Quebec,
  H3C 3J7, Canada.
• TP (1 514) 343-6880 Fax [phone]