Measuring and Comparing Skills and Skill Levels Not as easy as it looks Professor Ewart Keep ESRC Ce - PowerPoint PPT Presentation

1 / 36
About This Presentation
Title:

Measuring and Comparing Skills and Skill Levels Not as easy as it looks Professor Ewart Keep ESRC Ce

Description:

ESRC Centre on Skills, Knowledge & Organisational Performance, ... L2 in the Fitness industry are 60 hours! ... Reports, articles, magazines 52% Foreign ... – PowerPoint PPT presentation

Number of Views:45
Avg rating:3.0/5.0
Slides: 37
Provided by: SKO66
Category:

less

Transcript and Presenter's Notes

Title: Measuring and Comparing Skills and Skill Levels Not as easy as it looks Professor Ewart Keep ESRC Ce


1
Measuring and Comparing Skills and Skill Levels
Not as easy as it looks?Professor Ewart
KeepESRC Centre on Skills, Knowledge
Organisational Performance,University of
CardiffKeepEJ_at_Cardiff.ac.uk
2
YOUR STARTER FOR 10!
  • Q1. What is the lowest number of learning hours
    associated with a technical certificate in a
    current English apprenticeship?
  • Q2. What is the pass mark for a good (Grade C)
    pass in GCSE maths?
  • Q3, At Level 2 in NVQ, what is the range of
    learning hours (between highest and lowest hours
    normally required)?
  • Q4. What of your current skills, knowledge,
    competences are formally certified?
  • CLUE The world is not as it seems!

3
3 REASONS FOR ASSESSMENT MEASUREMENT
  • Formative assessment in order to diagnose
    weaknesses in ongoing learning, and to guide
    future progress
  • Summative/terminal certification of what has been
    achieved at the end of a period/element of
    learning
  • Comparative systemic measurement

4
THE ACTORS
  • Individuals
  • Employers (a very varied category)
  • The state and policy makers
  • International bodies (EU, OECD)

5
INTERNATIONAL MEASURES
  • In the past, economists had great difficulty in
  • finding datasets that would allow them to measure
  • skill with any degree of accuracy, particularly
  • across countries. They were therefore often
  • thrown back on using proxies for skills, such as
  • Years of schooling
  • Participation rates for given age cohorts
  • Earnings
  • None of these is a very good.

6
PROBLEMS WITH THESE MEASURES
  • Years of schooling tell us nothing about the
    quality of the education or what, if anything,
    was learned. Ditto participation rates, e.g.
    Italian higher education.
  • Participation rates may reflect labour market
    regulation systems rather than skill need, demand
    or usage patterns. Germany has a very high level
    of intermediate skills because licence-to-practice
    regulations mean that many jobs can only be
    accessed by people with L3.
  • The idea that earnings can be taken as being in
    more or less direct relationship to skill is a
    triumph of hope over reality.

7
CURRENT METHODS
  • Qualifications achieved
  • Occupation as a proxy for the skills required.
    Problem of variations of skill level required
    within occupations, e.g. manager.
  • Tests PISA, IALS, ALLS, etc. Accurate but very
    expensive. OECDs 2009 PIAC survey of adult
    skills, will use tests of ICT, literacy and
    numeracy on adult workers in different countries.
  • Job requirements individual responses to
    surveys, e.g. Skills Survey.

8
THE MAIN CURRENT MEASURE
  • Although there are many different ways to measure
    skills, public debate and policy in the UK now
    centres around formalised assessment and
    certifications systems that generate
    qualifications (or sometimes parts thereof).
  • In some other countries this is less of an issue

9
QUALIFICATIONS - DIFFERENT ACTORS, DIFFERENT NEEDS
  • Individuals and employers want measurement/
  • certification as a means of signalling
    knowledge and skill that is for exchange in the
    labour market. BUT individuals want
    certification to cover a broad range of
    general/transferable learning and to be portable,
    whereas employers often want it to be narrow,
    task-specific and non-portable.
  • The state wants standardised output units that
    can form the KPIs for the public ET system, and
    skills stock measures for the national workforce

10
THE BIG PICTURE
  • The performance of a national VET system and
    skills policy can be conceived of in a number of
    ways
  • 1. Internal Performance
  • Efficiency, value for money, unit costs per
    output
  • Levels of output
  • Levels of participation
  • Inclusiveness
  • Quality of process and output
  • Ability to meet demand (from individuals and
    employers)
  • 2. The Impact of VET on
  • Economic Performance
  • Social Performance

11
ONE PERSONS OUTPUT MEASURE IS ANOTHER PERSONS
INPUT MEASURE
  • People who are interested in VET tend to see the
    results of education and training as outputs.
  • It is true that skills and qualifications are the
    outputs of the learning process.
  • HOWEVER, in economic terms, skills are inputs
    into the productive process (alongside a wide
    range of other inputs capital, RD, etc.).
  • In other words skills only generate an economic
    output/outcome when they are utilised within the
    productive process.
  • We will come back to this point

12
LEVELS OF MEASUREMENT
  • International
  • Region and sub-region
  • Sector and sub-sector (market segment within
    sector)
  • Particular age cohorts within a population
  • Particular types of skill, e.g. IT graduates
  • Occupational groups

13
COMPARING QUALIFICATIONS STOCKS - THE WORKFORCE
AS A WAREHOUSE
  • English policy is driven by a stockpiling
    approach
  • The labour market is viewed as a vast warehouse,
    within which boxes are stored one box for each
    worker (or potential worker).
  • The boxes are assigned labels denoting the
    highest qualification held by each worker.
  • Policy tends to view it as an unqualified good to
    ensure that the greatest number of boxes as
    possible have as high a number on them as
    possible.
  • If the proportion of boxes in your national store
    with high numbers on the side exceeds the
    proportion in another country, you are deemed the
    winner!

14
LEITCH ONWARD AND UPWARD
  • The Leitch Review recommends that by 2020 we try
    to be in the top 8 at every different level of
    skill across the OECD (as measured by
    qualifications in Basic Skills, and at Levels
    2,3,and 4.
  • The pile of boxes must grow, and Leitch wants to
    employers to certify far more of their in-company
    and informal training

15
CERTIFYING WORK-BASED CPD AND INFORMAL LEARNING
  • Unlikely to grow massively because
  • Costly
  • Complex
  • Often simply impossible

16
SOME PROBLEMS WITH THE WAREHOUSE APPROACH
  • This approach ignores what is actually in the
  • boxes, or what use it might be put to
  • Is the content of a Level 3 box in the UK the
    same as a Level 3 box in Germany?
  • Is one Level 3 box in the UK the same as another
    in terms of content?
  • Levels tell us nothing about the relevance of the
    qualification to the labour market. Is a
    vocational L3 better or worse than a L4 degree in
    Art History if the job the person occupies is one
    in retail management?

17
APPLES AND PEARS?
  • Huge efforts expended on trying to standardise
  • measures at ISCED levels across developed
  • world, but
  • UK vocational L3 is far narrower in content than
    a L3 in Germany or Norway.
  • PISA and foreign languages, history, etc.
  • Is a degree from a South Korean university the
    same as a degree from an Australian university?
    Same subject/different subject?

18
QUALIFICATIONS IN ENGLAND SAME LEVEL, SAME TYPE
(VOCATIONAL), VERY DIFFERENT ANIMAL!
  • It is unwise to assume that even in the narrow
    range of a
  • vocational award at the same level in the same
    country,
  • that what is inside the box is anything like the
    same.
  • Construction L3 1,220 notional hours
  • IT L3 1,475 to 2,400 notional hours
  • Animal Care L3 650 notional hours
  • Engineering L3 3,900 notional hours
  • Those are official figures. Unofficial industry
    figures for a
  • L2 in the Fitness industry are 60 hours! Compare
    with its
  • academic L2 equivalent 5 GCSEs at A-C grade.

19
QUALIFICATIONS WHAT DO THEY TELL YOU?
  • Usually they measure (more or less imperfectly)
  • how well someone has acquired and can
  • demonstrate particular knowledge and
    capabilities.
  • What they are often less good at telling you is
  • Whether acquisition of the qualification, via a
    particular course of learning, inflicts costs
    for example, by lessening originality or
    willingness to question basic premises?
  • Whether the individuals ability to further
    develop has been enhanced.

20
MORE THINGS QUALIFICATIONS DONT MEASURE
  • Qualifications only measure some parts of the
    skill spectrum. For
  • example, they rarely offer much measurement of
    generic or soft skills.
  • In the service sector, employers are increasingly
    concerned in the
  • recruitment process to obtain
  • Generic skills such as problem solving, and
    inter-personal skills such as communication and
    empathy.
  • Personal attributes (such as self discipline,
    loyalty, and punctuality), which may not be
    skills per se.
  • Aesthetic skills dress sense, deportment,
    style, accent, voice.
  • As one UK call centre employer put it, we
    recruit attitude.
  • These skills may be vital, but are usually
    uncertified and therefore hard
  • to count.

21
OTHER WAYS OF MEASURING PEOPLE-RELATED CAPITAL
  • Business research shows that the private sectors
    measurement of human capital often has little to
    do with the public qualifications system.
    Measures include
  • Human capital (vendor qualifications, appraisal,
    supply chain TQM measures)
  • Intellectual capital
  • Structural capital
  • Customer capital
  • Social capital

22
STRETCHING THE QUALIFICATIONS BOX
  • Employers say that the new Vocational Diplomas
    (14-19) must cover teamworking.
  • QCA/DfES dream up a written exam in teamworking!

23
DOES SKILL ONLY RESIDE IN THE INDIVIDUAL?
  • Most attempts to measure skill accumulation focus
  • on the skills held by the myriad of individuals
    in
  • the workforce.
  • However, modern conceptualisations of skill
  • suggest that some skills are held collectively by
  • work teams or communities within organisations.
  • These collective competencies suggest that an
  • organisations skills may add up to more than the
  • sum of its individual employees skills.

24
MEASURING SUPPLY NOT USAGE
  • International benchmarking exercises (like
    Leitch) make the vast assumption that if created,
    additional skills will automatically be used.
  • This may not always be the case.

25
MISSING QUESTIONS OF USAGE
  • Skills deliver enhanced organisational
    performance within
  • contexts set by the organisation. We know that
    different
  • forms of
  • Employee relations context
  • Work organisation context
  • Job design context
  • Will materially impact on how productively
    enhanced skills
  • can be deployed. Looking at the stock of skills
    tells us little
  • about how they are being deployed.
  • If the context is poor, skill usage may be
    inefficient, and
  • productivity benefits from skills smaller than
    expected.

26
THE UK CONTEXT
  • Chartered Institute of Personnel and Development
    study in
  • manufacturing
  • The current managerial approaches to job design
    are
  • counter-productive.the types of impoverished job
    we
  • see in manufacturing organisations up and down
    the
  • country represent waste, on an enormous scale, of
    the
  • resources, intelligence, skills and energy of
    those
  • required to perform them.
  • Recent Workplace Employee Relations Survey showed
    no
  • signs whatsoever that high performance work
    organisation
  • is spreading across UK economy. Take up has been
    static
  • since 1998.
  • More skills may therefore be wasted.

27
DECLINING TASK DISCRETION
  • If one element of being skilled is control over
    how you do
  • your job, the UK workforce is in trouble.
  • The 2nd Skills Survey (2001), showed that
  • Between 1986 and 2001 there was a 14 per cent
    decline in the proportion of workers who felt
    they had a great deal of choice over how they did
    their work.
  • The fall was sharpest for professional groups
    (knowledge workers), from 72 per cent in 1986 to
    just 38 per cent in 2001.
  • Rising skill levels, as proxied by
    qualifications, have
  • not been accompanied by a rise in job control,
    quite
  • the reverse. This begs questions about how
    skills get used.

28
POOR JOB DESIGN, POOR USAGE ISSUES FOR SKILLS
SUPPLY
  • 2001 (UNPUBLISHED) Scottish Adult Literacy
    research showed that
  • Reading skills
  • Across the Scottish workforce, the following
    percentages of workers
  • rarely or never use
  • Info from computers 34
  • Letters or memos 26
  • Bills, invoices, spreadsheets 35
  • Diagrams 31
  • Manuals, reference books 53
  • Reports, articles, magazines 52
  • Foreign language material 91
  • 27 of the workforce indicated rarely or never to
    five or more
  • items.
  • What price boosting adult literacy skills?

29
OVER-QUALIFICATION
  • In Britain, among 20-60 year old employees, the
  • proportions holding qualifications at levels
    higher than
  • those needed to obtain their current job were
  • 1986 29
  • 1992 33
  • 2001 37
  • Increasing skill supply and skill stocks may look
    good in
  • international comparison, but how many of the
    skills get
  • used?

30
Qualifications Demand and Supply, 2001
31
SKILLS AND WHAT ELSE MAKES THE DIFFERENCE?
  • Research tells us that skills is one among many
  • factors that impact on the performance of firms
    and
  • the economy as a whole. Unless skills forms part
    of a
  • balanced scorecard for the economy, we may miss
  • where the real weaknesses in
  • investment lie
  • RD
  • Innovation
  • Capital investment in plant and machinery
  • Infrastructure

32
THE SCOTTISH STORY
  • Over a 20 year period, Scotland has spent about
    18 more per head of population on VET than
    England.
  • Scotland has a larger proportion of its age
    cohort in higher education than England.
  • The qualifications held by the Scottish workforce
    are at levels that roughly might be reached in
    England in about 2020.
  • Is Scotlands economy performing better than
    Englands?
  • Is its GDP per head higher?
  • NO.
  • A cause for some heartache among Scottish policy
    makers.

33
LEITCH THE BIG ISSUE LEFT HANGING
  • Skills are a derived demandfor the supply of
    skill to turn from merely potential change in
    performance into a tangible increase in
    productivity, the available skills of the
    workforce have to be effectively utilised.
    People need to be in jobs that use their skills
    and capabilities effectively.

34
LESSONS FOR FUTURE RESEARCH
  • Dont obsess about stocks of qualifications.
    They can hide poor usage
  • and over-qualification.
  • Remember that qualifications are only a proxy for
    one part of the skills
  • spectrum.
  • Measures of generic skill formation and usage are
    very important.
  • Comparisons need to look at demand for skills.
    The production of
  • masses of graduates, for example, may be a waste
    of resource if their
  • skills are not actually needed in the labour
    market.
  • Comparisons need to consider usage of skill.
    This, in combination with
  • a range of other factors, is important to gauging
    how well the skills
  • being produced aid economic performance. Supply
    and demand might
  • be in balance, but if usage is inefficient, or
    the other drivers of
  • productivity (e.g. RD) are missing, the economic
    impact may be small.

35
MORE LESSONS
  • The non-economic benefits of learning need to be
    considered much
  • more carefully. The real spill-overs from
    learning may not be
  • economic, but social.
  • The sustainability and quality of learning
    provision within national
  • systems is important, not just volume
    throughputs.
  • Balanced scorecards are vital. Skills alone tell
    us little.

36
METHODOLOGY IS ADVANCING
  • OECD work
  • EU work SKOPE/RAND/Danish Technology Institute
  • In the UK Skills Survey 3 will provide an
  • enhanced time series on generic skills, skill
    usage, and
  • employers demand for skills.
  • Many UK agencies are using a wider range of data,
    in
  • something closer to a balanced scorecard
    approach.
  • Fixations on qualifications is starting to
    decline.
  • There is increasing interest in data on usage and
    demand.
Write a Comment
User Comments (0)
About PowerShow.com