Title: Measuring and Comparing Skills and Skill Levels Not as easy as it looks Professor Ewart Keep ESRC Ce
1Measuring and Comparing Skills and Skill Levels
Not as easy as it looks?Professor Ewart
KeepESRC Centre on Skills, Knowledge
Organisational Performance,University of
CardiffKeepEJ_at_Cardiff.ac.uk
2YOUR STARTER FOR 10!
- Q1. What is the lowest number of learning hours
associated with a technical certificate in a
current English apprenticeship? - Q2. What is the pass mark for a good (Grade C)
pass in GCSE maths? - Q3, At Level 2 in NVQ, what is the range of
learning hours (between highest and lowest hours
normally required)? - Q4. What of your current skills, knowledge,
competences are formally certified? - CLUE The world is not as it seems!
33 REASONS FOR ASSESSMENT MEASUREMENT
- Formative assessment in order to diagnose
weaknesses in ongoing learning, and to guide
future progress - Summative/terminal certification of what has been
achieved at the end of a period/element of
learning - Comparative systemic measurement
4THE ACTORS
- Individuals
- Employers (a very varied category)
- The state and policy makers
- International bodies (EU, OECD)
5INTERNATIONAL MEASURES
- In the past, economists had great difficulty in
- finding datasets that would allow them to measure
- skill with any degree of accuracy, particularly
- across countries. They were therefore often
- thrown back on using proxies for skills, such as
- Years of schooling
- Participation rates for given age cohorts
- Earnings
- None of these is a very good.
6PROBLEMS WITH THESE MEASURES
- Years of schooling tell us nothing about the
quality of the education or what, if anything,
was learned. Ditto participation rates, e.g.
Italian higher education. - Participation rates may reflect labour market
regulation systems rather than skill need, demand
or usage patterns. Germany has a very high level
of intermediate skills because licence-to-practice
regulations mean that many jobs can only be
accessed by people with L3. - The idea that earnings can be taken as being in
more or less direct relationship to skill is a
triumph of hope over reality.
7CURRENT METHODS
- Qualifications achieved
- Occupation as a proxy for the skills required.
Problem of variations of skill level required
within occupations, e.g. manager. - Tests PISA, IALS, ALLS, etc. Accurate but very
expensive. OECDs 2009 PIAC survey of adult
skills, will use tests of ICT, literacy and
numeracy on adult workers in different countries. - Job requirements individual responses to
surveys, e.g. Skills Survey.
8THE MAIN CURRENT MEASURE
- Although there are many different ways to measure
skills, public debate and policy in the UK now
centres around formalised assessment and
certifications systems that generate
qualifications (or sometimes parts thereof). - In some other countries this is less of an issue
9QUALIFICATIONS - DIFFERENT ACTORS, DIFFERENT NEEDS
- Individuals and employers want measurement/
- certification as a means of signalling
knowledge and skill that is for exchange in the
labour market. BUT individuals want
certification to cover a broad range of
general/transferable learning and to be portable,
whereas employers often want it to be narrow,
task-specific and non-portable. - The state wants standardised output units that
can form the KPIs for the public ET system, and
skills stock measures for the national workforce
10THE BIG PICTURE
- The performance of a national VET system and
skills policy can be conceived of in a number of
ways - 1. Internal Performance
- Efficiency, value for money, unit costs per
output - Levels of output
- Levels of participation
- Inclusiveness
- Quality of process and output
- Ability to meet demand (from individuals and
employers) - 2. The Impact of VET on
- Economic Performance
- Social Performance
11ONE PERSONS OUTPUT MEASURE IS ANOTHER PERSONS
INPUT MEASURE
- People who are interested in VET tend to see the
results of education and training as outputs. - It is true that skills and qualifications are the
outputs of the learning process. - HOWEVER, in economic terms, skills are inputs
into the productive process (alongside a wide
range of other inputs capital, RD, etc.). - In other words skills only generate an economic
output/outcome when they are utilised within the
productive process. - We will come back to this point
12LEVELS OF MEASUREMENT
- International
- Region and sub-region
- Sector and sub-sector (market segment within
sector) - Particular age cohorts within a population
- Particular types of skill, e.g. IT graduates
- Occupational groups
13COMPARING QUALIFICATIONS STOCKS - THE WORKFORCE
AS A WAREHOUSE
- English policy is driven by a stockpiling
approach - The labour market is viewed as a vast warehouse,
within which boxes are stored one box for each
worker (or potential worker). - The boxes are assigned labels denoting the
highest qualification held by each worker. - Policy tends to view it as an unqualified good to
ensure that the greatest number of boxes as
possible have as high a number on them as
possible. - If the proportion of boxes in your national store
with high numbers on the side exceeds the
proportion in another country, you are deemed the
winner!
14LEITCH ONWARD AND UPWARD
- The Leitch Review recommends that by 2020 we try
to be in the top 8 at every different level of
skill across the OECD (as measured by
qualifications in Basic Skills, and at Levels
2,3,and 4. - The pile of boxes must grow, and Leitch wants to
employers to certify far more of their in-company
and informal training
15CERTIFYING WORK-BASED CPD AND INFORMAL LEARNING
- Unlikely to grow massively because
- Costly
- Complex
- Often simply impossible
16SOME PROBLEMS WITH THE WAREHOUSE APPROACH
- This approach ignores what is actually in the
- boxes, or what use it might be put to
- Is the content of a Level 3 box in the UK the
same as a Level 3 box in Germany? - Is one Level 3 box in the UK the same as another
in terms of content? - Levels tell us nothing about the relevance of the
qualification to the labour market. Is a
vocational L3 better or worse than a L4 degree in
Art History if the job the person occupies is one
in retail management?
17APPLES AND PEARS?
- Huge efforts expended on trying to standardise
- measures at ISCED levels across developed
- world, but
- UK vocational L3 is far narrower in content than
a L3 in Germany or Norway. - PISA and foreign languages, history, etc.
- Is a degree from a South Korean university the
same as a degree from an Australian university?
Same subject/different subject?
18QUALIFICATIONS IN ENGLAND SAME LEVEL, SAME TYPE
(VOCATIONAL), VERY DIFFERENT ANIMAL!
- It is unwise to assume that even in the narrow
range of a - vocational award at the same level in the same
country, - that what is inside the box is anything like the
same. - Construction L3 1,220 notional hours
- IT L3 1,475 to 2,400 notional hours
- Animal Care L3 650 notional hours
- Engineering L3 3,900 notional hours
- Those are official figures. Unofficial industry
figures for a - L2 in the Fitness industry are 60 hours! Compare
with its - academic L2 equivalent 5 GCSEs at A-C grade.
19QUALIFICATIONS WHAT DO THEY TELL YOU?
- Usually they measure (more or less imperfectly)
- how well someone has acquired and can
- demonstrate particular knowledge and
capabilities. - What they are often less good at telling you is
- Whether acquisition of the qualification, via a
particular course of learning, inflicts costs
for example, by lessening originality or
willingness to question basic premises? - Whether the individuals ability to further
develop has been enhanced.
20MORE THINGS QUALIFICATIONS DONT MEASURE
- Qualifications only measure some parts of the
skill spectrum. For - example, they rarely offer much measurement of
generic or soft skills. - In the service sector, employers are increasingly
concerned in the - recruitment process to obtain
- Generic skills such as problem solving, and
inter-personal skills such as communication and
empathy. - Personal attributes (such as self discipline,
loyalty, and punctuality), which may not be
skills per se. - Aesthetic skills dress sense, deportment,
style, accent, voice. - As one UK call centre employer put it, we
recruit attitude. - These skills may be vital, but are usually
uncertified and therefore hard - to count.
21OTHER WAYS OF MEASURING PEOPLE-RELATED CAPITAL
- Business research shows that the private sectors
measurement of human capital often has little to
do with the public qualifications system.
Measures include - Human capital (vendor qualifications, appraisal,
supply chain TQM measures) - Intellectual capital
- Structural capital
- Customer capital
- Social capital
22STRETCHING THE QUALIFICATIONS BOX
- Employers say that the new Vocational Diplomas
(14-19) must cover teamworking. - QCA/DfES dream up a written exam in teamworking!
23DOES SKILL ONLY RESIDE IN THE INDIVIDUAL?
- Most attempts to measure skill accumulation focus
- on the skills held by the myriad of individuals
in - the workforce.
- However, modern conceptualisations of skill
- suggest that some skills are held collectively by
- work teams or communities within organisations.
- These collective competencies suggest that an
- organisations skills may add up to more than the
- sum of its individual employees skills.
24MEASURING SUPPLY NOT USAGE
- International benchmarking exercises (like
Leitch) make the vast assumption that if created,
additional skills will automatically be used. - This may not always be the case.
25MISSING QUESTIONS OF USAGE
- Skills deliver enhanced organisational
performance within - contexts set by the organisation. We know that
different - forms of
- Employee relations context
- Work organisation context
- Job design context
- Will materially impact on how productively
enhanced skills - can be deployed. Looking at the stock of skills
tells us little - about how they are being deployed.
- If the context is poor, skill usage may be
inefficient, and - productivity benefits from skills smaller than
expected.
26THE UK CONTEXT
- Chartered Institute of Personnel and Development
study in - manufacturing
- The current managerial approaches to job design
are - counter-productive.the types of impoverished job
we - see in manufacturing organisations up and down
the - country represent waste, on an enormous scale, of
the - resources, intelligence, skills and energy of
those - required to perform them.
- Recent Workplace Employee Relations Survey showed
no - signs whatsoever that high performance work
organisation - is spreading across UK economy. Take up has been
static - since 1998.
- More skills may therefore be wasted.
27DECLINING TASK DISCRETION
- If one element of being skilled is control over
how you do - your job, the UK workforce is in trouble.
- The 2nd Skills Survey (2001), showed that
- Between 1986 and 2001 there was a 14 per cent
decline in the proportion of workers who felt
they had a great deal of choice over how they did
their work. - The fall was sharpest for professional groups
(knowledge workers), from 72 per cent in 1986 to
just 38 per cent in 2001. - Rising skill levels, as proxied by
qualifications, have - not been accompanied by a rise in job control,
quite - the reverse. This begs questions about how
skills get used.
28POOR JOB DESIGN, POOR USAGE ISSUES FOR SKILLS
SUPPLY
- 2001 (UNPUBLISHED) Scottish Adult Literacy
research showed that - Reading skills
- Across the Scottish workforce, the following
percentages of workers - rarely or never use
- Info from computers 34
- Letters or memos 26
- Bills, invoices, spreadsheets 35
- Diagrams 31
- Manuals, reference books 53
- Reports, articles, magazines 52
- Foreign language material 91
- 27 of the workforce indicated rarely or never to
five or more - items.
- What price boosting adult literacy skills?
29OVER-QUALIFICATION
- In Britain, among 20-60 year old employees, the
- proportions holding qualifications at levels
higher than - those needed to obtain their current job were
- 1986 29
- 1992 33
- 2001 37
- Increasing skill supply and skill stocks may look
good in - international comparison, but how many of the
skills get - used?
30Qualifications Demand and Supply, 2001
31SKILLS AND WHAT ELSE MAKES THE DIFFERENCE?
- Research tells us that skills is one among many
- factors that impact on the performance of firms
and - the economy as a whole. Unless skills forms part
of a - balanced scorecard for the economy, we may miss
- where the real weaknesses in
- investment lie
- RD
- Innovation
- Capital investment in plant and machinery
- Infrastructure
-
32THE SCOTTISH STORY
- Over a 20 year period, Scotland has spent about
18 more per head of population on VET than
England. - Scotland has a larger proportion of its age
cohort in higher education than England. - The qualifications held by the Scottish workforce
are at levels that roughly might be reached in
England in about 2020. - Is Scotlands economy performing better than
Englands? - Is its GDP per head higher?
- NO.
- A cause for some heartache among Scottish policy
makers.
33LEITCH THE BIG ISSUE LEFT HANGING
- Skills are a derived demandfor the supply of
skill to turn from merely potential change in
performance into a tangible increase in
productivity, the available skills of the
workforce have to be effectively utilised.
People need to be in jobs that use their skills
and capabilities effectively.
34LESSONS FOR FUTURE RESEARCH
- Dont obsess about stocks of qualifications.
They can hide poor usage - and over-qualification.
- Remember that qualifications are only a proxy for
one part of the skills - spectrum.
- Measures of generic skill formation and usage are
very important. - Comparisons need to look at demand for skills.
The production of - masses of graduates, for example, may be a waste
of resource if their - skills are not actually needed in the labour
market. - Comparisons need to consider usage of skill.
This, in combination with - a range of other factors, is important to gauging
how well the skills - being produced aid economic performance. Supply
and demand might - be in balance, but if usage is inefficient, or
the other drivers of - productivity (e.g. RD) are missing, the economic
impact may be small.
35MORE LESSONS
- The non-economic benefits of learning need to be
considered much - more carefully. The real spill-overs from
learning may not be - economic, but social.
- The sustainability and quality of learning
provision within national - systems is important, not just volume
throughputs. - Balanced scorecards are vital. Skills alone tell
us little.
36METHODOLOGY IS ADVANCING
- OECD work
- EU work SKOPE/RAND/Danish Technology Institute
- In the UK Skills Survey 3 will provide an
- enhanced time series on generic skills, skill
usage, and - employers demand for skills.
- Many UK agencies are using a wider range of data,
in - something closer to a balanced scorecard
approach. - Fixations on qualifications is starting to
decline. - There is increasing interest in data on usage and
demand.