ProLearnG Gendersensitive Evaluation - PowerPoint PPT Presentation


PPT – ProLearnG Gendersensitive Evaluation PowerPoint presentation | free to view - id: 9441d-ODJmZ


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

ProLearnG Gendersensitive Evaluation


developed in a participatory fashion, including all stakeholders and changeable ... analysis of content: examples, illustrations, stereotypes, role models, etc. ... – PowerPoint PPT presentation

Number of Views:58
Avg rating:3.0/5.0
Slides: 28
Provided by: elke2


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: ProLearnG Gendersensitive Evaluation

ProLearn-G Gender-sensitive Evaluation
  • Fellowship Project
  • Elke Dall
  • Draft 03.09.2004

supported by CSI KTH bmbwk
Draft for PROLEARN evaluation guidelines
  • This document will give a general outline on
    evaluation methodologies and processes and what
    can be understood as a gender-sensitive approach
    to evaluation and evaluation of gender-issues
    in the context of e-learning projects and
  • It will first given an outline of general goals
    of evaluation and evaluation processes. Then the
    possible relations between gender and evaluation
    and goals for the gender analytical approach. The
    presentation will deal with the development of
    objectives ? questions ? indicators ?
    methodologies ? data collection ? interpretations
    ? results ? actions for gender sensitive
    evaluation of e-learning resources.

Preliminary words to evaluation methods
  • The goals initiative ? evaluation ? learning ?
    change ? next initiative
  • The process evaluating the context (effects of
    and on individuals, organisations, communities
    the broader social, political, cultural and
    economic context), the product (e-learning
    resource) and the process (development, design,
    implementation) using qualitative and
    quantitative data
  • The focus complex inequalities result as gender
    interacts with power relations evaluation shall
    raise awareness towards these as simple actions
    can often avoid unequal treatment
  • The values evaluation is not value-neutral ? it
    is important to involve all stakeholders and feed
    back results to ensure validity and
  • The constraints affordability, time,
    reliability, politics, context sensitivity, etc.
    have to be considered in planning the evaluation

General goals of evaluation
  • The goals of an evaluation process are
  • To check if the project objectives have been met
    and resources wisely utilized as justification
    for the funding agency
  • To identify areas for improvement in a project or
  • To surface and resolve disagreements
  • To set priorities and goals
  • To clarify and tackle problems
  • To decide new strategic directions
  • To get feedback, appraisal and recognition
  • To celebrate achievements
  • To attract resources
  • Evaluation is not just done at the end of a
    project, it should be ongoing and part of the
    project plan from the beginning. Results can be
    used and disseminated, incorporated to
    organisational learning and published (formative
    or summative).

General goals of evaluation
  • The use of the results will depend on the
    objectives of the evaluation you have performed
  • Formative Evaluation involves collecting
    information about a prototype resource that will
    help in its development and ensure it works
    effectively. Emphasis at this point is typically
    on improving usability and refining content.
  • Summative Evaluation takes place at the end of
    the developmental cycle, in order to prove the
    success of the resulting resource. For teaching
    and learning resources, this may involve
    demonstrating that the resource has made a
    difference in some tangible way. This may be to
    show that users have learned something from it,
    or that it has brought about a beneficial change
    in practice.

Evaluation process
  • In the process of evaluation it is nesseary
  • to define the purpose of the evaluation (your
    values, your goal)
  • to define the evaluation objectives
  • to define the evaluation questions
  • to break them down to categories
  • to define indicators for each category to guide
    the data collection
  • to define the critical stakeholders
  • to identify the means to monitor and collect
    information from the stakeholders
  • to collect the data (qualitative and
  • to categorise of the findings according to the
    evaluation questions
  • to gather and document illustrating and
    supporting stories
  • to reflect on the findings and to extract lessons
  • to prepare the evaluation report and to reflect
    the information

Evaluation and gender
  • Evaluation processes involve one or more
    evaluators measuring a particular resource
    against a set of criteria ranging from some
    clear simple questions to relatively detailed
  • There are several possible relations between
    gender and evaluation
  • Gender-sensitive evaluation the process of
    evaluation shall be gender-sensitive
  • Evaluation as means for gender mainstreaming
    evaluation shall help to show problems and
    support projects in improving their gender
  • Evaluation from the gender point of view
    evaluation of the project performance in Gender
  • Research in the field of gender and evaluation
    basic research on the questions in this field
    (definition of gender-specific criteria and
    requirements for projects, development of methods
    to gather information)

Goals of the gender analytical approach
  • A gender analytical approach to evaluation
    provides a framework for identifying and
    analysing specific gender issues that determine
    the different impact ICT intervention as have on
    women and men.
  • The key elements are
  • gender equality
  • gender roles
  • practical gender needs and strategic gender
  • womens empowerment
  • gender and technology
  • The evaluation should be carried out in a team
    (gender expert, ICT expert, participation of key
    stakeholder groups, user participation members
    involved in the project from different levels).

Evaluation strategy layout
  • Detailed strategies for the process to carry out
    the evaluation, gathering data and the
    methodologies to be used have to be defined.
  • It is useful to use a form for the Testing
    Profile in order to summarize the strategy
    defined for the process (goals, objectives,
    indicators, methods and a timeline) before
    starting to gather the data. ()

a document containing an example for this
Gender Evaluation Testing Profile is also
provided in the framework of this
project. contact (Elke Dall)
Specific gender objectives
  • Possible objective for gender-sensitivity are
  • To ensure that womens practical and strategic
    gender needs are addressed in future programmes.
  • To understand how and why an ICT project has
    negatively (positively?) affected women in a
    community for social change.
  • To assess the gender equality in the project
    context and organisation.
  • To assess the gender sensitivity in the processes
    carried out to produce certain resources.
  • To assess gender aspects in the product
    (e-learning resources) themselves (e.g. from the
    point of view of didactics, design, language,

From objectives to questions
  • Definition of the specific gender issues to
    address in the evaluation
  • Check core thematic areas and the elements of a
    gender analytical framework
  • Define a clear goal for the evaluation
  • improvement of the product / the learning
  • improvement of the didactical and pedagogical
  • improvement of the set-up of the project (project
    organisation, project team, project
  • dealing with social inequity in the context
    (influence on the stakeholders, etc.)
  • Accordingly focus on context and/or process
    and/or product when defining the categories for
    gender analytical questions.

From objectives to questions
  • Context Analysis
  • Review of factors on individual, organisational,
    community, socio-economic, cultural and political
  • what gender issues can be identified in relation
    to the various stakeholders in the project
    context? are gender issues (GM goals) mentioned/
    requested/ favoured by the founding body? is what
    way can the project contribute to positive social
    change in its context? is there a way for the
    project to extend its scope so it can contribute
    to positive social change? (e.g. allow women
    groups to distribute the learning resource for
    free, attract more women through more targeted PR
    and advertising of the course, etc.)
  • Respect political issues when evaluating
  • form the team to carry out the evaluation and to
    implement its results gender-balanced
  • make sure, the evaluation is helping to build
    support and not causing irritation (consider
    group dynamics, etc.)

From objectives to questions
  • Definition of evaluation questions
  • breat down the evaluation objectives into
    categories () and articulate questions related
    to gender and e-learning
  • explicitly define the queries, dont select too
    many questions, dont formulate them too vague
  • consider womens participation in the project,
    development of strategies within the project to
    respond to gender issues, roles in
    decision-making in the project, project
    strategies and content, …

a xls-docment containing an extensive list of
categories and possible questions will also be
provided in the framework of this project contact (Elke Dall)
Example questions for different categories
  • These categories and questions are just
    illustrative examples ()
  • vision and strategy (stated goals and objectives)
  • are gender-equality and equal opportunities as
    goals defined and mentioned in the project vision
    or objectives?
  • is Gender Mainstreaming (GM) defined as a
    strategy in the project to achieve gender
  • were women or groups of women identified as
    specific beneficiary groups?
  • team and project management
  • what was the level of women and mens
    participation in project activities?
  • are men and women equally paid for equal
  • are gender-biased assignments of tasks avoided
    (not following stereotypes)?
  • are women in management positions?
  • are both gender involved in design,
    implementation and evaluation?
  • project communication
  • are informal structures of communication
    transparent and opened?
  • are decision-making structures transparent?
  • is the team exchanging knowledge (e.g.
    peer-to-peer-teaching) and working jointly on
    didactics, technology and evaluations?
  • gender competence and qualification
  • are gender issues a topic in the project team?
  • does anybody have the designated responsibility
    for Gender Mainstreaming?

a xls-docment containing an extensive list of
categories and possible questions will also be
provided in the framework of this project contact (Elke Dall)
Example questions for different categories
  • These categories and questions are just
    illustrative examples ()
  • participatory design / user needs
  • are users involved in the development and design
    (respecting gender-parity)?
  • are the different needs of user groups reflected
    in design and development of the learning
    materials? (family situation, competences in the
    use of technology, hardware conditions, financial
    and economical conditions, time constraints,
    previous learning experiences, motivation to use
    multi-media learning systems, etc.)
  • is the development of the material content-driven
    (by didactic and pedagogical considerations) and
    not technology-driven?
  • technical support
  • are the users qualified to use the technology?
    are means to acquire the qualification available?
  • is support available and can it be obtained
    through different channels telephone, e-mail,
    fora, FAQ-lists, coaching, "help"-texts, etc.?
  • is help context sensitive and adapted to
    different situations of use?

a xls-docment containing an extensive list of
categories and possible questions will also be
provided in the framework of this project contact (Elke Dall)
Example questions for different categories
  • These categories and questions are just
    illustrative examples ()
  • success / satisfaction
  • are the stakeholders (users or beneficiaries,
    team members, funding institutions, etc.)
    satisfied with the project results / products /
    materials offered? (sex-disintegrated data!)
  • what were the expected and unexpected results of
    the project (in relation to gender issues)?
  • topics and their presentation
  • are interests of both genders respected?
  • are topics presented using interdisciplinary
  • are men and women equally and balanced
  • are role models for women and men represented?
  • are examples gender-neutral?
  • didactics
  • is a training for "how to learn with e-learning"
  • is the relevance of the acquired knowledge
    clearly demonstrated?
  • are introductory notes (to content, technology
    and design) and summaries (for each module and
    separate parts of the material) available in
    order to quickly assess the content?
  • are there suggestions made how to plan the
  • are different learning styles and knowledge
    levels respected?
  • is groupwork, partnerwork and self-organised
    learning encouraged and in a good balance?
  • is interdisciplinary application of the content
    encouraged and required?

a xls-docment containing an extensive list of
categories and possible questions will also be
provided in the framework of this project contact (Elke Dall)
Example questions for different categories
  • These categories and questions are just
    illustrative examples ()
  • technology
  • is the necessary equipment clear (hardware,
    software, system used, internet access etc.)?
  • is the compatibility with other systems secured
    (e.g. different hardware, different displays,
  • is technology adaptable and personalisation
  • communication
  • are first-time users welcomed (in a
    gender-sensitive way)?
  • are contacts between students and teachers and
    among students possible and encouraged?
  • are the learners supported to build a community?
  • are technical terms explained and not assumed as
    being known?
  • is feedback constructive and related to the
  • language use
  • is stereotypic use of language avoided?
  • is discrimination (against women, but also
    against age, race, religion, minorities, …)
  • is gender-sensible language used in all texts of
    the project?
  • are users addressed in a personal way using
    female and male or gender-neutral forms?
  • are both genders visible in the language used?
  • graphical design

a xls-docment containing an extensive list of
categories and possible questions will also be
provided in the framework of this project contact (Elke Dall)
From questions to indicators
  • Tools to capture the information about the issues
    defined have to be selected and adapted
    indicators are specific (explicit) and
    objectively verifiable measures of results or
    changes brought about by a project
  • Indicators to guide the information gathering
    process have to be developed and selected
  • pointers, numbers, facts, opinions or perceptions
    that look into and measure changes of specific
    conditions or situations (but not too many)
  • quantitative or qualitative
  • SMART specific, measurable, achievable,
    realistic, time-bound
  • sex-disaggregated
  • easy to use and to understand, clearly defined,
    technically sound
  • should be established during the project planning
    phase, be linked to the project goals and
    objectives and relevant to the needs of the user
  • developed in a participatory fashion, including
    all stakeholders and changeable during the
    process of evaluation

Gender indicators
  • Examples for indicators
  • number of mentioned gender issues in official
    statements -gt analysis of project documents,
    objectives, proposals, website, etc.)
  • percentage of women in different levels and at
    different project activities
  • relations between salaries of women and men
  • formal decision-making structures vs. informal
    structures (methods questionnaire, interviews,
    satisfaction, etc.)
  • way of participatory design, inclusion of
  • satisfaction of the users, success rate,
    drop-out, etc.
  • analysis of content examples, illustrations,
    stereotypes, role models, etc.
  • analysis of language nr. of generic masculine
    expressions used, nr. of sexist expressions,
    stereotypes, etc.
  • analysis of curriculum time consumption, nr. of
    possibilities of adaptation of content and
    presentation of the content (respecting learning
    styles, knowledge levels, etc.)
  • communication channels nr. of contacts between
    different groups, quality of feedback, etc.

From indicators to a methodology
  • In order to gather data on these gender
    indicators, methodologies have to be chosen and
    instruments have to be set up.
  • There should be a mix of methodologies chosen to
    gather the data, they should be participatory and
    ensure collection of sex disaggregated data.
  • Data and information gathering tools and
    methodologies are e.g.
  • surveys and questionnaires (e.g. electronic
  • interviews (with stakeholders)
  • discussions or focus groups (with staff,
    beneficiaries, etc.)
  • observations
  • system data, log files
  • records and statistics
  • documents analysis (papers, reports,
    correspondence, minutes of meetings, etc.)
  • online discussion areas for reflections and
  • stories (accounts that reveal the stakeholders
    perspective about the project)
  • … with their respective strengths and weaknesses.

From the methodology to the questions
  • The questionnaire / interview questions should be
    a combination of open questions and closed
  • Designing the evaluation questionnaire it has to
    be considered
  • how are the questions constructed?
  • when are they asked?
  • who will reply to them?
  • how is the information received validated and
  • what will be done with the information collected?
  • Different types of questons can include
  • measurement (e.g. how many male / female students
    have successfully completed the module? exploring
    cost factors, etc.)
  • comparison (e.g. made with more conventional
    means of teaching and learning)
  • exploration (how was the learners experience
    during the course? subjective information on
    attitudes, motivational factors, etc.)

Data collection
  • For instance, data could be
  • quantitative data
  • analysis of statistical data
  • number of women and men involved
  • number of women and men trained
  • logfiles
  • user questionnaires (e.g. on their situation
    regarding use of time, money, etc.)
  • drop-outs, success rates
  • Qualitative data
  • interviews and questionnaires answered by
    relevant stakeholders
  • project manager
  • team members
  • teachers, moderators, tutors
  • users, students, learners
  • others if applicable
  • analysis of texts (project descriptions,
    proposals, presentations like website, etc.)
  • lessons learned
  • gathered through interviews

From evaluation results to actions
  • The results shall be interpreted according to the
    categories and indicators.
  • The data gathered and answers given shall lead to
    interpretations and then to results of the
  • The results shall be reflected critically and
    suggested actions shall be considered.
  • Then it it is necessary to put evaluation results
    to work!
  • A dissemination plan shall detail the activities
    and the timeline for sharing evaluation results
    with different stakeholders.
  • The presentation of the results depends on the
    audience to which it is addressed and should be
    carefully considered. The requirements,
    recommendations and comments of the audience
    should be integrated.
  • Who are the key stakeholders that will receive
    evaluation results?
  • What will they want to know? How will they use
    the data?

Critical stakeholders
  • Evaluation cannot include all groups that have
    participated, benefited or not benefited from the
    activities ? important stakeholders have to be
    selected and included in the evaluation process
  • internal (project executors including project
    staff and management) and external (project
    beneficiaries) stakeholders
  • direct (those who are/or were directly involved
    in project activities) and indirect stakeholders
    (those who did not participate in the project and
    may or may not have been affected by it)
  • other organizations that do similar work can also
    be seen as stakeholders (offering important
    insights and sector commentary providing a
    broader, but focused, context)
  • project beneficiaries /users (the primary target
    group for the e-learning initiative) are integral
    to the process of discovering and analyzing
    gender and e-learning issues.
  • In some contexts, a formal report of the
    evaluation results may be required, while in
    others a more informal presentation of the
    outcome may be more appropriate.

Possible actions
  • Evaluation Results shall result in learning and
  • Research and Critical Understanding Evaluation
    results can point to areas of your work that
    require additional research starting a learning
    process that will build your organisations
    critical expertise in a particular field of work.
    Evaluation results can also be used to test and
    advance analytical frameworks that inform gender
    equality and e-learning.
  • Sharing Best Practices and Lessons Learned share
    information about what gender issues in
    e-learning have come up. Using evaluation results
    as material to document best practices and
    lessons learned and sharing this information
    builds up a pool of needed reference material.
  • Project Planning and Design Evaluation results
    should inform future project design and
    implementation. Using evaluation results from
    completed projects as starting points for
    developing new projects maximises the
    organisations learning about its work. It also
    increases the likelihood that the next project
    will be more successful and satisfying for the
  • Resource Mobilisation Evaluation results can be
    used to acquire projects and funding. Evaluation
    reflects a track record and experience of working
    in a particular area. At a broader level,
    evaluation results can demonstrate the need for
    resources to be committed to gender and

From actions to a broader audience
  • Useful ways of disseminating the evaluation
    results are
  • Advocacy and Lobbying inputs in national,
    regional or global policy debates as research
  • Public Education and Networking communicating
    and building relationships with constituencies
    (communities, other organisations, agencies,
    funding bodies, etc.) A participatory approach to
    evaluation requires feedback of results to all
    communities involved with a project.
  • Basis for publishing articles in newsletters,
    popular and academic journals and annual reports
    and presentation in networking forums (electronic
    discussion groups or face to face meetings)
  • Marketing and Public Relations advocate for the
    work to raise awareness about the project, etc.
    Using evaluation results in publicity materials
    pamphlets, web sites, press releases, etc can
    raise the profile by establishing a rigorous
    approach to work. Evaluation results can also be
    used to provide media with "evidence" and story
    ideas related to gender issues.

  • This document is mainly based upon the following
    sources and own adaptations
  • Cook, J. Evaluating Learning Technology
    Resources, Learning Technology Support Service.
    University of Bristol, UK. 1999.
  • Creanor, Linda / Glasgow Caledonian University
    E-Learning Guides. 7. Evaluating E-Learning.
  • GEM Gender Evaluation Methodology for internet
    and ICTs developed by APC WNSP, 2001.
  • Wienold, Kirsten Evaluation multimedialer Lern-
    und Informationssysteme. Gender Mainstreaming in
    der Evaluation. 2002.
  • Wiesner, Heike, Marion Kamphans, Heidi Schelhowe,
    Sigrid Metz-Göckel, Isabel Zorn, Anna Drag,
    Ulrike Peter, Helmut Schottmüller Gender
    Mainstreaming in "Neue Medien in der Bildung"
    Leitfaden. 2004. http//dimeb.informatik.uni-breme
  • Wiesner, Heike, Heidi Schelhowe, Sigrid
    Metz-Göckel, Marion Kamphans, Ulrike Peter,
    Helmut Schottmüller, Claudia Kedenburg, Anja
    Tigges, Kirsten Wienold, Marc Jelitto, Hannah
    Cho-Heinze "GM-Styleguide" Gender Mainstreaming
    im Kontext Neuer Meiden. 2003. http//www.medien-b
  • Wood, Peregrine Building Gender Considerations
    Into ICT Evaluation Work. APC WNSP. 1997.