Impact Assessment Study of eGovernment Projects: Methodology and Some Preliminary Results

1
Impact Assessment Study of eGovernment
ProjectsMethodology and Some Preliminary
Results

• Subhash Bhatnagar
• Indian Institute of Management, Ahmedabad
• (subhash_at_iimahd.ernet.in)
• and
• Advisor e-government, Information Solutions Group
  (Informatics Program)
• World Bank, Washinton DC
• sbhatnagar_at_worldbank.org

2
Study Objectives

• Review the status of impact assessment of
  eGovernment projects in developing countries
• To develop and present an analytical framework to
  conduct analysis on the impact of e-delivery of
  services.
• To study selected e-delivery projects, assess
  their impact, and identify key determinants of
  economic, organizational, and social impact.

3
Study Team

• Indian Institute of Management, Ahmedabad (IIMA)
• Subhash Bhatnagar, Rama Rao, Nupur Singh, Ranjan
  Vaidya, Moushami Mandal
• London School of Economics
• Shirin Madon, Matthew Smith
• eGov Practice Group
• Deepak Bhatia, Jiro Tominaga, Soren Giggler
• Reviewers
• WBI, EAP, SAR, OECD, GICT,PREM, DEC
• Sponsors
• ISG,EAP,IIMA

4
Learning from Past Assessments

• Variety of approaches have been used-client
  satisfaction surveys, expert opinion,
  ethnographic studies
• Client satisfaction survey results can vary over
  time as bench mark changesneed for
  counterfactuals
• Often studies have been done by agencies that may
  be seen as being interested in showing positive
  outcome
• Lack of credibility of results-different studies
  of the same project show very different outcomes
• Lack of rigor in sampling-results can not be
  easily generalized
• Lack of rigor in controlling for external
  influence-need for counterfactuals ignored.
• Lack of a standard methodology-making it
  difficult to compare projects
• Hardly any projects do a benchmark survey

5
Critique of Existing Frameworks

• Biased towards quantification of short term
  direct cost savings- quality of service,
  governance and wider impacts on society not
  studied.
• Conceptual in nature-hardly any frameworks have
  been applied to assess impact of real projects
• Variety in delivery models has not been
  recognized. Impact is a function of the delivery
  model and the nature of clients being served
• Practical issues of paucity of data have not been
  taken into account-particularly in a developing
  country context where baseline surveys are not
  done and ME systems are weak

6
Proposed Framework

• Small budget exploratory study- focus on
  retrospective assessment of e-delivery
  systems(B2C and B2B)
• Balanced approach between case study and
  quantitative analysis
• Recognizes that some part of the value to
  different stakeholders can not be monetized
• Understand how inputs lead to outputs and
  outcomes in different project contexts
• A practical methodology that can be used for
  designing bench mark surveys, ME systems and
  prospective evaluation of Bank projects in
  countries with various delivery models and
  paucity of data

7
Measurement Framework
8
Projects of e-delivery of Services

• Issue of land titles in Karnataka (Bhoomi) 180
  Kiosks, 18 million titles issued
• Property registration in Karnataka (Kaveri) 230
  offices
• Computerized Treasury (Khajane) 240 locations
• Property Registration in Andhra Pradesh AP 400
  offices. 5.7 million documents, 3.6 million
  encumbrance certificates, 2 million market
  valuation slips
• eSeva center in Andhra Pradesh 250 locations in
  190 towns, Used by 3.5 million citizens,
• eProcurement in Andhra Pradesh
• Ahmedabad Municipal Corporation (AMC) 14 Civic
  Service Centers
• Computerized Inter State Check Posts in Gujarat
  8 locations
• eProcurement in Chile (Comprasnet)
• Income Tax on-line in Chile

9
Methodology Used for Assessment

• Selected 10 mature, wide scope and scale projects
  of e-delivery of services.
• Collected data through structured survey from
  clients, employees, supervisors using
  counterfactuals ( for old non computerized
  delivery and new e-delivery system)
• Survey instrument customized to each project,
  adapted in local language
• Sampled 240 clients, 30 employees randomly from
  locations stratified by activity levels and
  remoteness
• Collect data on investments, operating costs,
  activity levels, revenues, employee strength from
  agencies.
• Develop a case study-organizational context,
  process reform, change management.

10
Profile of AMC Respondents

• Partial data of 139 (out of 240) citizens
• 40 graduates, 50 schooled, 10 barely literate
• 16 workers, 50 business, 10 white collar
  workers, 24 supervisor/management
• Aware for 28 months, Average Income Rs 7000 pm
• Urban population 38 within 1-3 kms of a center,
  25 4-6 Kms, 20 lt 1 Km
• Payment of property tax, birth/death registration

11
Desirable Attributes in Service Delivery at AMC
Centers

• Less time and effort 48
• Less corruption 41.7
• Less cost to the citizen 29.5
• Greater Transparency 27.3
• No need for agents 20.1
• Good complaint handling 19.4
• Convenient time 16.5
• Privacy and security 15.1
• Waiting facilities 12.2
• Helpful attitude of civil servants 10.8
• Error free transactions 9.4
• Fair treatment 8.6
• Accountability 7.9
• More predictable outcome 5.4
• Clarity/simplicity of procedures 5.4

12
AMC Impact on Client Costs
13
Client Perceptions (Ratings on a 5 Point Scale)
14
Attitude to eGovernment
15
Preliminary Observations

• Overall Impact
• Significant positive impact on cost of accessing
  service
• Strong endorsement of eGovernment
• Variability across different service centers of a
  project
• Reduced corruption-outcome is fragile
• Any type of system break down leads to corruption
• Agents play a key role in promoting corruption
• Private operators also exhibit rent seeking
  behavior given an opportunity Systematizing
  queues and appointments helps prevent break down
• Small improvements in efficiency can trigger
  major positive change in perception about quality
  of governance.
• Challenges
• No established reporting standards for public
  agencies- In case of treasuries, the AG office
  has more information on outcome.
• What is the bench mark for evaluation-past
  performance or potential?
• Public agencies are wary of evaluation-difficult
  to gather data