ICT in Education Toolkit Version 2.0a
September 2006
Manage Users |   How to Use the Toolkit   Concept and Blue Print   Logout
  Welcome , NIUE
Toolkit Map Filing Cabinet Team Bookshelf Search Handbook Toolkit Library My Messages
Tool 6.1: Evaluation of ICT Interventions
  OVERVIEW
1 Classes of Evaluation
Class 1: Degree of Implementation
Class 2: Degree of Proper Use
Class 3: Degree of User Satisfaction
Class 4: Degree of Effectiveness
Class 5: Degree of Subsequent Application
Class 6: Degree of National Effect
2 Designs of Evaluation
3 Modes of Measurement of Evaluation
4 Management and Oversight of Evaluation
Reference Information
  Print This Tool

Toolbox 6:
Assessment and Subsequent Actions
6.1 Evaluation of ICT Interventions
6.2 Adjustment & Scaling Up
View Toolkit Map
View ICT for Education Handbook
 
 
  Class 5 Evaluations: Degree of Subsequent Application (Limited to Learning Objectives)
 


This Class of evaluation aims to determine the extent to which the ICT-Intervention is effective in preparing learners who subsequently apply the learned knowledge, attitudes, and skills in their later schooling, jobs, and social lives. If the ICT intervention is a pilot that is intended to test the implementability of the intervention or if it is limited to a specific technology, there may not be time to apply Class 5 before the pilot is modified and extended to a larger scale. However, if the pilot aims to change learner's subsequent application of knowledge, attitudes, or skill, then Class 5 Evaluation should be used as part of the evaluation of the pilot.

One of the claimed potential of ICTs is their ability to facilitate in the learners high level cognitive skills of application, problem solving, and learning how to learn. Learners may acquire considerable new capabilities as the result of an education or training intervention and yet not subsequently apply them. That can be attributed to many reasons:

  • The intervention had not trained the learners in how to make the applications and ultimately learn on their own;
  • The knowledge, attitudes, and skills learned are not relevant to the learners' subsequent lives;
  • Circumstances in the learners' subsequent lives make the application difficult or unrewarding.

An example of the latter would be when learners increase their creative problem-solving abilities but subsequently work for companies that are run autocratically by top managers who discourage creative problem-solving.

Class 5 evaluations address the hard question: Has the enhanced learning made a difference in the subsequent thinking and behavior of the learners? Class 5 evaluation is even more complex and difficult to conduct that Class 4 evaluation. It aims to discover whether the intervention caused changes in how the learners think and behave in their lives several years after completing participation in the intervention. That is difficult to determine because the extent of application will be affected not only by the capabilities developed during participation in the ICT-Intervention but also by the capabilities acquired over the learners' entire schooling and by the circumstances of their subsequent lives. In addition, Class 5 evaluation requires keeping track of the intervention group learners and the control group learners through their subsequent lives and securing their cooperation for additional data collection, both of which can be difficult. Once the evaluators lose track of 30 percent of the learners in either group, it is hard to know how representative the located learners are of the initial groups.

Class 5 evaluations are rare. There are several reasons for that. It is presumed that if people learn something, they will subsequently apply it, a presumption that is frequently not correct. Class 5 evaluations are difficult and expensive to conduct. Policymakers who supported (or opposed) a new ICT-Intervention rarely remain in office long enough to request such evaluations, and even when they do, their attention has often focused on other matters. Nevertheless, Class 5 evaluations can be valuable. They go beyond the objectives set for the intervention to its intermediate level goals. In other words, Class 4 evaluation addresses outputs and Class 5 addresses intermediate outcomes. In essence, Class 4 assesses "merit" and Class 5 assesses "value." The ultimate goals or outcomes are addressed in Class 6 evaluations.

Class 5 evaluations usually do not begin until 1 - 10 years after participants have completed an ICT-Intervention. Normally, these evaluations are a follow-up to Class 4 evaluations, using the same treatment and control groups of that evaluation.

The evaluation of the application can focus on many different questions. Each of these questions may be answered with multiple sources of data (see Section 3 below). A list of important questions that may be addressed appears below (Box 6.5). The Evaluation Team may select from them as appropriate, and add their own. Evaluation of application of knowledge, attitudes, and skills may also include questions about the learners' characteristics that might also affect learning application.

Box 6.5 - Questions to Determine Degree of Effectiveness

  1. To what extent did the learners using the ICT Intervention apply more or less than they would have otherwise in each of the subject-areas enhanced by the intervention?
  2. To what extent did the learners using the ICT Intervention apply more or less than they would have otherwise in: Memorization of information, retrieval and storage of information, exploration, application, analysis, evaluation, and constructing or designing?
  3. To what extent did the learners apply more or less than they would otherwise the ICT Intervention in communication skills: reading, writing, listening, and speaking?
  4. To what extent did the learners using the ICT Intervention apply more or less than they would have otherwise the cognitive skills of critical thinking, problem solving, applying knowledge and skills to new situations, and learning on their own.
  5. To what extent did the learners using the ICT Intervention apply more or less than they would have otherwise in technology utilization?
  6. To what extent did the learners using the ICT Intervention apply more or less than they would have otherwise in respect to lifelong learning?
  7. To what extent do the answers to the above Class 5 questions vary by geographic region; by socio-economic characteristics of the learners; by gender of the learners: and by the family, job, and community contexts of the learners?

Next

  Acknowledgements | Terms & Conditions

Copyright © 2006. All rights reserved.

Site designed and maintained by: Sonjara

UNESCO Bangkok JFIT InfoDev (World bank) AED Knowledge Enterprise