This class of evaluation aims to determine the extent to which the ICT Intervention is being implemented as intended.

Education innovations are often not fully implemented as intended. This could be due to the planners' having unreasonable expectations, infrastructure not performing as represented, contentware developers not producing as promised, funding not being adequately disbursed, educators not being satisfactorily trained in using the ICT intervention, or unexpected external factors interfering with implementation.

Early detection of incomplete implementation permits corrective actions. Large disparities may call for new management, review of implementation plans, and consideration of whether the project is so far off course that it should reset. In addition, if implementation is far from desirable, it often is wasteful to conduct higher-level Classes of evaluation, because poorly implemented innovations rarely prove to be notably effective.

Implementation can begin to be evaluated as the ICT intervention elements are in place, assessing whether they meet the planned specifications. Evaluation throughout the implementation rollout in schools or the community can provide early warnings of shortcoming and maximum opportunity to correct them. Evaluation after several years can indicate whether there has been decay in the implementation after the initial rollout.

The evaluation of implementation can focus on many different questions. Each of these questions may be answered with multiple sources of data (see Section 3). A list of important questions that may be addressed appears below (Box 6.1). The Evaluation Team may select from them as appropriate and add their own.

This class of evaluation aims to determine the extent to which the ICT-Intervention is being used as intended.

A reference point for this evaluation is the section in the ICT Policy Program Decision Document in Tool 2.2 dealing with "How the technology will be used to advance educational objectives." These include Learning and Instructional objectives set for the Project and its different components in terms of the following taxonomies:

ICT Interventions can be implemented well but still not be used as intended by users (teachers, facilitators, learners, administrators, etc.). They might not be used at all, used considerably less than intended, or used frequently for unintended purposes. That may be because the users
• misunderstood the guidance or instructions given,
• cannot make the intervention work as they were directed,
• are bored by the intended use and found alternative uses,
• are not convinced in the value of the ICt intervention, or
• were satisfied with the intended use which led them to find supplemental uses. For instance, a telecenter intended to facilitate communication and develop literacy skills might be used for the marketing of local crafts or e-mailed fraud schemes.

Early detection of low usage and mistaken usage might allow modifications that save the ICT Intervention from failure. Early identification of unintended positive uses may allow modifications that enhance those uses. Quick responses against potential abuses may avert adverse publicity and prevent closing of the intervention. In addition, if an ICT Intervention is not being used much, it is wasteful to conduct higher class evaluations, and if it is being used largely for unintended purposes, that would be important to know when planning the higher class evaluations.

Use of the ICT Intervention should be evaluated as different project components are put in place, including the contentware. Further evaluation throughout the phase-in of implementation allows early alerts to shortcomings and unforeseen opportunities. Class 2 evaluation is sometimes repeated several years after an intervention has been fully implemented to examine whether usage has evolved over time as a result of experience, training, comfort with ICT, new types of users, changing social contexts, or access to new types of resources.

The evaluation of the degree to which the ICT Intervention was used as intended can focus on many different questions. Each of these questions may be answered with multiple sources of data (see Section 3). A list of important questions that may be addressed appears below (Box 6.2). The Evaluation Team may select from them as appropriate and add their own.

This Class of evaluation aims to determine the extent to which the ICT-Intervention is pleasing or disappointing to users.

The subjective reactions to the ICT-Intervention of administrators, teachers, learners and other users are important indicators of their motivation to use the system and their likely persistence in using it. If many users are generally displeased with the intervention, the chances of it achieving the planned objectives are slim. Dissatisfactions may be caused by poor implementation or incorrect use of the ICT-Intervention but may also be inherent in the intervention itself.

Early assessment of dissatisfactions can help identify implementation failures that can be corrected, confusing guidelines or instructions that can be clarified, or aspects of the intervention that might benefit from improvement. Those aspects might pervade most of the ICT Intervention or may be limited to a few components.

Evaluation of users' degree of satisfaction with the ICT-Intervention may begin soon after implementation starts or it may be delayed for a year or so to allow for the intervention to be implemented and used as intended. Satisfaction is sometimes again evaluated after several years to see if it has changed as a result of experience, training, comfort with ICT, new types of learners or changing social contexts.

The evaluation of the degree of user satisfaction can focus on many different questions. Each of these questions may be answered with multiple sources of data (see Section 3 below). A list of important questions that may be addressed appears below (Box 6.3). The Evaluation Team may select from them as appropriate and add their own.

Note that Class 3 evaluations only assess teachers' and students' subjective assessment of the ICT's effect on student learning.

This Class of evaluation aims to determine the extent to which the ICT-Intervention is effectively fulfilling the educational objectives set for it. The reference point here is the set of educational objectives explicitly stated in ICT Policy Program Decision Document in Tool 2.2. They are expressed in one or more of the following:

• Expanding educational opportunities
• Increasing efficiency
• Enhancing quality of learning
• Enriching quality of teaching
• Facilitating skill formation
• Establishing and sustaining lifelong learning
• Improving policy planning and management
• Advancing community linkages (including Community centers)

Details of these objectives are described in ICT for Education: Analytical Review, Section 5.

A major subset of these objectives is developing intended knowledge, attitudes, and skills in the learners. Here a learner may be a school student, a worker, an adult lifelong learner, a teacher - if the ICT intervention is to improve teaching, an administrator -if the ICT intervention includes improvement of policy planning and management, or a member of a community in a community learning center.

Perfect implementation, widespread usage as intended, and high satisfaction on the part of users, do not assure that the ICT Intervention has been effective in fostering the intended educational objectives. Judging effectiveness requires a Class 4 evaluation, which is often referred to as "impact evaluations," "output evaluations," "outcome evaluations," or "summative evaluations." Class 4 evaluations address the first bottom-line: effectiveness. It examines whether the ICT-Intervention effectively met targeted educational objectives, including enhancing the quality of participants' learning. It may also broadly address whether the participants learned more of other things that had not been targeted. Finally, it may examine whether the added costs from the ICT-Intervention are justified by the extent and nature of fulfillment of these objectives.

While the first three classes of evaluation are directed at helping the ICT-Intervention developers and practitioners--as a means of refining the implementation of the intervention, the Class 4 evaluation is usually of more interest to policymakers and planners. The results of Class 4 evaluations can be used to help decide whether to

• modify the ICT intervention in hopes of making it more effective;
• expand the intervention to other geographic areas, grade levels, subjects, or target groups; or
• abandon the intervention as unsatisfactory.

The public, and even policymakers, often think that proof of effectiveness only requires objective measures showing that the targeted educational objectives have been achieved, and conversely, proof of ineffectiveness only requires objective measures showing the objectives have not been achieved. That is mistaken for several reasons. The central focus of Class 4 evaluations is to examine whether the ICT-Intervention caused gains or losses in respect to the educational objectives. The evaluators must know how beneficiaries not exposed to the intervention perform in respect to those objectives if they are to have a basis for determining whether the intervention caused gains (or losses). For instance, learners may make progress on the learning objectives from their normal school instruction, from out-of-school learning, and sometimes even from natural maturation. In addition, some shortcomings in evaluation procedures may upwardly or downwardly bias the results, such as administering a pre-test that then inadvertently prepares learners to do better on the post-test because they have been "sensitized" to the focus and procedures of the test. On the other hand, evaluation procedures themselves can sometimes be disruptive and adversely affect learning.

Consequently, Class 4 evaluations usually use both a "treatment group" and "control group," composed or selected to be as similar as possible, and then compare the learning outcomes in the two groups. Each group might be composed of multiple schools, many classes, and hundreds of learners. Class 4 evaluations also usually take "pre-measures" of the learning objectives administered before the learners begin participating in the intervention and then take "post-measures" after the learners have completed specified parts or all of the intervention, with identical measures administered at the same points of time to the control group. A "pre-measure" is taken just before learners begin an ICT-Intervention that is being evaluated. It indicates baseline knowledge, attitudes, and skills. A "post-measure" is taken just as learners complete the intervention. The best assessment of an intervention's effects is to compare the difference in the post-measures and pre-measures of the intervention group with the difference in the post-measures and pre-measures of a comparable control group subject that has been subject to measurement at the same times as the intervention group.

These arrangements require that Class 4 evaluations be planned well before the ICT intervention begins or at least before the studied treatment beneficiaries begin participating in the intervention. Because educational interventions are often not well implemented in their first year and have some early operational problems, it is usually desirable to plan on a two-year period of Class 1-3 evaluations and program refinement, before starting a Class 4 evaluation. The program should be operating stably and as expected before this type of evaluation is started. The Class 4 evaluation, however, always should start with a treatment group of beneficiaries who have not yet been exposed to the intervention and simultaneously with a control group. Then it must follow both groups until the treatment group completes participation in the intervention. In addition, it may revisit both groups a few years later to determine whether any initial gains from the intervention are retained by participants after completing the intervention.

The evaluation of the degree of effectiveness can focus on many different questions. Each of these questions may be answered with multiple sources of data (see Section 3 below). A list of important questions that may be addressed appears below (Box 6.4). The Evaluation Team may select from them as appropriate, depending on the stated educational objectives of the ICT Project, and add their own. Evaluation of effectiveness in respect to knowledge, attitudes, and skills may also include questions about the learners' characteristics that might also affect their performance.

This Class of evaluation aims to determine the extent to which the ICT-Intervention is effective in preparing learners who subsequently apply the learned knowledge, attitudes, and skills in their later schooling, jobs, and social lives. If the ICT intervention is a pilot that is intended to test the implementability of the intervention or if it is limited to a specific technology, there may not be time to apply Class 5 before the pilot is modified and extended to a larger scale. However, if the pilot aims to change learner's subsequent application of knowledge, attitudes, or skill, then Class 5 Evaluation should be used as part of the evaluation of the pilot.

One of the claimed potential of ICTs is their ability to facilitate in the learners high level cognitive skills of application, problem solving, and learning how to learn. Learners may acquire considerable new capabilities as the result of an education or training intervention and yet not subsequently apply them. That can be attributed to many reasons:

• The intervention had not trained the learners in how to make the applications and ultimately learn on their own;
• The knowledge, attitudes, and skills learned are not relevant to the learners' subsequent lives;
• Circumstances in the learners' subsequent lives make the application difficult or unrewarding.

An example of the latter would be when learners increase their creative problem-solving abilities but subsequently work for companies that are run autocratically by top managers who discourage creative problem-solving.

Class 5 evaluations address the hard question: Has the enhanced learning made a difference in the subsequent thinking and behavior of the learners? Class 5 evaluation is even more complex and difficult to conduct that Class 4 evaluation. It aims to discover whether the intervention caused changes in how the learners think and behave in their lives several years after completing participation in the intervention. That is difficult to determine because the extent of application will be affected not only by the capabilities developed during participation in the ICT-Intervention but also by the capabilities acquired over the learners' entire schooling and by the circumstances of their subsequent lives. In addition, Class 5 evaluation requires keeping track of the intervention group learners and the control group learners through their subsequent lives and securing their cooperation for additional data collection, both of which can be difficult. Once the evaluators lose track of 30 percent of the learners in either group, it is hard to know how representative the located learners are of the initial groups.

Class 5 evaluations are rare. There are several reasons for that. It is presumed that if people learn something, they will subsequently apply it, a presumption that is frequently not correct. Class 5 evaluations are difficult and expensive to conduct. Policymakers who supported (or opposed) a new ICT-Intervention rarely remain in office long enough to request such evaluations, and even when they do, their attention has often focused on other matters. Nevertheless, Class 5 evaluations can be valuable. They go beyond the objectives set for the intervention to its intermediate level goals. In other words, Class 4 evaluation addresses outputs and Class 5 addresses intermediate outcomes. In essence, Class 4 assesses "merit" and Class 5 assesses "value." The ultimate goals or outcomes are addressed in Class 6 evaluations.

Class 5 evaluations usually do not begin until 1 - 10 years after participants have completed an ICT-Intervention. Normally, these evaluations are a follow-up to Class 4 evaluations, using the same treatment and control groups of that evaluation.

The evaluation of the application can focus on many different questions. Each of these questions may be answered with multiple sources of data (see Section 3 below). A list of important questions that may be addressed appears below (Box 6.5). The Evaluation Team may select from them as appropriate, and add their own. Evaluation of application of knowledge, attitudes, and skills may also include questions about the learners' characteristics that might also affect learning application.

This Class of evaluation aims to determine the extent to which the ICT-Intervention is effective in contributing to the nation's developmental goals. If the ICT intervention is a pilot that is intended to test the implementability of the intervention or if it is limited to a specific technology, there may not be time to apply Class 6 before the pilot is modified and extended to a larger scale. Moreover, it is hard to expect a noticeable effect on the nation's development goals generated by ICT interventions that are of short pilot life or small-scale.

In developing countries, large investments in ICT Interventions are usually undertaken with the hope that they will contribute to the country's development. Even when a Class 4 evaluation shows that all the educational objectives have been effectively fulfilled and a Class 5 evaluation indicates that an intervention has substantially increased the application of learned knowledge, attitudes, and skills, these results do not ensure contributions to national development. It may be that the educational objectives fulfilled and the applied capabilities were not the ones needed by the country or it may be that other factors countered these effects. A Class 6 evaluation seeks to determine whether the ICT intervention ultimately contributed to national development, including economic development, human resource development, poverty alleviation, and gender equity.

This is the most complex level of evaluation to conduct, because it necessarily must cover a long time-span over which many other factors will be affecting national development, both boosting and depressing it, and thus it is very difficult to determine the unique effects of the ICT Intervention. Usually these evaluations are based on case-study methods that examine many types of information from many sources, including longitudinal national indicator data, documentary records, and expert opinion.

Class 6 evaluations are rare because of the long period of time that must pass, their complexity, and because interest in a given intervention fades with time. They are important, however, because they examine whether an ICT-Intervention has contributed to its ultimate goals.

The evaluation of national effect can focus on many different questions. Each of these questions may be answered with multiple sources of data (see Section 3 below). A list of important questions that may be addressed appears below (Box 6.6). The Evaluation Team may select from them as appropriate, and add their own.