This Class of evaluation aims to determine the extent to which the ICT-Intervention is effectively fulfilling the educational objectives set for it. The reference point here is the set of educational objectives explicitly stated in ICT Policy Program Decision Document in Tool 2.2. They are expressed in one or more of the following:

• Expanding educational opportunities
• Increasing efficiency
• Enhancing quality of learning
• Enriching quality of teaching
• Facilitating skill formation
• Establishing and sustaining lifelong learning
• Improving policy planning and management
• Advancing community linkages (including Community centers)

Details of these objectives are described in ICT for Education: Analytical Review, Section 5.

A major subset of these objectives is developing intended knowledge, attitudes, and skills in the learners. Here a learner may be a school student, a worker, an adult lifelong learner, a teacher - if the ICT intervention is to improve teaching, an administrator -if the ICT intervention includes improvement of policy planning and management, or a member of a community in a community learning center.

Perfect implementation, widespread usage as intended, and high satisfaction on the part of users, do not assure that the ICT Intervention has been effective in fostering the intended educational objectives. Judging effectiveness requires a Class 4 evaluation, which is often referred to as "impact evaluations," "output evaluations," "outcome evaluations," or "summative evaluations." Class 4 evaluations address the first bottom-line: effectiveness. It examines whether the ICT-Intervention effectively met targeted educational objectives, including enhancing the quality of participants' learning. It may also broadly address whether the participants learned more of other things that had not been targeted. Finally, it may examine whether the added costs from the ICT-Intervention are justified by the extent and nature of fulfillment of these objectives.

While the first three classes of evaluation are directed at helping the ICT-Intervention developers and practitioners--as a means of refining the implementation of the intervention, the Class 4 evaluation is usually of more interest to policymakers and planners. The results of Class 4 evaluations can be used to help decide whether to

• modify the ICT intervention in hopes of making it more effective;
• expand the intervention to other geographic areas, grade levels, subjects, or target groups; or
• abandon the intervention as unsatisfactory.

The public, and even policymakers, often think that proof of effectiveness only requires objective measures showing that the targeted educational objectives have been achieved, and conversely, proof of ineffectiveness only requires objective measures showing the objectives have not been achieved. That is mistaken for several reasons. The central focus of Class 4 evaluations is to examine whether the ICT-Intervention caused gains or losses in respect to the educational objectives. The evaluators must know how beneficiaries not exposed to the intervention perform in respect to those objectives if they are to have a basis for determining whether the intervention caused gains (or losses). For instance, learners may make progress on the learning objectives from their normal school instruction, from out-of-school learning, and sometimes even from natural maturation. In addition, some shortcomings in evaluation procedures may upwardly or downwardly bias the results, such as administering a pre-test that then inadvertently prepares learners to do better on the post-test because they have been "sensitized" to the focus and procedures of the test. On the other hand, evaluation procedures themselves can sometimes be disruptive and adversely affect learning.

Consequently, Class 4 evaluations usually use both a "treatment group" and "control group," composed or selected to be as similar as possible, and then compare the learning outcomes in the two groups. Each group might be composed of multiple schools, many classes, and hundreds of learners. Class 4 evaluations also usually take "pre-measures" of the learning objectives administered before the learners begin participating in the intervention and then take "post-measures" after the learners have completed specified parts or all of the intervention, with identical measures administered at the same points of time to the control group. A "pre-measure" is taken just before learners begin an ICT-Intervention that is being evaluated. It indicates baseline knowledge, attitudes, and skills. A "post-measure" is taken just as learners complete the intervention. The best assessment of an intervention's effects is to compare the difference in the post-measures and pre-measures of the intervention group with the difference in the post-measures and pre-measures of a comparable control group subject that has been subject to measurement at the same times as the intervention group.

These arrangements require that Class 4 evaluations be planned well before the ICT intervention begins or at least before the studied treatment beneficiaries begin participating in the intervention. Because educational interventions are often not well implemented in their first year and have some early operational problems, it is usually desirable to plan on a two-year period of Class 1-3 evaluations and program refinement, before starting a Class 4 evaluation. The program should be operating stably and as expected before this type of evaluation is started. The Class 4 evaluation, however, always should start with a treatment group of beneficiaries who have not yet been exposed to the intervention and simultaneously with a control group. Then it must follow both groups until the treatment group completes participation in the intervention. In addition, it may revisit both groups a few years later to determine whether any initial gains from the intervention are retained by participants after completing the intervention.

The evaluation of the degree of effectiveness can focus on many different questions. Each of these questions may be answered with multiple sources of data (see Section 3 below). A list of important questions that may be addressed appears below (Box 6.4). The Evaluation Team may select from them as appropriate, depending on the stated educational objectives of the ICT Project, and add their own. Evaluation of effectiveness in respect to knowledge, attitudes, and skills may also include questions about the learners' characteristics that might also affect their performance.