No matter how well the ICT interventions were designed and no matter how precise the implementation plans were drawn, in most cases there will be problems and surprises during implementation. It is, therefore, essential to put in place as early as possible, mechanisms to monitor the process of implementation. Tool 6.1 recommends three classes of monitoring and evaluation:

• Class 1- Degree of Implementation (as intended)
• Class 2 - Degree of Proper Use (as intended)
• Class 3 - Degree of User Satisfaction

These classes of evaluation should help identify issues faced during implementation. The flow chart below summarizes the process of monitoring and modification of implementation.

Monitoring of Implementation may identify two types of issues:

• Issues of Implementation

These issues include:

• Faulty implementation of some aspects of the Project. For instance, certain prerequisites or co-requisites may not be in place; hardware is there but there is lack of software or trained personnel.
• Improper use of the ICT interventions. Some components of the project are not used as was intended in the Project scenario
• Inadequate satisfaction with the way the Project is implemented. For instance, the users have not been adequately oriented or trained.

The response to this type of issues should focus on improving the process of implementation itself. After improvements, the same type of monitoring instruments should be applied to ensure that these implementation issues have been taken care of.

• Issues of Planning

Sometimes problems of implementation are derived from weaknesses in implemention plans. Monitoring of implementation may reveal faults in areas such as the kind of selected hardware, the type of contentware, the modality of personnel training, the timing of introducing different interventions, the degree of synergy among the different components, the level of risk associated with different plans.

The response to this type of issues requires modifications in the implementation plans to remedy the detected weakness and faults in them. Then implementation proceeds on the basis of the modified plans, and the whole reiterative process of implementation monitoring, improvement and modification will be applied until implementation reaches an adequate and intended level.

Once the implementation processes have been debugged and the ICT intervention is adequately in place long enough to produce results, evaluation of its effectiveness (Tool 6.1, Class 4) should be applied to determine the extent to which the ICT-Intervention is effectively fulfilling the educational objectives set for it.

Effectiveness evaluation may produce favorable results. But it may indicate that the degree of effectiveness is inadequate.

Assuming adequate implementation, the response to inadequate levels of effectiveness and impact should trigger a re-examination of the design and content of the ICT Intervention itself. This exercise may unearth some of the following problems and weaknesses in the ICT Intervention structure:

• ICT potential was erroneously applied to some components
• Comparative added value of technologies was not properly employed
• The educational, learning or instructional objectives were not appropriate
• The internal pre-requisites and co-requisites were not identified and designed
• The external pre-requisites, co-requisites and constraints were not adequately recognized and considered

The response to these issues may range between limited modifications of specific components (if effectiveness is partially unsatisfactory) and total redesign of the ICT Policy intervention. (See chart below.)

If an ICT Intervention is deemed implementable and effective on a limited scale (pilot or small scale), this may lead to a one of three decisions:

• Continuance of the ICT Program under the same conditions
• Replication of the ICT Program in specific institutions or geographic areas
• Scaling up the program in terms of more coverage, educational objectives, more grade levels, more curricular subjects, etc…

Design of Replication

Replication may not require any significant modifications in the design of the ICT interventions. What will be necessary is a reformulation of the implementation plan. (See below.)

Design of Scaling Up

Scaling up is not a mere expansion or multiple duplication of the pilot or small-scale project. The difference is not limited to size and scope. There are structural considerations that must be incorporated into the design and implementation plans.

• What may not have been efficient at a small scale level may prove to be efficient now, such as online courses or educational TV.
• There should be attempts to achieve economies of scale by reaching levels of critical mass of users to lower unit costs.
• Trade-offs between ICT interventions and other educational inputs and measures should be explored. For instance would a broad application of an ICT-enhancing learning model lead to changes in classroom organization, resulting in a change in time-on-task and teacher-student ratios?
• Does the country have adequate managerial, technical and financial capabilities to support the broad project? Are the national power and ICT infrastructures satisfactory?

Thus the scaling up of the ICT Intervention cannot be an expansion of the ICT Program formulated in Tool 2.2 for a pilot or small-scale project. Scaling up requires going back to the drawing board and starting once again by applying the Tools in the Toolkit that focus on preparation for and formulation of ICT Policy Interventions.

In light of the considerations cited above, the application of these Tools should be done with more caution and diligence because the stakes are higher and the investments are riskier.

Planning for Broad Application

Planning for broad application of ICT Interventions – whether duplication or scaling up – requires a reapplication of the appropriate planning tools in Boxes 3-5.

Large scale planning involves unique issues as well as advantages. For instance,

• Infrastructure may involve regulatory reform, special deals with providers, and preferential treatment because of size of use.
• For hardware provision, it may be advantageous to consider local production or adaptation.
• For contentware, the country may be able to get better licensing deals and produce contentware at reasonable unit costs.
• Because of the size of personnel training, efficient and unconventional models may be applied such as a combination between e-training and face-to-face instruction.

On the other hand, more attention needs to be paid to implementation structures and mechanisms including risk assessment and management of change strategies and mechanisms.

Evaluation of Broadly Applied ICT Intervention

Planning for evaluation of large scale interventions will require more elaborate designs and measures, through application of Tool 6.1.

Additionally, Evaluation Classes 5 and 6 (Tool 6.1) that may not work well in pilot or small-scale projects may work well here. If one of the claimed objectives of the ICT Project is to facilitate in the learners' high level cognitive skills of application, problem solving, and learning how to learn, then Tool 6.1, Class 5, needs to be applied after a sufficient maturation period. Finally, depending on the objective, scope and design of the ICT Project, it may be necessary to apply Class 6 of Tool 6.1 to determine the extent to which the ICT Intervention is effective in contributing to the nation's developmental goals.

The results of these evaluations may lead to a number of responses:

• Lowering the expectations from the ICT intervention in meeting any or both objectives
• Modifying the design of the large scale intervention
• Making adjustments in the way the ICT intervention is applied.