Evaluating and Refining Implementation

1. Were the original activities and timelines accurate?
2. Has implementation progressed according to schedule?
3. Have the professional development activities been delivered?
4. Do the activities and timelines require adjustment?

These questions are designed to determine the degree to which the logistics of the plan have been delivered.

The quality of the implementation uses a progress-monitoring approach. A number of evaluation instruments have been developed both to “progress monitor” implementation and to evaluate the integrity of the implementation. Click on Self-Assessment of Problem Solving Implementation to download one example of this type of instrument. Most of these progress-monitoring instruments used for evaluating degree of implementation use a “level of implementation” approach. The school-based leadership team (SBLT) has the primary responsibility for completing this assessment instrument. The instrument can be completed by the group as a group activity. When this approach is used, consensus must be achieved on the rating of each implementation item. An alternative method can be used in which each member of the SBLT completes the instrument individually. The group then convenes to come to consensus on an evaluation level for each item.

The Self-Assessment of Problem Solving Implementation can be completed multiple times per year. Typically, this instrument is completed two, three, or four times each year. Data from the survey are then used both to evaluate the implementation plan and to make decisions about which aspects of the implementation process to emphasize.

Implementation integrity is assessed by evaluating the degree to which the steps of Response to Intervention (RTI) are actually implemented and are evidenced in the process (e.g., problem-solving meetings) and products (e.g., reports). The degree to which the four major steps in the RTI process (problem identification, problem analysis, intervention, and response to the intervention) are implemented with integrity can be assessed using a Problem Solving Checklist and a Critical Components Checklist. The Problem-Solving Checklist is an observation instrument that is used during problem-solving meetings. The Critical Components Checklist is used to evaluate the degree to which the RTI steps are evidenced in intervention reports. Research indicates that positive student outcomes are strongly related to the number of steps in the RTI process that were actually implemented.

In summary, implementation evaluation provides the staff in a building with evidence of how well the model is being implemented and informs the professional development process. The SBLT can use the information from the evaluation process to assess the effectiveness of the professional development activities and to identify areas that require additional attention. Finally, the evaluation information can be given to the staff to provide them with feedback on how well the process is being implemented. When these implementation evaluation data are related to student outcomes, the staff will have direct evidence that their decision to implement this process resulted in positive outcomes for all students.