Ensuring the Fidelity of the “I” in RtI

October 28, 2013 3:00 PM - 4:00 PM • Christina M. Marco-Fies, D.Ed.

About this Talk

The Talk has Concluded.

Scroll below for questions and answers.

 

A multi-tier system of supports that uses students’ response to intervention to determine progress and to inform instructional decisions requires that the instruction and interventions that are delivered be implemented with high degrees of fidelity to achieve meaningful student outcomes. Fidelity of implementation or “treatment integrity” is a critical element of the RtI process. Dr. Christina Marco-Fies answered your questions about how to determine the level of treatment integrity to ensure an intervention is implemented as planned, intended, or originally designed.

Read more about Christina M. Marco-Fies, D.Ed.



Transcript

Cassie
What type of professional development do you suggest to ensure the observers know what to look for in each of the Tiers?
Christina M. Marco-Fies, D.Ed.
Observers completing the fidelity assessments should be very familiar with the instruction or intervention that is being implemented. Having an understanding of the teacher’s guide, script, or manual is important. Perhaps the observer was trained with the implementers or the observer may have been the person who provided the training. He or she must be familiar enough with the instruction to know if the components were or were not evidenced. The specificity of the assessment tool can also ensure accurate fidelity is collected. When the components of instruction or intervention are clearly defined in observable terms, the observer will more accurately be able to determine if the component was present.
Carol Ratliff
I am a school psychologist and often find myself helping teacher's plan interventions for struggling students. Many teachers want to utilize aides who are not trained in implementing interventions. What would you recommend as a "best practice" to ensure aids are properly trained and carry out the interventions with fidelity? Or would you even recommend the use of aides in general?
Christina M. Marco-Fies, D.Ed.
Due to the resources available in some schools, an aide may be the only person available for providing an intervention and training for this person is a necessity to ensure fidelity of instruction. Whether the person implementing the intervention is an aide or a classroom teacher, he or she must be provided with training. Conducting fidelity assessments over time, both directly through observation and indirectly through self-report, can assist with ensuring that the intervention is implemented with fidelity. Feedback can be provided to the interventionist, whether that person is an aide or a teacher, and that information can be used to improve fidelity of the instruction. As long as you have the knowledge and understanding of the intervention, you as a school psychologist could complete the direct observations, or the observations could be completed by another teacher, a specialist, or an administrator.
Pam labellarte
Where is the best place to locate what is defined as "fidelity" for a specific intervention or reading program?
Christina M. Marco-Fies, D.Ed.
Some interventions and programs may have a fidelity checklist included within the teacher’s guide or manual; however, this is not the case for all interventions or programs. For instances in which you do not have a publisher-created fidelity checklist, one would need to be developed. This process would require the key components to be listed and defined in observable terms. Examples of treatment integrity protocols from a variety of sources are available on this website.
Dean Kaminski
Does fidelity mean that the instruction must only be provided by the classroom teacher/specialist? What if that person were absent for the day, left detailed plans, and the substitute teacher provided the intervention. Does that still count as a day of intervention?
Christina M. Marco-Fies, D.Ed.
Depending on the training and expertise of the substitute teacher and the level of fidelity of the intervention delivery on that particular day, the lesson could potentially “count as intervention.” The substitute teacher would have needed training in the administration of the particular intervention and been fluent with delivery. In addition, that particular day of instruction would need to be implemented with a high degree of fidelity. To utilize resources available, some interventions in schools are being provided by instructional assistants or volunteers. Again, the training provided with the particular intervention is extremely important. In addition, frequent assessment of and feedback regarding fidelity is needed. You also bring up a good point in regards to fidelity. Not only is it important to assess to ensure that the key components of the program are being provided, but it is also important to ensure that the program is occurring for the amount of time and at a level of frequency that is required. If the teacher is absent, the student is absent, the class arrives late from the previous lesson, etc., then the fidelity of the intervention is compromised. Interventions can easily be inappropriately discarded without taking these factors into consideration.
Laura Campbell
What can you do if the RTI process has been compromised because an intervention wasn't delivered as specified?
Christina M. Marco-Fies, D.Ed.
Research suggests that interventions do not necessarily need to be implemented with 100% integrity in order to lead to student gains (McCurdy & Watson, 1999); however, higher levels of integrity typically result in greater outcomes (i.e., Noell, Gresham, & Gansle, 2002). Nonetheless, if the intervention is not implemented with 100% integrity, positive (or negative) outcomes could be a result of any missing (or added) components. This could lead to discarding an intervention that may have the potential to be effective. So if data collection demonstrates than an intervention has not been implemented with integrity, then training and support should be provided to the interventionist. The use of scripts and/or modeling can be effective tools in providing support. By frequently providing the teacher with updates of treatment integrity data and student outcomes data, the interventionist will have the opportunity to be responsive to the feedback. If training and support has been provided and factors such as the complexity of the intervention, teacher perceptions, or the appropriateness of fit for the student appear to be the reason for a lack of integrity and student progress, then a change in intervention program might be appropriate at this time.
Jim Wattie
Who should assess whether an intervention is being implemented with fidelity?
Christina M. Marco-Fies, D.Ed.
There are several options of personnel who have the ability to assess intervention fidelity. First, the interventionists can self-assess the fidelity of the intervention being provided. This may aid with reflection on the implementation of the program, but self-assessment may not be objective. A second possibility is another teacher. This allows an interventionist to be assessed objectively and in a non-evaluative manner. Third, specialists with expertise in the particular intervention could assess the fidelity. For example, a reading coach, reading specialist, or school psychologist with knowledge of the intervention could be utilized. Finally, administrators could serve as an assessor of fidelity. Principals, assistant principals, or other administrative personnel appear to be an obvious choice for conducting fidelity checks. Careful attention needs to be paid to the purpose of the observation, focusing on improving fidelity of the instruction, rather than connecting the observations to teacher evaluation.
Marilyn
Please share your opinion on utilizing a research-based intervention program to meet students' needs for an hour and using the regular common core curriculum as extension for a half hour.
Christina M. Marco-Fies, D.Ed.
Based on a response to intervention model, all students should receive Tier 1/core curriculum instruction. Those students who are not meeting expectations after receiving core instruction should be provided with Tier 2 and/or Tier 3 intervention in addition to the core instruction. For example, if a district’s core reading curriculum consists of 90 minutes daily reading instruction based on the common core standards, all students would receive 90 minutes of reading instruction. A student who is not meeting expectations after receiving the 90 minutes of core reading instruction might be provided with a Tier 2 intervention for an additional 30 minutes of instruction. This student would then have a total of 2 hours of reading instruction per day, 90 minutes of core and 30 minutes of targeted intervention. Although RtI has the flexibility of allowing schools to determine tiered instruction, implementing RtI with fidelity would indicate using interventions as supplements to core instruction. Perhaps there may be instances when the core curriculum is at such a higher level than the student’s instructional level that participation in the core curriculum is unreasonable for a student; however, making the decision to exclude a student from core instruction should be done with caution because in many instances, the student is still expected to reach proficiency with grade level standards. Additional information related to Tiered Instruction and Intervention in an RtI model can be found in the Tiered Instruction/Intervention section of this website.
joan grim
How can individual classroom & school fidelity data be scaled up for a district & state?
Christina M. Marco-Fies, D.Ed.
On a large scale, the fidelity of RtI implementation can be assessed at a state or district level. Some states (i.e., Colorado and Florida) have created RtI/Problem Solving implementation assessments that can be found on the Checklists and Forms page of this website. If interested in assessing specific components of instruction, such as state standards or core curriculum implementation, states or districts could create fidelity assessment tools by identifying the key components of interest and implementing direct or indirect measures of data collection across districts. The process would be very much the same as how a classroom or school fidelity assessment would be created and implemented, with direction coming from the state or district level rather than the school level.
Noreen Smyth-Morrow
Once a teacher is well-versed in an intervention, how necessary is it for a fidelity checklist be completed?
Christina M. Marco-Fies, D.Ed.
One may assume that once an interventionist has mastered implementing an intervention, fidelity assessments would no longer be necessary; however, implementation drift may occur. Teachers may not maintain implementation without structured follow-up (Noell, Duhon, Gatti & Connell, 2002). Certain components may be missed or shaped over time by the interventionist if continual fidelity assessment is not occurring. The more frequently interventionists are provided with fidelity data feedback, the higher the degree of implementation fidelity (i.e., Jones, Wickstrom, & Friman, 1997). Therefore, although the intervention with high fidelity may not need to be assessed for integrity as frequently as an intervention being implemented with low fidelity, this assessment should continue to be performed to ensure consistency of implementation over time. When using a student’s response to intervention as a method for decision making, we must demonstrate that the intervention was implemented as intended before we can use progress monitoring data to make decisions.
Roxanne Camus
How do teachers find the "right" interventions?
Christina M. Marco-Fies, D.Ed.
Finding the “right” intervention is really finding what works for a particular student or group of students. The first step would be to assess the student to identify needs. Next, an intervention should be selected that is shown to be effective for teaching the particular needs exhibited by the student or group of students. Then, the intervention should be implemented with fidelity. In addition to academic or behavioral progress monitoring data being collected for each student, fidelity assessments must occur to ensure that the intervention is implemented with integrity. Too often, an intervention is discarded because the student data may show a lack of progress and the assumption is that the intervention is not working; however, the intervention may not have been implemented correctly. Without having fidelity data available, educators cannot be sure whether the progress or lack of progress is due to the intervention or other factors. If an intervention is implemented with integrity and if the student(s) is making adequate progress, then the “right” intervention has been found (at least until the next review of data). If an intervention has been implemented with integrity, but the student(s) did not exhibit growth based on the progress monitoring data, then a change in intervention may be needed. By looking at the progress of the other students in the group, the length of time the student has received the intervention, any additional individual data that have been collected, and hypotheses as to why the intervention is not working, educators can make a decision as to whether a change in intervention is required. If the intervention has not been implemented with integrity, then it cannot be assumed that the student is not responding if there is a lack of progress. Rather, support should be provided to the interventionist to improve the fidelity of implementation, especially if the intervention has been shown to be effective for teaching the particular target skill in similar situations.
Sue Beres
What are the best ways to ensure instruction is being issued with fidelity?
Christina M. Marco-Fies, D.Ed.
To ensure that instruction is being delivered with fidelity, the integrity must be assessed. The most commonly used fidelity measures can be categorized as direct measures and indirect measures. Direct measures are direct observation of the behavior in the classroom. In this type of fidelity measure, instruction is observed and coded for treatment integrity. The observation can occur through direct observation, audio taping, or videotaping of the instruction. In contrast, indirect measures can be assessed after the instruction occurs and include self-reports, rating scales, interviews, checklists, Likert scales, lesson plan reviews, and permanent products. Teachers can provide this information through self-reports or through tangible features of the instruction. For the best assessment of fidelity and to reduce the limitations presented by specific assessment methods, it is recommended to combine multiple methods of assessment (Sheridan et al., 2009). For example, the instruction should be evaluated through a direct observation by a reading coach and a self-report rating by the classroom teacher. If the results of the assessments indicate a lack of fidelity, then feedback and support should be provided to the person delivering the instruction.
Noreen Smyth-Morrow
How do you make a functional fidelity check, other than a self-report? 
Christina M. Marco-Fies, D.Ed.
In addition to a self-report of fidelity, instruction and intervention can be directly observed. The observation could occur live in the classroom, via audio tape, or via video tape. First, a detailed task list of the intervention must be created. Then the components of the intervention should be defined in observable terms. Next, each component should be rated as occurring or not occurring during the observation period to calculate a percentage of treatment integrity. Finally, the integrity data can be graphed to display the fidelity over time. There are several examples of treatment integrity protocols from a variety of sources available on this website.
Kim Riley
In a Title I school, where students face many challenges in and outside of school, what is the best strategy for ensuring fidelity of interventions that are implemented in Tier 3?
Christina M. Marco-Fies, D.Ed.
There is not much difference in how the fidelity of intervention may be assessed at a Title 1 school versus a non-Title 1 school. Perhaps different funding opportunities may be available to provide the staffing resources for conducting fidelity assessments, but the methods of assessment (i.e., direct measures, indirect measure, a combination of the two) would be the same. When making decisions regarding student progress, in addition to the fidelity and student progress monitoring data, the specific challenges unique to the students may need to be considered. Examples may include truancy, language barriers, lack of resources, etc. For instance, truancy is a factor that definitely impacts the fidelity of instruction. If the student does not attend school on a regular basis, then the intervention is not being implemented as intended because the student is missing the instruction. If attendance is not considered as a factor, a decision may be made that the intervention is not working for this particular student, when in reality, there is a potential for it to work if the student was in school. Rather than discarding the intervention, supports should be put in place to improve the student’s attendance in collaboration with the student and family.
Michelle
How, as a parent, can I determine if my son's intervention was delivered with fidelity? What questions should I be asking his teachers to determine that what they are doing is working?
Christina M. Marco-Fies, D.Ed.
As a parent, you can and should have communication with your child’s teachers regarding his progress. First, you can ask teachers what types of supports and interventions they are providing your son. This may include specific programs or resources and the skills that they are targeting. Next, you can ask how often and for how long your son receives the intervention and how the school determines that the intervention is implemented as intended. This may occur through observations from the principal or other school staff or perhaps the teacher keeps documentation or rates him/herself in regards to the implementation. Finally, you can ask for data to show the progress that your son has made. This data may be displayed in a graph to show where your son started and how he performed over time with the particular skill that was being targeted. Based on this information, the teacher should be able to help interpret the data do determine if sufficient progress is being made and what recommendations should follow.
joan grim
How are schools planning to gather fidelity data across different content & different classroom constellations with accuracy?
Christina M. Marco-Fies, D.Ed.
On a large scale, some schools are choosing to gather instructional fidelity data by having teams of trained observers (i.e., teachers, administrators, school psychologists, coaches) complete “walkthroughs.” The walkthroughts consist of trained observers entering each classroom in a building for approximately 5 minutes each, searching for particular “look-fors” during the instruction. The “look-fors” may include particular teaching strategies or components of school initiatives. This data can be shared with the entire staff to improve overall instruction. On a smaller scale, teachers can be paired or work with instructional coaches to complete fidelity checks of instruction in particular subject areas or for particular interventions. The fidelity assessment tool must be developed based on the key components required for the instruction or intervention, and feedback should be provided to the individual teacher regarding his or her fidelity. How a school decides to collect fidelity data really depends on the goals of the assessment and the resources available for implementation.
Tanya Milanowski
Who exactly will be the individual to truly ensure the intervention is being done with integrity as the principal would be the only one able to truly redirect the staff to do something?
Christina M. Marco-Fies, D.Ed.
Although the principal would be the person to redirect staff in more of an evaluative and disciplinary manner, the idea of this model is that providing teachers with fidelity feedback is done in more of a coaching, consultative, and non-evaluative manner. The whole premise is that the school as a whole is improving instruction for all students and it is a team effort. By creating assessment tools that clearly identify the key components of the intervention, teachers are aware of expectations prior to a fidelity observation. Then, a peer, coach, or specialist can conduct the fidelity check and provide feedback to the teacher regarding the fidelity of the intervention. If the intervention is not implemented with high fidelity, the observer is then able to provide support to the teacher to ensure that future implementation is done with higher integrity. Principals may also perform in this role; however, they must be careful to conduct fidelity observations in a supportive rather than evaluative manner.
C. Schargen
What are the RTI Indicators?
Christina M. Marco-Fies, D.Ed.
In general, RtI includes a comprehensive assessment system, a range of research-based instruction and interventions, and the use of a problem-solving model to make instructional decisions (Shinn, 2008); however, different states and researchers may use different language or identify more or less components of an RtI model. Some examples of tools that can be used to assess fidelity with the indicators can be found in the Checklists and Forms section of this website.
Michelle Braun-Burget
How can progress be efficiently documented?
Christina M. Marco-Fies, D.Ed.
When referring specifically to intervention fidelity progress, the data gathered through fidelity checks can be graphed over time. The simplest way to create a graph would be through Microsoft Excel or a similar type of program that has graphing capabilities. The number or percentage of key components of the intervention that were implemented during the lesson would be on the Y axis and the date of the lesson would be on the X axis. Some schools have purchased on-line data warehouse systems and, depending on the system, there may be capabilities to house fidelity data. Frequently sharing fidelity data in this format, along with student rate of improvement data from progress monitoring assessments, can be a very effective in promoting integrity of intervention implementation and assist with instructional decision making.
Sue McLaughlin
How much flexibility is there in how an intervention is delivered before the integrity is affected?
Christina M. Marco-Fies, D.Ed.
Currently, there is not a standardized generic treatment integrity instrument available to collect fidelity of implementation data. Therefore, we do not know to what extent an intervention protocol can have deviations before significantly impacting the student outcomes (Gresham et al., 2000). In general, higher levels of fidelity are typically equated to greater outcomes (i.e., Noell, Gresham, & Gansle, 2002), but it has been reported that positive behavior changes can occur when the integrity is 60%-65% (McCurdy & Watson, 1999). Modifications to the intervention protocol likely lead to lower student outcomes, but the modifications could also lead to improvements in student performance as well. In addition, the treatment components being assessed may have differing levels of importance toward student outcomes. So if the interventionist is implementing the more effective components and not implementing the less effective ones, a low level of treatment integrity may still result in student gains.
Kim Riley
What are the best uses of Title I funds when focusing on fidelity?
Christina M. Marco-Fies, D.Ed.
Direct observation of instruction can be time consuming and resource intensive. Staff members need to be available to complete fidelity observations and provide supports to teachers who are not implementing strategies with high fidelity. Even if self-report measures are being primarily used for assessing fidelity, some direct observation is encouraged. One good use of Title 1 funds may be for hiring instructional specialists or instructional coaches to assist with aiding teachers and interventionists in improving instruction through fidelity assessment and support.
Shelby Jasmer
I am a district level administrator who struggles with building administrators who pull the reading interventionist from their students for other non instructional activities, any suggestions?
Christina M. Marco-Fies, D.Ed.
If the reading interventionists are being pulled for other activities during times that they are scheduled to be intervening with students, the integrity of the intervention is being compromised. There are always going to be emergency situations or absences that will impact the consistency of intervention implementation; however, if the intervention is not being implemented on a regular basis, then decisions cannot be made in regards to student progress and students are not receiving the supports that have been determined necessary. First, ensure that the building administrators understand the importance of fidelity and the link that high fidelity has with student outcomes. If there are other buildings within your district where interventions are being implemented consistently, compare the student progress in those buildings with the buildings that are not implementing the interventions consistently and see if there are differences. Share this information with the administrators. Second, look at the reasons why the reading interventionists are being pulled. Are there alternative solutions to meet the needs of the building administrators and allow interventionists to meet with the students who need the intervention? By supporting the building administrators with solutions to the problem of non-instructional activity coverage, they may be more likely to allow interventionists to remain in their classrooms during that time.
Elizabeth
What systems can we put into place that will ensure fidelity?
Christina M. Marco-Fies, D.Ed.
The fidelity of instruction should be assessed throughout all tiers of an RtI framework. The system in place will depend upon the current systems and resources available within the particular school. Therefore, the type of fidelity assessment utilized (i.e., direct, indirect, combination) and the person completing the fidelity assessment (i.e., teacher, coach, specialist, principal) will vary from district to district and building to building. Once the fidelity data is collected, it should be shared with the teachers and interventionists in order to improve fidelity. Depending on the types of instructional meetings available within the district, the format of sharing the data could vary. The fidelity data should be taken into consideration when reviewing student progress monitoring data and making instructional decisions. As an example, at Tier 1, perhaps a team of trained teachers complete frequent walkthroughs of core instruction during their planning time, looking for specific key components of the core curriculum. Feedback is provided to teachers about the core instruction as a whole at faculty meetings and improvements in fidelity can be made based on the data. At Tier 2, perhaps the reading coach completes direct observations of intervention implementation and provides fidelity feedback to individual teachers during their planning time. At Tier 3, perhaps a specialist (reading specialist, school psychologist, etc.) completes direct observations of intervention implementation and provides fidelity feedback to individual teachers during their planning time. In between the direct observations, self-assessments may be conducted for continual fidelity monitoring, especially if the fidelity during the direct observation is low. Then, when meeting to review student progress monitoring data to make decisions about instruction, the fidelity data is reviewed and factored into decision making.




Related Reading from RTINetwork.org:


Additional Resources: