Data-Based Decision Making

November 12, 2008 3:00 PM - 4:00 PM • Matthew K. Burns, Ph.D.

About this Talk

Join our expert, Matthew K. Burns, as he answers your questions about problem-solving decisions made in each tier and the various decision-making rules used in each tier within an RTI model. Dr. Burns will also offer tips and suggestions on using data to make decisions about instruction, intervention, and eligibility.



Transcript

Tammy Dunaway
Do you feel it is practical or realistic to expect mainstream teachers to implement or supervise interventions (RtI)? How can they teach all of their students (sometimes with large class sizes) and truly help the learning disabled? Is it fair to put this additional pressure on teachers who do not specialize in special education? Do you think they really understand the mind of the Learning Disabled? Even if they are sympathetic, kindhearted, and ethical teachers it is easy to become impatient, judgmental, and exhausted by the challenges of teaching students with LD if it is not their area of expertise - especially when combined with teaching the general population!
Matthew K. Burns, Ph.D.

The mainstreaming question for children with learning disabilities is a topic for a differt debate. I'll address the first part of the question, which is classroom teachers implementing interventions. I do think it is both practical and realistic. Tier 2 interventions have to be set up systematically so that they are part of the daily routine and somewhat easily selected and implemented. In other words, Tier 2 has to run like a well-oiled machine. Tier 3 is more individualized and requires teachers to implement interventions more unsystematically. That can be problematic but can be addressed in a high performing problem-solving team.

Susan Nelson
Who is responsible for collecting data on students within the RTI framework? Realistically there will be multiple people involved with each student so who should take the lead on organizing and analyzing data?
Matthew K. Burns, Ph.D.

Of course, the building principal is ultimately responsible. However, there should be a data management team (DMT) who handles the data. The DMT should be two or three people (usually two) who know how to use data. That is usually the school psychologist and a teacher. There job is to make sure the data are collected in a standardized manner, to present the data in a consumable format, and to participate in the grade level team meeting at which the data are discussed.

Susan Nelson
How often should a team meet to review student data?
Matthew K. Burns, Ph.D.

The grade level team should look at the data at least once each month.

Paloma Sanchez
Can math data and reading data be collected in the same way?
Matthew K. Burns, Ph.D.

I'm not sure what the question is. Yes, math data can be collected as a group and in fact is advantageous over oral reading fluency in that respect. We also need to collect both general outcome measures and subskill measures.

Paloma Sanchez
For a student who already has an IEP, but is now struggling in areas not identified in the IEP, should data be collected in the general education setting or in special education where other issues are being addressed?
Matthew K. Burns, Ph.D.

I suggest general education. I also suggest that students with IEPs participate in general education benchmark assessments.

Paula Taylor
Especially at the middle and high school level, how do we convince teachers that they must use data to inform instruction, rather than relying on what they feel or what they think they know?
Matthew K. Burns, Ph.D.

With data! Get the data to them as quickly as possible and show them the high correlation with what they see in the classroom and state test scores. I also talk a great deal about efficiency and false positive/negatives. By efficiency I mean that I can get more useful information in a 3 minute assessment as I can with many much longer assessments or even from directly working with the kid for a period of time.

Your question is probably contextualized within a larger one that addresses buy in etc. I suggest checking out Ervin et al. (2006) article in School Psychology Review. We also included chapters from major implementation sites in our Handbook of Response to Intervention, which might be helpful too.

Joseph LaMelza
Is there research indicating the number of data points to consider before deciding a change in intervention at Tier II? Besides CBM, are there other measures that have been found to be valid and reliable in predicting future success at Tier I and easily lend themselves to universal screening?
Matthew K. Burns, Ph.D.

Yes. Ted Christ at the University of Minnesota (Christ, 2006) found that eight (8) weeks of data are needed, assuming data are collected twice a week using appropriate standardization, in order for the rates of growth to be reliable enough for decisions. If you are collecting data less frequently, then more weeks are needed. Certainly within eight (8) weeks one could conclude that the intervention is working or that a different intervention should be tried within Tier II, but I would not make a resource decision (e.g., try a Tier III intervention) until enough data are collected for a reliable decision.

Suzanne Gries
What are the best methods for gathering data for 3rd and 4th grade readers? It seems like oral reading fluency is the quickest and easiet. Also, How do you identify an intensive reader in 3rd and 4th grade?
Matthew K. Burns, Ph.D.

You answered your own question. It is very tough to beat ORF for 3rd and 4th graders. ORF correlates highly with just about any indicator of good reading (comprehension, prosody, etc.). The difference is that an ORF assessment takes just a few minutes whereas direct assessments of the other constructs take much longer and don't really tell you much that you didn't already know.

As for identifying kids who need interventions, I suggest looking at class wide problems first, but then using a resource allocation model in which the lowest 20 percent receive a tier 2 intervention. There is no research base to support a triage approach in which the lowest five (5) percent go right to tier 3. Thus, I suggest starting with tier 2 for just about every kid. Certainly some exceptions can be made, but I do not recommend a systematic triage.

Lou Ann NOrris
Are the AIMSWeb probes considered to be adequate and appropriate means to collect data to monitor progress of a learning disabled child? Also, would there be another data collection method that would be more beneficial to determine the progress of a L.D. child?
Matthew K. Burns, Ph.D.

Yes. AIMSweb doesn't do everything I would like it to do, but the probes are well constructed. I also think the data management system is good. My only recommendation for assessments of a child who is LD is to monitor progress in the skill being taught and don't just rely on ORF, unless reading fluency is the target of the intervention.

Bobbie Miller
What do we do with a student who has transferred from another school with a folder of data that doesn't match our "system" of data-taking. Do we have to start all over?
Matthew K. Burns, Ph.D.
That is a policy decision. Much like transfers with children with IEPs, we have to look at the data that were collected in the previous schools and see if those data address our criteria, needs, etc. My hope is that those data could be converted to a rate of growth index that can continue to be monitored so as not to start all over again with data collection. However, the school will need to examine the data and see if they can be used.
Lisa Ray
Our school is planning professional development days centered on data-based decision making. What are the critical points we should cover?
Matthew K. Burns, Ph.D.

Here are some suggestions: standardized administration, use of reliable data, identifying class wide problems by examining class medians, identifying students who need a Tier II intervention, determining rate of growth, evaluating rate of growth, and using a data-decision making framework.

Jonathan Marks
How do you set benchmarks for acceptable progress if a student is below grade-level?
Matthew K. Burns, Ph.D.

Great question! You can do it normatively. Compute slopes for an entire grade then compute the mean and standard deviation. Your target would then be within one standard deviation of the grade-level mean. Alternatively, you could develop criterion-referenced goals by computing the benchmark assessment criteria based on the relationship between assessment data and state test score. Then determine the rate of growth necessary to maintain that level. For example, let's assume that a student who scores a 50 in the fall, 70 in the winter, and 85 in the spring (those numbers are made up) would likely pass the state (80% accurately predicting that he would pass). A student would have to grow at a rate of 35 words/minute across 32 weeks to maintain those levels. That is a rate of growth of 1.09 words/minute per week. That could serve as the criterion.

Jonathan Marks
What is the most efficient way to record data when dealing with large groups of children at one time?
Matthew K. Burns, Ph.D.

A good data management system (e.g., AIMSweb). Also, some schools record the data as they go (e.g., laptops at the data-collection stations) or some write the scores on an alphabetic printout and have clerical support staff enter them later.

Amanda Holsclaw
What connection do you see between RTI and AYP?
Matthew K. Burns, Ph.D.

A direct one! The federal regulations for LD identification explicitly mention that our primary variable of interest is wether or not a student will "make progress sufficient enough to meet state-approved results." That is a euphimism for passing the state test. Also, if we contextualize RtI as the systematic use of assessment data to most effeciently allocate resources in order to improve student learning (Burns & VanDerHeyden, 2006), then RtI is primarily a tool to enhance student learning for ALL students; which is the spirit of AYP as well.

Marcia Dean
How do we select interventions based on student data?
Matthew K. Burns, Ph.D.
This is a great question with an easy answer that would be difficult to answer here. First, I suggest assessing students who are identified as struggling learners in the National Reading Panel areas. There are some good systems to do so. For younger students you could use nonsense word fluency to assess phonics, and something like phoneme segmentation fluency for phonemic awareness. There is also Renaissance Learning's Star Early Literacy assessment that assesses all areas of the NRP. Finally, other measures such as the Comprehensive Test of Phonological Processing looks at phonics, phonemic awareness etc. After identifying the root of the problem (i.e., comprehension, fluency, phonics, or phonemic awareness), the intervention can directly address the deficit.
Layla Martinez
What are the most useful sources of student data?
Matthew K. Burns, Ph.D.

Oral reading fluency for grades 2 through 8. MAZE is probably best for grades 9 through 12, and the early skill indicators (e.g., letter naming fluency etc.) for younger students. However, I also suggest using group comprehension measures such as the Measures of Academic Progress and I know Ed Shapiro has found the 4Sight measures useful in Pennsylvania.

Ken Holmes
We're planning to begin implementing RTI next year, but resources are extremely tight. Do we need to hire someone to manage the data collected and run reports for our data team meetings?
Matthew K. Burns, Ph.D.

No! You need a person or two to be the data management team who will then present the information to your grade-level team. A school psychologists can do this, but so could other people on staff who have training, expertise, and interest in data (e.g., know the difference between median and mean).

jill frodsham
Do you need to use discrepency with peers when working with Tier III to determine special education or is the data from the interventions enough to justify special education?
Matthew K. Burns, Ph.D.

That is not an easy one to answer. A comprehensive evaluation for special education eligibility is required by law, is a student's right, and is best practice. However, what makes up that evaluation is determined by the multidisciplinary evaluation team. The team might very well determine that they need (of course) parental information, developmental history, etc., but that those data with student response data are all that are needed to determine eligibility. However, there may be other sources of data that are needed and those should be collected. It is not unusual to use a standardized assessment of reading (e.g., Kaufman Test of Educational Achievement) or an adaptive behavior scale. Vision screenings, etc., would be used if there are questions about those areas.

The bigger answer is do we need to do a discrepancy model, and the short answer is no - you do not. However, until a district is READY to use RtI data for eligibility decision, I suggest using the traditional approach. A district is ready when they have DATA to support that what they are doing is RtI, that they are implementing it correctly, and that the interventions are occurring. Only then can the data be used for LD identification. I would so much rather a district use discrepancy approaches than do a poor job of using RtI data.

Anonymous
You meniton AIMSweb, are there any other data management programs you would recommend? If so, why? And what features do you look for in your software?
Matthew K. Burns, Ph.D.

I mentioned AIMSweb because they currently have cornered the RtI market. AIMSweb ORF probes are well constructed and the data management system is easy to use. However, there are others. I encourage people to look at mClass by Wireless Generation, and Star Early Literacy by Rennaissance Learning. Edcheckup and Easy CBM also have good systems. I would look for a system that can a) give you data for each student, a class median, and a grade-level average, b) provides a graph of student progress, c) provides a slope of growth for that rate of progress, and d) very important - allows you to use other sources of data within the same management system.

Cara Kraft
What are some specific questions data management team members should answer when meeting to discuss student data? How do these differ from tier to tier?
Matthew K. Burns, Ph.D.

  1. Do we have a classwide problem?
  2. What students need a tier 2 intervention (after the classwide problem is gone or didn't exist to begin with)?
  3. Are the students making sufficient progress in tier 2?
  4. What students need a tier 3 intervention or are there students that we should refer to the problem-solving team (as part of tier 3)?
  5. Are there student that should be evaluated for special education eligibility?

Nancy Thomas Price
Should there be multiple sources of data from screening measures and progress monitoring measures, from which to make decisions?
Matthew K. Burns, Ph.D.

Yes! Especially for screening. I also suggest monitoring progress with a general outcome measure (e.g., oral reading fluency) AND a measure of the skill in which the intervention is occurring (e.g., monitoring progress with nonsense word fluency for a phonics intervention). However, only use reliable data. Using psychometrically inferior tools results weakens the assessment systems. Many schools are using informal reading inventories like the Developmental Reading Assessment to assess reading for all kids. Those data might be very helpful to a classroom teacher, but they are not reliable enough to inform a resource-allocation decision-making framework. Moreover, assessment tools should also meet basic psychometric requirements as well (i.e., result in reliable data and valid decisions).

Debra Yasutake
Teachers at my school often ask what data to collect. Besides state and district-wide assessments and Aimsweb results, what suggestions would you give them?
Matthew K. Burns, Ph.D.

Remember, the RtI framework is an approach to resource allocation. CBM, district-wide, and state tests should be plenty of data to inform those decisions. Classroom teachers certainly would need different data, but those would likely not inform the RtI framework.

Once a student is identified as a struggling reader with these data, I then suggest sampling phonics, phonemic awareness, oral reading fluency and comprehension (the latter two are likely already collected). Those data will help determine WHAT to do. I also suggest looking at the accuracy of the skill. A child who works quickly but inaccurately, one who is slow an inaccurate, and one who is accurate and slow would likely require different interventions for the same skill (e.g., phonics).

Nancy Thomas Price
Is there a resource that details and / or reviews appropriate screening measures and progress monitoring measures, and also, is there a resource that instructs teachers how to design their own?
Matthew K. Burns, Ph.D.
Yes. Please check out the National Center on Student Progress Monitoring website for more information. You can access a chart that details and reviews progress monitoring tools at Review of Progress Monitoring Tools on the National Center on Student Progress Monitoring website.
Nancy Thomas Price
When you talk about data driving the types of decisions made, do you mean, decisions such as what should happen next? Can you elaborate?
Matthew K. Burns, Ph.D.

I am referring to identifying class-wide problems, identifying students for Tier 2 interventions, deciding if students are making sufficient progress (and the intervention is working), and should a Tier 3 intervention be attempted. Does that answer your question?


That concludes our RTI Talk for today. Thanks to everyone for the thoughtful questions and thanks to our expert, Dr. Matthew K. Burns, for his time today.


Related Reading from RTINetwork.org:


Related Reading:

  • Burns, M. K. & Gibbons, K. (2008). Response to intervention implementation in elementary and secondary schools: Procedures to assure scientific-based practices. New York: Routledge.
  • Burns, M. K., Jacob, S., & *Wagner, A. (2008). Ethical and legal issues associated with using response-to-intervention to assess learning disabilities. Journal of School Psychology, 46, 263-279.
  • Burns, M. K., & Senesac, B. K. (2005).  Comparison of dual discrepancy criteria for diagnosis of unresponsiveness to intervention.  Journal of School Psychology, 43, 393-406.
  • Fuchs, L. S. (2003). Assessing intervention responsiveness: Conceptual and technical issues. Learning Disabilities: Research & Practice, 18, 172-186.
  • Silberglitt, B. & Hintze, J. M. (2007). How much growth can we expect? A conditional analysis of R-CBM growth rates by level of performance. Exceptional Children, 74, 71-84.


Additional Online Resources: