Hi my name is Rob Pennington, I am the assistant director of training at the Kentucky Autism Training Center. I am excited to be here. I started my teaching career in 5th grade, 1981 when I was a peer tutor for a classroom of students with moderate to severe disabilities. Since that time I have served in a variety of capacities from classroom teachers to district resource personnel to job coach, community coach, to researcher. And so I am really excited to be here again and talk about continuous data collection.
Today we are going to talk a little about presenting a rationale for why we should conduct formative assessments. Then we are going to describe important quantifiable qualities of behavior. So what type of things are we looking for when we measure behavior? We're going talk a little about how to do it. And then we're going to talk about some ways to interpret the data once we have it.
So when I start this conversation about continuous data collection, I like to start first with that question what is teaching? So if you take a moment to think about it yourself, several pictures might come to you. Maybe you standing in front of a classroom talking. Often times when I ask my students in a classroom they often say things like sharing knowledge or giving information to other folks.
Webster dictionary version says it's "to cause to know something". Well if I am walking down the street and I step in a pothole and fall down you'll learn not to walk in that pothole. But I am not directly manipulating the environment. So I really enjoy Julie Vargas' definition a little bit better. Design circumstances that change the way others behave.
When we talk about teaching we need to realize that teaching is not synonymous with presenting. When I am doing this webinar right now I am not actually delivering instruction, I am presenting. I'm giving you information. It's only when I have assessed that your behavior has changed that you've learned something, that I can say that I have been teaching. The assessment of whether teaching has actually occurred involves the documentation of changes in student behavior. Welcome data collection!
When we talk about assessment we're talking about two broad categories. Summative Evaluation, which is the documentation of progress or change of performance by comparing an individual's performance between two points in time. So essentially when you are looking at achievement tests, for instance, the scores between 5th and 6th grade is a measurement of two points of time in a student's performance. When we talk about Formative Assessment, we're looking at frequent and repeated measurement of performance on functional curriculum-based tasks assessed in natural conditions over time. So we're taking repeated measurements on the same behavior across time.
So why do we conduct formative assessments? Well we can't rely on our memory. I have a hard time remembering what I did a few minutes ago sometimes. You may feel the same way. It may also be dependent on the type of day that you had. For example, let's say we had a student that has been engaged in problem behavior for several weeks. And then he has a, on the final day, a day in which he engages in problem behavior for thirty seconds. But in that thirty seconds, trying to move the student or support the student, your finger accidently gets in his mouth and he bites down. So you are going to say "wow" that is the worst day, but in reality the student engaged in much less frequent problem behavior on that day. So again we cannot rely on our memory. When we have formative assessment we can clearly observe patterns and performance. We can look at changes in behavior over time. In addition we can correlate those changes in behavior with interventions that we have delivered, or changes in the environment. So we can actually take a look at the relationship between those variables within the environment and student performance. Finally we can make, not finally, but we can make frequent decisions about instructional practices. So for instance, if we look at student progress and we have several data points we can say "wow" the student is just sitting there not making performance. I better change what I am doing. Finally we can conduct our, we can document our efforts. As so, we have a demonstration of what we have done as an educator towards improving student outcomes, what we have done as an educator towards improving student outcomes.
Let's talk about getting started with data collection. First you must select a behavior to record. And you want to define this behavior in terms of physical qualities. Sometimes we got caught in the trap of trying to predict what students are thinking. Realistically, we want to make sure that two naive observers can determine whether or not a behavior is occurring. Let me give you a quick example. Often times I work in school context and I will see the objective "Student is on task". Well for one teacher "on task" may be that the student is sitting forward looking at the teacher, they're orientated towards the speaker, they're "on task". Well you and I know realistically that we can be performing all sorts of behaviors that indicate "on task". But we are not "on task" we're thinking about something else. But for another student "on task" may be mean that they're actually pencil to paper, they're writing something down. "On task" may be measuring whether or not they comprehended what was said to them, that they were attending. So several different observers, can have different definitions or ideas about the concept of "being on task". So again, when we are measuring something we want to make sure we are looking at a behavior and we want to make sure that it is defined in terms of physical quality.
Let's take a little quiz here. Take just a moment and read through this list. Which of these would you say are clear examples of behavior, that two naïve observers would agree whether or not this behavior is happening?
Well, let's take a look. Hits...most people can define whether or not somebody is hitting. Certainly we want to be able to operationalize whether a push is a hit, what type of force, but then again most people would agree if a student hit another student. Points to...again pointing to a card on a table during discrete trial instruction is very clear for most of us. Let's go back to this one...attends. When we talked about that a minute ago, much like "on task" behavior. Attending is really not a behavior. Now if we want to address orienting towards a speaker, certainly. If we want to address being able to answer comprehension questions after listening to some sort of verbal stimuli, certainly. But, attending is not a behavior. Comprehending or understands...again we see these a lot as our behavioral objectives but sometimes in school context, written in objectives. Again, a student will form a behavior to demonstrate his comprehension. He may answer a wh-question. He may select a picture card from a field of three. But, comprehending and understanding is not a behavior. It's defined in terms of its physical dimensions. Looks...we can make a decision whether or not a student is looking at certain stimuli. Here's a tricky one....reads. Most of us, because of its common vernacular, would say yes, reading is a behavior. But I would say it's still very confusing. Do we mean silent reading? Can we read without performing any sort of behavior that's observable? Certainly. And so what we want to do is look at things like "will state the word vocally". Maybe if we're doing reading comprehension, will answer wh-questions after reading a passage. Again, in terms of a measurable behavior, reading is very difficult. States, yes of course, states is an observable behavior. Participates? No, we can observe whether somebody is participating in an activity but we do not want to say that, especially in a behavioral objective, that we are measuring participation. We want to say how are they participating. For example you may be sitting there now and if I said "Could you participate in a cheesecake eating contest?" Well, you might get 100% on that, right? But then if I said go fix my carburetor in my car, you might get a 0%. I wasn't measuring participation in that context. I was measuring other skills. So of course you've probably mastered that cheesecake eating skill but not necessarily the carburetor rebuilding skill. And then finally, initiating. We're not sure where initiates starts so we prefer to say things like "will respond within a certain amount of time". That's what we mean by initiating. Will start the task. So, I hope this clarifies the differences between good examples of behavior that we want to measure and maybe some weaker ones.
When we start with data collection, we first want to be able to select which dimension of behavior is relevant to record. For instance, during talk outs, it's perfectly relevant to say let's do number of talk outs in class or interruptions. Let's say if we look at our peers in a general education context and we see that they have an average of 10 to 15 social interactions per hour we may want to look at our target student and measure the number of interactions that this student makes per hour. So again, being able to select the dimensions is very important. Second, we need to select the appropriate tool. And that's what we are going to talk about in these next few moments.
The first dimension of behavior we are going to look at is the number of times a behavior occurs. When we talk about numbers, it's a simple count of the number of times that a behavior or event occurs within a certain context, right? So, it's important to know that when we're observing the number of times a behavior occurs, it's only appropriate when the duration of observation periods are equal in length. So for example, if I'm looking at number of call outs that Johnny makes during class. And first day I go in and take data and I only take for 30 minutes and he engages in that behavior 10 times in that 30 minutes, well, if I come the next day and I watch for an hour and he engages 10 times in that hour, those are not equal. I cannot compare those. The expectation would be that if he did it 10 times in 30 minutes, then he would do 20 minutes in an hour. So again, if we're going to use number and we're going to compare number, we're going to compare data for two different days, we have to observe that behavior for the same amount of time. Let's talk about some ways that we can use number data. Ok, let's talk for a minute about establishing rate. Once we have a number we can establish a rate. This is useful when observation periods differ in length. So let's talk about our previous example. Johnny exhibits call outs 10 times out of every 30 minutes, about once every three minutes, right? So, if I want to compare that subsequent data collection session that lasted 60 minutes, and let's say it happened 15 times in that 60 minutes, well I can say then it happens once every four minutes. So, I can actually make a comparison using rate. If you look at the formula for determining rate, it's the number of times a behavior occurs over the number of minutes in an observation period. What system do we use to record frequency or number of occurrences? One method we can use is event recording. That's essentially just tallying each time a response occurs. Take a look at the data sheet below. On the left side I have the date and the length of observation. Remember this is very important because we cannot compare observation periods that are not equal unless we are determining a rate. The next column over, I have the counts so there's a simple tally mark, and the third column I have a total. Then if you look at the last column I went ahead and calculated a rate to make sure that if there are any sessions to which the duration of the observation period is different, I can compare those as well.
In this next example I want you to watch a video. I would like for you to take a minute, practice your new event recording skills. I'd like for you to tally every time I engage in an ear touching response. I'm going to touch my ear. One, if I do this...that's two...so anytime my hand leaves my ear and then re-engages my ear it's going to be a separate occasion. So go ahead, take a minute and watch and let's see how you do.
Video: Why don't we start out by introducing ourselves. I'm Rob Pennington and I work at the Kentucky Autism Training Center and I'm also a professor in the Department of Special Education. I teach classes here and I work on training issues, like we develop training and content and those type of things. That's kind of what I do. I'm Katie Carnazzo and I work with Rob at the Training Center and I train teachers and I go out and work hard at setting up model sites in Jefferson County. I travel around the state and I'm going to Hazzard tomorrow and I'm very excited about that. It's only state travel at this point, right? I only work part-time. I have two kids, I'm a mom.
And we're back? How did you do? You should have caught 15 times that I touched my ear. And if you want to go ahead and calculate the rate for that it was a 50-minute, excuse me, 50-second observation session. So just divide 15 by 50 seconds and that should come up with a rate for you per minute.
We're going to talk now about another set of methods for determining how often a behavior occurs. We're going to talk about time sampling methods. In general, time sampling methods involve defining an observation period then dividing it into smaller and equal intervals of time and then you record the occurrence/non-occurrence of the target behavior during or at the end of each interval. There are three different ways that these can be conducted. First, whole interval recording, so look at the example below. You have an interval, one minute. So, we've taken this one minute period of time and we've divided it into six ten-second intervals. During whole interval recording, if the behavior occurs throughout the entire interval then you would put a check. For instance, let's say I'm measuring yelling behavior. I'm going to count with my fingers the number of seconds. We start...go... (Rob is yelling)...new interval. Because I screamed or yelled through that whole interval I'm going to put a check in that box. If I would have stopped at anytime there would not be a check in that box. You want to use this method for periods of time in which you're expecting the student will engage in this behavior for long periods of time. For instance we did a study involving elopement with a young child with autism. Not that he was running off to getting married but what he was doing was actually running out of the classroom. And so we were actually measuring the amount of time that he was on the carpet. We expected him to be on the carpet for long periods of time so we used whole interval recording. Another time sampling method is partial interval recording and this is the same, this is very similar except for the fact that if the behavior occurs at any point during the interval then you would put a plus. So again, we start, we start the watch, I am going to count with my hands one one-second, two one-second, three ahh, four one-second, five one-second, six, I would put a mark in that box because it occurred at least one time during that interval. And then I would move to the next interval. Okay, so partial interval recording is another way that we can, is another time sampling method for recording whether or not a behavior occurs.
Finally we are going to talk about momentary time sampling. Momentary time sampling is very useful for individuals in that it doesn't require you to observe the student for the entire time. So I may, in this example below, if you look at the one-minute observation period, I would have a timer or maybe use what we call a motivator, which is a device that you can get at habitchange.com. It's like a little vibrating device that will cue you to moments in time. I would set that for every ten seconds. I would do my, if I'm a classroom teacher, I would do my job as soon as that would vibrate I would look up, see if the student was engaged in the problem behavior at ten seconds. I would mark if he was or not. And so this is a little bit different. Whereas the other two time sampling methods involve you observing the student the entire time, at this point you only have to observe the student at the end of each interval. And so again, momentary time sampling can be very effective especially for some of our general ed teachers. And one thing I want to note I have talked a little bit about observing problem behavior. These can also be used to observe behaviors that we want to increase, that we want to happen. I want to show you quickly there is another example of a partial interval recording system and this is for a student that may, that we actually used that was engaged in elopement so again, running out of the class behavior and aggression. And so in this interval, this is for a special education teacher, not this interval but this example, this is for a special education teacher, we divide the day up into ten-minute intervals. And as you look on the top of the graph, we have places to put the date, on the left side we have a place that we can put the, that we have the time intervals and then if the behavior engages, excuse me if the student engages in aggression or elopement during any time in that interval, she would just mark the box. Again this is much easier for the teacher than, you know prior to this she was writing down every, every time the student ran and so again she is just making a single mark during that time interval. So again another way to use partial interval recording.
So there are other dimensions of behavior that we might want to record. For instance, duration, how long the behavior lasts. So for instance we might want to look at the amount of time Jessica engages in a work task or the amount of time Oliver engages in a tantrum. Certainly a decrease in the duration of a tantrum can be perceived as an improvement, so let's say that Oliver on Monday engages in a tantrum for 45 minutes but on Tuesday he engages in a tantrum for 5 minutes. If we were taking frequency data it would both say one, there was no change in behavior, but if we were taking duration data we would certainly see that change in behavior.
So how do I conduct duration recording? There are really two different ways that we can do this. One, we can use total duration. So during total duration we have a stopwatch. Every time the student engages in the problem behavior we start that watch. As soon as they stop, we turn that behavior, we stop that stopwatch. When they start again we do not reset the stopwatch we just start it up again. So what that gives us is a total duration that the student was engaged in the problem behavior. More useful is what we call DPO, or duration per occurrence. So if you look below at the data sheet, this is a duration per occurrence data sheet. We start the stopwatch when the student engages in the problem behavior and then we stop it and we record how long that occurrence lasted. Then the next time, so we reset the stopwatch, the next time the student engages in the problem behavior we start it, we stop it and we record that. What's nice about this is not only can we get duration data, we can get the average duration data, we can also get a frequency count and so we can have a lot more information about the student. So if we are going to be collecting the data anyways, it is better to have as much information as possible about the problem behavior.
Another measure that, another dimension of behavior that we might want to record is latency, the amount of time it takes to begin a response once a cue or direction has presented. This may be useful for determining whether or not a student is improving performance when starting tasks. Often times we talk about things like off task behavior but really we are looking at, really we have several components of off task behavior or not doing work. Is the student not getting started, is the student not able to persist on a task, does the student even know what they are supposed to be doing? So often times we will start a program where given a familiar set of tasks the student has mastered we will actually measure latency to see, measure the latency between given the request "get started" and the task. And so this might be a good application of this measurement system. Another way if you consider latency is how long does it take for students to engage in social contexts, so once you see the increased proximity of a peer as our cue, how long does it take for the student to issue a greeting or deliver a greeting? So that might be another way that we look at latency as a measure.
How do I record latency? Latency recording essentially begins much like duration recording. As soon we deliver the cue, we start our watch, as soon as the student initially starts the task then we stop our watch. Again look below here is an example of a latency recording data sheet.
Up to now we have talked about a couple different dimensions of behavior and how to record those. Now we're going to spend a few minutes talking about taking their data during instruction. Certainly certain instructional procedures have their own data collection methods but in general we have two different ways to deliver, excuse me, we have two different ways to record data during instruction. We have trial recording and then we have task analytic recording. When you're doing trial recording you're scoring a child's response that occurs within a set response interval at the presentation of the cue or directive. So if I was to hold up this yellow card to you and I said what color? 1 2 3 4 5 yellow, I was recording whether or not you said yellow within that five seconds. So typically you would get a plus if you said yes or a minus or some other, whatever I chose to mark if you had an incorrect response. Then I'd probably mark a no response if you did not respond at all. Data, when we collect these data they are often recorded as number or percent of correct responses. So if I give you five responses and you get three correct then that's 60%.
Again talking about calculating percent. Percent is the number of times a behavior occurs per total number of opportunities for the behavior to occur multiplied by 100. This is most often used in trial instruction but it is important to note that when we give different numbers of opportunities it can really skew our data. For instance, if you're delivering instruction and you provide the student with two opportunities and they get one correct you're looking at 50%. The day before you may have given the student five opportunities, they got one correct; you're looking at 20%. Same number of opportunities that as you varied the excuse me, same number of correct responses but as you varied the number of opportunities your data changed significantly. So just be aware that when at all possible when comparing those data there was the same number of opportunities.
Take a look at this trial recording example. Here's a data sheet that is often used with constant time delay. On the left side we have our stimuli. So representing a penny, a nickel, a dime, a quarter. The next two columns indicate the student's responses, so in the B column these are responses that occur before the prompt. In constant time delay we have a constant interval that we would wait before delivering the prompt. A column is the student's response that occurs after the prompt. So in these examples, in the first example if you look there are six correct responses before the prompt out of a total of 12 trials so we're looking at 50% correct responses.
Next we're going to take a few minutes to look at task analytic data collection. During task analytic data collection an observer records an individual's response to each step of a chained behavior. How do I collect task analytic data? First we want to conduct the task analysis, we will break these chained tasks into their simpler responses. We will record those steps on the left side of the data sheet so for instance if we're washing our hands we will turn on the water step 1. Step 2 get the soap. Step 3 put our hands under the water. Then we'll observe this, after this is recorded on the data sheet we will observe the individual performing the task and then we'll record the student's response for each step and usually we're going to record whether they did it correctly, incorrectly or what type of prompt that we used to get them to perform it correctly.
So let's take a look at this data sheet. If you see this for putting on a single sock, so on the left side we have grasp sock, put toe in sock. Step 2 pull sock over toes. Step 3 pull sock over heel. Step 4 pull sock over ankle. On the first day the student was only able to perform the first step correctly. So out of 5 possible steps, he had 20% of the steps completed and as you see as we progress through the week the student finally got to 100% of the steps completed. So 5 out of 5 is 100%.
Up to now we've spent a few minutes talking about different ways that we can collect data. I'd like to make a few suggestions about ways to make sure that you have some reliable scoring when taking data, that is making sure that you are consistent as the data collector. We want to make sure that it's the student's behavior that changes not your behavior as their observer. So first it is very important as we mentioned earlier that you have a very clear description of the target behavior. The clearer it is the less likely you are to drift from your definition. For instance if you are working with a student who is exhibiting aggressive behavior just over time you may become used to those aggressive behaviors and you may perceive them to be less aggressive. So you really want to define what behavior does the student have to exhibit to be labeled as conducting or exhibiting aggression. This is also very important if there's multiple people collecting data. So if you're working with teacher assistants, if you're at home and you're working with another family member and you're recording data, you both have to agree on whether or not the behavior occurred. Second, make sure that everybody is trained. Often times when I'm conducting assessment in my classroom, I'm asking my teacher assistant to do it, we'll first do some role play then I'll have them work with the student and I'll observe them and give them feedback. Not until they, not until their scoring and my scoring agrees up to 100% before I let them go to take data on their own. Again, good training is an important key. Another one is frequently taking reliability data. What the means is when somebody else is scoring a student's response, intermittently pop in there and score at the same time. See if you have an agreement. We're hoping for all to at least have an 80% agreement. 100% agreement is the best but anywhere between 80% and 100% agreement should be good enough to continue with instruction. Finally use history training and what the means is some students when you're collecting data may respond to having a new observer or having data collected on them. So we want to make sure that students are familiar with new people in the environment. So if you're observing them have a clipboard with you before you start observing the behavior, let them get used to that a couple sessions. If you're using a video camera to record make sure that video camera is in the classroom. If you have a new observer recording have them come in the classroom before recording data to familiarize themselves with the students.
Up to this point again we've talked a little bit about different dimensions of behavior, ways to collect data, and now we're going to talk about displaying it. I like to use graphs and in my coursework here at the University of Louisville, we highly recommend that all of our students use graph student formative data. The reason we do this is because it provides an ongoing and objective summary of student performance. Every day teachers are asked to plot their data points on their graph. It gives us an imminent and in-depth view of their students' performance and it allows them to make moment-by-moment decisions related to their instructional practices. Finally and most importantly I think it communicates very clearly the intervention story to others. So what that means is looking at a data sheet with a lot of numbers on it may be very hard to communicate the information to other folks. So if you're sitting in an IEP meeting and you want to share that with a parent or you want to share it with another team member, they have to take the time to look at that data and then make some determinations. If you can give them a clear graph of that data it is immediately more evident whether the student is making progress or not and so again it's a great communication tool. Something else that is really great about graphs is when you show a person the graph everybody has their own decision. Again it is not one person on a team saying this student is doing better or this student is performing as expected. Everybody looks at that data and gets to make their own subjective evaluation whether or not that student is making progress.
Let's look at the next picture. In this image we have a teacher construct a graph. In this example they were actually using what is called simultaneous prompting. If you take a look at it they have a clear, there are a couple components of this graph that are very important. On the left side, we know the measure that they are graphing. So we need to know if it's the difference between the number of responses or percent of responses that is very clear and at the bottom we call that the abscissa we are looking at the days so the student, the teacher will write the number of days in which instruction occurred. Each data point depicts the student's performance on that date. A couple of other things that are very important if you look at the graph, there is a line drawn between the baseline and the intervention session so we can make a clear comparison between those two different conditions. So before instruction and then after instruction. Again very clear documentation. If you take a look at the graph you can make a determination yourself whether or not the student made progress.
Take a look at this example of a graph. In this graph the teachers using the system of least prompts to teach a chained task. So this is task analytic data collection and they are actually using it for toothpaste. In this graph if you look the teacher has the raw data on the top half of the data sheet and at the bottom she has a built in graph. So again another clever way teachers can go directly with their pencil from right in the raw data right down to the data point. Again this is graphed daily. The teacher can immediately assess whether their student made progress from the day before.
Helpful hints. Make sure as I mentioned before that you graph your data daily. Again this gives you a clear opportunity to take a look at your student's performance on this day as compared to the day before or the few days before. To make this easier make sure that you graph your data, you keep your graph close to your raw data again, very, it's much more practical than having to run across the room to get your data from somewhere else. A lot of folks are using technology to graph their data and certainly that is a great way to do it. But I often will keep a hand drawn piece of graph paper right there with my data sheet so I'm using hand graphed data collection, excuse me, if I'm using handwritten data collection procedures it just takes one more motor response to put that data point in and then later I'll translate that to a pretty electronic graph. And finally when possible use a percentage key. What this might do is say for instance you are delivering nine trials. It's a little more difficult to calculate percentage than if you had five trials and so you might want to write down what percentage is three responses and what percentage is six responses, what percentage is five correct responses.
Okay so up to now we've talked a little bit about recording data, how to display that data. Now I'd like to talk a little bit about decision making for data based decision making for instruction. Again, sometimes we get in situations that are really great data collectors but they sit in our big 3-ring binders and we don't do anything with it. So I'm going to give you a little bit of information about how to make some decisions after looking at your data.
So essentially after reviewing data a teacher is given, a teacher, a parent, any sort of professional is given three broad options. First you can keep on trucking, you can keep on moving, things are proceeding as planned, student's making progress don't change a thing. Second you might need to change the intervention or so you're looking at it and you're saying wow my, this student is not making progress. Another decision you might need to make if the student does not appear to make adequate progress is you might want to change the instructional target. And so again for example, last week I was in the school in which they were doing reading comprehension and so the student was listening to a reading passage, or actually listening comprehension. It was a reading comprehension activity right. And so the student had, the teacher was reading a full passage to the student and then the student was to select pictures to answer WH questions. Well the student, the teacher was frustrated because the student wasn't making adequate progress. And so after doing, so we conducted some further assessment and realized that the student really wasn't even, the student did not have some of the prerequisite skills of being able to select a picture based on just a single word sample. So certainly hearing 40 words strung together was much too complex of a skill. So we went back to very basic simple discrimination training for the student and now the teacher is seeing progress. Again sometimes we want to change our instructional target to make sure that we are seeing progress.
The first decision-making tool we are going to discuss today are aim lines. Aim lines are visual aids used to assist in progress monitoring. When using Aim lines, we are drawing a line on top of our graph and then we are evaluating student performance based on the adherence of that data path to that line. So there's our line, we want our data to follow along that line. So how do we draw these Aim lines? Well first we draw a line through the intersection of the mid-date and the mid performance of the first three training days. So when we start instruction those first three days we will try to figure out where are, where's their average performance across those three days. That is where we will plot our data point. Then we are going to make a determination upon where do we think that the student, when should the student acquire the skill and meet criterion. So let's say if the student has typically mastered similar types of skills in about four weeks we are going to go along our graph twenty days, go up to our criterion and place a point there. So then we are going to draw a line between those two data points and that will be our data path.
Take a look at this example on a simple graph here. So in the first three days of instruction with this student, the student had 10% correct responses, 20% and 15%. So if you see where we plotted our first line it was on that second day, that's the mid-date right and then we went for the middle, the median of the data set and so we're looking at 15. So go up, go over to that second data point, go up to 15 and draw our data point. If we go again, the same example, we expect the student to acquire the skill in four weeks. And so we went out twenty days, we drew our second data point; we drew a line to connect those two. So there's our Aim line.
When determining our Aim dates there are a couple of considerations we want to make. First we want to look at students' acquisition of similar skills. So we are going to look at previous performance on skills that were similar to the ones we are instructing now and look at how long it took him to master those skills. It will help us determine our Aim date. Second of all we want to make sure that we are following that student's performance and looking at how long it takes them to get criterion. So we want to take a look at making sure that we are making changes as we go along. So we're going to give you some examples in a minute of when you might want to adjust your criterion based on student performance. And second we want to make sure that we're not setting our Aim dates too far in the future. These are looking at the short term. We are not looking at beginning of the year, end of the year because we want to be able to make clear changes in instruction much more frequently. We talked a little bit earlier in this lecture about the difference between summative and formative assessment. One of the big disadvantages to summative assessment is if you take, lets say you take an assessment in October and you're gong to test again in May to see the student's performance, well if you test again in May and the student hasn't made any progress you've wasted the student's time for those eight months, eight or so months. And so you want to make sure that we're taking frequent data and that we're graphing data frequently so if our student is not making progress we are not wasting any time delivering ineffective instruction.
Okay based on using an Aim line, Wolery, Bailey, and Sugai, have some basic suggestions. So first if the student's progress approximates or exceeds the Aim line that means that data path is hanging around that line that we drew and we want to make no change, we are right on target. If the student is making steady progress but it's well below the Aim line, our line is here and the student's progress is here, it's not up near the line but it's below then we are going to change our Aim date. And so we may adjust it out a little bit which will flatten it out and then our student will be along that Aim date. If the data show the student is performing some but not all of the tasks successfully we may want to go back as we discussed earlier and move back to an easier version of the skill. Let's take a look at some sample graphs here. So in this example we have the student response, our Aim line is in blue, our student's performance is in black and they are hanging pretty close to that Aim line. So take one second and make a decision. What would you do? In this example we've continued with instruction. The student is hanging pretty close to that Aim line, they're making progress, we're all right with that. Next in this example, what would you do? Well since the student is making progress up to a point and then has a complete drop-off in skills, we may have lost student interest or we may have lost their motivation so we might want to adjust some of our instructional practices. In this example what should we do? Well, if you notice the student is making progress but it is starting to kind of veer away from the Aim date so we want to make sure that if this is a continued kind of pattern we might adjust our Aim date in this one.
What if the data show that the student is not performing any part of the task correctly? Well we first want to make, we might want to try a different instructional procedure. We may not be using a controlling prompt. There may not be reinforcement. We may not have conducted a preference assessment. The student may not find the reinforcer reinforcing. What if data indicate high rates of errors but with some correct? Again we might want to go back and teach our prerequisites, step back and teach a prerequisite skill.
What if data indicate a high proportion of correct responses? Well if the student is at criterion or almost at criterion we want to move on to a new phase of learning. That means we will move beyond acquisition and make sure that the student generalizes these skills to other areas and contexts. We want to make sure that the student can use these skills with sufficient fluency and to make sure that they are functional. What if the student has met criteria for accuracy and fluency? Make sure to move on to a new skill.
Another tool that I find very useful are data based decision rules by Diane Browder. Essentially we are making, when using these rules we are making decisions about instruction, about students' performance and whether or not to change our instruction every two weeks. So for this to be useful we want to make sure that we have at least six data points to review. So that means you're taking data on a particular skill at least three times a week. So first based on these data we may be able to use visual inspection and so in this example it is clear that the student is making progress. Sometimes it may not be so clear to us. We may have some variable data. Our data paths may look very, may have very limited stability. And so when the pattern is not clear, the teacher can use phase means. And a phase mean is the total value of data points divided by the number of the data points within a phase.
So again when using Diane Browder's decision based rules if the student reaches criterion within two weeks we're going to develop a plan to maintain and extend performance. So again we are making sure that they can generalize these skills to other contexts and they can emit it with a, with proficiency so that it's functional in important environments. If the student makes no progress and all data points are at zero, make no change for two weeks so you wait two weeks and then after two more weeks, re-write the instructional plan to address a simpler skill. So sometimes our students do take awhile to make progress so in this, when the student has made no progress whatsoever you want to wait a total of four weeks and then if there is no progress you want to re-write the instructional plan. If the trend is accelerating by 5% that means the phase mean is 5% higher than baseline for the past two-week period make no changes. Again some of our students make progress at different levels. As long as they are progressing by at least 5% every two weeks that is sufficient to go forward. If our trend is flat or accelerating by less than 5% that means our student has kind of flat lined in their performance. You are going to see their data path look very flat. You want to improve antecedents or prompting strategies to increase independent responses. So that means that you want to take a look and see if you are using constant time delay, if you're using a controlling prompt, if you are using prompts consistently, if you are using prompts that are effective getting students to respond. So again evaluate your prompting methods. If your trend is decelerating that means it starts going up and then it tanks and goes down you want to consider reinforcement, often times the student is losing motivation.
That concludes our webinar on using formative assessment. I want to thank you for your time. Again my name's Robert Pennington and I'm at the University of Louisville so if you ever have any questions feel free to contact me. I like to leave you with some images of some great resources that are available to you and things of which I pulled some of the information from today's webinar from. First we have Applied Behavioral Analysis by John Cooper, Timothy Herron and William Heward. Curriculum Assessment for Students with Moderate and Severe Disabilities, Diane Browder and Effective Teaching: Principles and Procedures of Applied Behavioral Analysis for Exceptional Students by Wolery, Bailey, and Sugai. Again these are great resources that you can find available on the web and in your local bookstores. And you should be able to take any concept, any of the concepts we discussed today a little bit further and deeper with any of these texts. Have a great day and thanks again for your time.
While selection and administration of assessment measures is important, it is the interpretation, integration, and sharing of data that directly lead to program development. Test scores can be useful with careful interpretation and when considered in conjunction with other relevant information gathered through observations, interviews, record reviews, etc. The information gathered during the assessment process is used to create a comprehensive snapshot of the student’s current strengths and challenges.
This section lists resources for interpreting and integrating assessment results, including (a) defining key terms, (b) interpreting data in a family-friendly manner, and (c) writing reports that integrate data across disciplines.
National Center on Student Progress Monitoring
A website that provides webinars and on-line training in progress monitoring.
A website that offers information on data collection and guidance on how to use data to make decisions regarding program and instructional planning. The site also includes evidence-based interventions and information about data collection.