Does your manual or published materials specify validated decision rules for when changes to instruction need to be made?
Yes
Specify the decision rules:
aimswebPlus applies a statistical procedure to the student’s progress monitoring scores in order to provide empirically-based guidance about whether the student is likely to meet, fall short of, or exceed his/her goal. The calculation procedure (presented below) is fully described in the aimsweb Progress Monitoring Guide (Pearson, 2012). aimswebPlus users will not have to do any calculations—the online system does this automatically. The decision rule is based on a 75% confidence interval for the student’s predicted score at the goal date. This confidence interval is student-specific and takes into account the number and variability of progress monitoring scores and the duration of monitoring. Starting at the sixth week of monitoring (when there are at least four monitoring scores), the aimswebPlus report following each progress monitoring administration includes one of the following statements:
A. “The student is projected to not reach the goal.” This statement appears if the confidence interval is completely below the goal score.
B. “The student is projected to exceed the goal.” This statement appears if the confidence interval is completely above the goal score.
C. “The student is projected to be near the goal. The projected score at the goal date is between X and Y” (where X and Y are the bottom and top of the confidence interval). This statement appears if the confidence interval includes the goal score.
If Statement A appears, the user has a sound basis for deciding that the current intervention is not sufficient and a change to instruction should be made. If Statement B appears, there is an empirical basis for deciding that the goal is not sufficiently challenging and should be increased. If Statement C appears, the student’s progress is not clearly different from the aimline, so there is not a compelling reason to change the intervention or the goal; however, the presentation of the confidence-interval range enables the user to see whether the goal is near the upper limit or lower limit of the range, which would signal that the student’s progress is trending below or above the goal.
A 75% confidence interval was chosen for this application because it balances the costs of the two types of decision errors. Incorrectly deciding that the goal will not be reached (when in truth it will be reached) has a moderate cost: an intervention that is working will be replaced by a different intervention. Incorrectly deciding that the goal may be reached (when in truth it will not be reached) also has a moderate cost: an ineffective intervention will be continued rather than being replaced. Because both kinds of decision errors have costs, it is appropriate to use a modest confidence level.
Calculation of the 75% confidence interval for the score at the goal date:
Calculate the trend line. This is the ordinary least-squares regression line through the student’s monitoring scores.
Calculate the projected score at the goal date. This is the value of the trend line at the goal date.
Calculate the standard error of estimate (SEE) of the projected score at the goal date, using the following formula:
〖SEE〗_(predicted score)= √((∑_i^k▒(y_i-y ́_i )^2 )/(k-2))×√(1+1/k+(GW-(∑_1^k▒w_i )/k)^2/(∑_i^k▒(w_i-(∑_1^k▒w_i )/k)^2 ))
where k = number of completed monitoring administrations, w = week number of a completed administration, GW = week number of the goal date, y = monitoring score, y’ = predicted monitoring score at that week (from the student’s trendline).The means and sums are calculated across all of the completed monitoring administrations up to that date. Add and subtract 1.25 times the SEE to the projected score, and round to the nearest whole numbers.
What is the evidentiary basis for these decision rules?
The decision rules are statistically rather than empirically based. The guidance statements that result from applying the 75% confidence interval to the projected score are correct probabilistic statements, under certain assumptions: The student’s progress can be described by a linear trend line. If the pattern of the student’s monitoring scores is obviously curvilinear, then the projected score based on a linear trend will likely be misleading. We provide training in the aimsweb Progress Monitoring Guide about the need for users to take non-linearity into account when interpreting progress-monitoring data. The student will continue to progress at the same rate as they have been progressing to that time. This is an unavoidable assumption for a decision system based on extrapolating from past growth.
Even though the rules are not derived from data, it is useful to observe how they work in a sample of real data. For this purpose, we selected random samples of students in the aimsweb 2010–2011 database who were progress-monitored on either Reading Curriculum-Based Measurement (R-CBM) or Math Computation (M-COMP). All students selected scored below the 25th percentile in the fall screening administration of R-CBM or M-COMP. The R-CBM sample consisted of 1,000 students (200 each at of Grades 2 through 6) who had at least 30 monitoring scores, and the M-COMP sample included 500 students (100 per Grades 2 through 6) with a minimum of 28 monitoring scores. This analysis was only a rough approximation, because we did not know each student’s actual goal or whether the intervention or goal was changed during the year.
To perform the analyses, we first set an estimated goal for each student by using the ROI at the 85th percentile of aimsweb national ROI norms to project their score at their 30th monitoring administration. Next, we defined “meeting the goal” as having a mean score on the last three administrations (e.g., the 28th through 30th administrations of R-CBM) that was at or above the goal score. At each monitoring administration for each student, we computed the projected score at the goal date and the 75% confidence interval for that score, and recorded which of the three decision statements was generated (projected not to meet goal, projected to exceed goal, or on-track/no-change).
In this analysis, accuracy of guidance to change (that is, accuracy of projections that the student will not reach the goal or will exceed the goal) reached a high level (80%) by about the 13th to 15th monitoring administration, on average. The percentage of students receiving guidance to not change (i.e., their trendline was not far from the aimline) would naturally tend to decrease over administrations as the size of the confidence interval decreased. At the same time, however, there was a tendency for the trendline to become closer to the aimline over time as it became more accurately estimated, and this worked to increase the percentage of students receiving the “no change” guidance.