aimswebPlus Reading

Area: Oral Reading Fluency

Cost

Technology, Human Resources, and Accommodations for Special Needs

Service and Support

Purpose and Other Implementation Information

Usage and Reporting

aimswebPlus™ is a subscription-based tool. There are three subscription types available for customers:

aimswebPlus Complete is $8.50 per student and includes all measures.

aimswebPlus Reading is $6.50 per student and includes early literacy and reading measures.

aimswebPlus Math is $6.50 per student and includes early numeracy and math measures.

Test accommodations that are documented in a student’s Individual Education Plan (IEP) are permitted with aimswebPlus. However, not all measures allow for accommodations. Oral Reading Fluency (ORF) is an individually-administered, timed test that employs strict time limits, in part, to generate rate-based scores. As such, valid interpretation of national norms, which are an essential aspect of decision-making during benchmark testing, depend on strict adherence to the standard administration procedures.

These accommodations are allowed for Oral Reading Fluency: enlarging test forms and modifying the environment (e.g., special lighting, adaptive furniture).

NCS Pearson, Inc.
Phone: (866) 313-6194

www.aimsweb.com

www.aimswebPlus.com

Training manuals are included and should provide all implementation information.

Pearson provides phone- and email-based ongoing technical support, as well as a user group forum that facilitates the asking and answering of questions.

aimswebPlus is a brief and valid assessment system for monitoring reading and math skills. Normative data were collected in 2013-14 on a combination of fluency measures that are sensitive to growth as well as new standards-based assessments of classroom skills. The resulting scores and reports inform instruction and help improve student performance in Grades 2–8, and the early literacy and early numeracy measures provide ecologically valid and developmentally appropriate information about foundational reading and math abilities for students in Kindergarten and Grade 1.

Students read one story in each Progress Monitor test session. The fictional texts were written by educators with a close knowledge of the kinds of writing typically encountered by students at different grade levels. Twenty unique progress monitoring (PM) forms per grade are available; PM testing conducted at teacher-determined intervals.

While the Kindergarten and Grade 1 measures are administered individually, most of the Grades 2 through 8 measures can be taken online by entire classes. Once testing is complete, summary or detailed reports for students, classrooms, and districts can be immediately generated, and the math and reading composite scores can be used to estimate the risk to students or classes for meeting end-of-year goals. aimswebPlus reports also offer score interpretation information based on foundational skills for college and career readiness, learning standards, and other guidelines, Lexile® and Quantile® information, and recommendations for appropriate teaching resources.

Raw score and percentiles scores (based on grade norms) are provided. Local norms are also available.

ORF is a timed measure that assesses fluency of oral reading skills. Performance is reported on words read correctly. Words not reached within 60 seconds are not considered “incorrect;” they are ignored for the purpose of reporting the number of words read correctly.

 

Reliability of the Performance Level Score: Convincing Evidence

Alternate form reliability coefficients for Oral Reading Fluency (ORF) represent the reliability of the mean of the pair of reading passages comprising each benchmark period. The Spearman-Brown Prophecy formula was used to estimate reliability of the mean reading rate from the correlation of reading rates for the two passages.

Type of Reliability

Grade

n (range)

Coefficient Range

Coefficient Median

SEM

Alternate form

1

1346

0.95–0.96

0.96

6.01

Alternate form

2

1741

0.95–0.97

0.96

7.28

Alternate form

3

1580

0.95–0.96

0.96

7.60

Alternate form

4

1633

0.95–0.96

0.95

8.12

Alternate form

5

1643

0.95–0.95

0.95

8.91

Alternate form

6

1317

0.94–0.96

0.95

8.29

Alternate form

7

959

0.94–0.95

0.94

9.62

Alternate form

8

850

0.94–0.96

0.95

8.06

 

Reliability of the Slope: Convincing Evidence

Validity of the Performance Level Score: Unconvincing Evidence

aimswebPlus Reading ORF Score Predictive Validity Coefficient, by Grade and Criterion Measure

Criterion

Grade

N

Correlation

Gender Percentage

Race Percentage

Unadjusted

Adjusted1

F

M

B

H

O

W

ITBS

1

82

0.53

0.65

41

59

25

25

17

33

ISAT

3

113

0.69

0.75

47

53

2

28

20

49

ISAT

4

230

0.66

0.69

56

44

4

39

10

47

ISAT

5

250

0.64

0.68

48

52

4

22

13

61

ISAT

6

332

0.69

0.70

58

42

9

14

12

65

ISAT

7

179

0.63

0.65

44

56

12

12

7

68

ISAT

8

202

0.58

0.64

46

54

10

11

6

74

NWEA

2

218

0.73

0.72

52

48

2

23

21

53

NWEA

3

128

0.69

0.67

45

55

2

25

20

52

NWEA

4

150

0.62

0.68

53

47

4

28

16

52

NWEA

5

125

0.72

0.72

48

52

3

30

18

48

NWEA

6

141

0.55

0.61

52

48

4

21

12

63

MAP - GLA

3

226

0.65

0.62

55

45

24

2

2

72

MAP - GLA

4

229

0.74

0.71

49

51

32

1

5

62

MAP - GLA

5

157

0.50

0.46

50

50

42

0

7

50

MAP - GLA

6

77

0.60

0.62

56

44

27

2

6

65

MAP - GLA

7

124

0.66

0.70

46

54

41

4

0

55

STAAR

3

205

0.50

0.56

56

44

10

49

14

27

STAAR

4

266

0.55

0.60

44

56

8

52

10

29

STAAR

5

152

0.55

0.64

53

47

9

57

3

31

aimswebPlus RC

2

1329

0.65

0.65

49

51

15

26

10

48

aimswebPlus RC

3

1241

0.62

0.62

51

49

12

25

10

53

aimswebPlus RC

4

1306

0.53

0.53

50

50

12

24

10

54

aimswebPlus RC

5

1185

0.57

0.57

52

48

15

24

11

50

aimswebPlus RC

6

972

0.53

0.53

50

50

16

24

9

51

aimswebPlus RC

7

646

0.49

0.49

51

51

16

26

9

50

aimswebPlus RC

8

639

0.45

0.45

53

47

13

27

10

51

aimswebPlus CA

8

1026

0.69

0.69

50

50

12

22

8

58

1 correlation adjusted for range restriction

aimswebPlus Reading ORF Score Concurrent Validity Coefficient, by Grade and Criterion Measure

Criterion

Grade

N

Correlation

Gender Percentage

Race Percentage

Unadjusted

Adjusted1

F

M

B

H

O

W

ITBS

1

  82

0.68

0.75

41

59

25

25

17

33

ISAT

3

113

0.70

0.76

47

53

2

28

20

49

ISAT

4

230

0.72

0.77

56

44

4

39

10

47

ISAT

5

250

0.69

0.72

48

52

4

22

13

61

ISAT

6

332

0.66

0.69

58

42

9

14

12

65

ISAT

7

179

0.60

0.64

44

56

12

12

7

68

ISAT

8

202

0.56

0.63

46

54

10

11

6

74

NWEA

2

218

0.71

0.72

52

48

2

23

21

53

NWEA

3

128

0.69

0.69

45

55

2

25

20

52

NWEA

4

150

0.67

0.75

53

47

4

28

16

52

NWEA

5

125

0.74

0.72

48

52

3

30

18

48

NWEA

6

141

0.57

0.61

52

48

4

21

12

63

MAP - GLA

3

226

0.53

0.53

55

45

24

2

2

72

MAP - GLA

4

229

0.73

0.72

49

51

32

1

5

62

MAP - GLA

5

157

0.55

0.52

50

50

42

0

7

50

MAP - GLA

6

77

0.55

0.50

56

44

27

2

6

65

MAP - GLA

7

124

0.61

0.63

46

54

41

4

0

55

STAAR

3

205

0.51

0.56

56

44

10

49

14

27

STAAR

4

266

0.57

0.61

44

56

8

52

10

29

STAAR

5

152

0.58

0.63

53

47

9

57

3

31

aimswebPlus RC

2

1523

0.62

0.62

49

51

15

26

10

48

aimswebPlus RC

3

1441

0.62

0.62

51

49

12

25

10

53

aimswebPlus RC

4

1540

0.55

0.55

50

50

12

24

10

54

aimswebPlus RC

5

1468

0.55

0.55

52

48

15

24

11

50

aimswebPlus RC

6

1042

0.51

0.51

50

50

16

24

9

51

aimswebPlus RC

7

796

0.52

0.52

51

51

16

26

9

50

aimswebPlus RC

8

726

0.47

0.47

53

47

13

27

10

51

aimswebPlus CA

8

1026

0.74

0.74

50

50

12

22

8

58

1 correlation adjusted for range restriction 

Predictive Validity of the Slope of Improvement: Data Unavailable

The predictive validity of the Oral Reading Fluency (ORF) slope was assessed using the correlation of the annual ORF ROI (ORFROI) with a spring criterion (CritSpring) test scores, after controlling for fall ORF (ORFFall) performance. The model used is shown here:

〖Crit〗_spring= Intercept+ (β_1 )×〖ORF〗_Fall+ (β_2 )×〖ORF〗_ROI+ ε

A positive and statistically significant β_2 indicates that for a given fall ORF score, students with higher ORF ROIs had higher spring Criterion scores. 

Grade 1 Criterion: Iowa Test of Basic Skills Total Reading Score

Grades 2–8: aimswebPlus Reading Comprehension score

aimswebPlus Reading Comprehension (RC) is an online, untimed (non-speeded) reading measure that assesses literal and inferential comprehension. It includes three literary and three informational passages, each with four questions aligned to the Common Core State Standards of Reading.

RC is used exclusively for screening (benchmarking) and is not part of the progress monitoring system. There are three RC benchmark forms (fall, winter, and spring), each with 24 items. Scores are reported on a developmental scale that spans Grades 2 through 8.

Predictive validity of the fall to spring rate of improvement, Oral Reading Fluency

Grade

n

b2

SE

T

p

1

82

10.6

2.02

5.2

<0.01

2

1289

7.6

0.87

8.7

<0.01

3

1203

6.2

1.00

6.2

<0.02

4

1202

5.4

0.80

6.7

<0.03

5

1126

6.3

1.02

6.1

<0.04

6

834

5.8

1.08

5.4

<0.05

7

525

-1.4

1.33

-1.1

0.28

8

494

2.4

1.42

1.7

0.09

 

Disaggregated Reliability and Validity Data: Data Unavailable

Alternate Forms: Partially Convincing Evidence

What is the number of alternate forms of equal and controlled difficulty? 20 per grade

To maximize the equivalency of the alternate test forms used for progress monitoring, each form was developed from the same set of test specifications (i.e., test blueprints). These specifications address:

Passage length, based on word count (250 words for Grades 1 and 2, 300 words for Grade 3; 350 words for Grades 4 through 8).

Grade-appropriate number of syllables and sentences per 100 words, based on the Fry readability formula.

Review process (reviewed by aimsweb staff and either returned to author (with edits) for revision or rejected if the subject matter did not have wide cultural appeal).

For Grade 1, the first 60 words were either highly decodable or common sight words.

Twenty-four alternate forms per grade were developed and administered to students from across the U.S. For this study, each student read aloud six on-grade level ORF forms. The 24 forms per grade were divided among 12 sets. Each set included the winter benchmark form as an anchor form, and blocks of five additional forms drawn from the 24 alternate PM forms. Each block of five was assigned to two of the 12 sets, with the order of the five PM forms reversed across the two sets they appeared in. In each set, the anchor form was always the first form administered. For example, Set 1A = Winter, PM1, PM2, PM3, PM4, PM5; while Set 1B = Winter, PM5, PM4, PM3, PM2, PM1. This approach was used to control for order effects and sampling variation.

Sets were randomly assigned to students by spiraling sets within grade at each testing site.

Form equivalency is further evaluated by comparing the mean difficulty of each form. Two methods are used to describe comparability of form difficulty: effect size and percentage of total score variance attributable to form.

The effect size (ES) for each form is the mean of the form minus the weighted average across all forms divided by the pooled SD:

 ES=((x_i-X ̅ ))/〖SD〗_pooled

Effect sizes less than 0.30 are considered small. Most effect sizes were less than 0.10–0.25 for the ORF forms, depending on grade.

The percentage of the total score variance attributable to test form was computed by dividing the between form variance by the pooled within form variance plus between form variance. The percentage of test score variance attributable to forms was less than 1%–4%, depending on grade.

Grade

Measure

Form

n range

Mean

SD

ES

1

ORF

7

55

64.7

37.9

0.05

1

ORF

8

45

68.9

39.0

0.07

1

ORF

9

44

67.8

32.9

0.04

1

ORF

10

44

67.0

36.6

0.02

1

ORF

11

42

62.3

35.7

0.12

1

ORF

12

22

61.8

27.7

0.13

1

ORF

13

42

69.9

24.7

0.10

1

ORF

14

22

70.9

38.4

0.13

1

ORF

15

20

68.2

32.7

0.05

1

ORF

16

20

68.9

32.3

0.07

1

ORF

17

44

68.2

36.6

0.05

1

ORF

18

41

68.4

36.5

0.06

1

ORF

19

21

65.8

27.7

0.01

1

ORF

20

55

60.2

38.8

0.18

1

ORF

21

54

68.2

39.3

0.05

1

ORF

22

20

63.4

34.2

0.08

1

ORF

23

54

62.5

37.7

0.11

1

ORF

24

42

68.9

34.0

0.07

1

ORF

25

63

62.5

30.8

0.11

1

ORF

26

42

67.8

36.2

0.04

     

Mean

66.3

34.5

0.08

     

SD

3.15

4.16

 

 

 

 

 

Percentage variance:  0.78%

 

 

Grade

Measure

Form

n range

Mean

SD

ES

2

ORF

7

63

73.8

39.1

0.06

2

ORF

8

63

67.1

35.9

0.23

2

ORF

9

63

75.8

37.5

0.01

2

ORF

10

64

66.6

44.5

0.24

2

ORF

11

64

83.9

43.2

0.20

2

ORF

12

62

81.6

37.9

0.14

2

ORF

13

62

69.7

36.4

0.16

2

ORF

14

63

84.4

41.4

0.21

2

ORF

15

59

67.1

36.1

0.23

2

ORF

16

62

78.0

36.3

0.05

2

ORF

17

63

71.1

41.3

0.13

2

ORF

18

62

85.2

39.5

0.23

2

ORF

19

62

84.7

43.8

0.22

2

ORF

20

59

74.9

33.6

0.03

2

ORF

21

62

75.0

38.5

0.03

2

ORF

22

59

70.3

33.6

0.15

2

ORF

23

64

78.2

42.3

0.06

2

ORF

24

63

81.2

37.3

0.13

2

ORF

25

64

65.9

43.2

0.26

2

ORF

26

63

86.1

40.8

0.26

     

Mean

76.0

39.1

0.15

     

SD

6.94

3.33

 

 

 

 

 

Percentage variance:  3.77%

 

 

Grade

Measure

Form

n range

Mean

SD

ES

3

ORF

7

65

106.0

40.8

0.13

3

ORF

8

68

104.7

34.8

0.09

3

ORF

9

68

106.1

26.7

0.13

3

ORF

10

65

103.5

37.1

0.06

3

ORF

11

65

98.5

38.6

0.08

3

ORF

12

64

95.8

35.7

0.16

3

ORF

13

68

96.2

33.5

0.15

3

ORF

14

66

105.9

33.8

0.12

3

ORF

15

61

101.0

35.7

0.01

3

ORF

16

65

101.4

37.1

0.00

3

ORF

17

66

108.3

34.8

0.19

3

ORF

18

64

104.9

39.7

0.10

3

ORF

19

66

100.8

35.5

0.02

3

ORF

20

61

95.0

34.6

0.18

3

ORF

21

64

109.0

38.4

0.21

3

ORF

22

61

88.7

36.4

0.36

3

ORF

23

66

96.5

32.6

0.14

3

ORF

24

64

112.6

36.7

0.31

3

ORF

25

61

93.4

36.8

0.22

3

ORF

26

68

100.5

37.2

0.03

     

Mean

101.4

35.8

0.13

     

SD

5.98

2.98

 

 

 

 

 

Percentage variance:  2.60%

 

 

Grade

Measure

Form

n range

Mean

SD

ES

4

ORF

7

69

113.2

38.7

0.20

4

ORF

8

65

115.5

37.3

0.14

4

ORF

9

70

128.8

29.4

0.19

4

ORF

10

65

125.3

44.1

0.11

4

ORF

11

65

121.4

45.9

0.01

4

ORF

12

65

126.5

44.0

0.14

4

ORF

13

65

116.4

40.8

0.12

4

ORF

14

65

115.5

39.4

0.14

4

ORF

15

70

120.2

35.3

0.02

4

ORF

16

65

118.7

41.4

0.06

4

ORF

17

65

117.1

38.1

0.10

4

ORF

18

61

125.2

38.4

0.10

4

ORF

19

65

130.0

43.4

0.22

4

ORF

20

69

119.4

42.3

0.05

4

ORF

21

65

125.5

36.2

0.11

4

ORF

22

70

114.5

37.8

0.17

4

ORF

23

61

123.1

37.1

0.05

4

ORF

24

70

122.9

34.4

0.04

4

ORF

25

61

115.8

37.4

0.14

4

ORF

26

65

128.5

43.5

0.19

     

Mean

121.2

39.2

0.12

     

SD

5.27

4.00

 

 

 

 

 

Percentage variance:  1.68%

 

 

Grade

Measure

Form

n range

Mean

SD

ES

5

ORF

7

52

141.3

38.9

0.30

5

ORF

8

54

126.2

35.8

0.11

5

ORF

9

56

125.4

36.3

0.13

5

ORF

10

54

124.3

32.6

0.16

5

ORF

11

54

141.1

35.7

0.29

5

ORF

12

52

135.7

37.4

0.15

5

ORF

13

52

119.8

33.8

0.28

5

ORF

14

56

129.4

38.9

0.02

5

ORF

15

52

137.6

37.3

0.20

5

ORF

16

56

125.7

32.9

0.12

5

ORF

17

52

123.6

35.3

0.18

5

ORF

18

52

137.7

40.8

0.20

5

ORF

19

57

125.7

41.5

0.12

5

ORF

20

56

120.3

36.1

0.27

5

ORF

21

54

139.8

39.1

0.26

5

ORF

22

56

133.4

37.6

0.09

5

ORF

23

57

140.2

40.1

0.27

5

ORF

24

56

129.5

38.0

0.02

5

ORF

25

52

126.8

36.4

0.09

5

ORF

26

56

120.8

35.9

0.25

     

Mean

130.2

37.0

0.18

     

SD

7.44

2.44

 

 

 

 

 

Percentage variance:  3.65%

 

 

Grade

Measure

Form

n range

Mean

SD

ES

6

ORF

7

66

156.2

41.6

0.09

6

ORF

8

68

147.3

33.9

0.15

6

ORF

9

66

157.4

39.8

0.13

6

ORF

10

68

155.1

35.6

0.06

6

ORF

11

64

157.8

34.0

0.14

6

ORF

12

64

145.4

29.3

0.20

6

ORF

13

66

152.8

42.3

0.00

6

ORF

14

65

147.8

37.7

0.13

6

ORF

15

66

158.4

40.1

0.15

6

ORF

16

66

143.8

39.3

0.24

6

ORF

17

66

154.3

44.0

0.04

6

ORF

18

68

151.7

38.9

0.03

6

ORF

19

67

158.1

32.3

0.14

6

ORF

20

64

159.4

32.1

0.18

6

ORF

21

67

150.3

38.6

0.07

6

ORF

22

66

144.1

38.7

0.23

6

ORF

23

64

146.5

34.7

0.17

6

ORF

24

66

145.5

36.0

0.20

6

ORF

25

67

160.5

35.0

0.21

6

ORF

26

65

162.4

37.3

0.26

     

Mean

152.7

37.1

0.14

     

SD

6.01

3.75

 

 

 

 

 

Percentage variance:  2.40%

 

 

Grade

Measure

Form

n range

Mean

SD

ES

7

ORF

7

51

145.5

38.4

0.19

7

ORF

8

55

139.3

26.6

0.35

7

ORF

9

51

153.7

39.7

0.02

7

ORF

10

55

150.5

44.1

0.06

7

ORF

11

55

156.5

38.3

0.09

7

ORF

12

51

153.5

38.9

0.01

7

ORF

13

51

162.1

39.9

0.24

7

ORF

14

51

165.2

42.9

0.32

7

ORF

15

57

157.8

43.5

0.13

7

ORF

16

55

144.0

41.8

0.23

7

ORF

17

55

151.9

47.9

0.03

7

ORF

18

55

151.3

46.8

0.04

7

ORF

19

55

147.5

28.3

0.14

7

ORF

20

55

165.2

40.0

0.32

7

ORF

21

51

158.1

38.6

0.13

7

ORF

22

55

137.2

34.8

0.41

7

ORF

23

55

156.8

35.8

0.10

7

ORF

24

51

153.4

34.5

0.01

7

ORF

25

51

153.0

39.7

0.00

7

ORF

26

57

156.2

34.6

0.08

     

Mean

152.9

38.8

0.15

     

SD

7.55

5.37

 

 

 

 

 

Percentage variance:  3.44%

 

 

Grade

Measure

Form

n range

Mean

SD

ES

8

ORF

7

58

161.8

42.4

0.25

8

ORF

8

56

152.1

32.5

0.01

8

ORF

9

58

160.2

42.8

0.21

8

ORF

10

56

143.3

31.2

0.20

8

ORF

11

56

150.8

42.2

0.02

8

ORF

12

56

151.1

42.2

0.01

8

ORF

13

56

140.3

26.4

0.28

8

ORF

14

58

152.3

38.3

0.02

8

ORF

15

53

159.3

47.2

0.19

8

ORF

16

56

152.6

43.2

0.03

8

ORF

17

58

151.4

40.5

0.00

8

ORF

18

52

146.7

38.6

0.12

8

ORF

19

52

150.6

47.7

0.02

8

ORF

20

56

138.3

41.5

0.33

8

ORF

21

53

161.8

48.2

0.25

8

ORF

22

52

151.7

43.4

0.00

8

ORF

23

56

153.9

52.9

0.06

8

ORF

24

56

154.1

33.2

0.06

8

ORF

25

53

144.5

43.8

0.17

8

ORF

26

56

154.6

37.2

0.07

     

Mean

151.6

40.8

0.12

     

SD

6.5

6.4

 

 

 

 

 

Percentage variance:  2.35%

 

 

Sensitive to Student Improvement: Convincing Evidence

Describe evidence that the monitoring system produces data that are sensitive to student improvement (i.e., when student learning actually occurs, student performance on the monitoring tool increases on average).

Sensitivity to improvement was assessed by demonstrating that annual performance gains were statistically significant and moderate in size as expressed in fall standard deviation units. A gain expressed in SD units that exceeds 0.3 can be considered moderate (see Cohen, J., 1988. Statistical Power Analysis for the Behavioral Sciences (Second Edition). Lawrence Erlbaum Associates.)

Fall and spring benchmark mean, SD, paired-sample t, and annual gain represented as fall standard deviation units, Oral Reading Fluency by grade

 

Mean

SD

N

Paired t

p

Gain/SD

Grade

Fall

Spring

Fall

Spring

1

41.0

74.4

22.02

21.95

2000

50.8

<0.01

1.11

2

73.2

104.7

36.41

41.30

2000

86.0

<0.01

0.87

3

96.0

122.8

37.98

36.88

2000

71.4

<0.01

0.71

4

114.2

136.1

36.32

39.55

2000

56.6

<0.01

0.60

5

131.1

152.1

39.83

42.95

2000

51.0

<0.01

0.53

6

148.6

158.0

37.06

38.13

2000

17.8

<0.01

0.26

7

150.8

170.2

39.29

39.70

2000

34.9

<0.01

0.49

8

153.7

158.4

36.05

36.47

2000

7.4

<0.01

0.13

 

End-of-Year Benchmarks: Convincing Evidence

Are benchmarks for minimum acceptable end-of-year performance specified in your manual or published materials?

Yes

Specify the end-of-year performance standards:

aimswebPlus allows users to select from a range of end-of-year targets the one that is most appropriate for their instructional needs. The targets are based on spring reading or math composite national percentiles by grade level. Twelve national percentile targets ranging from the 15th through the 70th percentile are provided, in increments of 5.

For Grades 3 through 8, it is recommended that users select the spring percentile that most closely aligns to the overall percentage of students below proficient on state reading/math tests. This is the percentage considered at risk. For example, if the percentage of students below proficient on the state test is 20%, the recommended end-of-year benchmark is the 20th percentile. Likewise, if the percentage of students below proficient on the state test is 60%, the recommended end-of-year benchmark is the 60th percentile.

Because passing rates on state assessments are fairly consistent across grades, the percentage of students at risk in Kindergarten through Grade 2 is likely to be very similar to the percentage at risk in Grade 3. As such, aimswebPlus recommends using the percentage of students below proficient on the Grade 3 state reading/math tests as the end-of-year benchmark for students in Kindergarten through Grade 2. For example, if the percentage of students below proficient on the state test in Grade 3 is 30%, the recommended end-of-year benchmark for students in Kindergarten through Grade 2 is the 30th percentile.

If these percentages are not available, aimswebPlus recommends using the 25th percentile as the end-of-year benchmark.

Fall and winter benchmark cut scores are derived automatically by the aimswebPlus system. The cut scores are based on empirical research of the relationship between fall/winter scores and spring benchmarks. Two cut-scores are provided: one corresponding to a 50% probability of exceeding the spring benchmark, and the other corresponding to an 80% probability of exceeding the spring benchmark. Fall or winter scores above the 80% probability cut score are deemed low risk; fall or winter scores between the 50% and 80% cut scores are deemed moderate risk; and fall or winter scores below the 50% probability cut score are deemed high risk. These three levels correspond to the RTI tiers reported in the aimswebPlus system.

What is the basis for specifying minimum acceptable end-of-year performance?

Norm-referenced

Specify the benchmarks:

Percentage of students below proficient level on state test.

What is the basis for specifying these benchmarks?

Norm-referenced

If norm-referenced, describe the normative profile:

Demographic Characteristics of the aimswebPlus Oral Reading Fluency Norm Sample, Grades 1 Through 8

   

Sex

Race

SES (F/R lunch)

Subject

Grade

F

M

B

H

O

W

Low

Mod

High

ORF

1

0.50

0.50

0.13

0.25

0.10

0.51

0.32

0.32

0.36

ORF

2

0.49

0.51

0.15

0.26

0.10

0.48

0.32

0.32

0.36

ORF

3

0.51

0.49

0.12

0.25

0.10

0.53

0.32

0.32

0.36

ORF

4

0.50

0.50

0.12

0.24

0.10

0.54

0.32

0.32

0.36

ORF

5

0.52

0.48

0.15

0.24

0.11

0.5

0.32

0.32

0.36

ORF

6

0.50

0.50

0.16

0.24

0.09

0.51

0.32

0.32

0.36

ORF

7

0.51

0.49

0.16

0.26

0.09

0.5

0.32

0.32

0.36

ORF

8

0.53

0.47

0.13

0.27

0.10

0.51

0.32

0.32

0.36

Representation: National

Date: 2013–2014

Number of States: 12 (Grade 1), 18 (Grades 2-8)

Regions: 5

Gender: 50% male, 50% female

SES: Low, middle, high, free and reduced lunch

ELL: 10%

Rates of Improvement Specified: Convincing Evidence

Is minimum acceptable growth (slope of improvement or average weekly increase in score by grade level) specified in your manual or published materials?

Yes

Specify the growth standards:

aimswebPlus provides student growth percentiles (SGP) by grade and initial (fall and winter) performance level for establishing growth standards. An SGP indicates the percentage of students in the national sample whose seasonal (or annual) rate of improvement (ROI) fell at or below a specified ROI. Separate SGP distributions are computed for each of five levels of initial (fall or winter) performance to control for differences in growth rate by initital performance level. 

When setting a performance goal for a student, the system automatically generates feedback as to the appropriateness of the goal. An SGP < 50 is considered Insufficient; an SGP between 50 and 85 is considered Closes the Gap; an SGP between 85 and 97 is considered Ambitious; and an SGP > 97 is considered Overly Ambitious. aimswebPlus recommends setting performance goals that represents rates of growth between the 85th and 97th SGP. However, the user ultimately determines what growth rate is required on an individual basis.

What is the basis for specifying minimum acceptable growth?

Norm-referenced

If norm-referenced, describe the normative profile.

Demographic Characteristics of the aimswebPlus Oral Reading Fluency Norm Sample, Grades 1 Through 8

   

Sex

Race

SES (F/R lunch)

Subject

Grade

F

M

B

H

O

W

Low

Mod

High

ORF

1

0.50

0.50

0.13

0.25

0.10

0.51

0.32

0.32

0.36

ORF

2

0.49

0.51

0.15

0.26

0.10

0.48

0.32

0.32

0.36

ORF

3

0.51

0.49

0.12

0.25

0.10

0.53

0.32

0.32

0.36

ORF

4

0.50

0.50

0.12

0.24

0.10

0.54

0.32

0.32

0.36

ORF

5

0.52

0.48

0.15

0.24

0.11

0.5

0.32

0.32

0.36

ORF

6

0.50

0.50

0.16

0.24

0.09

0.51

0.32

0.32

0.36

ORF

7

0.51

0.49

0.16

0.26

0.09

0.5

0.32

0.32

0.36

ORF

8

0.53

0.47

0.13

0.27

0.10

0.51

0.32

0.32

0.36

Representation: National

Date: 2013–2014

Number of States: 12 (Grade 1), 18 (Grades 2-8)

Regions: 5

Gender: 50% male, 50% female

SES: Low, middle, high, free and reduced lunch

ELL: 10%

Please describe other procedures for specifying adequate growth:

To get the most value from progress monitoring, aimswebPlus recommends the following: (1) establish a time frame, (2) determine the level of performance expected, and (3) determine the criterion for success.  Typical time frames include the duration of the intervention or the end of the school year. An annual time frame is typically used when IEP goals are written for students who are receiving special education.  For example, aimswebPlus goals can be written as follows: In 34 weeks, the student will compare numbers and answer computational problems to earn of score of 30 points on Grade 4 Number Sense Fluency forms.

The criterion for success may be set according to standards, local norms, national norms, or a normative rate of improvement (ROI). For example, the team may want to compare a student’s performance to district/local norms, which compares the student’s score to his or her peers in the context of daily learning.

For normative ROIs, aimswebPlus uses student growth percentiles to describe these normative rates of improvement. Within the aimswebPlus software, the user enters the goal date and moves a digital slider to the desired ROI.  As the slider moves, it provides feedback about the strength of the ROI: Insufficient, Closes the Gap, Ambitious, or Overly Ambitious. Users are encouraged to use the Ambitious (85th – 97th SGP) for students in need of intensive intervention.

Decision Rules for Changing Instruction: Convincing Evidence

Does your manual or published materials specify validated decision rules for when changes to instruction need to be made?

Yes

Specify the decision rules:

aimswebPlus applies a statistical procedure to the student’s progress monitoring scores in order to provide empirically-based guidance about whether the student is likely to meet, fall short of, or exceed his/her goal. The calculation procedure (presented below) is fully described in the aimsweb Progress Monitoring Guide (Pearson, 2012). aimswebPlus users will not have to do any calculations—the online system does this automatically. The decision rule is based on a 75% confidence interval for the student’s predicted score at the goal date. This confidence interval is student-specific and takes into account the number and variability of progress monitoring scores and the duration of monitoring. Starting at the sixth week of monitoring (when there are at least four monitoring scores), the aimswebPlus report following each progress monitoring administration includes one of the following statements:

A. “The student is projected to not reach the goal.” This statement appears if the confidence interval is completely below the goal score.

B. “The student is projected to exceed the goal.” This statement appears if the confidence interval is completely above the goal score.

C. “The student is projected to be near the goal. The projected score at the goal date is between X and Y” (where X and Y are the bottom and top of the confidence interval). This statement appears if the confidence interval includes the goal score.

If Statement A appears, the user has a sound basis for deciding that the current intervention is not sufficient and a change to instruction should be made. If Statement B appears, there is an empirical basis for deciding that the goal is not sufficiently challenging and should be increased. If Statement C appears, the student’s progress is not clearly different from the aimline, so there is not a compelling reason to change the intervention or the goal; however, the presentation of the confidence-interval range enables the user to see whether the goal is near the upper limit or lower limit of the range, which would signal that the student’s progress is trending below or above the goal.

A 75% confidence interval was chosen for this application because it balances the costs of the two types of decision errors. Incorrectly deciding that the goal will not be reached (when in truth it will be reached) has a moderate cost: an intervention that is working will be replaced by a different intervention. Incorrectly deciding that the goal may be reached (when in truth it will not be reached) also has a moderate cost: an ineffective intervention will be continued rather than being replaced. Because both kinds of decision errors have costs, it is appropriate to use a modest confidence level.

Calculation of the 75% confidence interval for the score at the goal date:

Calculate the trend line. This is the ordinary least-squares regression line through the student’s monitoring scores.

Calculate the projected score at the goal date. This is the value of the trend line at the goal date.

Calculate the standard error of estimate (SEE) of the projected score at the goal date, using the following formula:

〖SEE〗_(predicted score)= √((∑_i^k▒(y_i-y ́_i )^2 )/(k-2))×√(1+1/k+(GW-(∑_1^k▒w_i )/k)^2/(∑_i^k▒(w_i-(∑_1^k▒w_i )/k)^2 ))

where k = number of completed monitoring administrations, w = week number of a completed administration, GW = week number of the goal date, y = monitoring score, y’ = predicted monitoring score at that week (from the student’s trendline).The means and sums are calculated across all of the completed monitoring administrations up to that date. Add and subtract 1.25 times the SEE to the projected score, and round to the nearest whole numbers.

What is the evidentiary basis for these decision rules?

The decision rules are statistically rather than empirically based. The guidance statements that result from applying the 75% confidence interval to the projected score are correct probabilistic statements, under certain assumptions: The student’s progress can be described by a linear trend line. If the pattern of the student’s monitoring scores is obviously curvilinear, then the projected score based on a linear trend will likely be misleading. We provide training in the aimsweb Progress Monitoring Guide about the need for users to take non-linearity into account when interpreting progress-monitoring data. The student will continue to progress at the same rate as they have been progressing to that time. This is an unavoidable assumption for a decision system based on extrapolating from past growth.

Even though the rules are not derived from data, it is useful to observe how they work in a sample of real data. For this purpose, we selected random samples of students in the aimsweb 2010–2011 database who were progress-monitored on either Reading Curriculum-Based Measurement (R-CBM) or Math Computation (M-COMP). All students selected scored below the 25th percentile in the fall screening administration of R-CBM or M-COMP. The R-CBM sample consisted of 1,000 students (200 each at of Grades 2 through 6) who had at least 30 monitoring scores, and the M-COMP sample included 500 students (100 per Grades 2 through 6) with a minimum of 28 monitoring scores. This analysis was only a rough approximation, because we did not know each student’s actual goal or whether the intervention or goal was changed during the year.

To perform the analyses, we first set an estimated goal for each student by using the ROI at the 85th percentile of aimsweb national ROI norms to project their score at their 30th monitoring administration. Next, we defined “meeting the goal” as having a mean score on the last three administrations (e.g., the 28th through 30th administrations of R-CBM) that was at or above the goal score. At each monitoring administration for each student, we computed the projected score at the goal date and the 75% confidence interval for that score, and recorded which of the three decision statements was generated (projected not to meet goal, projected to exceed goal, or on-track/no-change).

In this analysis, accuracy of guidance to change (that is, accuracy of projections that the student will not reach the goal or will exceed the goal) reached a high level (80%) by about the 13th to 15th monitoring administration, on average. The percentage of students receiving guidance to not change (i.e., their trendline was not far from the aimline) would naturally tend to decrease over administrations as the size of the confidence interval decreased. At the same time, however, there was a tendency for the trendline to become closer to the aimline over time as it became more accurately estimated, and this worked to increase the percentage of students receiving the “no change” guidance.

Decision Rules for Increasing Goals: Convincing Evidence

Does your manual or published materials specify validated decision rules for when changes to increase goals?

Yes

Specify the decision rules:

The same statistical approach described under Decision Rules for Changing Instruction (GOM 9 above) applies to the decisions about increasing a goal. aimswebPlus provides the following guidance for deciding whether to increase a performance goal:

 If the student is projected to exceed the goal and there are at least 12 weeks remaining in the schedule, consider raising the goal.

What is the evidentiary basis for these decision rules? 

See GOM 9 evidentiary basis information above.

Improved Student Achievement: Data Unavailable

Improved Teacher Planning Data Unavailable