Academy of READING

Study: Torlaković (2011)

Torlaković, E. (2011). Academy of READING® Efficacy Study: Whitehall City Schools – Special Education Report.
Descriptive Information Usage Acquisition and Cost Program Specifications and Requirements Training

The Academy of READING, formerly known as the AutoSkill Component Reading Subskills program, is an intensive and comprehensive Tier II and Tier III intervention tool that focuses on strengthening the root skills of reading. Completely web-based since version 5 was released in 2005, the Academy of READING features age-appropriate content, interfaces, and graphical elements that create an optimal learning environment for students as they build foundational reading skills.

Design Principles and Features

  • Personalized Learning –Using test data the program identifies performance goals and creates a Personalized Training Plan
  • Automaticity
  • Explicit Instruction –Systematic and direct instruction is conducted in the five key pillars recognized by the National Reading Panel:
    • Phonemic Awareness
    • Phonics
    • Fluency
    • Vocabulary and Comprehension
  • Positive Feedback
  • Assessment Options – Academy of READING includes automated online tests that take a snapshot of each student’s reading abilities relative to grade level.
  • In-product reporting can be accessed at any time, from any workstation on the network, through the browser-based management system.
  • Spanish-language support
  • RtI Tools—The RtI Package can be added to the Academy of READING for a single, web-based solution that connects the key components of RtI

 

 

Academy of READING is intended for use in grades 2 through high school. It is designed for use with students with disabilities (including learning disabilities and behavioral disabilities), English language learners and any student at risk of academic failure. The academic area of focus is reading (including phonological awareness, phonics/word study, comprehension, fluency, and vocabulary).

The program has been in use since 1985, in 15 different countries, including: Canada, USA, US Virgin Islands, Bahamas, Jamaica, St. Lucia, St. Thomas, West Indies, Montego Bay, South Africa, Australia, UK, Scotland, England, Northern Ireland, and Republic of Ireland. In the USA only, it is currently in use in over 2950 schools.

 

 

Where to obtain:

EPS/School Specialty Literacy and Intervention
625 Mount Auburn Street
Cambridge, MA 02138
Phone: 617-547-6706
Toll Free: 1-800-225-5750
Web Site: eps.schoolspecialty.com

Cost: Academy of READING is available as per-school perpetual licenses or as an annual per-student subscription, with the following pricing and fulfillment:

  • Perpetual License Model- Perpetual product pricing is based on a sliding scale of concurrent users: 5, 10, 20, 30, and unlimited.
  • We can provide schools and districts customizable pricing, based on their needs.
  • Schools receive an email with links to activate their licenses and an educator’s kit with wall charts, checklists, and supplementary activities.
  • In Year 1, technical support and maintenance is included in the software cost. In Year 2 and beyond, schools can optionally purchase a support and maintenance program.
  • Volume discounts can also be applied for large, multi-school orders.
  • Annual per student Subscription Model –Licenses must be renewed annually for continued use of the program.
  • Technical support and maintenance is included in the annual subscription price.
  • Volume discounts may apply to large, multi-school orders.
  • Schools receive an installation kit with the software DVD and installation guide to install on the school or district server and an educator’s kit, which contains wall charts, checklists, and supplementary activities.
  • Professional Development- is provided with the purchase of the Academy of READING program and includes one day of teacher training, one day of live start up with the students as well as two days of in class coaching/best practice and data review.  Teacher instructor resource guides are also provided for up to 10 teachers.

 

 

It is recommended that the Academy of READING is used 20-30 minutes per session, three to five days a week.

The program includes a highly specified teacher’s manual.

Academy of READING is a browser-based instructional application that can run on a WAN or LAN and requires a T1 or similar network infrastructure. It can also be hosted by School Specialty Intervention.

 

 

The instruction training includes one full day (6 hours) of hands-on training to learn functionality of the program and three days of on-site mentoring and coaching with students (teachers not pulled out).

The minimum qualifications of instructors are that they must be paraprofessionals.

The current version of the Instructor’s Resource Guide has been field tested for the past 4 years.

Additional days of professional development are available including advanced reading workshops and on-site days of consultation. Technical support is included in the software price in Year 1 and can optionally be purchased for subsequent years.

 

 

Participants: Convincing Evidence

Sample size: 75 students (38 program, 37 control)

Risk Status: Students were pre-tested with the Gates MacGinitie Reading Test (GMRT). This test is a norm-referenced group-administered reading survey. It is widely used and well-recognized to identify students at academic risk due to their poor reading skills.

Demographics:

 

Program

Control

p of chi square

Number

Percentage

Number

Percentage

Grade level

  Kindergarten

 

 

 

 

 0.99

  Grade 1

 

 

 

 

 

  Grade 2

 2

50% 

50% 

 

  Grade 3

1

33%

2

67%

 

  Grade 4

43% 

57% 

 

  Grade 5

57% 

43% 

 

  Grade 6

50% 

50% 

 

  Grade 7

44% 

56% 

 

  Grade 8

60% 

40% 

 

  Grade 9

 8

53% 

47% 

 

  Grade 10

50% 

50% 

 

  Grade 11

38% 

63% 

 

  Grade 12

 

 

 

 

 

Race-ethnicity

  African-American

11

44%

14

56%

0.55

  American Indian

       

 

  Asian/Pacific Islander

       

 

  Hispanic

6

67%

3

33%

 

  White

16

46%

19

54%

 

  Other

5

63%

3

38%

 

Socioeconomic status

  Subsidized lunch

31

53%

28

47%

0.31

  No subsidized lunch

7

39%

11

61%

 

Disability statusa

  Speech-language impairments

         

  Learning disabilities

       

 

  Behavior disorders

       

 

  Intellectual disabilities

       

 

  Other

38

49%

39

51%

 

  Not identified with a disability

       

 

ELL status

  English language learner

5

83%

1

17%

0.08

  Not English language learner

33

47%

38

54%

 

Gender

Female

12

46%

14

54%

0.69

Male

26

51%

25

49%

 

a All students included in the current study were identified as having a learning disability. However, data on the specific disability was not collected.

Training of Instructors: The 34 instructors involved in the program had been teaching for a range of 2 to 40 years. They had been teaching at their current school for a range of 1 to 32 years. Twelve of the teachers held a Bachelor’s degree, 21 held a Master’s degree, and 1 held a Doctorate degree. Following initial training all instructors had access to technical support throughout the study.

Design: Partially Convincing Evidence

Did the study use random assignment?: Yes.

If not, was it a tenable quasi-experiment?: Not applicable.

If the study used random assignment, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures used as covariates or on pretest measures also used as outcomes?: No.

If not, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures central to the study (i.e., pretest measures also used as outcomes), and outcomes were analyzed to adjust for pretreatment differences?: Not applicable.

Were the program and control groups demographically comparable at pretreatment?: Yes.

Was there attrition bias1 ?: No.

Did the unit of analysis match the unit for random assignment (for randomized studies) or the assignment strategy (for quasi-experiments)?: Yes.

1 NCII follows guidance from the What Works Clearinghouse (WWC) in determining attrition bias. The WWC model for determining bias based on a combination of differential and overall attrition rates can be found on pages 13-14 of this document: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

 

Fidelity of Implementation: Unconvincing Evidence

Describe when and how fidelity of treatment information was obtained: The Academy of READING® automatically collects and stores information concerning time spent in the program, training, percent of program completed, and skills mastered. Study facilitators also collected qualitative reports from teachers on-site to ensure treatment fidelity.

Provide documentation (i.e., in terms of numbers) of fidelity of treatment implementation:

Teachers received implementation guidelines before the beginning of the study that asked them to use the program for a minimum of 30 minutes, three to five times a week.

In Academy of READING instructional benchmarks are identified by the Placement Test that is built into the program and presented to students at their first login.  The Placement test evaluates students’ various reading skills.  Based on the Placement test score, lessons needed for each skill to close the gap to allow a student to reach their grade level are identified.  In order to complete a lesson, a student needs to have a score of at least 85-96% correct for that lesson, depending on the skill that is being taught.  The way that lessons are mapped into sessions is that every time a student logs into the program, they will continue with the lesson that they were last working on or will be presented with the following lesson if the previous lesson was completed with 85% (or higher) accuracy.

Number of lessons for each student is identified by their performance on the Placement test, in other words, the number of lessons for each skill area depends on students’ proficiency.  Percentage of program completed is a variable automatically stored in an online database and represents percentage of assigned lessons completed by each student.  In one semester (or one school year), a less proficient student, with the same number of lessons completed, the same time in program, and time on task will have much smaller percentage of program completed than a more proficient student, since the amount of work they need to do in order to close the gap and be at their grade level is much larger than the same of a more proficient student.  Time in program represents time on task plus time spent on computer delivered instruction plus time spent on tutorials plus time spent on motivational components, while time on task represents the time a student spent working on actual lessons. Therefore, the best measure of fidelity for Academy of Reading is time on task which is on average 50% of time in program.  In addition, if there is no activity, a student is automatically logged out of the program after 5 minutes.

School Specialty Literacy and Intervention set a requirement of 3 sessions per week (recommended in the Academy of READING’s best implementation model and required by the study design), which makes 57 total sessions over the course of the intervention (19 weeks). Thirty eight treatment students averaged 43.79 sessions, which identifies levels of fidelity at 77%.  Fourteen of the treatment students met this objective.  The other twenty-four averaged between 21 and 55 sessions.

The 38 treatment students averaged 2.3 sessions/week (SD=0.93). Fourteen of them completed at least 3 sessions per week.  The other twenty four completed between 1 and 2.9 sessions.

Group

N

Total Sessions Required

Avg. Sessions Completed

Avg. Session/Week

Fidelity

Treatment

38

57

43.79

2.30

77%

Control

30

0

0

0

0

EPS/SSLI also set a requirement of 30 minutes time in program per session, which translates into 15 minutes time on task per session over the course of the intervention (19 weeks, 57 sessions).  Treatment students averaged 12.21 minutes on time on task per session, which identified levels of fidelity at 85% (see the following table).

Group

N

Total Time on Task (hr)

Time on Task / Week (min)

Treatment

26

11.59

12.21

Control

25

0

0

A student cannot move from one level or one skill to another until all the required exercises are successfully completed and the necessary skills are acquired.  The program automatically stores this information for each student in a database which is referred to as number of skills mastered. 

For this particular study the number of average skills acquired for the experimental group was provided in the initial study report (copy of the table below).  

Academy of READING Fidelity Measures by Group

Group

N

Skills Mastered

Treatment

38

21.5

Control

30

0.0

 

Measures Targeted: Convincing Evidence

Measures Broader: Partially Convincing Evidence

Targeted Measure Score type & range of measure Reliability statistics Relevance to program instructional content

Academy of Reading (AoR) Placement Test

Grade Level Equivalent (GLE), 0-12

Test-Retest Coefficient = 0.77

Close paragraph test of Academy of READING® material.

Gates-MacGinitie Reading Test (GMRT) Total

GLE, 0-12

Test-Retest Coefficient = 0.89

Provides an overall index of students reading ability and indexes skills trained in Academy of READING®.

System to Enhance Educational Performance (SEEP)

Word Count Per Minute (WCPM), 0-165

Test-Retest Coefficient = 0.94

Reading fluency level assessed using one-minute timed reading.

Academy of Reading (AoR) Oral Reading Fluency Benchmark Assessment (ORFBA)

WCPM, 0-169

Test-Retest Coefficient = 0.92

Reading fluency level assessed using one-minute timed reading.

Note. Maximum scores for STEEP and AoR ORFBA are based on scores from the current sample. The maximum values for these tests are theoretically infinite.  

Broader Measure Score type & range of measure Reliability statistics Relevance to program instructional content

Gates-MacGinitie Reading Test (GMRT) Vocabulary

GLE, 0-12

Test-Retest Coefficent = 0.77

Students are not directly trained on building Vocabulary skills in the Academy of READING®. Therefore, this measure indicates the amount of transfer between specific training in the Academy of READING® and other reading related skills.

Gates-MacGinitie Reading Test (GMRT) Comprehension

GLE, 0-12

Test-Retest Coefficient = 0.83

One of the goals of the Academy of READING® is to develop students’ comprehension abilities. However, they are not directly trained on this skill.

Ohio Achievement Assessments (OAA) Reading Scaled

Scaled Score, 0-432

Test-Retest Coefficient = 0.21

The Ohio Achievement Assessment is a state-wide assessment of reading ability administered to students in grades 3 through 8. The test includes a variety of question formats and types that are not directly trained in the Academy of READING®.

Ohio Achievement Assessments (OAA) Reading Raw

Raw Score, 0-420

Test-Retest Coefficient = 0.33

The Ohio Achievement Assessment is a state-wide assessment of reading ability administered to students in grades 3 through 8. The test includes a variety of question formats and types that are not directly trained in the Academy of READING®.

Note. Maximum scores for STEEP and AoR ORFBA are based on scores from the current sample. The maximum values for these tests are theoretically infinite.

Number of Outcome Measures: 8 Reading

Mean ES - Targeted: 0.36*

Mean ES - Broader: 0.50*

Effect Size:

Targeted Measures

Construct Measure Effect Size
Reading Academy of READING Placement Test 0.72*
Reading GMRT Total Score 0.35
Reading STEEP 0.17
Reading  Academy of READING ORFBA 0.19

Broader Measures

Construct Measure Effect Size
Reading GMRT Comprehension 0.26
Reading GMRT Vocabulary 0.39
Reading OAA Scaled 0.70*
Reading OAA Raw 0.63

 

Key
*        p ≤ 0.05
**      p ≤ 0.01
***    p ≤ 0.001
–      Developer was unable to provide necessary data for NCII to calculate effect sizes
u      Effect size is based on unadjusted means
†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Visual Analysis (Single Subject Design): N/A

Disaggregated Data for Demographic Subgroups: Yes

Targeted Measures

Construct Measure Effect Size
Reading Academy of READING Placement Test - Special Education 0.72*
Reading GMRT Total Score - Special Education 0.35
Reading STEEP - Special Education 0.17
Reading  Academy of READING ORFBA - Special Education 0.19

Broader Measures

Construct Measure Effect Size
Reading GMRT Comprehension - Special Education 0.26
Reading GMRT Vocabulary - Special Education 0.39
Reading OAA Scaled - Special Education 0.70*
Reading OAA Raw - Special Education 0.63

 

Key
*        p ≤ 0.05
**      p ≤ 0.01
***    p ≤ 0.001
–      Developer was unable to provide necessary data for NCII to calculate effect sizes
u      Effect size is based on unadjusted means
†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Disaggregated Data for <20th Percentile: No

Administration Group Size: Individual

Duration of Intervention: 30 minutes, 3-5 times a week, 13 weeks

Minimum Interventionist Requirements: Paraprofessional, 6 hours of hands-on, training; 3 days of on-site, training (teachers not, pulled out)

Reviewed by WWC or E-ESSA: WWC & E-ESSA

What Works Clearinghouse Review

Adolescent Literacy Protocol

EffectivenessNo studies of Academy of READING® that fall within the scope of the Adolescent Literacy review protocol meet What Works Clearinghouse (WWC) group design standards. Because no studies meet WWC group design standards at this time, the WWC is unable to draw any conclusions based on research about the effectiveness or ineffectiveness of Academy of READING® on adolescent readers. Additional research is needed to determine the effectiveness or ineffectiveness of this intervention..

Studies Reviewed: N/A

Full Report

Beginning Reading

EffectivenessAs of July 2007 no studies of Academy of READING® were found that fell within the scope of the Beginning Reading review protocol and met WWC design standards. Therefore, the WWC is unable to draw any research based conclusions about the effectiveness or ineffectiveness of Academy of READING® to improve outcomes in this area.

Studies Reviewed: N/A

Full Report

 

Evidence for ESSA

No studies considered met Evidence for ESSA's inclusion requirements for Elementary or Secondary Reading.

Other Research: Potentially Eligible for NCII Review: 2 studies

Kerr, J. (1993). Computerized reading skills remediation for reading-disabled adolescents: A compensatory supplement to instructional strategies. University of Toronto (Canada). ProQuest Dissertations and Theses, 198 p. Retrieved from http://ezproxy.lakeheadu.ca/login?url=http://search.proquest.com/docview/304080604?accountid=11956
 

Thomas, A. & Clapp, T. (1989). A comparison of computer-assisted component reading skills training and repeated reading for adolescent poor readers. Canadian Journal of Special Education, 5, 135-144.