Academy of MATH

Study: Torlaković (2011)

Torlaković, E. (2011). Academy of MATH® Efficacy Report – Westwood Elementary School.
Descriptive Information Usage Acquisition and Cost Program Specifications and Requirements Training

The Academy of MATH is a standards-based, online intervention tool that focuses on strengthening the root skills of mathematics. It uses a systematic approach to build fluency on foundation math skills in 10 content strands aligned to NCTM and Common Core state standards. For each skill area and level, students attempt to build conceptual knowledge (terms), computational fluency (operations), and strategic competence (word problems).

Design Principles and Features

  • Personalized Learning –Using test data the program identifies performance goals and creates an Individualized Training Plan. While training, the Responsive Intervention Engine monitors progress and adjusts content to accelerate or remediate as required.
  • Positive Feedback
  • Assessments – The online placement test creates an automatically scored snapshot of each student’s math skills relative to grade level. Using testing, educators can: administer group testing, capture baseline data for the start of each session, and determine independent and non-independent levels for each content area.
  • In-product Reporting –Automated reports can be accessed at any time, from anywhere on the network, through the browser-based management system.
  • Spanish Language Support
  • Response to Intervention– Individualized Training Plans (ITP) are given to each child and are kept current by the progress monitoring system. The Academy Management System tracks students testing and training results.

Academy of MATH Content Strands

  • Number & Operations
  • Algebra
  • Measurement
  • Geometry
  • Data Analysis & Probability
  • Apply understanding of probability to predict the likelihood of events

Academy of MATH is intended for use in grades 2 through high school. The program is intended for use with students with disabilities (including learning disabilities and behavioral disabilities), English language learners, and any student at risk of academic failure. The academic area of focus is math (including computation, concepts, word problems, algebra, geometry, and fractions).

The program has been in use since 1990, in 8 different countries, including: the United States, Canada, the Bahamas, Jamaica, St. Lucia, Singapore, the United Kingdom, and the Virgin Islands. In the USA only, it is currently in use in approximately 1,930 schools.

Where to Obtain: 
EPS/School Specialty Literacy and Intervention
625 Mount Auburn Street
Cambridge, MA 02138
Phone: 617-547-6706
Toll Free: 1-800-225-5750
Web Site: eps.schoolspecialty.com

Cost: Academy of MATH is available as per-school perpetual licenses or as an annual per-student subscription, with the following pricing and fulfillment: 

  • Perpetual License Model - Perpetual product pricing is based on a sliding scale of concurrent users: 5, 10, 20, 30, and unlimited.
  • We can provide schools and districts customizable pricing, based on their needs.
  • Schools receive an email with links to activate their licenses and an educator’s kit with wall charts, checklists, and supplementary activities.
  • In Year 1, technical support and maintenance is included in the software cost. In Year 2 and beyond, schools can optionally purchase a support and maintenance program.
  • Volume discounts can also be applied for large, multi-school orders.
  • Annual per student  Subscription Model –Licenses must be renewed annually for continued use of the program.
  • Technical support and maintenance is included in the annual subscription price.
  • Volume discounts may apply to large, multi-school orders.
  • Schools receive an installation kit with the software DVD and installation guide to install on the school or district server and an educator’s kit, which contains wall charts, checklists, and supplementary activities.
Professional Development is provided with the purchase of the Academy of READING program and includes one day of teacher training, one day of live start up with the students, as well as two days of in class coaching/best practice and data review.  Teacher instructor resource guides are also provided for up to 10 teachers.

It is recommended that the Academy of MATH is used 20-30 minutes per session, three to five days a week.

The program includes a highly specified teacher’s manual.

Academy of MATH is a browser-based instructional application that can run on a WAN or LAN. It can also be hosted by EPS/School Specialty Literacy and Intervention.

Training is required for the instructor. The instruction training includes one full day (6 hours) of hands-on training to learn functionality of the program and three days of on-site mentoring and coaching with students (teachers not pulled out).

The minimum qualifications of instructors are that they must be paraprofessionals. The program does not assume that the instructor has expertise in a given area. 

The current version of the Instructor’s Resource Guide has been field tested for the past 6 years.

Additional days of professional development are available including advanced mathematics workshops and on-site consultation days. Technical support is included in the software price in Year 1 and can optionally be purchased for subsequent years.

 

Participants: Convincing Evidence

Sample size: 51 students (26 program, 25 control)

Risk Status: Students in grade 2 who scored in the bottom 30th percentile on the STAR Math Assessment were identified as at risk for academic failure. Students in grades 3 and 4 who were categorized as ‘Basic’ or ‘Below Basic’ in overall mathematics proficiency on the DEAMPZ were identified as at risk for academic failure.

Demographics:

Grade level

PROGRAM Number

PROGRAM Percentage

CONTROL Number

CONTROL Percentage

p of chi square

  Kindergarten

 

 

 

 

 

  Grade 1

 

 

 

 

 

  Grade 2

 6

46% 

54% 

 

  Grade 3

10

48%

11

52%

 

  Grade 4

10

59% 

41% 

 

  Grade 5

 

 

 

 

 

  Grade 6

 

 

 

 

 

  Grade 7

 

 

 

 

 

  Grade 8

 

 

 

 

 

  Grade 9

 

 

 

 

 

  Grade 10

 

 

 

 

 

  Grade 11

 

 

 

 

 

  Grade 12

 

 

 

 

 

Mean Age

       

0.73

Race-ethnicity

  African-American

4

67%

2

33%

 

  American Indian

 

 

 

 

 

  Asian/Pacific Islander

 

 

 

 

 

  Hispanic

4

44%

5

56%

 

  White

18

50%

18

50%

 

  Other

 

 

 

 

0.68

Socioeconomic status

  Subsidized lunch

23

52%

21

48%

 

  No subsidized lunch

3

43%

4

57%

0.64

Disability status

  Speech-language impairments

 

 

 

 

 

  Learning disabilities

 

 

 

 

 

  Behavior disorders

 

 

 

 

 

  Intellectual disabilities

 

 

 

 

 

  Other

6

55%

5

46%

 

  Not identified with a disability

20

50%

20

50%

0.79

ELL status

  English language learner

4

44%

5

56%

 

  Not English language learner

22

52%

20

48%

0.67

Gender

Female

16

62%

11

42%

 

Male

10

42%

14

58%

0.16

 

Note. Student's specific disabilities were not identified. Percentages may not add up to 100 due to rounding.

Training of Instructors: A certified Academy of MATH® and RTI implementation specialist was chosen to provide the site with training and support consistent with typical Academy of MATH® services to schools. Professional development specialists equipped teachers with product knowledge and tools to implement the interventions effectively. Students received hands-on guidance, and teachers learned to monitor and manage student training. At the end of this phase, teachers:

  • Understood the theory and benefits of the programs
  • Assessed students’ mathematics proficiency
  • Monitored and managed student learning effectively

Trainers from Professional Development started the program with students to model best practices and ensure that teachers were engaged with the implementation. At the end of this phase, teachers:

  • Articulated the benefits of the programs to students
  • Created excitement for student learning in the programs
  • Started student training and interpreted Placement Test results

Ongoing classroom coaching and implementation support helped teachers analyze data and apply appropriate intervention strategies for each student. This ensured ongoing program fidelity with on-site classroom coaching and supplemental materials. During these classroom visits, teachers were coached on:

  • Monitoring student progress
  • Analyzing and assessing data
  • Intervention strategies
  • Classroom strategies

Design: Convincing Evidence

Did the study use random assignment?: Yes.

If not, was it a tenable quasi-experiment?: Not applicable.

If the study used random assignment, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures used as covariates or on pretest measures also used as outcomes?: Yes.

If not, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures central to the study (i.e., pretest measures also used as outcomes), and outcomes were analyzed to adjust for pretreatment differences?: Not applicable.

Were the program and control groups demographically comparable at pretreatment?: Yes.

Was there attrition bias1 ?: No.

Did the unit of analysis match the unit for random assignment (for randomized studies) or the assignment strategy (for quasi-experiments)?: Yes.

1 NCII follows guidance from the What Works Clearinghouse (WWC) in determining attrition bias. The WWC model for determining bias based on a combination of differential and overall attrition rates can be found on pages 13-14 of this document: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

 

Fidelity of Implementation: Unconvincing Evidence

Describe when and how fidelity of treatment information was obtained: The Academy of MATH automatically collects and stores information concerning the amount of time students spend in the program, time spent training, percentage of the program completed, and skills mastered. Study facilitators also collected qualitative reports from teachers on-site to ensure treatment fidelity.

Provide documentation (i.e., in terms of numbers) of fidelity of treatment implementation:

Teachers received implementation guidelines before the beginning of the study that asked them to use the program for a minimum of 30 minutes three to five times a week.

In Academy of MATH instructional benchmarks are identified by the Placement Test that is built into the program and presented to students at their first login. The Placement test evaluates students’ various math skills. Based on the Placement test score, lessons needed for each skill to close the gap and allow a student to reach their grade level are identified. In order to complete a lesson, a student needs to have a score of at least 85-96% correct for that lesson, depending on the skill that is being taught. The way that lessons are mapped into sessions is that every time a student logs into the program, he will continue with the lesson that he was last working on or will be presented with the following lesson if the previous lesson was completed with 85% (or higher) accuracy.

Number of lessons for each student is identified by his performance on the Placement test, in other words, the number of lessons for each skill area depends on students’ proficiency. Percentage of program completed is a variable automatically stored in an online database and represents percentage of assigned lessons completed by each student. In one semester (or one school year) a less proficient student, with the same number of lessons completed, the same time in program, and time on task will have much smaller percentage of program completed than a more proficient student, since the amount of work he needs to do in order to close the gap and be at his grade level is much larger than the same of a more proficient student. Time in program represents time on task plus time spent on computer delivered instruction plus time spent on tutorials plus time spent on motivational components, while time on task represents the time a student spent working on actual lessons. Therefore, the best measure of fidelity for Academy of Math is time on task which is on average 50% of time in program.  In addition, if there is not activity, a student is automatically logged out of the program after 3 minutes.

The 26 treatment students averaged 2.7 sessions / week (SD=0.69). Nine of them completed at least 3 sessions per week (recommended in the Academy of MATH’s best implementation model and required by the study design). EPS/SSLI set a requirement of 3 sessions per week which makes 54 total sessions over the course of the intervention (18 weeks). Treatment students averaged 49 sessions (SD=13.08) which identifies levels of fidelity at 90%. Nine of the treatment students met this objective. The other seventeen averaged between 31 and 47 sessions.

EPS/SSLI also set a requirement of 30 minutes time in program per session which translates into 15 minutes time on task per session over the course of the intervention (18 weeks, 54 sessions). Treatment students averaged 15.85 minutes per session, which identified levels of fidelity at over 100% (see the following table).

Group

N

Total Time on Task (hr)

Time on Task / Week (min)

Skills Mastered

Treatment

26

14.27

15.85

47

Control

25

0

0

0

A student cannot move from one level or one skill to another until all the required exercises are successfully completed and the necessary skills are acquired. The program automatically stores this information for each student in a database which is referred to as number of skills mastered. For this particular study the number of average skills acquired for the experimental group is provided in the table above.

Measures Targeted: Partially Convincing Evidence

Measures Broader: Partially Convincing Evidence

Targeted Measure Score type & range of measure Reliability statistics Relevance to program instructional content

Academy of MATH® Placement Test

Grade Level Equivalent (GLE), 0-12

Test-Retest Coefficient = 0.55

Directly tests content trained in the Academy of MATH® with different questions than those used during training.

Broader Measure Score type & range of measure Reliability statistics Relevance to program instructional content

Diagnostic Online Math Assessment (DOMA)  Numbers and Operations

GLE, 0-12

Test-Retest Coefficient = 0.85

Incorporated novel problems in areas students trained in Academy of MATH® but in novel format and structure.

Diagnostic Online Math Assessment  (DOMA) Measurement

GLE, 0-12

Test-Retest Coefficient = 0.78

Incorporated novel problems in areas students trained in Academy of MATH® but in novel format and structure.

Diagnostic Online Math Assessment (DOMA) Fractions

GLE, 0-12

Test-Retest Coefficient = 0.24

Incorporated novel problems in areas students trained in Academy of MATH® but in novel format and structure.

 

Number of Outcome Measures: 4 Math

Mean ES - Targeted: 0.58*

Mean ES - Broader: 0.29

Effect Size:

Targeted Measures

Construct Measure Effect Size
Math Academy of MATH Placement Test 0.58*

Broader Measures

Construct Measure Effect Size
Math DOMA Numbers and Operations 0.37
Math DOMA Measurement 0.38
Math DOMA Fractions 0.11

 

Key
*        p ≤ 0.05
**      p ≤ 0.01
***    p ≤ 0.001
–      Developer was unable to provide necessary data for NCII to calculate effect sizes
u      Effect size is based on unadjusted means
†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Visual Analysis (Single Subject Design): N/A

Disaggregated Data for Demographic Subgroups: No

Disaggregated Data for <20th Percentile: No

Administration Group Size: Individual , Small Group, (n=3-5)

Duration of Intervention: 30 minutes, 3 times a week, 20 weeks

Minimum Interventionist Requirements: Paraprofessional, 6 hours of hands-on, training; 3 days of on -site, training (teachers not pulled out)

Reviewed by WWC or E-ESSA: No

What Works Clearinghouse Review

This program was not reviewed by What Works Clearinghouse.

 

Evidence for ESSA

This program was not reviewed by Evidence for ESSA.

Other Research: Potentially Eligible for NCII Review: 0 studies