Early Numeracy Intervention Level 1

Study: Bryant, et al. (2011)

Bryant, D. P., Bryant, B. R., Roberts, G., Vaughn, S., Hughes, K., Porterfield, J., & Gersten, R. (2011). Effects of an early numeracy intervention on the performance of first-grade students with mathematics difficulties. Exceptional Children, 78(1), 7-23.
Descriptive Information Usage Acquisition and Cost Program Specifications and Requirements Training

The Early Numeracy Intervention (ENI) Level 1 program was developed to provide teachers with evidence-based instructional materials for teaching students who struggle with primary level numeracy concepts and procedures, while applying these skills to mathematics problem solving. The ENI Level 1 program provides instructional materials designed to remediate and strengthen numerical competence for students who need supplemental, intensive, ongoing mathematics instruction. The targeted program concepts and procedures were identified based upon investigations conducted with students in the primary grades and discussions with teachers, curriculum specialists, and content experts.

The lessons are aligned with the Common Core State Standards in Mathematics.

ENI Level 1 is intended for use in first grade or with students performing at about the first grade level. It is designed for use with any student at risk of academic failure. The academic area of focus is math, with emphasis on comprehensive aspects (includes computation/procedures, problem solving, and mathematical concepts), as well as developing early number sense.

Where to obtain:

Psycho-Educational Services
5114 Balcones Woods Dr. Suite 307-163
Austin, TX 78759

Phone: 1-512-699-9381

Website: http://www.psycho-educational.com

Cost:

$189 per kit. One kit will serve one teacher and all students in their classroom.

The kit includes a spiral bound Teacher’s Manual (10 units, 14 lessons per unit) and CD-ROM, which contains the Teacher Masters, Student Masters, and Unit Checks (for progress monitoring) for each unit.

ENI Level 1 is designed for use with small groups of four to five students.

ENI Level 1 is administered for 25-30 minutes per session. 3-4 sessions are recommended per week, over a course of 19-20 weeks.

The program includes a highly specified teacher’s manual. No special technology is required.

The minimum qualification for ENI Level 1 instructors is that they be paraprofessionals. The program does not assume that instructors have expertise in a given area.

An instructional manual is available for practitioners. The instructions can be read in less than an hour. The manual discusses the components of the program and specific features of the intervention.

The training materials have been field-tested with the target population (students who have identified mathematics difficulties).

Instructors who require additional technical or professional support may consult the Psycho-Educational Services website or direct their questions to the President and Chief Executive Officer of the company. 

 

Participants: Convincing Evidence

Sample size: 224 first grade students across 10 elementary schools. (771 students were initially screened. The final sample was comprised of 139 program students and 64 control students.)

Risk Status: Two main considerations drove sample selection: (a) maintaining sufficient power and (b) reliably assessing risk.  Of the initial pool of students (N = 771), the lowest 35% (n = 269) was identified as being “at-risk” based on an initial administration of the Texas Early Mathematics Inventories-Progress Monitoring measures (TEMI-PM; University of Texas System & Texas Education Agency, 2007a; refer to the Measures for further details about this test) in the fall (September). Students were administered four additional TEMI-PM probes (alternate forms of the original measure used for student selection) over a 3-week period to determine whether there were false positives among the initial pool of students. False positives are a particular concern given the generally “chaotic” nature of early achievement and the increased possibility of falsely identifying students as being “at-risk” when they were merely distracted, anxious, or unfamiliar with the testing protocols. Growth modeling (with continuous outcomes and auto-correlated residuals) was used to estimate case-level factor scores for intercept and slope for each of the 238 cases using PLUS 4.1 (preliminary analyses suggested a statistically significant positive trend in scores over time, on average thus, a growth model approach was preferred over a confirmatory factor model). Intercept was conceptualized as the last of the four additional TEMI-PM measures (beyond the TEMI-PM used to initially identify the lowest 35%). Estimated time 4 scores were used to make final sample selection.  The cut score was selected based on the probabilities of diagnostic accuracy (i.e., likelihood ratio [LR]) derived using receiver operator curve (ROC) analysis. Using this procedure, 14 students were found to be false positives and were eliminated from the sample.

A concern with accuracy and the need to maintain an adequate sample size both influenced our sampling strategy. Preliminary power analyses suggested a sample size of 240, with 160 in the treatment condition and 80 in the comparison group. The initial pool of eligible students was only 238, so our strategy was to identify students who clearly were not at risk, based on their estimated score at time 4 and a very conservative risk threshold (LR: negative of 0.70). The final sample (n = 224:151 treatment and 73 control) identified for treatment and control conditions was associated with a minimal detectable effect size of approximately 0.40, assuming 0.80 power and 45 instructional groups with five students in each group.

Demographics:

 

Program

Control

p of chi square

Number

Percentage

Number

Percentage

Grade level

  Kindergarten

 

 

 

 

 

  Grade 1

139

 

 64

 

 0.00

  Grade 2

 

 

 

 

 

  Grade 3

 

 

 

 

 

  Grade 4

 

 

 

 

 

  Grade 5

 

 

 

 

 

  Grade 6

 

 

 

 

 

  Grade 7

 

 

 

 

 

  Grade 8

 

 

 

 

 

  Grade 9

 

 

 

 

 

  Grade 10

 

 

 

 

 

  Grade 11

 

 

 

 

 

  Grade 12

 

 

 

 

 

Race-ethnicity

  African-American

 

26.6

 

21.5

0.006

  American Indian

 

 

 

 

 

  Asian/Pacific Islander

 

3.6

 

6.2

0.091

  Hispanic

 

33.0

 

40.0

0.003

  White

 

36.6

 

32.3

0.002

  Other

 

 

 

 

 

Socioeconomic status

  Subsidized lunch

 

50.4

 

52.3

0.000

  No subsidized lunch

 

49.6

 

47.7

0.000

Disability status

  Speech-language impairments

 

 

 

 

 

  Learning disabilities

 

 

 

 

 

  Behavior disorders

 

 

 

 

 

  Intellectual disabilities

 

 

 

 

 

  Other

 

 

 

 

 

  Not identified with a disability

139

100

64

100

0.000

ELL status

  English language learner

 

 

 

 

 

  Not English language learner

 

 

 

 

 

Gender

Female

 

43.9

 

44.6

0.000

Male

 

56.1

 

55.4

0.000

Training of Instructors: The research team included two full- time intervention coordinators and five graduate research assistants (GRAs) who were doctoral and master’s students in the Department of Special Education; all of the GRAs held teaching credentials or were completing a teaching certification program. This research team was also responsible for conducting the intervention.

At the beginning of the academic year, the principal investigator provided a 3-hour training on the intervention lessons and accompanying instructional materials. This training consisted of an explanation of the content and review and modeling of systematic instruction. Following this training, the research team practiced the lessons with one another. Prior to intervention, the tutors taught a lesson and received feedback from experienced tutors who were using the same lessons with a group of students. Throughout the school year, training sessions were conducted before each intervention unit (seven total sessions). Tutors were visited weekly and team meetings were held to solve issues. 

Design: Convincing Evidence

Did the study use random assignment?: Yes

If not, was it a tenable quasi-experiment?: Not applicable

If the study used random assignment, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures used as covariates or on pretest measures also used as outcomes?: Yes

If not, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures central to the study (i.e., pretest measures also used as outcomes), and outcomes were analyzed to adjust for pretreatment differences? Not applicable

Were the program and control groups demographically comparable at pretreatment?: Yes

Was there attrition bias1No

Did the unit of analysis match the unit for random assignment (for randomized studies) or the assignment strategy (for quasi-experiments)?: Yes

1 NCII follows guidance from the What Works Clearinghouse (WWC) in determining attrition bias. The WWC model for determining bias based on a combination of differential and overall attrition rates can be found on pages 13-14 of this document: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

 

Fidelity of Implementation: Convincing Evidence

Describe when and how fidelity of treatment information was obtained: Each tutor was observed by research staff for three sessions during the 19-week intervention to assess the quality (i.e., fidelity) of specific implementation performance indicators. Quality of Implementation (QoI) indicators included the degree to which tutors (a) followed the scripted lessons (e.g., modeling, guided practice, independent practice); (b) implemented the features of explicit, systematic instruction (e.g., pacing, error correction); (c) managed student behavior (e.g., use of reinforcers and redirection); and (d) managed the lesson (e.g., use of timer, smooth transitions between booster lessons).

Provide documentation (i.e., in terms of numbers) of fidelity of treatment implementation: Performance indicators were rated on a 0-to-3 point scale, in which 0 = Not at All, 1 = Rarely, 2 = Some of the Time, and 3 = Most of the Time. Results were shared with the tutors and areas in need of further training and recommendations for improved performance were discussed. Results on the QoI showed average ratings exceeding 2.5 in all areas, with no single rating of <2.0. The majority of ratings were 3.0. These results across tutors show that there was a high degree of fidelity in the implementation of the booster lessons.

Measures Targeted: Convincing Evidence

Measures Broader: Convincing Evidence

Targeted Measures:

Targeted  Measure Reliability Statistics Relevance to Program Focus Exposure to Related Content Among Control Group

Texas Early Mathematics Inventories-Progress Monitoring (TEMI-PM)

The TEMI-PM consists of four forms (A, B, C, D). There are four subtests: Magnitude Comparisons, Number Sequences, Place Value, and Addition/Subtraction Combinations.

Magnitude Comparisons: Immediate test-retest reliability with alternate forms coefficients for Form A with Forms B, C, and D ranged from 0.70 to 0.78 (median =0.74). Correlating the spring Form A MC score with the Total Mathematics score from the SAT-10 yielded a concurrent validity coefficient of 0.64.

Number Sequences: Immediate test-retest with alternate forms reliability coefficients for Form A with Forms B, C, and D ranged from 0.74 to 0.82 (median =0.76). Spring intercorrelations between Form A NS scores and SAT-10 Total Mathematics score resulted in a concurrent validity coefficient of 0.60.

Place Value: Intercorrelation coefficients between Form A and Forms B, C, and D ranged from 0.67 to 0.77 (median =0.71). Correlating Form A with the Total Mathematics score from the SAT-10 yielded a validity coefficient of 0.58.

Addition/Subtraction: Immediate test-retest with alternate forms reliability coefficients for Form A with Forms B, C, and D ranged from 0.78 to 0.86 (median = 0.80). We correlated the results of Form A ASC with the Total Mathematics score from the SAT-10, which resulted in a validity coefficient of 0.68.

Total Score: The Total Score was derived by summing the raw scores for MC, NS, PV, and ASC. Immediate test-retest with alternate forms reliability coefficients for Form A with Forms B, C, and D ranged from 0.83 to 0.88 (median = 0.86). We also correlated the spring Form A TOT with the Total Mathematics score of the SAT-10; the resulting concurrent validity coefficient was 0.72.

Targeted Outcome Measure: This measure assessed the intervention topics.

Students were taught similar skills and assessed through the district’s benchmark tests.

All students received the TEMI-PM.

Broader Measures:

Broader  Measure Reliability Statistics Relevance to Program Instructional Content Exposure to Related Content Among Control Group

Texas Early Mathematics Inventories-Outcome (TEMI-O)

The TEMI-O was considered to be a distal outcome measure because it assesses all of the state’s standards for first-grade instruction (our intervention focused only on the number and operation standards). The TEMI-O is composed of two subtests: Mathematics Problem Solving (MPS) and Mathematics Computation (MC). The TEMI-O Total Score reliability, as estimated using coefficient alpha, was 0.86 for Form A, 0.90 for Form B, and 0.92 for Form C. Examination of concurrent criterion-related validity was conducted by correlating TEMI-O Total Scores with the Total Scores obtained by students on the SAT-10 and by estimates of student mathematics abilities, as rated by their teachers. The coefficient for the SAT-10 and Form A for the TEMI-O Total Score was 0.61; teacher ratings correlated with TEMI-O Total Scores at 0.61. Both coefficients reflect positively on the validity of the TEMI-O scores.

Broader Outcome Measure: measures all of the state standards including the skills taught in the intervention. The TEMI-O was used as a generalization measure. 

Teachers are responsible for teaching the state’s mathematics standards and the district has benchmarks to assess progress. The TEMI-O is based on the state standards.

All students received the TEMI-O.

SAT-10

The mathematics portion of the SAT-10 includes the Mathematics Problem Solving (MPS) and Mathematics Procedures (MP) subtests in grade 1 with items that assess numeration, numerical sequencing, measurement, statistics, problem solving, and computation. A composite score (Total Mathematics) is also available. In the fall, students in grade 1 are administered the Stanford Early School Achievement Test 2 level, and in the spring, students are administered the Primary-1 level. In the current study, the SAT-10 mathematics subtests yielded internal consistency reliability coefficients that exceeded 0.80. The total score internal consistency reliability coefficient exceeded 0.90.

Broader Outcome Measure: measures all of the state standards including the skills taught in the intervention. The TEMI-O was used as a generalization measure.

The SAT-10 is a broad based measure that assesses topics in state's standard and now the Common Core State Standards for Mathematics.

All students received the SAT-10.

 

Number of Outcome Measures: 11 Math

Mean ES - Targeted: 0.46*

Mean ES - Broader: 0.21*

Effect Size:

Targeted Measures

Construct Measure Effect Size
Math TEMI-PM Magnitude Comparisons 0.21
Math TEMI-PM Number Sequences 0.53***
Math TEMI-PM Place Value 0.44**
Math TEMI-PM Addition & Subtraction Combinations 0.59***
Math TEMI-PM Total Score 0.55***

 Broader Measures

Construct Measure Effect Size
Math SAT-10 Mathematics Procedures 0.27
Math SAT-10: Mathematics Problem Solving 0.14
Math SAT-10 Total Score 0.20
Math TEMI-O Mathematics Computation 0.49**
Math TEMI-O Mathematics Problem Solving 0.00
Math TEMI-O Total Score 0.27

 

Key
*      p ≤ .05
**    p ≤ .01
***       p ≤ .001
–      Developer was unable to provide necessary data for NCII to calculate effect sizes
u      Effect size is based on unadjusted means
†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Visual Analysis (Single Subject Design): N/A

Disaggregated Data for Demographic Subgroups: No

Disaggregated Data for <20th Percentile: No

Administration Group Size: Small Group, (n=4-5)

Duration of Intervention: 25-30 minutes, 3-4 times a week, 19 weeks

Minimum Interventionist Requirements: Paraprofessional, Less than 1 hour of training

Reviewed by WWC or E-ESSA: No

What Works Clearinghouse Review

This program was not reviewed by What Works Clearinghouse.

 

Evidence for ESSA

This program was not reviewed by Evidence for ESSA.

Other Research: Potentially Eligible for NCII Review: 0 studies