Burst:Reading

Study: Pappas, York, Wang, & Richards (2015)

Pappas, S., York, A. E., Wang, Y., & Richards, K. (2015). Examining the efficacy of the Burst:Reading literacy intervention. Unpublished manuscript.
Descriptive Information Usage Acquisition and Cost Program Specifications and Requirements Training

Burst:Reading delivers highly differentiated reading instruction based on formative assessment data. Using mobile technology for assessment administration, sophisticated data-analysis algorithms to generate lesson plans for small groups, and engaging instruction. Burst:Reading puts every student on his or her most efficient path to reading. Teachers, coaches, specialists, and qualified volunteers deliver 10-day “Bursts” of instruction to small groups of students. These Bursts are generated by the system based on formative assessment results for each student.

Burst:Reading is intended for use in grades K-6. The program is intended for use with any student at risk of academic failure. The academic area of focus is reading (including phonological awareness, phonics/word study, comprehension, fluency, and vocabulary).

Where to obtain: 
Amplify Education, Inc.
55 Washington St., Suite 900 Brooklyn, NY 11201
Phone: 1-800-823-1969, Option 1
Website: www.amplify.com

Cost: $60.00 per student annual license fee.

The annual student license fee provides access to the digital intervention program, including customized curriculum modules and reporting. Teachers are able to use 10-day lesson sequences that are customized for their small groups of intervention students based on the formative assessment results of each student. Additional costs associated with the program include per student licenses to formative assessment (generally $14.90 per student), teacher kits ($215 for K-3; $195 for 4-6) and professional development and implementation support (varies based on nature of the implementation).

It is recommended that Burst:Reading is used in small groups of three to five students.

Burst:Reading takes 30 minutes per session with a recommended 5 sessions per week.

The program includes a highly specified teacher’s manual.

The program requires a computer with internet access in order to generate and view the 10-day sequences of instruction as well as reporting. Educators also need a mobile device for administration of the formative assessment measures.

Training is required for the instructor. Training beyond 8 hours is required if the user is not familiar with the formative assessment measures, which include DIBELS Next. The length of this training varies based on the implementation. During this training, educators learn to implement Burst:Reading with fidelity, including administering formative assessments, accessing sequences of lessons through the web-based interface, administering instruction, and monitoring success based on results reporting.

The minimum qualifications of instructors are that they must be paraprofessionals. The program does not assume that the instructor has expertise in a given area.

Training manuals and materials are available and have been designed to ensure that educators are prepared to faithfully implement the program. Feedback is regularly solicited on our training sessions during field use and adjustment to sessions and materials are made to continuously improve outcomes.

Additional follow-up sessions and coaching are available. Online technical support and toll-free phone technical support are available at no additional cost.

 

Participants: Convincing Evidence

Sample size: 4,022 (2,003 program, 2,019 control)

Risk Status: The Burst intervention algorithm determines individual student eligibility based on DIBELS Benchmark Status or risk category (i.e., Red, Yellow, and Green). Students who were identified as At Risk (Red) or Some Risk (Yellow) at the pre-test are considered eligible for Burst intervention. The table below provides the DIBELS Composite Scores corresponding to the 25th percentile and 30th percentile for each grade as determined in a national norming study conducted by Dewey, Kaminski, & Good (2014), as well as the percentage of students in the Treatment and Control groups whose scores fell below each of the percentile thresholds.

It is important to note that DIBELS Benchmark Status for each grade was not determined according to norm-referenced methodology; that is, risk was determined through prediction of later success on an outcome measure and not determined according to percentile ranks. Students who are categorized as Red (Well Below Benchmark) likely need intensive support. Their odds of achieving subsequent early literacy goals are approximately 10%-20% and those students are unlikely to achieve subsequent reading benchmarks unless provided with substantial, intensive instructional support. Students who are categorized as Yellow (Below Benchmark) means they likely need strategic support and the odds of achieving subsequent early literacy goals are roughly 40-60% for students with skills in this range. Students with scores in this range typically need strategic, targeted instructional support to ensure that they make adequate progress and achieve subsequent reading benchmarks (Good et al., 2013). Therefore, while it is not expected that all students eligible for Burst would fall below the 25th or 30th percentiles, all students eligible for Burst intervention were identified based on risk and need for additional instructional support.

 

Percentage of Students Below National Norm Percentile

Grade (DIBELS CS at Percentile)

Treatment

Control

25th Percentile

K (10)

49.70%

46.64%

1 (94)

62.96%

55.68%

2 (125)

84.88%

83.30%

3 (191)

78.48%

78.93%

30th Percentile

K (14)

62.46%

60.70%

1 (100)

71.01%

67.03%

2 (138)

97.29%

97.59%

3 (209)

91.14%

91.61%

 

 

Demographics:

 

Program

Control

p of chi square

Number

Percentage

Number

Percentage

Grade level

  Kindergarten

 666

29.25%

654

27.78%

0.04

  Grade 1

621

27.27%

643

27.32%

0.00

  Grade 2

516

22.66%

497

21.11%

0.05

  Grade 3

474

20.82%

560 

23.79%

0.10

  Grade 4

 

 

 

 

 

  Grade 5

 

 

 

 

 

  Grade 6

 

 

 

 

 

  Grade 7

 

 

 

 

 

  Grade 8

 

 

 

 

 

  Grade 9

 

 

 

 

 

  Grade 10

 

 

 

 

 

  Grade 11

 

 

 

 

 

  Grade 12

 

 

 

 

 

Race-ethnicity

  African-American

400

17.57%

424

18.01%

0.00

  American Indian

22

0.97%

31

1.32%

0.00

  Asian/Pacific Islander

6

0.26%

5

0.21%

0.00

  Hispanic

211

9.27%

356

15.12%

0.35

  White

1280

56.21%

1006

42.74%

0.32

  Other

358

15.71%

532

22.59%

0.27

Socioeconomic status

  Subsidized lunch

1332

58.50%

1187

50.42%

0.20

  No subsidized lunch

515

22.62%

597

25.36%

0.07

  Not specified 

430

18.88%

570

24.21%

0.18

Disability status

  Speech-language impairments

 

 

 

 

 

  Learning disabilities

 

 

 

 

 

  Behavior disorders

 

 

 

 

 

  Intellectual disabilities

 

 

 

 

 

  Other

254

11.16%

213

9.05%

0.14

  Not identified with a disability

1216

53.40%

1373

58.33%

0.12

   Not specified 807 35.44% 768 32.63% 0.05

ELL status

  English language learner

711

31.23%

460

19.54%

0.36

  Not English language learner

639

28.06%

658

27.95%

0.00

  Not specified 927 40.71% 1236 52.51% 0.29

Gender

Female

873

38.34%

898

38.15%

0.00

Male

1133

49.76%

1032

43.84%

0.14

  Not specified 271 11.90% 424 18.01% 0.29

Training of Instructors: Prior to implementing Burst, school personnel participated in a standardized training series that included a one-day on-site session hosted by Professional Development staff to prepare teachers or interventionists. This training followed a common “see one, do one” model in the class with students, so teachers could quickly learn, through context, how the Burst instruction should be delivered. Teachers implemented Burst instruction from August through May, or for the duration of their academic school year. Treatment schools also received one to two on-site support site visits by a member of the Educational Support Team. The support site visit served as an opportunity to observe Burst implementation and provide teachers with immediate constructive feedback.

Design: Unconvincing Evidence

Did the study use random assignment?: No

If not, was it a tenable quasi-experiment?: Yes

If the study used random assignment, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures used as covariates or on pretest measures also used as outcomes?: N/A

If not, at pretreatment, were the program and control groups not statistically significantly different and had a mean standardized difference that fell within 0.25 SD on measures central to the study (i.e., pretest measures also used as outcomes), and outcomes were analyzed to adjust for pretreatment differences?: No

Were the program and control groups demographically comparable at pretreatment?: No

Was there attrition bias1?: No

Did the unit of analysis match the unit for random assignment (for randomized studies) or the assignment strategy (for quasi-experiments)?: No

1 NCII follows guidance from the What Works Clearinghouse (WWC) in determining attrition bias. The WWC model for determining bias based on a combination of differential and overall attrition rates can be found on pages 13-14 of this document: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

 

Fidelity of Implementation: Partially Convincing Evidence

Describe when and how fidelity of treatment information was obtained: Members of the Amplify Educational Support Team (EST) conducted Burst Fidelity site visits to treatment schools during the 2014-2015 school year. The goal during each site visit was to observe one (or more) Burst sessions for each grade level. Although not always possible due to scheduling conflicts, efforts were made to observe different Teachers/Interventionists implementing Burst during each site visit. Fidelity data were collected via the Burst fidelity checklist, which was developed and piloted specifically for this study. The fidelity checklist includes 21 items that capture key components of Burst implementation rated on a three point Likert scale. EST members underwent a day-long training on using the checklist and demonstrated at least 85% inter-rater agreement prior to collecting data in the field. There were two methods for collecting data: either the Observer could complete the checklist in real time during or immediately following an observation or the Observer could use a structured notes guide during the observation and translate notes into ratings on the checklist immediately following the Burst observation.

Provide documentation (i.e., in terms of numbers) of fidelity of treatment implementation: The below table summarizes the ratings on six dimensions of program delivery. The observation covered 16 schools by 3 observers for K-3, 35 responses in total. The average number of students in Burst groups was 4.33 when observed. The ratings used a three-point scale (0-“None or almost none/poor”/1-“Some or fair”/2-“All or almost all/good”). The average ratings on all six dimensions generally ranged from about 1.43 to 1.91 with a mean of 1.76 (71.50% to 95.50% with a mean of 88.12% —well above the 75%).

Fidelity Questions

N

Mean*

SD

Percent Meet Fidelity

I. Teacher follows Burst as prescribed

 

 

 

 

Has all materials ready/prepared

35

1.71

0.46

85.50%

Uses the model/practice and instruction models

33

1.88

0.33

94.00%

Completes all parts of the instructions within the activity in order (e.g., Preview – Practice – Model – Wrap-Up)

35

1.89

0.32

94.50%

II. Teacher allows each student to demonstrate beginning skills’ acquisition before processing

 

 

 

 

Provides each individual student with an individual turn

35

1.89

0.32

94.50%

Ensures each student provides at least one accurate response within the activity

35

1.91

0.28

95.50%

III. Teacher employs management skills that support Burst

       

Transitions smoothly and quickly between parts of activities (e.g., transitions smoothly from Preview to Wrap-Up).

35

1.37

0.60

68.50%

Promotes student engagement and precludes disruptions

35

1.83

0.57

91.50%

Paces to fit student needs

34

1.59

0.61

79.50%

Verbally and/or nonverbally praises students for appropriate behavior (e.g., nonverbal behavior: teacher smiles at student, nods his/her head, etc.)

35

1.86

0.43

93.00%

IV. Teacher addresses individual student needs

       

Differentiates as needed using Support/Challenge activities

7

1.43

0.98

71.50%

Reinforces correct student responses (e.g., teacher says, "Yes, the sound is mmmmm")

34

1.85

0.50

92.50%

Provides corrective feedback for incorrect student responses

35

1.86

0.36

93.00%

Re-engages off-task students

22

1.73

0.63

86.50%

Encourages students who are unsure

33

1.88

0.48

94.00%

V. Teacher promotes active student participation

       

Engages students with expression

35

1.71

0.57

85.50%

Encourages students to be involved in activities

34

1.91

0.38

95.50%

VI. Across all activities within one lesson

       

Completes all activities in order (i.e., A, B, C, D, E)

35

1.66

0.48

83.00%

Summary

 

1.76

 

88.12%

*  (3-point scale of “None or almost none/poor”/“Some or fair”/“All or almost all/good”)

Measures Targeted: Convincing Evidence

Measures Broader: Convincing Evidence

Targeted  Measure

Reliability Statistics

Relevance to Program Instructional Content

Exposure to Related Content Among Control Group

Phoneme Segmentation Fluency (PSF)

Alternate form: 0.44-0.78; inter-rater: 0.95-0.98

PSF measures fluency in identifying the individual sounds in words. Students are asked to segment orally-presented words into individual phonemes and their score is the number of words they segment within one minute.

Burst’s Phonemic Awareness strand includes instruction in the following skills:

Onset-rhyme segmentation: students are taught to segment words into their onsets and rhymes.

Onset-rhyme blending: students are taught to combine onsets and rimes into words.

Phoneme segmentation: students are taught to segment words into their individual phonemes.

Phoneme blending: students are taught to combine individual phonemes into words.

Phoneme substitution: students are taught to manipulate the phonemes in words to make new words (e.g., cat, bat). 

Because students in Burst’s Phonemic Awareness strand are explicitly taught to segment words into onset-rhymes and phonemes, improvements in PSF, an indicator of critical phonemic awareness skills, are expected as a result of Burst participation.

The following is relevant to all five targeted outcome measures:

The control condition was considered a “business-as-usual” condition, meaning that schools implemented business-as-usual practices, which likely include district- or school-directed interventions. Interventions implemented within non-Burst control group schools are district/school-selected and may differ between districts as well as between schools within a district. We do not have information about the percentage of students in each school who participated in a different intervention, nor do we have specific information about what skills were the focus of such intervention programs. However, many early literacy intervention programs target phonological awareness, phonics/word study, and fluency skills; thus, it is likely that many students in the control group received similar instruction as that provided in Burst: Reading.

In addition, Control schools are granted access to Now What? Tools, a supplement to DIBELS: Next, which helps support struggling students. Now What? Tools include Item-Level Advisor, which highlights student error patterns in DIBELS: Next and makes instructional recommendations, Small Group Advisor, which suggests student groupings for intervention based on DIBELS: Next performance, and Home Connect, which generates letters that teachers can send to parents regarding their child’s DIBELS: Next performance and how they can help their children with additional reading practice at home.

 

Nonsense Word Fluency- Correct Letter Sounds (NWF-CLS)

Alternate form: 0.71-0.94; inter-rater: 0.99-1.00; test-retest: 0.76-0.90

In NWF, students are asked to read pseudo consonant-vowel-consonant (CVC) and vowel-consonant (VC) words that follow the rules for regularly-spelled words. The NWF measure results in two scores: CLS and WWR. NWF - CLS measures fluency in producing letter sounds in CVC and VC words; a student’s score is the number of letter sounds they correctly produce (whether blended into words or not) in one minute.

Burst’s Letter Sounds strand introduces students to the sounds that each letter in the alphabet makes. Instructional activities build students’ fluency and accuracy in identifying letter sounds within words.

Because students in Burst’s Letter Sounds strand are explicitly taught the most common sounds that letters make, improvements in NWF-CLS, an indicator of critical alphabetic principle skills, are expected as a result of Burst participation. 

 

Nonsense Word Fluency- Whole Words Read (NWF-WWR)

Alternate form: 0.90-0.97; inter-rater: 0.99-1.00; test-rest: 0.70-0.88

In NWF, students are asked to read made up consonant-vowel-consonant (CVC) and vowel-consonant (VC) words that follow the rules for regularly-spelled words. The NWF measure results in two scores: CLS and WWR. NWF - WWR measures fluency in blending letter sounds into CVC and VC words; a student’s score is the number of whole words they correctly read in one minute.

Burst’s Letter Sounds strand introduces students to the sounds that each letter in the alphabet makes. Instructional activities build students’ fluency and accuracy in identifying letter sounds within words.

Burst’s Sounding Out strand teaches students to decode regular words ranging from 2 to 5 letters long. The instruction builds up from simpler and shorter words with one-to-one letter-sound correspondences to longer words that include consonant blends and digraphs.

Because students in Burst’s Letter Sounds and Sounding Out strands are explicitly taught to sound out and blend regularly spelled words, improvements in NWF-WWR, an indicator of critical alphabetic principle skills, are expected as a result of Burst participation.

 

DIBELS Oral Reading Fluency- Words Correct (DORF-WC)

Alternate form: 0.88-0.98; inter-rater: 0.91-0.95; test-rest: 0.99-0.99

DORF measures accurate and fluent oral reading of connected text. DORF-WC is the number of words a student reads correctly in one minute. DORF also includes an observational checklist that characterizes student prosody, decoding strategies used, error patterns, and self-monitoring.

Burst’s Regular Words and Irregular Words strands teach students to identify common regular and irregular words with automaticity. Instructional activities build students’ fluency in recognizing common words in isolation and within text.

Burst’s Letter Combinations strand teaches students to identify and read words with common vowel combinations, consonant blends, and digraphs.

In Burst’s Fluency strand, students engage in repeated readings of connected texts to build accuracy and fluency.  Lessons also focus on reading sentence types (declarative, interrogative, exclamatory), prosody (pacing, punctuation, volume, rhythm, rhyme, expression), clear and accurate pronunciation, and tone (formal, informal, conversational).

Because students in Burst’s Irregular and Regular Words strands and Letter Combinations strands are taught to identify and read words with common sounds and irregular patterns with fluency and automaticity, and because students in Burst’s Fluency strand engage in repeated text reading for the purpose of building accuracy and fluency, improvements in DORF-WC are expected as a result of Burst participation.

 

DIBELS Maze (DAZE)

Alternate form: 0.81-0.93; inter-rater: 0.99-1.00

DAZE is a measure of reading comprehension that employs cloze procedures in which approximately every seventh word in a reading passage is deleted and replaced with three options for students to select from. The total score is the number of correct words a student selects minus half of the number of incorrect words the student selects in three minutes.

In Burst’s comprehension strand, students read literary and informational texts and focus on the following comprehension skills: setting a purpose for reading; making inferences; retelling and summarizing; identifying main ideas, essential messages and themes; activating prior knowledge; making predictions; identifying plot, story elements, sequencing, structure, and cause-effect relationships; using text features; and asking and answering questions. Because students in Burst’s Comprehension strand are explicitly taught skills and strategies for comprehending text, improvements in DAZE are expected as a result of Burst participation. 

 

 

Broader  Measure

Reliability Statistics

Relevance to Program Instructional Content

Exposure to Related Content Among Control Group

DIBELS Composite

Alternate form: 0.66-0.97; inter-rater: 0.81-0.94; test-rest: 0.97-0.99

DIBELS Composite Score is a combination of multiple DIBELS scores, which provides the best overall estimate of the student’s reading proficiency. The measures included in DIBELS Composite Scores are developmentally appropriate based on a student’s grade and time of year.

Because Burst provides instruction in the reading sub skills that are tested by the measures included in the DIBELS composite score, student composite scores are relevant measures of the overall success of the Burst intervention. 

The following is relevant to all broader outcome measures (i.e., DIBELS Composite Score, Letter Naming Fluency, and STAR Early Literacy Scaled Score):

The control condition was considered a “business-as-usual” condition, meaning that schools implemented business-as-usual practices, which likely include district- or school-directed interventions. Interventions implemented within non-Burst control group schools are district/school-selected and may differ between districts as well as between schools within a district. We do not have information about the percentage of students in each school who participated in a different intervention, nor do we have specific information about what skills were the focus of such intervention programs. However, many early literacy intervention programs target phonological awareness, phonics/word study, and fluency skills; thus, it is likely that many students in the control group received similar instruction as that provided in Burst: Reading.

In addition, Control schools are granted access to Now What? Tools, a supplement to DIBELS: Next which helps support struggling students. Now What? Tools include Item-Level Advisor, which highlights student error patterns in DIBELS: Next and makes instructional recommendations, Small Group Advisor, which suggests student groupings for intervention based on DIBELS: Next performance, and Home Connect, which generates letters that teachers can send to parents regarding their child’s DIBELS: Next performance and how they can help their children with additional reading practice at home.

STAR Early Literacy Scaled Score

Generic reliability: 0.78-0.86; split-half: 0.75-0.85; alternate form: 0.63-0.78

STAR Early Literacy is a computer-adaptive assessment of foundational reading skills (print concepts, phonological awareness, phonics and word recognition, and fluency), vocabulary, and numbers and operations (counting and cardinality, operations and algebraic thinking, and measurement and data). Assessment items from these domains are administered to students based on performance and developmental appropriateness, meaning that not all students are administered from items in all domains; the scaled score summarizes student performance across administered domains.

Because Burst provides instruction in the reading domains that are included in STAR Early Literacy, student scaled scores are relevant measures of the overall success of the Burst intervention.   

 

DIBELS Letter Naming Fluency (LNF)

Alternate form: 0.86-0.95; inter-rater: 0.99-1.00

LNF measures student fluency in naming upper- and lowercase letters. A student’s score is the number of printed letters named in one minute.

Burst does not include an explicit focus on letter naming. Burst’s Letter Sounds strand introduces students to the sounds that each letter in the alphabet makes but does not teach their names.

LNF is considered a broad indicator of risk for future reading failure: letter name knowledge is not a skill that leads directly to successful reading, but it is highly predictive of future reading success. Therefore, students in Burst are expected to show improvements on LNF, indicating general improvement in reading skills.

 

 

Number of Outcome Measures: 2 Prereading, 16 Reading

Mean ES - Targeted: 0.09*

Mean ES - Broader: 0.08*

Effect Size:

Targeted Measures

Construct

Measure

Effect Size

Prereading

PSF GK

0.07

Reading

NWF-CLS GK

0.14*

Reading

NWF-CLS G1

0.14*

Reading

NWF-WWR GK

0.29***

Reading

NWF-WWR G1

0.10

Reading

DORF G1

0.18**

Reading

DORF G2

0.18**

Reading

DORF G3

-0.05

Reading

DAZE G3

0.02

 

Broader Measures

Construct

Measure

Effect Size

Reading

DIBELS Composite Score GK

0.14*

Reading

DIBELS Composite Score G1

0.15*

Reading

DIBELS Composite Score G2

0.13

Reading

DIBELS Composite Score G3

-0.02

Reading

STAR Early Literacy Scaled Score GK

0.13*

Reading

STAR Early Literacy Scaled Score G1

0.13*

Reading

STAR Early Literacy Scaled Score G2

0.12

Reading

STAR Early Literacy Scaled Score G3

-0.02

Prereading

LNF GK

0.16**

 

Key
*       p ≤ 0.05
**     p ≤ 0.01
***   p ≤ 0.001
–      Developer was unable to provide necessary data for NCRTI to calculate effect sizes
u      Effect size is based on unadjusted means
†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Visual Analysis (Single Subject Design): N/A

Disaggregated Data for Demographic Subgroups: Yes

Targeted Measures: Low-achieving students fall below the 20th percentile on pretest DIBELS Composite Score.

Construct

Measure

Effect Size

Prereading

PSF GK

0.05

Reading

NWF-CLS GK

0.21*

Reading

NWF-CLS G1

0.08

Reading

NWF-WWR GK

0.49***

Reading

NWF-WWR G1

0.00

Reading

DORF G1

0.22*

Reading

DORF G2

0.17*

Reading

DORF G3

-0.09

Reading

DAZE G3

0.02

 

Broader Measures: Low-achieving students fall below the 20th percentile on pretest DIBELS Composite Score.

Construct

Measure

Effect Size

Reading

DIBELS Composite Score GK

0.16

Reading

DIBELS Composite Score G1

0.14

Reading

DIBELS Composite Score G2

0.14

Reading

DIBELS Composite Score G3

-0.06

Reading

STAR Early Literacy Scaled Score GK

0.13

Reading

STAR Early Literacy Scaled Score G1

0.20*

Reading

STAR Early Literacy Scaled Score G2

0.09

Reading

STAR Early Literacy Scaled Score G3

-0.12

Prereading

LNF GK

0.18

 

Key
*       p ≤ 0.05
**     p ≤ 0.01
***   p ≤ 0.001
–      Developer was unable to provide necessary data for NCRTI to calculate effect sizes
u      Effect size is based on unadjusted means
†      Effect size based on unadjusted means not reported due to lack of pretest group equivalency, and effect size based on adjusted means is not available

 

Disaggregated Data for <20th Percentile: No

Administration Group Size: Small Group, (n=4-6)

Duration of Intervention: 30 minutes, 5 times a week, Multiple weeks

Minimum Interventionist Requirements: Paraprofessional, 1-2 full days of training

Reviewed by WWC or E-ESSA: No

What Works Clearinghouse Review

This program was not reviewed by What Works Clearinghouse.

 

Evidence for ESSA

This program was not reviewed by Evidence for ESSA.

 

Additional Research Studies

This program has no additional research studies at this time.

Other Research: Potentially Eligible for NCII Review: 0 studies