More Research Needed: Does this Study Demonstrate the Positive Effects of a Voucher Program or the Negative Effects of Accountabaloney?

In Florida, public schools are graded, almost entirely based on student standardized test scores, and then either rewarded or punished. Test scores are used to make decisions that professional educators used to make: promotion/retention, remediation and, even, graduation. The punitive measures associated with Florida’s A-F School Grading system fall particularly hard on low income, children of color and their schools. Desperate for higher test scores, and higher school grades, schools have narrowed their curriculum, focusing almost entirely on tested subjects; students have lost access to music, arts, and, even, recess. Much of the school year is spent “getting reading” for  the state mandated assessments. It is rare to attend a Back-to-School night that does NOT mention the importance of state testing. Sometimes, schools resemble little more than test prep factories. Students and teachers are stressed to their limits. It is difficult to blame parents who leave these public schools for the hope of something better, perhaps less stressful, at a private school. Expansion of the Florida Tax Credit Scholarship (FTCS) program has allowed increasing numbers of students to qualify for a state funded voucher to offset the cost of private school tuition. Private schools that accept FTCS vouchers are required to administer one of several approved standardized tests but such data is not collected in anyway that would allow comparison to public school test results. There are no state mandates for how those scores should be used and they are certainly not used to assess the quality of the educational environment. Private schools celebrate their lively arts and athletics programs; they brag about their academics and their small class size. It is assumed by the State that if parents chose the private school, then it must be good enough.

Last week, the Urban Institute published a report claiming that students who participated in Florida’s Tax Credit Scholarship program were more likely to enroll in college than their peers who remained in their neighborhood public school. The report, “The Effects of Statewide Private School Choice on College Enrollment and Graduation,” by Matthew M. Chingos and Daniel Kuehn, was immediately celebrated and tweeted out by School Choice advocates, including Jeb Bush, his foundations and Step Up For Students, the organization that administers the Tax Credit Scholarship in Florida.

I started to read the report that day but got stuck on page 4. I am suspicious of non-peer reviewed studies to begin with, but I am super suspicious of those funded by such education reform giants as Jeb’s Foundation for Excellence in Education and the Walton Family Foundation.

I admit, I put the report down without reading it, figuring it was another all-too-familiar paid propaganda piece from the Ed Reform movement. I was inspired to take a second look after hearing Matt Chingos, one of the authors, interviewed  on the Tampa Bay Times’ Gradebook podcast. Mr Chingos stated that recent studies in Louisiana, Indiana and Ohio suggested that private school vouchers had a negative impact on test scores, lowering student achievement. His study was designed to see if private school vouchers might have longer term benefits, not evidenced by student performance on standardized test scores. He said this was the first study to look at whether voucher students were more likely to attend and graduate college. He claimed he chose the Florida Tax Credit Scholarship program because it is the biggest private school voucher program in the country.

The study was summarized in The Miami Herald:

Researchers looked at some 10,000 low-income students in Florida who attended private schools and compared them to some 50,000 children with similar backgrounds and academic performance who never participated in the program. Tax-credit scholarships are a type of voucher program where scholarships are funded by corporate donations by companies that then receive tax credits.

Among students who attended private school in grades 3-7, 45 percent enrolled in a public college in Florida within two years of their expected graduation, compared to 39 percent students in public school. For students who attended private school in grades 8-12, 48 percent went to college, compared to 42 who attended public high schools.

“It means that for the kids who took advantage of this choice on average it worked out OK,” said Matthew Chingos, a co-author of the report. He added, however, that participation in the program did not improve students’ graduation rates. “These results are positive, but you don’t go ‘oh wow’ just yet.”

During the podcast interview, Mr. Chingos suggested more study was needed but college enrollment rates appeared to be highest for students who remained in the voucher program longer and for certain subpopulations of students, specifically immigrant children. Mr. Chingos suggested that specific policy recommendations should not be made based on one study, but felt his study was important because it looked at a statewide voucher program’s long term outcomes for the first time. He admitted that more research was needed, given the reports from other states suggesting voucher students had lower test scores and graduation rates.

Florida’s Tax Credit Scholarship program does not collect student test score data in a way that would allow assessing the program’s impact on participating students’ achievement (test scores). (This is, of course, by design. Voucher advocates in Florida fought against test based accountability standards for voucher participants, making it impossible to compare public and private school student achievement in any apples-to-apples sort of way.) It is impossible to know whether the lowered student achievement seen with Louisiana, Ohio and Indiana voucher participants also occurs in Florida. It would be important, I think, to look at the impact of those states’ programs on college enrollment and graduation, to see if similar benefits are seen.

Several education advocates have written or been interviewed regarding what they saw as flaws in the study’s methodology and conclusions. In The Miami Herald, Samuel Abrams, director of the National Center for the Study of Privatization in Education at Teachers College, Columbia University, “criticized the study’s methodology as flawed, saying that students who had the energy and motivation to get accepted and remain at private schools may already have an edge compared to their peers in public schools.” Sue Legg, in her League of Women Voters Education blog, pointed out that voucher students who enrolled in private schools that were in existence before the Tax Credit Scholarship Program began were more likely to go to college and the likelihood that a voucher student would enroll in college declined as more vouchers students enrolled in a particular private school.  In other words, a voucher recipient enrolled in a newer private school, serving lots of voucher students, was less likely to enroll in college than a voucher recipient attending an established private school, with fewer fellow voucher students. She suggested:

No matter how the numbers are manipulated, private schools are no answer to improving student achievement. The students who succeed attend selective, well established private schools that will only enroll a few scholarship students. No doubt these children were carefully screened for admission.

I, too, have a few concerns regarding this study. For one, I wonder why “enrollment in college” was considered the only successful longterm outcome for a student. I wonder what happened to the 52% of voucher students and 58% of public school students that did not enroll in college. Many public schools provide vocational training for students, earning industry certifications that can make them ready to work in a trade or may inspire them to enroll in a trade school. Such students are ignored by this study. Also, many public schools offer Dual Enrollment classes, allowing students to earn college credits during high school. Some motivated high school students graduate high school take advantage of Dual Enrollment classes and graduate from high school with an Associates degree from their local college. Could some of the non-college enrolled students in this study also have had successful outcomes? This study assumes “no.”

Also, could the noted increased college enrollment for students enrolled at an established private school serving few voucher students be a result of the positive effects of smaller class size? Elite private schools all advertise their small class size. Could smaller class size during elementary school result in increased college enrollment later on? These questions should be asked and evaluated.

More important to me is the assumption of cause and effect in this report. It is assumed that the increased college enrollment is a positive effect of the voucher program. What if the current traditional public school system, with its focus on high stakes standardized testing and test based retention and graduation requirements (none of which are required in the private voucher schools), has a NEGATIVE effect on college enrollment? What if, the longer a student is exposed to Florida’s test and punish accountability system, the less likely he/she is to enroll in college? What if  mandatory 3rd grade retention reduces the likelihood a child with continue on to college? What if years of remedial reading and math classes, designed to improve performance on annual state testing, kills a student’s natural desire for learning and decreases their desire to attend college? What if the state mandated graduation requirements, Algebra 1 EOC and the 10th grade English Language Arts FSA (which private school students are not required to take or pass) are a deterrent to high school graduation and college enrollment?

Negative long term impacts from Florida’s Education Accountability System have already been documented. Earlier this year, Kathleen Jasper, et al, published a report looking at the long term effects of Florida’s test based mandatory third grade retention, a cornerstone of “Florida’s A+ Plan.” Researchers followed a cohort of students who scored a level 1 on the 3rd grade FCAT in 2003-2004, some of who were retained in 3rd grade and others who had the same FCAT scores but were promoted to 4th grade. The non-retained group were 14.7% more likely to graduate with a standard diploma than the retained group. Retention demographics showed approximately 6% of white students were retained that year while 20% of nonwhite students were retained. Of the students retained in 2003-2004, 69.8% were on free or reduced price lunch. Of course, students who take advantage of the FTCS voucher during third grade are not subject to mandatory, test based retention.

In the Urban Institute study, fewer public school students enrolled in college. Attributing the decreased college enrollment, in the  public school group, to the negative effects of a punitive, test based accountability system rather than the positive effect of vouchers, could explain many of the study’s findings. For example, college enrollment was significantly higher in immigrant children attending voucher schools. In public schools, English Language Learners are required to participate in state testing, in English, from day 1. Such testing may be frustrating and demoralizing to an English Language Learner. The Florida Standards Assessment in Math, with its reliance on “real world example” word problems, requires a significant proficiency in English to be successful. Perhaps immigrant children would be better served in a less high stakes, test focused environment?

For years, parents and educators have complained that Florida’s test focused accountability system has “sucked the joy out of the classroom.” When a test stressed, demoralized student uses a voucher to attend a private school, free from harsh accountability standards, are they able to rediscover the joy of learning? Could this explain why the data showed that students who participated in the voucher 3 or more years were significantly more likely to choose to enroll in college than those who returned to the test stressed schools after a year?

We have written, frequently, about the negative, so-called “unintended consequences” of Florida’s Accountability System. Does this study provide evidence that this system may have negative long term consequences on Florida’s public school students? We urge the study’s authors and the Urban Institute to consider whether this study demonstrates the positive effects of a voucher program or the negative effects of #accountabaloney.

We agree, more research is needed.