Student Achievement and FSA Validity: More Accountabaloney

A test is only valid if the results are used in a valid way. If a test’s results are used in anyway other than their intended use, the validity of that use must, also, be confirmed. This, essentially, sums up the lack of validity of the entire Florida Education Accountability system: tests that may have once been be valid for one use are now being used for a myriad of uses they were never intended for. Our children are sitting for tests whose results are used to determine promotion and graduation eligibility, rank and punish teachers, evaluate schools and districts and, because of school grades, determine local property values. Are these valid uses of the FSA? Has anyone bothered to check?

Following significant concerns regarding the validity of the new Florida State Assessment (FSA), legislators, during the 2015 session, ordered an independent verification of the new state assessment. Last summer, Alpine Testing Solutions, in partnership with EdCount, completed a partial, so called “independent”, verification of the psychometric validity FSA.  (You can read the report here, and our comments on it’s incomplete nature here). In the report, Alpine describes the relationship of validity to the intended use of assessment results.

“The process of evaluating an assessment and its associated validity evidence is directly related to the intended uses of the scores. Validity refers to these specific uses rather than a global determination of validity for an assessment program. As such, it is possible that the validity evidence supports one specific use of scores from an assessment while is insufficient for another.”

For example, a spelling test might be a valid way of measuring a student’s spelling abilities.  That same spelling test would tell you nothing about the child’s math skills, knowledge of U.S. History or ability to drive a car. Likewise, a student’s driver’s test is an insufficient measurement of that student’s spelling skills.

Table 2 (page 27 of the final Alpine report) provides a summary of these intended uses of the FSA (as provided to Alpine by the FLDOE) and notes the uses for which modifications have been made for 2014-15 as the first year of the program.

The stated intended uses of the FSA include: teacher evaluation, school and district grades, school improvement rating and state accountability.  The ONLY listed uses for students are 3rd grade promotion, graduation eligibility (for both grade ELA and Algebra EOC) and student grades (for the math EOCs which are worth 30% of the students course grade). Please note: nowhere in Table 2 is “provide student academic achievement and learning gains data to students, parents, teachers, or schools” mentioned, yet this is reported to be the primary purpose of the student assessment program. In fact, according to the chart, there are NO intended uses for individual students, at all, for grade level FSA testing from grade 4 to grade 9.

I want to repeat that: according to the Alpine Validity Study, there are NO intended uses for individual students, at all, for grade level FSA testing from grade 4 to grade 9!

So, not only did the Alpine Study fail to evaluate 11 of the 17 new FSA assessments and fail to assess whether any of the tests were fair, valid or reliable for vulnerable sub-populations (read more here), but, according to Table 2, it EVEN failed to evaluate  whether the results could be used for the (reported) primary purpose of the state assessment system: evaluating student achievement.

It bears repeating that the 2014 Florida Statewide Assessments (FCAT 2.0) Technical Report (released in 12/2014), on page 137, suggested that further studies were needed to verify the inference “that the state’s accountability program is making a positive impact on student proficiency and school accountability without causing unintended negative consequences.” (Emphasis mine.) It will be difficult, I think, to evaluate that if the validity of the use of individual student scores is never determined.

When school administrators tell you the FSA is about providing information about student achievement and learning gains, show them this.  Providing information about student achievement and learning gains does not appear to be a priority of the current system; ranking and punishing teachers and schools does. Ask the “powers that be” when is Florida’s Accountability system going to take a serious look at the appropriate use and the current misuse of test scores?  When will they verify whether the current system is making a positive impact on student proficiency (dropping ACT and NAEP scores suggest otherwise)?  When will the myriad of unintended consequences be addressed? Until then, this is not a valid accountability system; it is all a bunch of accountabaloney.