When an Adult Takes a Test… and Fails

I am continually amazed by how otherwise intelligent people can act so stupidly when they are arguing about something they like or don’t like.

Standardized testing is almost universally loathed by students, parents, teachers, and principals.  Of course, the simple reason is that

No one uses it how it’s supposed to be used!

But this article is about an otherwise intelligent school board member who doesn’t understand how do to things and why we do the things we do… plus he doesn’t even bother to attempt to learn about the things he’s rambling on about before rambling on about them.

Here’s the article: When an adult took standardized tests forced on kids

The first bit talks about the person who took the test.  The sad part is that he took it because he’s actually on the school board of the town.  This saddens me for reasons that will become apparent later.

Apparently this person manages the budget of a large organization, yet can’t see how math is important.  He has a bachelor of science degree and two masters and can’t read effectively.  This says a lot about something…

Let’s look at some specific points.

“It might be argued that I’ve been out of school too long, that if I’d actually been in the 10th grade prior to taking the test, the material would have been fresh. But doesn’t that miss the point? A test that can determine a student’s future life chances should surely relate in some practical way to the requirements of life. I can’t see how that could possibly be true of the test I took.”

It can be argued that, but it isn’t.  No one who actually understands the entire point of these tests would argue that.

Let me say this very carefully.  These tests are NOT to test real world knowledge and abilities.  These tests are NOT to test practical life skills.

These tests are designed very carefully.  It usually takes about 30 months to write the material for these tests.  The material goes through several committees, client review, multiple copy editors, psychometricians, and content specialists.

They are designed to test the student’s knowledge of the curriculum.  That’s all.  There is NOTHING ELSE that these tests should be used for… but we’ll get to that in a second.

If I’d been required to take those two tests when I was a 10th grader, my life would almost certainly have been very different. I’d have been told I wasn’t ‘college material,’ would probably have believed it, and looked for work appropriate for the level of ability that the test said I had.

BTW: That’s a link to 2nd grade tests… not the tenth grade tests.  Journalism fail.

If you manage the budget for a major company and still can’t handle basic algebra, maybe you aren’t college material.  However, through hard work you made it through college anyway.  There are WAY more factors on success rates in college than test scores.  With socioeconomic status being the main one and parent’s influence being the next.

That being said, in terms of success in college, the two biggest predictors from standardized tests are literacy skills and mathematics skills.

But let’s look at something here.  I don’t know of a single person, ever (having been a high school teacher, college counselor, and college assistant registrar) who has ever been told “You aren’t college material.”  That’s a fundamental lie.  And it is a very disingenuous think to say.

Go ahead, go to your local high school and ask the counselor if they have ever told a student that they are not college material.  It’s these people’s job to encourage students and support them.

Colleges are there for education.  They will never say “You aren’t college material”, now they might say, “Your test scores indicate you need some help in math or literacy and we have some classes that you can take to help with that.”

Education is not an either-or thing.  It is a large range of opportunity and training.

“It makes no sense to me that a test with the potential for shaping a student’s entire future has so little apparent relevance to adult, real-world functioning. Who decided the kind of questions and their level of difficulty? Using what criteria? To whom did they have to defend their decisions? As subject-matter specialists, how qualified were they to make general judgments about the needs of this state’s children in a future they can’t possibly predict? Who set the pass-fail “cut score”? How?”

This is about the most depressing thing in the whole article.  Presumably this idiot had to do some measure of research for his two masters degrees.  Yet, he can’t be bothered to pick up the phone and call the state education department who could tell him this information in about five minutes.

First of all, this has nothing to with a student’s “entire future”.  Again, education is a life long thing and some people are good at it and some aren’t.  Some people are good test takers, some are not.  Some people will be massively successful with and without college and education.   Some people will fail miserably even with a college education… like this guy.

Now, the details.  “Who decided the kind of questions and their level of difficulty?”  The state did.  The people who write the test wrote items and gave them an estimated difficulty.  Then the client reviewed them and made changes if needed.  Then a committee put together of teachers and content experts in that state reviewed the items… and made changes.  Then everyone (except the committee) did it again.

As to what kind of question, you might ask your teacher why they teach what they teach.  It’s called a ‘curriculum’ and most states mandate the curriculum for every course and every student in the entire state.  A school board member doesn’t know this?  sigh…

“Using what criteria? To whom did they have to defend their decisions?”  Same answer.  The client and the client committees are the ones to whom the decision is defended.  I’d be willing to bet that the curriculum was developed by dozens of educators and content experts from the state.  In many states, the curriculum must be approved by the entire state school board and either the governor or the state legislature.

“As subject-matter specialists, how qualified were they to make general judgments about the needs of this state’s children in a future they can’t possibly predict? Who set the pass-fail “cut score”? How?”

Let’s see, I’m a subject matter expert for a testing company.  I’ve been living and breathing my field of study for over 20 years.  I have over a hundred college hours in science and another 30+ in education.  I have taught at middle school, high school and adult learner levels.  I have been in this industry for three years and have been put in charge of the science portion of two state projects and a national project.  Let’s just say that I know what I’m talking about.

“Future”? Another disingenuous comment.  For most kids, the next level of education is… next year.  These tests are specifically designed to test how student’s have learned the current coursework.  Again, these are NOT for life skills or anything else… just the current curriculum.

“Who sets the pass fail cut score?” Another call to the state education office and you would learn that setting cut scores is a highly technical process that for most tests takes between 3 days and a full week, with a large committee of educators and non-teacher subject matter experts.  The whole show is run by a couple of people with doctorate degrees in a subject called psychometrics.  That’s the science of statistical analysis of test results.

This not just someone saying ‘the cut score is here’.

“I can’t escape the conclusion that decisions about the [state test] in particular and standardized tests in general are being made by individuals who lack perspective and aren’t really accountable.”

Which is plainly a steaming pile of horse cookies to anyone who keeps up with the news in education.  The decisions are being made with the help of advisory committees full of professional educators, subject matter experts, and statisticians.  Accountable?  Seriously?  Again, you just aren’t paying attention.

There you have it. A concise summary of what’s wrong with present corporately driven education change: Decisions are being made by individuals who lack perspective and aren’t really accountable.

No, there you have it, a guy who has no clue what’s going trying to explain why everything is wrong.  This is so typical it’s not even funny.

Those decisions are shaped not by knowledge or understanding of educating, but by ideology, politics, hubris, greed, ignorance, the conventional wisdom, and various combinations thereof. And then they’re sold to the public by the rich and powerful.

All that without so much as a pilot program to see if their simplistic, worn-out ideas work, and without a single procedure in place that imposes on them what they demand of teachers: accountability.

Hmm… this guy has never heard of field testing of items.  He obviously has no clue about the large body of scientific research that is conducted every year for the last few decades on education assessment.

I assure you, no one in my industry is rich and powerful.

Finally, as I said before, anyone who uses standardized test results given to students to hold teachers accountable is a moron.  These tests can only be used to hold one person accountable for their education: students.

If you use a toaster oven to cook a Thanksgiving turkey, it is an exercise doomed to failure.  Using tools for purposes they were never intended for is a failure of the person, not the tool.

You want to hold teachers accountable, then give THEM the tests.

The stupidity of the current test-based thrust of reform has triggered the first revolt of school principals.

Winerip writes: “As of last night, 658 principals around the state (New York) had signed a letter — 488 of them from Long Island, where the insurrection began — protesting the use of students’ test scores to evaluate teachers’ and principals’ performance.”

Of course, when you use a tool improperly, then you have failed on an epic scale.

Let’s face it.  Some kids, no matter what you do with them, will never ever learn.  It’s not the teachers’ fault, but someone has to be blamed and no one will ever blame students.

The article concludes

My school board member-friend concluded his email with this: “I can’t escape the conclusion that those of us who are expected to follow through on decisions that have been made for us are doing something ethically questionable.”

He’s wrong. What they’re being made to do isn’t ethically questionable. It’s ethically unacceptable. Ethically reprehensible. Ethically indefensible.

You are absolutely correct.  It is entire unethical to use a tool for a purpose it was never designed for.  But, since you people won’t hold the students accountable for learning, someone has to be to blame so the teachers and principals get it.

This concludes your lesson.  I would encourage you (and everyone reading this) to do a modicum of research into the standardized assessments in your state.  If you have general questions, I may be able to help.  If I can’t, I can probably find out how to find the information for you.

I’ll say one last thing about people who think that we don’t need advanced math skills or science skills.  You are all absolutely correct, most people don’t use math every day.  Everyone uses science principles almost everyday, but I digress.

The point is, if you were only educated at an elementary level, then what would happen that one time you actually do need the quadratic equation?  You wouldn’t even have the basis to understand it.

My boss says that content is dead.  You can find out anything on the internet, but with an elementary level education, how can you know what you are looking at it is even correct?

Yes, you can learn anything, if you can search for it.  But if you don’t even know what to search for?  How can you do anything, when you don’t know what exists?

What’s even scarier is thinking about all the critical decisions people make every day.  Everything from voting, to medicine, to dangerous actions (like cooking).  How can someone with only a elementary level of education function in our modern world?

Thanks Shosh!
This entry was posted in Culture, Education, Government, Society and tagged , , , . Bookmark the permalink.

5 Responses to When an Adult Takes a Test… and Fails

  1. JGB says:

    I’m far from convinced most state mandated NCLB tests rise to the standards you have laid out. Some tests are at least useful as you suggest for assessing the students in a limited, but controlled way. None of the mandated tests in my state provide any relevant statistical data to do any actual analysis. No reported variances, or standard deviations, error bars at all whether we talk about school level or student level. Short version the accountability and testing culture is pseudoscience, for largely the reason you explained it is being asked to do something it was not designed to do under the best circumstances, and in others it wasn’t rigorously designed to do anything.

  2. OgreMkV says:

    Then you need to talk to your state education office about correcting that. There are several vendors for standardized tests and some are higher quality than others.

    I would suggest that you might be surprised at the data that is generated, but just not reported to parents or educators. This is mainly due to budget restrictions.

    For my own company, we have a process called ‘data review’ where the company reps (content and statistics), the client, and often a committee look at the data generated from every single item on the exam. We can tell what socioeconomic group did better on the question, we can tell which gender/race did better on the question, we can tell who got it wrong and (sometimes) why.

    We can also see the correlation between how well the student did on the question and how well they did on the exam overall. This is very useful. For example, a student who got a very difficult question correct, but did poorly on the test overall very likely guessed (or happened to hit that little bit that they did know). On the other hand, if a large group of students did poorly on a specific item, while doing really well on the test overall, then we look at the item to see if it is confusing or have multiple correct answers.

    I’d also like to add that, at least in the case of most vendors, there is a ‘field test’ of questions. The questions are on the test, but it is their first showing and they are not counted towards the student’s score. However, the data is reviewed for those items very carefully and items that are too difficult are rejected.

    That’s one reason that I don’t believe this school board member. I happen to be familiar with the company that does at least part of the FCAT work and they don’t allow questions to be on the test for a grade if less than 30% of the students to field tested the question got it wrong.

    Again, none of this is guess work. I guess a few of the less careful vendors might have issues, but none of the major players do.

    Finally, I’d like to add that, if that information is not being made available to the students/parents/teachers, then the test cannot be used for its intended purpose. It can’t be used to correct deficiencies in learning… which is the goal for everyone.

    And I fully disagree that the educational assessment industry is pseudoscience. You obviously didn’t click on the link to the research. My company produces dozens of papers per year and educational researchers probably do more than that.

    However, anytime you deal with humans, you have to understand that things will be… less than perfect. 2 hydrogens and an oxygen make water every time. With people though, you can get different answers, even from the same person on the same day. It’s not an exact science, which is where statistics come into play.

    I would encourage you to contact your state education agency and find out about those exams. Ask to come to some of the committee meetings as an observer… you might be especially interested in the Technical Advisory Committee… this is the group that does all the things that you say don’t happen.

  3. JGB says:

    This misuse of the data is not the state education departments fault. It is a political problem, standardized testing stopped being about proper usage of the materials over a decade ago because of the misplaced political desire for accountability. As you said it is not used for it’s intended purpose, and adding the proper statistical information won’t change that it in a meaningful way. It is fundamentally being asked to do a job it cannot do (rate school quality). I do not deny that there is a rigorous truly scientific process for generating these questions, and such. But those methods are not generating a product to achieve the end that people believe the data to be useful for, hence my psuedoscience comment. I may take you up on the suggestion to make some public records requests and see how hard it is to find some of this information, which should be released along with the test results.

  4. OgreMkV says:

    I think we are agreed. The test isn’t used for its proper purpose (which is student achievement of the stated curriculum).

    I would add, I’ve been a college career counselor and if I had a nickel for every time I heard, “I’m not smart enough for engineering, I don’t like business, and I get sick at the sight of blood… so I’ll be a teacher because I like kids”. Not exactly the way to improve student achievement, but that’s another story.

    One more example from my teaching days. I worked in a hard core inner city school. Gang fights with firearms were a non uncommon occurrence. Out of 6 10th grade science teachers, my students had the best pass rate… at 33%. There were two teachers with single digit pass rates in their classrooms.

    However, my contract was not renewed and there’s was. The stated reason was ‘surplus’, but everyone knew it’s because I demanded that my students do homework and read the chapter ahead of time. My failure rate was 3 times what every other teacher was… and that’s just unacceptable for an inner city school.

    So not only are schools/districts/etc not using the tests for their intended purposes, but they (at least in one case and I suspect many more) are going against the actual results of the test.

  5. Pingback: Arguing about education via anecdote « My Track Record

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s