Asymmetrical Information

>>I'm in favor of tests in the sense that I believe in the proposition that if you don't measure something, it's awfully hard to manage or make policy for it.

Interesting. What's your view on racial quota systems?

I'm against them, because I think they don't do what they're supposed to do, and because I think it's legally very difficult to reconcile the doctrine that the government should treat all races the same, except certain groups that are special. If it had been a short term fix that would be one thing, but we're going on 40 years at this point with no mechanism to recognize when we have achieved the limits of government intervention. But I'm not sure what that has to do with primary and secondary school testing.

I don't know how we can fix schools if we don't have a meaningful measure by which to compare their performance longitudanally and latitudinally.

Education is no more important than eating. Imagine your statement cast as one about the food market: "I don't know how we can fix food-providers if we don't have a meaningful measure by which to compare their performance longitudanally and latitudinally."

That's just silly. As if we might compare Steve the Farmer's job performance against that of McDonalds as against Chez Snob as against my kitchen abilities. All are involved in food-production. All are incomparable. One size does not fit all. There is no reason to assume it does, or should.

Of course, food production in this country is not broken and does not need to be fixed. Food production is private. Education is public. Therein lies the solution if we care to see it.

Returning to the production of food: food production that involves the market is "tested", in a sense. It's tested on the market. Businesses that make money continue and if they are successful enough, may expand. Those that don't make money eventually fail. But this has nothing to do with any standardized test.

The solution to the education crisis is not standardized testing. It is competition - market testing. The solution is possible only with more liberty - ideally, completely privatizing education; less ideally but practically, instituting voucher programs. Educational socialism doesn't work; and it cannot work in the long run for the same reasons that socialism always doesn't work. Why do we accept as obvious that the Soviet Union failed economically, but then puzzle over the failure of our daycare prisons? If there's anything puzzling, it's that, not the failure of the schools.

With educational freedom, "we" don't need to test. "We" - the collective we - can leave the evaluation of results up to "us" - as individual parents. Tests will continue to be a part of that, of course, for some or even many schools and parents. But there are many other aspects of education that are not on standardized test, and cannot be.

Uh. I think the link's broken.

"Page Not Found

Thank you for visiting this Homestead Web site.
We cannot find the page you requested. Please check to make sure that you typed the correct address."

Most of High School is easy, that including tests, but ofcourse there are levels of difficulty. Though those that generally graduate in top of their class usually take the hardest classes and heaviest work loads (Algebra > Advanced C++ programming).

BTW I get "page not found" when I try the Kim Swygert link. Can you check it?

Fixed. It just goes to her page now; you have to scroll.

Yes, you can only manage what you measure. But that means you have to be very careful about what you measure, because that's what you wind up managing for.

Cheating is only part of the problem. The bigger problem is gaming: activity that improves the score on whatever measurement system is in place without actually improving outcomes. (See Andrew Grove's High-Output Management on the use of paired measures to defeat gaming.)

If we measure reading but not science or music, schools will drop science and music. If we measure reading with multiple-choice tests, schools will stop teaching anything about reading that isn't caputred on multiple-choice tests. As long as we insist on testing every student, the test we use will have to be cheap and therefore crude. (The obvious alternative would be to mix cheap census testing with expensive tests given to samples.)

This problem isn't merely theoretical. States that instituted high-stakes testing improved on their own test measures, but on average fell back on nationally-normed tests such as the NAEP.

When we try to bring accountability down to the school or classroom level, another problem arises: the sample sizes get so small that the signal is swamped by the noise, and we wind up rewarding or punishing mostly random variation.

The fact that a good testing program would be part of any serious attempt at schools management doens't mean that the testing program we have is better than nothing. Arguably, it's worse. And neither the polticians for whom "testing" is a mantra -- and who don't know or don't care about the complexities of measurement and incentive management -- nor the testing companies who collect their money whether their output is of any value or not have any real incentive to fix the problem.

More here.

There's an interesting analogy between school testing and corporate finance. In both cases, people are tempted to "game" the system to make numbers look good at the expense of actual performance. But we can't have public corporations without some kind of accounting numbers, and I suspect the same will turn out to be true for schools.

Excellent point. I've been self-employed most of my adult life, but have been in close contact with the operations of several S&P 500 firms, and the degree to which management at all levels engages in gaming to produce desired statistical outcomes that have a negligible relationship to real world results has been a near-constant source of amusement and/or frustration to me. Book-cooking is THE great American pastime, and I find it hard to believe it wouldn't be worse in a sector that has no suppliers of capital that eventually demand cold, hard, can't-be-gamed, cash in hand.

One of the advantages of increased dividend payouts would be some reduction in gaming, since cash payments are harder to play games with than abstract numbers such as "net income" and "retained earnings."

Now, if we could just think of the educational equivalent of dividends...

Indeed. A pile of cash, that one can spend as one pleases, is a fact, not a statistical measurement, and therefore not subject to gaming, beyond the relatively short term. I don't know how such discipline can be transferred to the education industry, short of complete privitization, which is not politically doable, even if other problems were solved.

>>But I'm not sure what that has to do with primary and secondary school testing.

Well, as in that case, it's a matter of not being able to manage or make policy for something if you don't measure it. Either you say that there should be no policies aimed at increasing racial equality at all, or you have to be prepared to measure the effect that those policies are having. And if that measurement is going to be used to guide your actions, then it's going to look very like a quota.

Not meaning to hijack your thread or anything, just trying to riff off your point about measurement to suggest that the position of being "in favour of some kind of help for minorities but against quota-based systems" is pretty much incoherent; any meaningful action is going to need measurements and any meaningful measurements are as close to quotas as makes no difference.

Well, but there are ways and there are ways. If you had a program aimed at making sure that, say, blacks made up 10% average admissions at top universities by providing intensive tutoring and associated services so that blacks had equivalent grades and SAT scores, I think most Americans would be in favor of it. If you aim to achieve the same measure by admitting 10% of blacks who apply no matter what their qualifications are, I think most Americans would be against it. For all the blather, current programs are closer to the latter than the former.

I tend to agree with Jane that some sorts of actions can be measureable but not quotalike, as is exemplified in her "intensive tutoring for blacks".

However I also think that such programs would be politically popular only until they were tried. Initially the proponents would be able to sell the program based on its purported large effect and low price. But time would show that the amount of money spent to get the effect desired would be far greater than originally thought. (If boosting educational achievement was cheap, people would do it privately. Only because it is expensive - not worth it to them - do they want the government to pay for it.)

Quotas are cheap, at least in terms of tax-cost. So they can remain politically popular. The real costs are inflicted diffusely on the general population. Quotas also are not possible via private action.

I think the program, if it worked, would have to be pretty damn expensive before people dropped support for it.

Two points.
1. Kleinman is just wrong about what the study he cites says, though his theoretical point may have some validity. It doesn't appear that he's actually read to the end of the study. Blogging press releases is fine, but he's smarter than that.

2. It may be that we can't use the current tests to manage effectively, but they may still have important political effects. As other posters have pointed out, testing is a substitute, and a poor one at that, for school choice. But, importantly, under the new law it is also the outcome, if schools fail to improve. From the squawking I've read in newspapers locally and nationally, lots of schools believe that, even with gaming, they will be unable to meet the new standards. Which means they'll be judged failing, and we'll move from one system of accountability to another. (But then maybe I'm just an optimist.)

Leonard: your 11:54 post is so dead-on it should be bronzed.

Private choice is an infinitely superior way of measuring a subtle and unquantifiable thing like the quality of education to testing. But testing is much superior to having no measurable comparisons of any sort and no school choice.

>> I'm in favor of tests in the sense that I believe in the proposition that if you don't measure something, it's awfully hard to manage or make policy for it.

There is only one valid measure of the success of schools - how well the students do in the world (or the next grade) after they graduate.

Its simply NOT possible to NOT measure the success of a school unless you were to kill all of the students at graduation. So it is always possible to make policy, one has only to pay attention to what teachers are saying about the preparation of students entering their classrooms.

What high stakes testing does instead is to subsitute a fairly bogus measure of success and elevate it in importance so that it becomes the only dominant measure of success. This is fundamentally stupid, but predictable. An easy and cheap test is preferred even though it measures the wrong things because once the bean counters have numbers to crunch, they are happy regardless of how little the numbers correspond to reality.

Anyone who understands education will tell you that high stakes testing does more harm that good. For good schools they are a nuisence, for marginal schools they distort things so that poor education turns into no education. And they become an excuse for not solving the real problems; which are that some of them simply don't have the resources to do a good job (or in some cases, the authority to remove problem children from the classroom).

That brings up an interesting point. Perhaps mid-career post-graduation earnings would be a better metric? Hell of a lag, but.....

I went to Stuyvesant High School, one of the best public high schools in the country. Can I tell you what made it that way? It wasn't the teachers, although they were good, and it wasn't the classes, although you did have a wide variety of stimulating courses and AP classes to choose from. It was the standard that our PEERS expected from us. It's the only time in my life I read the New York Times AND The Wall Street Journal EVERY DAY. Had I not, I would have been unable to keep up with the conversations. Now I detest studying (I am educated dispite myself, not due to any great desire to learn) but I was a teenager and wanted to fit in...and that's was how you fit in at Stuyvesant. And that's how you improve education in the US. Peer Pressure, not tests.

That and stop waisting time and resources teaching kids who will never do anything but work at McDonald's, and add more gifted programs so that kids who are pleanty smart can hang out with each other and get the peer pressure they so desperately need.

All those tests really do is make the school drill everyone for weeks at a time... it's like tyhe whole school became a class on the ACT's. And they rely on the smart kids to boost the score--- they threatened to punish the smart kids if they didn't reach a sufficent score on the standardized test. It's all crap, basically.

Point being: school choice. You'll be able to find the good schools by seeing which ones had a waiting list.

It's "longitudinally" not "longitudanally".

As a teacher, I've spent hours debating this issue. In my state, I have certain standards that I must follow. So, I make my lesson plans with the standards in mind. The standards do not dictate what I teach and I feel that my students do not suffer because I am somewhat concerned with them passing the mandatory tests in our state. I have yet to find a creative and useful lesson that did not meet state standards. It is the uneffective and boring lessons that fall by the wayside.

As a teacher, I've spent hours debating this issue. In my state, I have certain standards that I must follow. So, I make my lesson plans with the standards in mind. The standards do not dictate what I teach and I feel that my students do not suffer because I am somewhat concerned with them passing the mandatory tests in our state. I have yet to find a creative and useful lesson that did not meet state standards. It is the uneffective and boring lessons that fall by the wayside.

As a teacher, I've spent hours debating this issue. In my state, I have certain standards that I must follow. So, I make my lesson plans with the standards in mind. The standards do not dictate what I teach and I feel that my students do not suffer because I am somewhat concerned with them passing the mandatory tests in our state. I have yet to find a creative and useful lesson that did not meet state standards. It is the uneffective and boring lessons that fall by the wayside.

I'm in favor of testing. Particularly competency-testing of teachers. I've seen far too many grade school teachers who can't seem to master basic spelling and grammar skills.

Testing of the students, on the other hand, used to be done extensively. I'm not sure whether the results were ever fed back into the system, but does anyone else remember the (IIRC) Iowa Tests? In Florida, we now have mandatory testing of all elementary (dunno about higher) student bodies in public schools. The result, unfortunately, has been that the schools are training the children to take the test well, rather than educating the kids to the point that they will do well on the tests.

It seems to me that schools are now doing much less with much more than they did a few decades ago. Parents are being asked to help grade papers. Students have LOADS of homework to do. I don't know why we have to drill the kids so much to learn what required much less drilling in the past.

Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted. - Albert Einstein

Comments are Closed.