20 Years of Optional SATs

William C. Hiss ’66, Vice President for External Affairs
Prem R. Neupane ’05

Bates College, Lewiston, ME

A paper presented at the 60th Annual Meeting of the National Association for College Admissions Counseling

Milwaukee, WI
October 1, 2004

Image 1

(Comments below are keyed to slide #’s on the accompanying PowerPoint.)

Nineteen years ago this week, I gave with three wonderful colleagues the closing keynote session at NACAC, a talk entitled “Admissions as Ministry.” All four of us—Dick Moll, Betsy DeLaHunt, Zina Zacque and I—had considered the ministry and ended up in Admissions, and our talks drew out, often with piquant humor, the parallels between the two careers. I never said better why I made Admissions my life’s work, and still remember dozens of people coming up afterwards to hug us, crying. I realized on the plane home that my suit was ruined, from all that wet make-up.

That was the year after Bates made testing optional, and I think it no accident that three of us on that panel—myself, Dick at Bowdoin and Betsy at Sarah Lawrence—had led colleges to make testing optional. We had each thought a lot about an ethical issue: Are we helping young people with this process? Are we doing things that hurt them? We weren’t trying to point a finger at standardized testing as though it came from the forces of darkness; it hasn’t. But at least in our judgment at the time, testing was occupying too much emotional space, and kids were being hurt, either in self-esteem or in actual admissions decisions, in their access to higher ed. So we were going to try another tack, not with a statement of moral rectitude, but as an attempt to say, “How can we help kids to believe in themselves, to channel their time and energies into sluice gates of causes and values worth of their efforts?”

Hard to imagine, but this is the 20th anniversary of the decision at Bates to make SAT’s optional for admission, and we researched the issue for five years before recommending it to the Bates faculty, so in all, a 25-year project for this quite literal grey beard. SAT’s were made optional in 1984; all testing was made optional in 1990. This presentation is a 20-year retrospective study of the policy, and I would praise my co-author, Prem Neupane, a senior at Bates from Nepal, who did the statistical research you will see. That students like Prem can come from very different cultures, work in a second or third language, and have a scholarly paper read at a national conference before they finish their undergraduate degree at Bates, is one mark of the success of this policy.

From the outset, Bates decided to share its data and research on the policy. We have done major research projects at roughly 5-year intervals, and provided the data and articles to any press outlets or other colleges which asked for them, including a fine college represented here today. There has been amazing consistency of findings in our data over 20 years, and now some intriguing outcome data in our alumni.

The report is in 17 images on which I will have a few comments, but running like a scarlet thread through the data are three fundamental policy issues, on which organizations like NACAC should debate as national priorities.

(1) Does requiring the tests open or truncate access to higher education? Call this a marketing issue if you like–who will apply?—or an access issue—who is allowed to go to college and where will they go?

(2) How predictive are the tests? Are they consistently predictive across populations? Are they “standardized” because people take the same test, or because their predictive value is consistent? As you will see, we seriously question the latter argument.

(3) What are the definitions of intelligence and achievement which a college (or society) signals to its youth with such a policy? What are the career and graduate degree results of our policy?

Underneath all three of these issues is a fundamental question of social ethics and social policy: who gets to go to college, and what are the definitions of intelligence and achievement which a college, or a society, signals to young people by what it requires for admission. Many of us are deeply indebted to the work of two of America’s premier educational thinkers in this generation, Howard Gardner and Jonathon Kozol. Bates sees itself as being a small Petri dish of Howard Gardner’s work on the multiple definitions of intelligence, and I think Professor Gardner regards Bates as one of his small Petri dishes. I would acknowledge with profound gratitude his new gestalt on human intelligence. What you are about to see is the efforts of an “in the trenches” disciple of Gardner and Kozol.

Image 2

Some of you may know Woody Allen’s wonderful movie, “Annie Hall”, where he looks at the camera to talk about a fellow who thought he was a chicken. People tried to get his relatives to get him into some therapy to get over this obsession about being a chicken, and the relatives said, “Well, we would, but we need the eggs.”

At some levels, this discussion is about holding on obsessively, perhaps neurotically, to something that demonstrably doesn’t make any sense. We might as a gesture of health say to our young people, “You are not a chicken!”, and say to each other, “We do not need these eggs.”

It also is about being willing to take down a structure to see more clearly, to try a different way, which is the point of the Japanese haiku.

Image 3

Now we start to look at the first of our three principles, the access issue, or if you prefer, the marketing issue. The applicant pool at Bates has almost doubled, from 2200 to 4200, since we made testing optional, while admits and enrollees went up marginally.

A proposal for national debate, both at the college level and for the current “No Child Left Behind” emphasis on testing: Does testing truncate access and success more than it helps identify promise or achievement?

The most basic question for any admissions dean: Can you get a better class from twice as many applications? Of course you can, and on all the scales.

In this and previous studies, we asked statistical experts at Bates to check and critique our work. Michael Murray, a renowned international economist at Bates who designs national economies and central banking systems for third world countries, said to me, “Bill, you shouldn’t be comparing submitters and non-submitters!” I thought, “Oh no, into what statistical blind alley is Michael leading me.” He went on, “You should be comparing the enrolled non-submitters with the students you would have had to admit if you didn’t have 1500 non-submitter applicants from which to choose the very best.” He is right, of course, and at Bates and most colleges, that would comprise the entire wait list and a decent slice of the refuse pool.

Image 4

Women, international students, and U.S. multicultural students gained a lot, but all cells of our pool increased. We know have enough applications from abroad to fill the class twice over, and have no American citizens in the class. Many of the international applicants are some of the brightest people in their countries, and there is no question but that the influx of highly talented international students at Bates has turned up the intellectual thermostat for the whole college. The numbers of students of color and international students are still not large, but we have increased those populations by between two and four times.

Image 5

From 1984 to 1990 about a quarter of the Bates students entered with no SATs; when all testing was made optional in 1990, the percentage of students not submitting testing rose to the mid-to-high 30% range and stayed there.

From 1992 to the present, 129 (about 3% of the total enrolled students) SAT I Non-submitters submitted SAT IIs. We have significant volumes of AP’s, A levels, and IB’s, but most of them come in late in the senior year for placement and advanced credit use, so they are not part of this research.

Image 6

These data are a snapshot of use of the policy by students of color, and by gender. Measurably more women than men will use the policy, and use by non-whites is about 8% higher than for non-whites. Hispanic and Black students will use the policy at a rate about 10%-15% higher than the class averages.

But something important should be pointed out here. Optional testing is often assumed to be a device for an affirmative action policy, to open the admissions process from a narrow statistical review to a more complex and subtle reading. And it does that. But white students using the policy outnumber the students of color by about five to one. We have found that the policy appeals to all the subgroups of students which folk wisdom would tell you are the students not being much helped by standardized testing in admissions: women, rural and blue collar students, immigrants, learning disabled students, students with spike talents in something (arts, chemistry, athletics, debate, theatre, dance, political or campus leadership), and students who speak a second language, no matter what their ethnicity or citizenship. We found heavy percentages of non-submitters from Maine, because so many are rural or low-income, and have neither the money nor even the physical access to be coached for tests. But we also found an intriguing pattern of high percentages of non-submitters across the top of Maine, New Hampshire and Vermont: they turned out largely to be young people of French-Canadian heritage. They may have been US Citizens for several generations, but still speak French at home, and are carrying two grammars, vocabularies and syntaxes in their heads.

Image 7

Now we get to the second major point and heart of this report: What are the productive results of the policy? Over the 20-year history of the policy, the difference in Bates GPAs between submitters and non-submitters is .05 of a GPA point.

Image 8

And the difference in graduation rates is 0.1%. .05% of a GPA point, and one-tenth of one percent difference in graduation rates. On this we hang the national sluice gate system about who gets into college and where they go?

We could spend the rest of the time talking about these two slides, which are the heart of this report. In a word, in a college generally regarded as a highly demanding academic environment, non-submitters earn exactly the same grades, and graduate at exactly the same rates, as do submitters.

Let’s have a little thought experiment to expand this finding out to national access policy. In California, that bellwether state so often several years ahead of the rest of us. I refer you to Eugene Garcia’s report of several years ago on Hispanic admission to the public institutions in California. The U-Cal public university admission rate for Hispanic students has been over the years less than 4%, and Hispanic students comprise 50% of the K-12 school cohort. Does this pass a common sense test of access to a public university system, to have a 4% admit rate for 50% of the school population? Does it pass a test of social ethics? I am not pointing a finger at California, but asking a common sense question about our country: are we getting the students the education they need to be competitive? In California, it has always been an article of faith that the state colleges and extensive community college system will provide much wider access than the state universities. But with state budget cuts, the curriculum of the entire community college system in California was just reduced by 4%, stranding 117,000 students seeking access to the community colleges. In Maine, we have lost startling percentages of our manufacturing jobs to overseas competition. What will people do for employment who are turned away from various forms of higher education, which has been by far the major route to economic improvement?

The same question must be asked about No Child Left Behind, which is largely driven by standardized testing results. How much are we truncating our success rates by using testing?

Image 9

At Bates, Non-Submitters enter with very marginally lower academic ratings, and marginally higher academic ratings.

Image 10

On average, Submitters score about 90 points above the Non-Submitters in Verbal SAT, and 70 points above non-submitters in Math SAT, for a total SAT gap of 160 points. This TSAT gap has been amazingly stable for the entire history of the policy, and if there reasons for that, we cannot see them.

Image 11

Yet for both submitters and non-submitters, the Admissions Office is able to read folders accurately and make very accurate predictions of success at Bates.

It is sometimes said that an optional testing policy will only work at a small college able to read applicants individually and thoughtfully. With respect, I think this is nonsense. Lots of large research universities read folders just as carefully as small colleges. Another national policy and social ethics question: What are the public costs of not admitting students who would succeed, in order to run a simple, inexpensive admissions process driven by class ranks and testing? If Bates’ experience can be extrapolated to other kinds of institutions—one can wish that many colleges on Fairtest’s list of 700 colleges not requiring or de-emphasizing scores would publish more research on their policies—we may be throwing away as much as a third of our potential national talent.

Image 12

While testing seems to have some very basic correlation with GPAs, non-submitters seem to outperform submitters with the same SAT scores, but for both groups, the lines are pretty flat, because virtually everyone is succeeding.

Image 13

And now to the third major point: how do wider definitions of achievement and intelligence play out in students’ choice of majors, careers and graduate fields?

There are some modest over-weights toward submitters in math and sciences, and corresponding over-weights to non-submitters in social sciences.

Image 14

The patterns of distribution by majors are intriguing, and I have made four groupings of majors in this slide. First, in three majors generally regarded as among the toughest at Bates—chemistry, biological chemistry and neuroscience—the percentages of submitters and non-submitters are very close. Second, in only three of our 32 majors is there a clear imbalance toward submitters: Math, Philosophy and Physics, but remember that when we get to the next slide. Third is a grouping of majors that folk wisdom would suggest are places that would reward imagination, intuition, unconventional thinking, interest in other unexplored culture, new ways of viewing experience, and the like. In this group—African American Studies, American Cultural Studies, Art, Classical and Medieval Studies, Theatre, Women and Gender Studies, and Self-designed majors—there are patterns of non-submitters being equally or strongly represented. And in our largest majors—Biology, Economics, English, History, Political Science and Psychology—there are only modest trends by submitters or non-submitters.

Image 15

This slide was something of an experiment and dense with data, but intriguing. We separated by majors, and graphed GPA difference against SAT difference. In the large majority of departments, submitters and non-submitters are within .1 of a GPA point of the mean. Remember Math, Philosophy and Physics, the departments with many more submitters? Two of them have 200 point differences in SATs, but only math has a larger than average GPA difference.

At Bates, taking a double major is a sign of intellectual ambition. There are 313 double majors: 108 Non-Submitters, 205 Submitters, just about the ratio of non-submitters to submitters.

Image 16

With now a 20-year time line, we can begin to look at graduate degrees and career outcomes, and there are some fascinating patterns. In general, there is very little evidence of submitters and non-submitters having different career tracks, with one glaring exception, which you will see.

In creative or human service fields like the arts, broadcasting, or education, non-submitters are represented at slightly higher rates, while the opposite is true in data processing and scientific or technical fields.

Image 17

In this slide, we divided career fields by both submitter status and by gender. Perhaps this image looks a lot better than it would have 30 years ago, but one conclusion that jumps out is that is not submitter or non-submitter that shapes more than a few career decisions, but still gender. The shapes of the graphs from top to bottom are amazingly parallel.

Image 18

Another interesting snapshot of outcomes. We isolated alumni in several specific fields, including two highly competitive fields, CEOs, (including founders, managing directors, heads of corporate divisions, etc.), and financial analysts/advisors (stockbrokers, hedge fund types, etc.). The percentages of submitters and non-submitters are about the same. So that’s good news. Not so with lawyers and doctors.

Image 19

And here is the glaring exception. Bates alumni earn graduate degrees at quite high rates: about 70% of all Bates alumni will earn at least one graduate degree. At the Master’s Degree level, the percentage of submitters and non-submitters are quite close. But in fields that require another standardized test for admission, there are big, visible gaps between submitters and non-submitters: MBAs, PhDs, MDs and JD’s. I mean this as a honest and not a rhetorical question: are these the best, or just the best test-takers? Let that question go proxy for a lot of what we need to understand better than we do.