The last time the anti-affirmative-action Supreme Court case Fisher v. University of Texas hit the news, it was because former Justice Antonin Scalia had put his foot in his mouth about it again. “There are those who contend that it does not benefit African Americans to get them into the University of Texas, where they do not do well, as opposed to having them go to a less-advanced school, a slower-track school where they do well,” Scalia said. “One of the briefs pointed out that most of the Black scientists in this country don’t come from schools like the University of Texas. They come from lesser schools where they do not feel that they’re being pushed ahead in classes that are too fast for them.”

Multiple sources have confirmed that Abigail Fisher didn’t meet the University of Texas’ qualifications and would not have been accepted even if she were a person of color.

I find myself wondering why progressives accepted the core of his argument at all. The acceptance of his value system, that only the application of these values was wrong, is evident when we compare what he said to how it was reported.

Headlines included “Justice Scalia Suggests Blacks Belong at ‘Slower’ Colleges” (Mother Jones), “Scalia says most black scientists went to ‘lesser’ colleges” (Politifact) and “Justice Scalia: Minorities better off in ‘less-advanced’ schools” (New York Post). Newspapers and social media users alike responded with wholly justified anger and with lists of Black scientists who graduated from elite universities, from personal anecdotes to the trump card of public science educator and celebrity Neil deGrasse Tyson’s Master’s degree from the University of Texas.

Scalia was unambiguously racist, but he got more wrong than racism, and we collectively failed to hold him accountable.

Every rebuttal depended on reinforcing his stated values: that the definition of a good school is one that goes fast. A “less-advanced” school is one that is “slower-track.” In identifying the problem as “Scalia said that Black people deserve slow, inferior schools,” thus making the rebuttal “Here are reasons Black people are fine at fast, superior schools just as they are,” Scalia’s definitions of good and bad in education were left untouched. No one challenged the assertion that a “lesser” school is one where students “do not feel that they’re being pushed ahead in classes that are too fast for them.” If a school must be too fast for some of its students in order to be a good school, the choice to abandon students becomes proof of quality and rigor. Someone feeling that a class is moving too fast for them is a failure not on the school’s part, but on the individual student. If a student feels that their classes are too fast for them, that must mean they do not deserve a good education, because a good education is one that goes fast. Fast enough to leave some students feeling like failures or literally failing, fast enough that people are left behind.

By accepting Scalia’s value system wholesale when we criticize his racist application of it, we are choosing to abandon people who need anything beyond the absolute minimum of what a professor has left time to provide. We choose to underserve people whose previous educational institutions have failed them — and who made it to an “elite” school like Oberlin.

Who is most likely to be underserved by their school district? Poor people in poor communities — overwhelmingly people of color, specifically Black people. Scalia misattributed the gap in who is effectively served by a “fast-paced” education to race, and to inherent ability by linking it to race, banking on a long tradition of anti-Black discrimination. The problem he failed to identify is not specific to Black people or to people of color in general, and its common cause is both externally imposed and solvable. For someone who never had a chance to take AP classes, who is a first-generation student or whose ability to put in extra work outside of class is constrained by working their way through college, a class built for people with none of those disadvantages will deliberately leave that person behind. How can we call that a virtue in good conscience?

Affirmative action in college admissions can help compensate for the gap in resource allocation as far as letting people from historically marginalized populations get their foot in the door. But it means nothing if schools fail to actually work for their students and less than nothing when authority figures take that willful disenfranchisement as a sign that underprivileged people just don’t belong. Making it into a so-called elite college is supposed to be an immense achievement for anyone; doing it without the “necessary” background is phenomenal. And the reward for this is a struggle not to be left behind.

Again: How can we call this a virtue?

As long as we unquestioningly reassert that a fast-paced education is a good one, it’s an excuse to fail the people who need support the most. It’s an excuse for professors to ignore their own failings, because if the mark of superiority is for classes to be too fast for some people, faculty abandonment of students who need help is just a sign of a professor’s educational prestige. And the people who need more support than what a professor trying to rush through a class is willing to provide are likely to be already marginalized. It’s not about a person’s race or ability, whether or not a given speaker chooses to assume that the two are tied.

It’s about an institution seeing the option to support, uplift and educate and choosing laziness and willful marginalization instead.