I appreciate that Pedro took the time to write an extensive response to my initial post taking a position against signing the open letter that is currently circulating. Now it is my turn. Before getting to his specific rebuttal points, I offer some context.

I am assuming good faith among the signers; I do not believe their intent is to harm. I do not say this to patronize, but because I understand that some see simply signing the letter as justification to attack the signers, and I disagree. Indeed, I am not attacking the signers’ views at all because — due to the ambiguity of the letter — I do not know what their views are. My focus is on the text of the letter, not the views of the signers.

I am generally a supporter of free speech, even when it makes people uncomfortable. I often find myself disagreeing with “cancellations”, and defending my position against friends arguing for them. I consider the letter and Pedro’s rebuttal to be reasonable debate; there is nothing in them that I find out-of-bounds for discussion. In fact, I can easily imagine versions of the open letter that I would have signed. However, I oppose signing this particular letter because of its ambiguity and absolutism, as I’ve already explained in my previous post.

I have the easier side of this argument: my goal is to show that the letter can be misused — that there exist plausible interpretations of the letter that people should not support. The writers on the other hand need to demonstrate that all plausible interpretations of the letter are acceptable to a potential signer. And unlike the letter-writers, since I am not asking people to sign anything, but only to refrain from signing, the only harm in my posts being ambiguous is that they may fail to convince. But Pedro’s rebttal, and other commentary I’ve seen, shows a misunderstanding of my objections, so I’ll try to clarify them here. I aim to point out natural interpretations of the letter that I reject, and which provide reasons not to sign on. As before, I’ll tackle one bullet at a time, but I’ll try to provide more specific examples of the letter’s problems.

The Bullets

1. Scientific work should be judged on the basis of scientific merit, independent of the researcher’s identity or personal views.

In my first speculative interpretation of bullet 1 as addressing scholarship submitted for publication, I argued that peer review is expressly designed to provide independent judgment of such work. Pedro rejected this argument by discussing “attacks for what they publish after it’s published, people being fired, ostracized, antagonized, and more.” So, it seems that his intended meaning of the ambiguous point here is the broader interpretation I explored immediately after, of judgment of the scholars themselves.

So let’s consider this broader point. Pedro and I may find ourselves in partial agreement here. I will readily agree with Pedro that there are cases where academics have faced excessive retaliation when their scientific work is inconvenient. Perhaps like me Pedro will sign the Standing with Dr. Timnit Gebru petition which argues that denying publication of her research and firing her was not an appropriate response to the disagreement about her scientific findings.

But while we might agree that Timnit’s firing (and some others that have been in the news) was excessive, Pedro seems to argue that every response is excessive. This is a fallacy of overgeneralization: the existence of overreactions does not mean that every reaction is an overreaction. There are cases where even “meritorious science” should have consequences for the scientist.

Consider for example the infamous Tuskegee Syphilis Study. This study asked the scientifically “meritorious” question of how syphilis affects African Americans. The experimenters deceived the subjects and left them untreated or mistreated, and many died of the disease. We now recognize the severe moral violations in this study; they informed the Belmont Report which guides many institutional review boards (IRBs) overseeing the use of humans as experimental subjects. Research that violates the Belmont principles is forbidden by our institutions no matter how meritorious the scientific questions are. And a scientist who violates these principles should be judged for doing so.

So what exactly does the open letter mean when it speaks of “judging on scientific merit”? Do they mean that the Tuskegee experiment was actually justified because it gathered scientifically useful data? Or do they mean that IRBs should be abolished, with the decision about appropriate ethical boundaries left to the scientists? Or do they consider it obvious that ethical review is part of “scientific merit”? And if the last, then what exactly are the bounds of “scientific merit” and what is being excluded? These interpretations are widely varying, but all consistent with bullet 1. Which is a signatory supporting?

The AMA’s Code of Medical Ethics also addresses this issue. Like me, and unlike the letter, it recognizes that there is no black and white answer, but instead that complex considerations must be weighed against each other in deciding how scientific work should be judged.

Pedro goes on to suggest that there is a contradiction between my objections to bullet 1 and my argument that we must avoid racism and sexism in judging scientific work. But this is another fallacious overgeneralization: while I agree that it is not valid to judge scientific work based on the researcher’s race and gender, there is no contradiction in simultaneously arguing that it is appropriate to include other considerations, such as ethical concerns, in such judgments.

Pedro then addresses the Nature Communications work on mentorship by women, and objects that it is unfair that this work was retracted (by the authors) only because it got so much attention which led to the identification of scientific flaws. If Pedro has some technique for ensuring that all scientific work receives equal scrutiny, I’d love to know it because I think many of my papers haven’t gotten the attention they deserved. I’m not sure what remedy Pedro is advocating here (and it certainly isn’t clear from bullet 1). Should we limit the number of people permitted to view a given article? Or should people not be permitted to question the science in papers they read if they disagree with the conclusions? Should authors not be permitted to retract? Pedro also claims this is an example where “the science is attacked because of the scientist” but goes on to remark that another paper by the same authors was not attacked, refuting his own claim. Overall, I couldn’t find a clear argument to rebut here.

Pedro then disagrees with my argument that it will sometimes be appropriate to exclude scientists from a conference even if we accept their work to be published there. I described the specific example of banning someone who has publicly stated that certain races or genders are inferior. Pedro asserts this is “letting objections to the person’s views affect the reception of the scientific work.” To make this concrete, let’s consider Ludwig Bieberbach, a talented mathematician and a Nazi who made the racial inferiority argument that “the spatial imagination is a characteristic of the Germanic races, while pure logical reasoning has a richer development among Romanic and Hebraic races” and “was enthusiastically involved in the efforts to dismiss his Jewish colleagues.” (Yes, I know this will generate piles of “David says Pedro Domingos is a Nazi” quotes from people who can’t read, but I think it is a useful test case.) He was dismissed from his position in 1945 because of his Nazi work, but was invited to lecture in 1949 by others who “considered Bieberbach’s political views irrelevant to his contributions to the field of mathematics”. So you can see that the disagreement between me and Pedro isn’t new. Bieberbach’s work is of course cited, and there’s a conjecture named after him. But, if he were alive today, would we want him at our conferences? I would say no. The letter, at least in Pedro’s interpretation, seems to say yes — so I cannot sign it.

Pedro finally argues that “the whole point of Principle 1 is to separate judgments of the science from judgments of the scientist.” Can we really do this? Long ago I served on various STOC and FOCS program committees. In those days the page limits meant that a lot of math got left out of the proof “sketches” that were submitted. At one of our committee meetings, one of our greatest researchers told me they were opposed to double-blind review because “you need to know who wrote the paper in order to decide whether you trust the proof sketches”. I disagreed with this particular argument, but it continues to be the case that a great deal of assessment of scientific work relies at a minimum on our trust in the author. When I read a paper describing execution of code that yielded certain data, I don’t run the code and I don’t compare its output to the data claimed. I trust the author. If this author had been convicted of research misconduct in the past, I would demand a higher standard of proof that they did the work they claimed. This would certainly be a case of judging science dependent on the scientist, and I think it would be appropriate.

2. Discussion and debate in the scientific community must be free of prior restraint as to topic or viewpoint.

I turn now to bullet 2 on prior restraint. Pedro claims “Karger says that this principle is vacuously satisfied” and that this shows a “shocking naivete.” He must have failed to notice my “taken literally” qualifier. My “shocking” statement was proposed only to point out that since nobody is literally prevented from publishing on the internet, the letter writers must mean something more than this. I then proceeded to argue against these broader interpretations.

In particular, I observed that conferences have always exercised prior restraint on which topics they will accept. Pedro appears to accept this point. But at the same time he objects to the restraint imposed by the NeurIPS ethics review. I’m not sure why Pedro says “Karger claims that (for example) NeurIPS’s requirement of broader impact statements and ethics reviews is not really a restraint” when in fact I described it as “a pretty clear prior restraint.” But overall Pedro seems to be saying that topic constraints are valid but that ethics review is an invalid restraint. This is inconsistent with the absolutist message of the letter. Bullet 2 does not say that discussions must be free of invalid restraints (which everyone would agree with, but would push off the entire debate into which restraints are invalid); instead it levels a blanket objection to any restraint. This would include the restraints on topic that we all expect. So how can anyone signing this letter understand exactly what kinds of restraints are covered by the principle? The letter writers might have a specific set of restraints in mind, but they remain hidden there.

3. No individual should suffer harassment or attack based on their personal or political views, religion, nationality, race, gender, or sexual orientation.

Finally we come to bullet 3 on attacks. Again we agree about a good part of this, but we disagree about the inclusion of personal or political views as grounds. Here, Pedro raises a valid objection to something I worded poorly. In my post, I explained that “If someone declares that a certain group is inferior, I want that “personal or political view” to be “attacked”. Or let’s consider another example: someone at a conference expressing their “personal view” that a particular scientist is sexy. I stand by the assertion that such views should lead to attack. But I then argued that these attacks “should not be ad hominem — -they should attack the individual on the grounds of the views they hold.” This was poorly worded; the attack should not be on the grounds of the views they hold, but on the grounds of their expression of those views.

What is the difference? Well, once you go from merely holding views to expressing them, that expression impacts those around you. Looking to my specific examples, I have heard from many researchers (especially students), of the psychological toll it takes to be told that you are inferior or unwanted or sexy. It burdens every interaction with concern about how one is being perceived and a pressure to be perfect. It discourages scientists from doing their best work. It drives people from the field. It is bad for science. Most important, it is hurting someone. Indeed, I would say these kinds of assertions are perfect examples of the attacks that bullet 3 claims should be off limits. I therefore believe it is entirely justified to respond to such attacks. Of course we can split hairs over whether and when such responses are themselves attacks. Is telling someone their comment was sexist an attack? What if it actually was sexist? Is telling them they are sexist an attack? Is refusing to have dinner with them an attack? Even if such actions are declared to be attacks, I think there are situations in which they are appropriate responses, and again the letter offers no guidance on the scope of “attack” that it rejects.

Balancing Harms

Pedro finishes by addressing my argument that we need a better balance between false positives and false negatives on accusations of racism and sexism. Here Pedro agrees with my high-level point but disagrees on the direction in which we need to shift. Pedro claims that we are tilted strongly in the direction of (imagining) “racism and sexism everywhere”. I have the opposite perspective: while I have seen a few eminent researchers like Pedro, and a few students, called out as racist or sexist for their expressed views, it seems that just about every female or minority student or researcher I have spoken to can tell me about their personal experiences of sexism and racism that have gone unaddressed in our field. Indeed, I suspect it is the rarity of call-outs that has made them newsworthy enough to be collected as anecdotal evidence of a “cancel culture”, while the experiences of those facing racism and sexism are so common that they go unremarked. To me it appears that our unwillingness to allow even the minimum of false positives has led to an astronomical cost in false negatives.

Pedro’s main anecdote here is from his own experience of being called a racist, sexist, misogynist bigot. This is unpleasant; I know because I too have been called these things. Not as often as Pedro, but enough to know that I’d like to avoid the experience. I’ve even modified my behavior a little bit to reduce the chances of it happening again; I think of it as “reflection and self-improvement.” But in the end, Pedro and I are pretty well situated to handle these kinds of attacks: we have prestige, strong social ties in academia, money, and tenure. False positives cost us relatively little. Contrast this with a smart, talented female undergraduate who dropped my class this semester because she faced sexism from the male students she needed as collaborators. For her the cost was very high. If we at the apex can reduce harm to our vulnerable members at the cost of a degree of discomfort for ourselves, deserved or not, that’s a trade worth making.

Conclusion

In sum, I cannot support a letter that takes such a black-and-white view on principles while simultaneously presenting significant ambiguity on exactly what those principles are. Pedro asks “where I draw the line.” As a scientist I get it; we like to draw lines and define absolute boundaries. But while our science may be precise, we do that science in a real world that is much more complicated. The relative costs of tolerating or rejecting certain positions, speech or behaviors depend on the speaker, the audience, the topic, and many other factors. The value of freedom in speech and scientific inquiry is high, sufficient to outweigh many other costs, but it is not infinite. And when we accept the costs that come with free inquiry, we have a responsibility to spread those costs fairly over our community, instead of letting them fall entirely on our most vulnerable members.

Lest some argue that my arguments reject signing any statement of principles, here’s an example of something I could sign: the World Wide Web Consortium’s code of conduct. It covers some of the same ground as the current letter — for example, like bullet 3, it rejects attacks based on “gender, gender identity and gender expression, sexual orientation, disability (both visible and invisible), mental health, neurotype, physical appearance, body, age, race, socio-economic status, ethnicity, caste, nationality, language, or religion.” Notably, parallel to my arguments above, it does not include “personal or political views” in this list. But it does expect members to “be inclusive and promote diversity” and asserts that “diversity of views and of people powers innovation, even if it is not always comfortable.” This matches a core theme of the open letter. But crucially, instead of laying out diversity of views as an inviolable principle, the code recognizes that it is a value that can be in tension with and overridden by others, and that such tensions ought to be resolved with empathy, in favor of those who would be most harmed. Unlike the current letter, the code’s precision and detail give me a better understanding of what I would be supporting. I’d be happy to sign a letter committing to the principles of this code.

I respect many of the signatories as brilliant academics. And count some of them as friends. I even agree with most of the petition taken literally or “in theory” (bar one crucial problem I’ll mention below). However, I strongly disagree with the practical implications of the message in the context of today’s discourse.

I’ll start with the meat of the petition, three bullet points I’ll quote here:

1. Scientific work should be judged on the basis of scientific merit, independent of the researcher’s identity or personal views,

2. Discussion and debate in the scientific community must be free of prior restraint as to topic or viewpoint, and

3. No individual should suffer harassment or attack based on their personal or political views, religion, nationality, race, gender, or sexual orientation.

On first reading, all three of the statements seem innocuous, even admirable articulations of the need to treat all people equally and permit free speech. Indeed, yesterday I posted that I agreed with the first two bullets and only differed on the third. But reflecting further, the blanket nature of all these bullets covers far too much.

Let’s start with bullet 1, “scientific work should be judged on its merits, independent of the researcher’s identity or personal views.” This is absolutely true but is also not in need of defense. This principle is why our conferences use blind peer review: when we review scientific work, we don’t know who the authors are, which forces us to judge the work entirely on its merits. So, if this problem is solved, why does the petition raise it? I can imagine two likely interpretations, both of which I reject.

First, the petition may be addressing situations (faced by some petition authors) where a respected scientist makes statements that harm others and therefore attract opprobrium from the community. But this is a judgment of the scientist, not the science, and so bullet 1 is irrelevant.

Second, the petition may be addressing controversies like the recent one where Nature Communications published a work claiming negative effects of mentorship by female scientists that attracted a tremendous amount of negative attention and was ultimately retracted by the authors. But bullet 1 is irrelevant here as well. Although attention was drawn to the work by people who disliked its social implications, the retraction was due to a realization that the science in the paper was flawed. That flawed science made it past peer review, a failing of the system. One could argue that it’s somehow unfair that this paper was subjected to more careful scientific review, but it certainly was the science, not the “personal views of the authors” that led to the revised judgment to retract.

To give the petition the best possible light, let me raise one other possibility where bullet 1 might be relevant, at least in theory. Suppose that some scientist is continuing to do great work but has for example stated publicly that a certain race or gender is inferior. Suppose this individual submits their great work to a conference. Should the conference refuse to judge that work? I could see that as going too far. But I think it would be perfectly appropriate for the conference to accept the work but deny the scientist the opportunity to attend the conference to present it. Let some proxy come to make the presentation. Preventing attendance by such an individual would be a reflection of our desire to support and respect the feelings of the maligned group, not a judgment of the science. If the scientist were uncomfortable with a proxy presentation, they could still post their work on Arxiv, and scientists would have, and I am sure would fulfill, an obligation to cite it *regardless* of the author’s personal views.

This last example points at the last subtext of bullet 1 that I want to reject: that when science is good, it is our obligation to respect or admire or fete that scientist who did the work. That’s just not true. If the person who did the great science is a jerk, I can say so. I can call them out for it. I can shun them. None of that judges the science.

I turn to bullet 2, “Discussion and debate in the scientific community must be free of prior restraint as to topic or viewpoint.” Again, taken literally this statement is vacuously satisfied. There is no prior restraint, as there is nothing that can stop me from saying whatever I want in my academic community, posting whatever I want on Facebook, or writing whatever I want in a conference or journal submission. So to understand this bullet we have to broaden the interpretation.

One possible interpretation goes back to what I already discussed and dismissed with bullet 1: confusing rejection of the scientist with rejection of the science. Another is that the authors are attacking statements that warn that certain kinds of submissions will be rejected. For example, the NSF states that proposals submitted without a “broader impact statement” discussing the effect of the research on society will be rejected. Similarly, the NeurIPS conference has begun requiring a statement in each paper about potential broader impacts including ethical aspects. Of course these don’t literally restrain my discussion, but it does tell me certain kinds won’t be accepted or published. But there’s nothing new about that: every call for papers describes the subject matter they are looking for, with the implicit guidance that papers about other subject matter will be rejected. That’s a pretty clear “prior restraint as to topic.” So there’s some fatal ambiguity in this bullet, where the authors are failing to explain what kinds of prior restraint they consider acceptable.

I turn finally to bullet 3, “No individual should suffer harassment or attack based on their personal or political views, religion, nationality, race, gender, or sexual orientation.” Again on first reading this seems an admirable defense of equality. But there’s one injection that makes me completely reject this statement as given: “attack based on personal or political views.”

I cannot think of any circumstance in which it would be appropriate to attack someone based on religion, nationality, race, gender, or sexual orientation. These are innate, uncontrollable individual characteristics whose diversity we should always treasure or at minimum accept. And harassment, at least in its current incarnation on the internet, is explicitly seeking to harm an individual much as physical violence would be, and should be rejected in the same circumstances (i.e., almost always).

But agreeing with all these aspects still leaves “attack based on personal or political views” and I strongly approve of this activity in the right circumstances. If someone declares that a certain group is inferior, I want that “personal or political view” to be “attacked”. Indeed, this petition is doing that — -attacking those who would violate the bullets. Obviously, such attacks should not be ad hominem — -they should attack the individual on the grounds of the views they hold, not because they are short or smell bad or whatever. But an attack is justified. So I firmly reject bullet 3 as written.

Now I want to broaden the context. Why was this petition drafted? I know that several of the signatories have faced significant backlash for some of their statements. So I have to suspect that the impetus for this petition is, in the end, a desire to prevent those kinds of backlashes. I can’t support that. I consider some of that backlash justified and some of it unjustified. But overall, I support the increased pushback over speech that is occurring in our field.

Those of us who work in information retrieval, or scientific experimentation more broadly, understand the concept of false positives and false negatives in detection. If you’re have a test for something like a positive Covid infection, that test might sometimes fail to detect an infection — -that’s a false negative — -and may sometimes declare an infection that doesn’t exist — -that’s a false positive. Such errors are generally unavoidable, since no science is perfect. But, you can often trade off one for the other by shifting the “sensitivity threshold” of your test. If you make the test more positive, you’ll detect more Covid infections — -some real (you’re reducing false negatives) and some imagined (you’re increasing false positives). If you make it less sensitive, you’ll do the reverse, decreasing false positives at the cost of more false negatives. You have to decide on the relative “cost” of false positives (making someone quarantine unnecessarily) versus false negatives (infecting more people with Covid) to decide where to put the threshold.

We face a similar situation with racism, sexism, and other isms. For a long time, we ignored them. Our detection threshold was way too low, and people offended with impunity. We had a huge amount of false negatives, failures to detect isms when they were on display. One of the strongest justifications for our failure to respond was “innocent until proven guilty”, declaring that there was some ambiguity in the offense, that we couldn’t be sure it was racism, that there are other interpretations. In other words, we refused to tolerate any false positives at all.

Imagine if we applied this rule to Covid, saying we can’t quarantine anyone, because it would be simply unfair to force anyone uninfected to quarantine. Forcing such a quarantine might be unfair at the individual level, but it’s the right choice to benefit society.

The same holds true for the isms. Our rejection of false positives, our refusal to push back on ambiguous isms, has directed all the damage at specific groups in our community, such as minorities and women. That is unfair at a huge scale. The shift we’re seeing now, being more sensitive to these isms when we see them, is a significant improvement, a way to shift some of the burden of those false negatives off of these historically targeted groups. It comes at a cost: those of us (like me) who have historically been untouchable *and* given the benefit of the doubt are suddenly being hit by false positives — -we didn’t mean it, but they’re coming after us anyway. But at least some amount of that cost is worth bearing. A system with no false positives, only false negatives, is not an optimally configured system. We have to balance out the impact of our imperfect assessments, making sure that everyone shoulders a part of the burden.