Administered by:
In the field of cybersecurity, a distinction is made between the "blue team" task of building a secure system, and the "red team" task of locating vulnerabilities in such systems. The blue team is more obviously necessary to create the desired product; but the red team is just as essential, given the damage that can result from deploying insecure systems.
The nature of these teams mirror each other; mathematicians would call them "dual". The output of a blue team is only as strong as its weakest link: a security system that consists of a strong component and a weak component (e.g., a house with a securely locked door, but an open window) will be insecure (and in fact worse, because the strong component may convey a false sense of security). Dually, the contributions to a red team can often be additive: a red team report that contains both a serious vulnerability and a more trivial one is more useful than a report that only contains the serious issue, as it is valuable to have the blue team address both vulnerabilities. (But excessive low-quality reports can dilute attention from critical issues.)
Because of this, unreliable contributors may be more useful in the "red team" side of a project than the "blue team" side, though the blue team can still accommodate such contributors provided that the red team is competent enough to catch almost all of the errors that the contributor to the blue team might make. Also, unreliable red team contributions only add value if they _augment_ the output of more reliable members of that team, rather than _replace_ that output, and if their output can be effectively filtered or triaged by more experienced red team members. (1/3)
The blue team / red team distinction extends beyond cybersecurity to many other disciplines as well. In software engineering, for instance, "blue teaming" might correspond to the generation of new computer code, while "red teaming" would consist of such tasks as quality assurance and testing of such code. In mathematics, "blue teaming" could involve coming up with speculative ideas to solve a math problem, while "red teaming" checks the arguments for formal errors, and also raises heuristic objections to a blue team approach being viable. (See also my discussion about "local" and "global" errors in mathematics at https://terrytao.wordpress.com/advice-on-writing-papers/on-local-and-global-errors-in-mathematical-papers-and-how-to-detect-them/ ).
I like to refer to these two teams in mathematics as the "optimists" and "pessimists"; in my experience, the strongest collaborations arise when there is a roughly equal split between the optimists and pessimists in the collaborations. (Depending on the collaboration, I myself have sometimes played the optimist, sometimes the pessimist, and sometimes a mixture of both.) (2/3)
Many of the proposed use cases for AI tools try to place such tools in the "blue team" category, such as creating code, text, images, or mathematical arguments in some semi-automated or automated fashion, that is intended for use for some external application. However, in view of the unreliability and opacity of such tools, it may be better to put them to work on the "red team", critiquing the output of blue team human experts but not directly replacing that output; "blue team" AI use should only be permitted up to the capability of one's "red team" to catch and correct any errors generated. This approach not only plays to current AI strengths, such as breadth of exposure and fast feedback, but also mitigates the risks of deploying unverified AI output in high-stakes settings.
In my own personal experiments with AI, for instance, I have found it to be useful for providing additional feedback on some proposed text, argument, code, or slides that I have generated (including this current text). I might only agree with a fraction of the suggestions generated by the AI tool; but I find that there are still several useful comments made that I do agree with, and incorporate into my own output. This is a significantly less glamorous or intuitive use case for AI than the more commonly promoted "blue team" one of directly automating one's own output, but one that I find adds much more reliable value. (3/3)
@tao I did talk to someone working on quality control at a large newspaper. And they use LLMs exactly for the red team approach, not for the blue team. This works well, he reports!
@tao As many have said before me, an LLM is often like talking to yourself in a mirror.
That would probably work for research purposes.
Or you can just write something down,
or draw some doodles, or talk to your dog about your work. All of these things engage useful parts of the brain, and feel more natural, which is important.
In a pinch, a graduate student will do if you don't have a dog. Or in the worst case, a cat, perhaps. Though they have their own agendas. Probably an LLM would be better than a cat ...
@tao I think a lot of bloggers are coming to a similar conclusion. LLMs are a great way to feed your first-draft arguments into a wood chipper to see how strong they are, but no one is interested in reading a blog post written by an LLM.
Interesting to see the writing process analogized to coding software and solving math problems.
@tao Several (many?) open source projects with bug-bounty programs have reported being inundated, even overwhelmed, by "excessive low-quality reports" generated using LLMs.
https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/
https://thenewstack.io/curl-fights-a-flood-of-ai-generated-bug-reports-from-hackerone/
> "the risks of deploying unverified AI output in high-stakes settings."
the vibe coding geniuses responsible for the Tea App debacle could have benefited from this advice
https://www.404media.co/women-dating-safety-app-tea-breached-users-ids-posted-to-4chan/
@tao also fwiw i found AI pretty helpful doing things like writing documentation for my mastodon timeline algorithm
@tao It seems to me the current products are tuned in a way that favors blue-team functions: they are upbeat, talkative extroverts, sycophantic glad-handers who want to make everyone happy.
Today Claude suggested my code was doing two things in the wrong order and offered to make a change that updated a line of code with an identical line. When I objected that its proposed edit didn't change anything, it said “you're absolutely right!”
But I was absolutely wrong. Its proposal had been to remove a line and add back an identical one *in a different position*, fixing the out-of-order operations, just as it had suggested. I had misread the diff. It was entirely my mistake—up to the point when Claude was faced with my misunderstanding and recanted instantly. Then it became a joint mistake.
I've tried to get Claude to be less accommodating, not just for practical reasons but because its default Ned Flanders personality irritates me. It's curiously resistant to this.
I think for best red team operations, we would need LLMs that had been trained to be sour, cranky pessimists who take satisfaction in pointing out errors. But it seems that the blue-team personalities are more attractive and make flashier demos.
@mjd @tao
Worth noting that effective blue team AND red team work each requires having a balance of both confidence and openness to being wrong.
Going back to "reliability": what's crucial for blue team work is not "reliable delivery" but "reliably secure output of a team". Individuals being wrong is expected, but together we identify those mistakes, and we're open to recognizing them together (without losing self-confidence nor respect in each other). But individuals don't have to be "reliable" in the sense of always producing useful output.
Realistically, most security teams do a mix of blue and red team things, much like mathematicians in collaboration : )
@isotopp Perhaps, I haven't tried it.
@isotopp No, I found its relentlessly performative negativity very tiresome. It's like a sullen teenager.
@isotopp Later on in the conversation it has softened up, to the point where I suspect that OpenAI is trying to manipulate me into believing that I have somehow won it over.
@tao yes, as a software engineer I find even the very latest AI autocomplete to be a pain and a distraction. It's like trying to engage in a conversation with a person over your shoulder continually trying to end your sentences. Super annoying.
However, once I have written something it may be useful in suggesting ways to improve the written code.
@tao - Logicians are starting to think harder about this blue team / red team distinction. Constructive logic, the logic of verification, codifies some of the blue team principles you enunciate. But co-constructive logic, which seems to be a more recent topic of interest, is the logic of refutation. It codifies some of the red team principles.
Mike Shulman has recently written about a form of logic that combines the constructive and co-constructive aspects. His paper is rather technical, but here's a revealing quote:
"Intuitionistic logic is often explained informally (e.g. in [TvD88a]) by the so-called Brouwer–Heyting–Kolmogorov (BHK) interpretation, which explains the meaning of the logical connectives and quantifiers “pragmatically” in terms of what counts as a proof of them. [...] Practicing constructive mathematicians, however, have found that it is often not sufficient to know what counts as a proof of a statement: it is often just as important, if not more so, to know what counts as a refutation of a statement. [...] To repeat, the problem is that the BHK interpretation and resulting intuitionistic logic privilege proofs over refutations. [...] This suggests a BHK-like explanation of logical connectives in terms of both what counts as a proof and what counts as a refutation. We now explore what such an explanation might look like."
Studying this formally, as Mike is doing, might be useful for AI safety, etc.
@johncarlosbaez @tao Interestingly in my PhD thesis (Chapter 8), I introduce a new diagrammatic proof system for bi-intuitionistic logic called "bubble calculus", that combines literally red and blue bubbles encoding proofs and refutations! (unfortunately I mixed up Tao's color code)
By the way, there's "Devil's advocate".
Not that I expect it to catch on.
@johncarlosbaez There are been a lot of nice works with this perspective in logic, e.g. in game semantics or linear logic, but I don't know other ones that relate it to mathematics like Mike Shulman does here. He gives a great perspective!
@johncarlosbaez Shulman's paper “Linear Logic for Constructive Mathematics” takes this view also, that linear logic can be understood in a sort of BHK-like interpretation that distinguishes proofs from refutations, and in which (for example):
“A refutation of P⊸Q is a proof of P together with a refutation of Q.”
Shulman has a blog post about the paper: https://golem.ph.utexas.edu/category/2018/05/linear_logic_for_constructive.html
@johncarlosbaez Oh, whoops! That's the same paper. I hadn't seen it since he changed the title.
@mjd - but thanks, that blog post is a good place to start.
@tao I'm curious whether this is an intentional re-framing or evolution of what you've written about thinking in terms of game semantics here https://mathoverflow.net/questions/38639/thinking-and-explaining/38882#38882
Perhaps it's a similar idea phrased in terms computer programmers have more familiarity with?
@tao Believe it or not, the most interesting part of your thread to me was your statement that the best collaborations probably have a roughly equal number of people from both groups. That's a statement that's much more profound that it seems at first glance, and I'll be chewing on that for a while.
@tao IMHO red/blue team is bullshit newspeak... newbies! just a few years ago there were only white/grey/blackhat hackers and admins/sysops and they did the main hacking job alone and not in a team...