1.5M ratings
277k ratings

See, that’s what the app is perfect for.

Sounds perfect Wahhhh, I don’t wanna
nostalgebraist
nostalgebraist

I was just looking over the various people who’ve translated FFVI into English, and it was fun seeing what people did with the opening narration.

Classic 1994 Woolsey:

Long ago, the War of the Magi reduced the world to a scorched wasteland, and Magic simply ceased to exist.

1000 years have passed… Iron, gunpowder and steam engines have been rediscovered, and high technology reigns.

But there are those who would enslave the world by reviving the dread destructive force known as “Magic”.

Can it be that those in power are on the verge of repeating a senseless and deadly mistake?

Tom Slattery’s translation for the GBA version, supposed to be more accurate on the whole:

The ancient War of the Magi… When its flames at last receded, only the charred husk of a world remained. Even the power of magic was lost…

In the thousand years that followed, iron, gunpowder, and steam engines took the place of magic and life slowly returned to the barren land.

Yet there now stands one who would reawaken the magic of ages past and use its dreaded power as a means by which to conquer all the world.

Could anyone truly be foolish enough to repeat that mistake?

Unofficial translation by Sky Render, which is widely viewed as awkward although more literally “faithful” than Woolsey (not least by the translator themselves, cf. their later LP of it):

Long ago, humans battled one another, and the world became a scorched wasteland. The power of “magic” simply vanished.

1000 years have passed… Iron, gunpowder, and steam engines have been developed. Machinery has been revived to replace magic.

The great magic war faded into a legend. But the power of “magic” has been revived secretly, by the powerful military empire, run by a man who wants to rule the world.

Would this man be willing to destroy the world again for his own greed…?

(Unfortunately, Lina Darkstar never translated this part.)


I can’t tell how much of this is nostalgia, but the Woolsey version seems best to me here.  For one thing, “dread destructive force” is a good phrase (though so is Slattery’s “charred husk of a world”), and I like “high technology reins.”

Just as importantly, the Woolsey translation avoids the blunt moralizing of the other two (which I figure must have been in the original script, since it’s in two out of three translations).  The other two mention an individual villain, and finish on a note of “fuck that guy,” which fees like putting the cart before the horse – it’s like a movie that starts by telling you “there will be a bad guy in this movie, and you should hate him, because he’s evil.”

Woolsey generalizes it to an unnamed plural group, and his last sentence lacks the exasperation of the other two.  It’s not “how could this one guy possibly be so evil?”, it’s “have today’s rulers failed to learn the lessons of history?”, which is a much more interesting question.  “Can it be that those in power are on the verge of repeating a senseless and deadly mistake?” sounds like something you could easily hear in real-life political discussion; the others sound like we’re reading someone’s story outline.

lostpuntinentofalantis

I don’t know if I was primed by this post, but I went and looked at the original JP intro (this PS1 one on youtube, the relevant bits start around 5 minutes) And I’m not sure where the moralizing from the other two comes from. If I had to translate (as a terrible translator[0]) the last line I’d do something like…

Could it be that man would be willing to repeat the mistakes of the past?

(Without having played FF6 I’m sure this is shit; the original language had 人 as the subject, which could be one of: Person, people in general or a specific group of people)

I personally have Many Thoughts about the aesthetic impedance mismatch that often happens between the Eastern Fiction and Western Fiction and am inclined to attribute the hamhanded moralizing to the non-Woolsey translations being Too Christian for their own good.

[0] source: me

effortpost 俺の日本語少し変ですね japanese WHAT DO YOU MEAN CANNIBALS CAN BE SYMPATHETIC\
sinesalvatorem
sinesalvatorem

The fact that trying to italicise a word that begins with ‘w’ but failing to pull my finger off the ctrl key in time has killed about 5% of all the effort posts I’ve ever written is a major quality of life drain and Tumblr needs to stop being terrible and either save your work or warn if you’re closing a tab with writing.

lostpuntinentofalantis

as a wise person who has written like 10 effortposts total, sometimes I just randomly save draft does this helping???

shitpost clearly sinesalvatorem should learn emacs and write the tumblr integration
notthedarklord42

bayes: a kinda-sorta masterpost

nostalgebraist

@lostpuntinentofalantis

I don’t think the fact that humans are bad at thinking up logical implications is a very strong argument against bayes, in the same way that “But Harold, you said you loved Chocolate earlier!” is an argument against preferences.

So, I will agree that there’s this non-monotonic thing. This is indeed a very good point against using Bayes as a mental tool! I am not disagreeing with that!

What I do disagree with is the idea that it’s ipso facto problematic. I think the correct way to do this is throw out your first estimate as a preliminary one, and then use the other logical implication questions as a way to generate a battery of knowledge in a kinda organic fashion. To use the original “California succession” thing, let’s say I think it’s unlikely, so I throw out 98% as my likelihood, then some else asks me the “USA still together” so I also generically throw out 98% but A HA!!!!!! THIS SEEMS WRONG, because the set of situations involving the US together but California leaving seems I dunno small or whatever, so I end up adjusting the probabilities as,  repeating until I’ve thought of all “relevant” probabilities.

But logically speaking isn’t this troublesome? Isn’t it terrible that in theory an adversary can choose a sequence of questions which allows them to set my probabilities? Well, not really. My claim is that thoughts of these logical implication things provide information because humans are really bad at accessing all the information they have, and that, yeah sure if the adversary controls how a person accesses their information, of course the person is screwed? So you hope that people have good internal “implication generating”  machinery, such that by the time that they have worked through a bunch of subset questions, they have dumped out all relevant information, and the ordering effects are washed out.

Which is a much more elaborate way of saying “guys stop throwing out random probabilities and sticking to them if you don’t have good intuition/facts doing cognitive work aaaaaaaahh”

I guess I can agree that nothing I said above is specifically motivated by Bayes, except for this vague feeling of “well, shit it turns out I’m actually really bad at incorporating all relevant information” and I think it’s really just unavoidable.

I don’t think this is a problem with humans, I think it’s much more fundamental.  The real issue is that these kinds of “obviously nested” statements have a “easy to check, hard to find” property, like with NP-complete problems.

Let’s define “A is obviously nested in B” as “if you describe both A and B to me, it’ll be immediately obvious to me that A is sufficient but not necessary for B.”  And let’s define an “obviously nested pair” as A, B where one is obviously nested in the other.

The “US in 2100″ statements mentioned earlier are all obviously nested pairs with one another.  But the ones mentioned are just a few examples; there are infinitely many statements of the same form, asking about slightly bigger or smaller regions of the US, that also form obviously-nested pairs with all other such statements.

And that whole infinite chain is just one “direction” in hypothesis space.  You can think about any other subject – existence of various markets and sub-markets (will candy be sold?  will lollipops?), demographics and sub-demographics, scientific ideas and special cases thereof, you name it – and produce an infinite obviously-nested chain like this.

In finite time (much less polynomial time), you can only explicitly think about some vanishingly small subset of these statements.  Yet you implicitly know infinitely many facts about them (about each chain, in fact, of which there are infinitely many).  There’s no way to sit down and think enough beforehand that all of the obvious-nesting information has been dumped out into an explicit representation (and that representation would take infinite space anyway).

Now, maybe there is a way to handle this in practice so that it doesn’t hurt you too much, or something.  Such a theory would be very interesting, but as far as I know it doesn’t exist, and it would have to exist for us to begin talking about how a finite being could faithfully represent its implicit knowledge in a prior.

(This is a human problem in the sense that you could make a machine which would lack all this implicit knowledge.  That machine would not have this problem, but it would know less than we do, so we’d be throwing away information if we tried to imitate it.)

notthedarklord42

Yet you implicitly know infinitely many facts about them (about each chain, in fact, of which there are infinitely many).  There’s no way to sit down and think enough beforehand that all of the obvious-nesting information has been dumped out into an explicit representation (and that representation would take infinite space anyway).

Now, maybe there is a way to handle this in practice so that it doesn’t hurt you too much, or something.

This sounds like a natural continuity/limits problem. It does seem like there could be infinite nesting like this, and that you do know information about each step of the chain. However, I’m not sure this necessarily needs infinitely many facts to describe, perhaps an overarching fact could sum them up, or the facts get ‘smaller’ as the chain does, so that together they form a finite total fact. Thinking about the obvious-nesting information sounds very much like taking a limit.

The geographic example has very literal continuity, with larger and smaller regions of the US. I’m actually quite surprised there isn’t such a theory already! Hypothesis space, even when infinite, is continuous, and that makes a big difference.


On a separate note, I’m not convinced that we couldn’t make do with a model where we only consider a finite universe, with discrete rather than continuous space. That would mean you could not take infinitely many different regions of the US. And it would mean that only finitely many events could possibly occur in a given time period, which intuitively seems like saying there will only be finitely many such different chains of hypotheses to worry about.

While it seems a bit artificial at times, I don’t think it’s too unreasonable to allow a theory like this to only cover finite cases, not when the finite case can approximate the infinite case arbitrarily closely. Then it seems we could reasonably represent our priors. 

lostpuntinentofalantis

I am a bad pun blog and I endorse this message as elaborating on my “eh it probably converges” intuition earlier.

I think we can afford to agree to disagree unless @nostalgebraist can help me intuition pump this a bit further on why doing the subset enumeration problem doesn’t (eventually) converge.

I will say that this substantially downgraded my belief that Bayes is complete; there is much more work to be done, and I think it’s totally reasonable to call out the “unfounded intuition” parts of *the bayes memeplex* from the more proper Edwin and Eliezer’s Excellent Adventure canon.

Source: nostalgebraist bayes cw effortpost baited and outbayesed
nostalgebraist

bayes: a kinda-sorta masterpost

nostalgebraist

I have written many many words about “Bayesianism” in this space over the years, but the closest thing to a comprehensive “my position on Bayes” post to date is this one from three years ago, which I wrote when I was much newer to this stuff.  People sometimes link that post or ask me about it, which almost never happens with my other Bayes posts.  So I figure I should write a more up-to-date “position post.”

I will try to make this at least kind of comprehensive, but I will omit many details and sometimes state conclusions without the corresponding arguments.  Feel free to ask me if you want to hear more about something.

I ended up including a whole lot of preparatory exposition here – the main critiques start in section 6, although there are various critical remarks earlier.

Keep reading

lostpuntinentofalantis

This isn’t convincing to me (and I guess everything of this genre isn’t convincing to me) because, like, it seems to me that the infinite hypothesis thing is just a problem for every kind of thinking?
You can claim that frequentist tools only work in limited domains or whatever, but in my mind all you’ve done is swept the “oh no what if I didn’t think of relevant hypothesis??!??” problem into the “well yeah you’re going to get burnt by this if you use it out of bounds”.

To (ab)use the tool analogy, it turns out that all human made tools cannot survive in the middle of a supernova, and yes you’re technically correct that all the omnitool fanboys have been overselling the utility of omnitool usage in Exotic Space Environments, but the fact that all the non-omnitools have warnings about “cannot be used in supernovae” is not going to convince me that omnitools don’t exist, or are necessarily worse in all cases.

nostalgebraist

If you’re talking about Section 7, I’m not just saying that “there might be relevant hypotheses you hadn’t thought of,” I’m saying that it’s really hard to encode what you do know in a prior without throwing away some information.

In jadagul’s examples with the different regions in 2100, you already know (before you think about any of it) that those statements have a certain logical implication structure.  But you only start thinking about each relation as the relevant statement is brought to your attention.  Like, if you ask someone those questions in a non-monotonic order, they’ll have to take care to squeeze some probabilities inside others they’ve already stated, and this will make things clearly depend on the order of asking.  (In my example, the person said “94.5%” because they know they needed something between 94 and 95, even though they were giving whole-number answers at first, and would have given a whole number answer to the intermediate case if asked about it first.)

(BTW I once actually asked these questions sequentially to a rationalist meetup group as a way of making this point)

So the problem isn’t “your knowledge is finite” but “you can’t encode exactly what you know (and nothing else) in a prior, or at least I know of no way to do it.”

You could say this is just another thing warning that should go on the label, but it suggests that we’re actually using the wrong representation for our prior knowledge, and so we have a “garbage in, garbage out” type problem: Bayes is somehow failing to capture what we know, and we don’t (AFAIK) have any bounds or guarantees on what problems this will or won’t cause.  Whereas in the frequentist procedures, we can at least describe what it would look like for a human to use them correctly, and guarantee certain things for that human.

lostpuntinentofalantis

I don’t think the fact that humans are bad at thinking up logical implications is a very strong argument against bayes, in the same way that “But Harold, you said you loved Chocolate earlier!” is an argument against preferences.

So, I will agree that there’s this non-monotonic thing. This is indeed a very good point against using Bayes as a mental tool! I am not disagreeing with that!

What I do disagree with is the idea that it’s ipso facto problematic. I think the correct way to do this is throw out your first estimate as a preliminary one, and then use the other logical implication questions as a way to generate a battery of knowledge in a kinda organic fashion. To use the original “California succession” thing, let’s say I think it’s unlikely, so I throw out 98% as my likelihood, then some else asks me the “USA still together” so I also generically throw out 98% but A HA!!!!!! THIS SEEMS WRONG, because the set of situations involving the US together but California leaving seems I dunno small or whatever, so I end up adjusting the probabilities as,  repeating until I’ve thought of all “relevant” probabilities.

But logically speaking isn’t this troublesome? Isn’t it terrible that in theory an adversary can choose a sequence of questions which allows them to set my probabilities? Well, not really. My claim is that thoughts of these logical implication things provide information because humans are really bad at accessing all the information they have, and that, yeah sure if the adversary controls how a person accesses their information, of course the person is screwed? So you hope that people have good internal “implication generating”  machinery, such that by the time that they have worked through a bunch of subset questions, they have dumped out all relevant information, and the ordering effects are washed out.

Which is a much more elaborate way of saying “guys stop throwing out random probabilities and sticking to them if you don’t have good intuition/facts doing cognitive work aaaaaaaahh”

I guess I can agree that nothing I said above is specifically motivated by Bayes, except for this vague feeling of “well, shit it turns out I’m actually really bad at incorporating all relevant information” and I think it’s really just unavoidable.

bayes cw effortpost bayes of our lives
nostalgebraist

bayes: a kinda-sorta masterpost

nostalgebraist

I have written many many words about “Bayesianism” in this space over the years, but the closest thing to a comprehensive “my position on Bayes” post to date is this one from three years ago, which I wrote when I was much newer to this stuff.  People sometimes link that post or ask me about it, which almost never happens with my other Bayes posts.  So I figure I should write a more up-to-date “position post.”

I will try to make this at least kind of comprehensive, but I will omit many details and sometimes state conclusions without the corresponding arguments.  Feel free to ask me if you want to hear more about something.

I ended up including a whole lot of preparatory exposition here – the main critiques start in section 6, although there are various critical remarks earlier.

Keep reading

lostpuntinentofalantis

This isn’t convincing to me (and I guess everything of this genre isn’t convincing to me) because, like, it seems to me that the infinite hypothesis thing is just a problem for every kind of thinking?
You can claim that frequentist tools only work in limited domains or whatever, but in my mind all you’ve done is swept the “oh no what if I didn’t think of relevant hypothesis??!??” problem into the “well yeah you’re going to get burnt by this if you use it out of bounds”.

To (ab)use the tool analogy, it turns out that all human made tools cannot survive in the middle of a supernova, and yes you’re technically correct that all the omnitool fanboys have been overselling the utility of omnitool usage in Exotic Space Environments, but the fact that all the non-omnitools have warnings about “cannot be used in supernovae” is not going to convince me that omnitools don’t exist, or are necessarily worse in all cases.

bayes cw effortpost bayes of wrath

Oh. Goku is definitely liberal. Wants to never work and get hand outs for everything…

Vegeta is far more conservative, believes in Sanctitty of Marriage, called Bulma a lewd women when she aaa being herself.

Hard to say about the Half Saiyans.

Although Mirai No Trucks is DEFINITELY conservative. Reason for that is simple. After he killed 17 and 18, he was instrumental in rebuilding the world. Liberalism can only destroy, never build, so that was definitely a capatalist society.

quotes shitpost checkmate shounens death cw anime cw
phenoct

Kabbalah of Strings

empresscelestia

The day the sky cracked open and Archangel Uriel’s machinery was revealed to the world was a humbling day for most natural scientists. Suddenly, they were confronted with the fact that their descriptions of reality were but crude caricatures of the world.

There was, however, one notable exception.

String theory, whose connection to physical reality was tenuous at best, flourished in the kabbalistic era.

“YES, THAT’S DEFINITELY ON THE RIGHT TRACK.” the archangel boomed when asked about how gravitation was implemented consistently in his machinery. “TO BE HONEST, I JUST NEVER GOT THE PURELY BOSONIC CASE TO WORK.“ He gave pause, a crowd of disheveled physicists hanging on his words.

“But none of the anomaly and tachyon-free fermionic strings give us anything resembling particle physics! We want to describe the world!” a voice called out from the crowd. Uriel looked contemplative. “THE WORLD IS EMBODIED BY THE SEPHIRAH MALKUTH WHOSE GEMATRIA IS THE PERFECT NUMBER 496. IT IS ALSO THE REAL DIMENSION OF THE LARGEST EXCEPTIONAL LIE GROUP E8.” the angel’s voice boomed, fear and anticipation sweeping over the crowd. “IF YOU WISH TO UNDERSTAND THE WORLD, YOU’LL HAVE TO TACK-E-ON!”

A string is not unlike a serpent. To ascend the body of Adam Kadmon from the foot of Malkuth it needs something to bind and couple to. This kabbalistic role is played by the Kalb-Ramond B field. Including it in the calculation is like remembering the force provided by the tree itself. But not all paths are permitted and not all strings are pure enough to ascend the Pillar of Mercy. In the end, the condition in which all sicknesses are cured, all anomalies cancelled, is exactly that the theory’s gauge group have dimension 496. Thus was born the Type I, HO, and E8 x E8 Heterotic strings. To many, however, this sudden multiplicity was a problem.

The physicists returned to the Archangel with this question. Uriel smiled, “ANSWER THIS: WHY ARE THERE SO MANY FACETS OF GOD?”. Unlike before, this comment was met with a wave of puzzled faces. Uriel looked around uncomfortably. “LOOK, I DON’T WANT TO JUST SPOIL THE WHOLE THING. JUST USE YOUR BRANES.”

The theories of superstrings must be formulated in ten dimensions, in correspondence with the ten sephirot. To understand them, because of their mathematical difficulty, requires great wit. Perhaps it was no surprise that Edward Witten was the one who pierced through the veil and just really clarified everything for us mortals.

“It’s always been somewhat of a mystery why we compactify on Calabi-Yau threefolds, six dimensional manifolds, but we know the shattering of the vessels involved seven sephirot.” he spoke, standing in front of a powerpoint with exactly those words on it and the equation “6=7?”

“Essentially, what I’d like to argue today, is that to understand the relation between the unshattered vessels and our four dimensional space-time, we should regard time as an aspect of the sephirah Da’at. The nature of Da’at is the unity of the sephirot and, though it is often conflated with Keter, for these purposes we should regard it as a distinct aspect of Divinity.”

“With that in place, I think it’s clear we should view our superstring theories as being various aspects, or limits, of a unified eleven dimensional theory.” he spoke and the slide changed to a crudely drawn Tree of Life with various mathematical procedures connecting each sephirah.

At the top of the Tree sat this new mystical theory, M-theory for short, in correspondence with the sephirah Keter. It is sublime and incomprehensible to man – and yet all other sephirot or theories flow from it. Further developments, such as the equivalence between M-theory and the Matrix Model with only a time direction, would only solidify the correctness of this picture.

To descend is to distort the theory, pulling and wrapping its parts in dizzying geometric arrangements. “Each connection is still a duality! Though it may be hard to see the true equivalence of these theories it is no harder than understanding the whole of God by any of His aspects.”

The true heart and mind of the superstring theories, type IIA, takes the seat of Binah. The feminine nature of Binah can be understood from its origin from the mother theory, M-theory, wrapped on a circle – the metaphorical ‘circle of life’.

By twisting, T-dualizing, the warmth of Binah we find the fiery judgement of Gevurah. This is the type IIB theory at its strongest coupling. However, to be consistent with God’s forgiving nature, we find a highly non-trivial constraint on the type IIB string; self-duality. By S-dualizing we find, again, the type IIB string, but this time at weak coupling and in correspondence with the sephirah of Hod, the foot of Adam Kadmon, which ensures good placement when using his hand.

The relation between Netzah – the other foot of Adam Kadmon, which represents conquering and victory over one’s problems – and the subdued and steady foot of Hod, is merely a change of orientation. Similarly, when one takes an “orientifold” of the type IIB theory, one obtains the type I theory, folded across the body.

At the heart of superstring theory, and the heart of Adam Kadmon, beats supergravity. Beneath the complexity and opacity of Keter lies simple beauty; the sephirah Tiferet. This explains the role of supergravity in capturing the beauty and lessons of M-theory, converting them to all the other superstring theories. It is a splendid image – a hologram, if you will – of the foundation Yesod from which God has made the world.

External image

Yesod is the channel through which the light above is delivered to the world, Malkuth. It is the superstructure of the world and is thus a supersymmetric quantum field theory. Thus we arrive at perhaps the most powerful discovery yet; the kabbalistic connection to the (2,0) theory of M5-branes. In fact the connection runs deeper. Yesod represents Kabbalah. Or rather, it represents our understanding of Kabbalah. The meaning of Kabbalah is that, to understand God we must understand the world, and to understand God is to know his True Name. The rest, as they say, is just commentary.

Thus to understand the (2,0) theory is to know the Shem HaMephorash.

Perhaps it is not surprising then that the (2,0) theory has no known name; no Lagrangian description. There is still hope however! Other Names can be understood as simplifications, shortenings and rearrangements, of Shem HaMephorash. Thus by studying simplifications –compactifications, if you will – of the (2,0) theory we may discover other Holy Names. Hence we derive the kabbalistic correspondence between them and superconformal field theories. The most famous of which, the N=4 super Yang-Mills, is identified with the tetragrammaton.

For a while, this ‘string-string’ correspondence generated a lot of excitement within the applied kabbalah world and the first class of sacred theories, class S, were quickly enumerated. As it turns out, understanding supersymmetric gauge theories is considerably harder than simply speaking potential names.

Though interest has waned, it is believed that strings will play a role in the end of days. We should recall the Archangel’s words: to understand the world we must include E8. What this means remains a mystery.

-Prof. Juan Maldacena, Institute for Advanced Kabbalistic Study

External image
Source: empresscelestia
nostalgebraist

on MIRI’s “Logical Induction” paper

nostalgebraist

When I first saw this paper, I said it looked impressive, and seemed a lot more substantial than MIRI’s other work, but I never really looked at it in much detail.  In the past week I’ve been having an extended conversation with a friend about it, and that spurred me to read it more closely.  I now think it’s much less impressive than it seems at first glance.

It occurred to me that the criticisms I made to my friends might be of wider interest, so I’ll write a post about them.

Summary:

The authors state something they call the “logical induction criterion,” meant to formalize a kind of ideal inference.  To satisfy this criterion, an inference procedure only needs to have certain asymptotic properties in the infinite-time limit.  Rather than being sufficient for good inference but too strong for practical computation (as the paper suggests), the criterion is too weak for good inference: it tolerates arbitrarily bad performance for arbitrarily long times.

The easiest way to see why this is problematic in practice is to consider the criterion-satisfying procedure constructed in the paper, called LIA.  Speaking very roughly, LIA makes a countably-infinite list (in unspecified/arbitrary order) of all the “mistakes” it could possibly make, and at discrete time n, does a brute force search for a set of beliefs which avoids the first n mistakes in the list.

Depending on the ordering of the list, LIA can make an arbitrarily bad mistake for an arbitrarily long time (some mistake has to go in slot number 3^^^3, etc.)  Nonetheless, LIA can be proven to asymptotically avoid every mistake, since for every mistake there is some time N at which it begins to avoid it.  Thus, LIA enjoys a very large number of nice-sounding asymptotic properties, but it converges for very different reasons than most algorithms one is used to hearing about: rather than converging to nice properties because it moves toward them, it simply exploits the fact that these properties can only fail in countably many ways, and ticks off those ways one by one.

Thus, LIA is like an “author-bot” which generates all possible character strings in some arbitrary order, and at each step consults a panel of readers to weigh in on its best work so far.  One could argue that this bot’s “asymptotic best work” will have any aspect of writerly brilliance imaginable, but that does not mean that the bot satisfies some “ideal writing criterion,” and indeed we’d expect its output in practice to lack any aspects of writerly brilliance.  (LIA differs from author-bot in that it has an objective, if very slow, way to find its “best work so far.”  But garbage in means garbage out.)

More detail under the cut

Keep reading

lostpuntinentofalantis

Thanks for doing this.

Do you think that this type of overstatement is due to incompetence? (The statement you made about ensemble methods seems particularly bad to me, and would be understandable in light of the “MIRI doesn’t actually have machine learning experts, just math dudes” criticism)

effortpost ai risk MIRI is a friend who is good at cooking

On asymmetry in defense against sociopath humans and Unfriendly AI

[Epistemic Status: No evidence for, just vague intuitions]

This post was inspired by a recent SlateStarCodex comments thread where someone explicitly stated that they believe that Unfriendly AI wouldn’t necessarily be a problem, because we already have defenses against sociopaths and antisocial humans (e.g. crime syndicates, foreign governments, griefers in Counterstrike). Namely, that whatever we use to defend ourselves against bad human actors would apply equally well against bad AI actors.

I don’t think this is true.

First off, I think this denies the idea that culture as a social technology is doing work to reduce the number of bad actors. The potential pool of bad actors is much smaller than it could be because people who otherwise would be good Trojan creators instead hope to be hired by Apple, or they view math as a higher calling, or they are pro social enough that they become a white hat instead. Or that our systems are mostly focused on punishment for humans, in that most of techniques are social engineering ones for finding criminals, like looking on known black hat forums for slipups (the Silk Road guy) or by cross correlating background information (there was a KrebsonSecurity blogpost where the the Mirai bot creator was traced via his taste in anime). These work because of information leakage that is specific for humans and not general.

Now I’m not saying that no general techniques like network analysis or various types of fingerprinting won’t work, but that I haven’t seen any acknowledgement that human specific defenses are a Thing at all, or that there is anything other than fully general defense techniques against bad actors. I’m willing to change my mind if, for example a security researcher comes and tells me “yeah, we use sophisticated math techniques that are fully general defenses way more often than social engineering techniques and those SE techniques are rare rather than common“.

Second off, it’s not clear to me that actions humans do when antisocial are uncorrelated and varied enough that they cover the entire gamut of actions that an AI would. For example, if terrorists really do try and pick the most effective terror actions instead of the actions that allow terrorist organizations to propagate (see gwern’s post on how terrorism is not about terror). Or if it turns out the reason why there aren’t supervillain corporations is because CEOs that are successful also happen to be prosocial, because brains are built as signalling machines and morals are correlated with good decisions due to neural architecture reasons. It would seem very surprising to me if it turns out that despite a lot of things correlating with each other in psychology (like dark triad traits and being poor) that these traits turn out to be fully general as opposed to a quirk of how our brains work. I don’t think there’s existing evidence (other than showing me that human brains are far more diverse and heterogeneous at high enough rates that it’s unlikely that there are any possible minds “in the gaps”).

Lastly, and this is just a feeling, but the people talking about defense just don’t seem to have a security mindset. Where instead of thinking “what could go wrong” they think “what goes right so that I can stop thinking about this problem”. They aren’t really like security researcher/funnyman James Mickens who thinks that security online is like being choked out constantly by an unseen adversary. Or that if you’re getting attacked by the Mossad, THERE IS NOTHING YOU CAN DO YOU’RE GONNA GET MOSSADED ON. These AI skeptics don’t seem to know that security isn’t just cryptography, that “side channel attacks” exists or “the protocol is secure but all implementations are flawed” happen more often than we’d like. There’s this general attitude which treats defenses deployed long after exploit discover as victories and characterizes large scale ransomware attacks as “well that isn’t an example of world domination!″. I admit that this is a mixture of a straw man and tone argument, but I want people engaging with me to know my antipathy is partially impression based and to forgive me for my stubbornness if we do engage.

All this is not to say that I think they’re necessarily wrong (I do agree with them that e.g.

@slatestarscratchpad

‘s claim that “getting a Jupiter brain and cracking RSA with quantum computing” is vacuous). Just that these claims seem undersourced compared to how confident they seem to be.

effortpost ai risk x risk serious cw not anime