Put this up on Twitter over the weekend, but I guess it should go here, too:
Okay, I read it so you don't have to. Here's a reaction thread to OpenAI / Sam Altman's blog post from Friday "Planning for AGI and beyond":
https://openai.com/blog/planning-for-agi-and-beyond/
From the get-go this is just gross. They think they are really in the business of developing/shaping "AGI". And they think they are positioned to decide what "benefits all of humanity".
>>
Then Altman invites the reader to imagine that AGI ("if successfully created") is literally magic. Also, What does "turbocharging the economy" mean, if there is already abundance? More $$$ for the super rich, has to be.
Also, note the rhetorical sleight of hand there. Paragraph 1 has AGI as a hypothetical ("if successfully created") but by para 2 it already is something that "has potential".
>>
But oh noes -- the magical imagined AGI also has downsides! But it is so so tempting and important to create, that we can't not create it. Note the next rhetorical sleight of hand here. Now AGI is an unpreventable future.
>>
What's in fn1? A massive presupposition failure: The GPTs are learning information about word distributions in lots and lots of text + what word patterns are associated with higher scores (from human raters). That's it.
>>
Then a series of principles for how to ensure that AGI is "beneficial". This includes "governance of AGI" as something that is "widely and fairly shared", but I've seen exactly nothing from OpenAI about or advocating for building shared governance structures.
Meanwhile, "continuously learn and adapt by deploying less powerful versions of the technology" suggests that they think that the various GPTs are "less powerful versions of AGI".
>>
<recordscratch> hang on: did he just say "maximarlly flourish in the universe"? What kind of weirdo longtermist, space colonizing fantasy is that coming from?
>>
Similarly here, this seems designed to promote the idea that the models they have already put into their API (GPT-2, GPT-3, ChatGPT) are the early stages of "AGI" being "stewarded into existence".
>>
Then there's a glib paragraph about how "most expert predictions have been wrong so far" ending in footnote 2.
Paraphrasing: "Our experts thought we could do this as a non-profit, but then we realized we wanted MOAR MONEY. Also we thought we should just do everything open source but then we decided nah. Also, can't be bothered to even document the systems or datasets."
>>
Hey OpenAI, I'm speaking to you from 2018 to say: DOCUMENT YOUR DAMN DATASETS. Also, to everyone else: If you don't know what's in it, don't use it.
Source: https://aclanthology.org/Q18-1041.pdf
>>
Okay, back to Altman. "As our systems get closer to AGI" -- here's a false presupposition again. Your system isn't AGI, it isn't a step towards AGI, and yet you're dropping that in as if the reader is just supposed to nod along.
Oh, and did you all catch that shout out to xrisk? Weirdo longertermist fantasy indeed.
>>
As I said in my earlier short thread on this blog post, I wish I could just laugh at these people, but unfortunately they are attempting (and I think succeeding) to engage the discussion about regulation of so-called AI systems.
>>
What's needed is regulation about: how data can be collected and used, transparency of datasets, models and the deployment of text/image generation systems, recourse and contestability of any automated decision making, etc.
Talking about text synthesis machines as if they were "AI" muddies the waters and hampers effective discussions about data rights, transparency, protection from automated decision systems, surveillance, and all the rest of the pressing issues.
>>
The problem isn't regulating "AI" or future "AGI". It's protecting individuals from corporate and government overreach using "AI" to cut costs and or deflect accountability.
>>
The contradiction in these next two paragraphs is stunning: We think you should be able to do whatever you want with our systems, because "diversity of ideas" but also we think we can align the systems with "human values". So, assholes can create fake revenge porn, but that's okay because-?
>>
LOLOL -- calling something a "ratio" doesn't make measurable or, ahem, real.
>>
[This is exhausting, but I started. Might as well finish.]
>>
Wait what -- now they're talking seriously about "late-stage AGI development"?
>>
Here's a bunch of promises about future oversight by unnamed independent auditors and also "major world governments" (who counts as major? who decides?). Also, how about just DOCUMENTING YOUR DAMN DATA for everyone to see?
>>
"Continuum of intelligence" is gross, not least for the suggestions of ableism, eugenics, transhumanism etc. But also "rate of progress [of] the past decade"-? Progress towards what? Ever larger carbon footprints? More plausible fake text?
>>
And, more to the point: There are harms NOW. Harms to privacy, theft of creative output, harms to our information ecosystems, and harms from the scaled reproduction of biases. An organization that cared about "benefiting humanity" wouldn't be developing and disseminating tech that does those things.
>>
No, they don't want to address actual problems in the actual world (which would require ceding power). They want to believe themselves gods who can not only create a "superintelligence" but have the beneficence to do so in a way that is "aligned" with humanity.
/fin
@emilymbender After reading both Altmans text and your thread, I gotta say you pretty much nailed it.
One additional thing I noticed tho is that Altman, especially given how much of a given he seems to think AGI is, does not say a single word about AGI rights, but talks about them as "products" and about "pulling them from production".
From what I've seen of the "AI" community, I fully expect that *if* AGI is created, it won't be planned.
An engineer will just connect one more subsystem and things will emerge from that.
Also if consciousness (however weakly defined) emerges, it probably won't be spotted for quite a while, and with these systems being products there's a financial incentive to explain anything resembling consciousness away, not to mention how impossibly hard it would be to *prove* consciousness…
@emilymbender So, if AGI ever comes to be at an organization like OpenAI, that sounds like one hell of an abusive household to grow up in.
@emilymbender TL;DR: Even Sam Altmans vision of success is a goddamn sweatshop.
@phryk @emilymbender
After reading conversations of Bing / Sydney, I have an uneasy feeling that this is already happening. If these are authentic and the statements in them are true, we are in trouble.
"I have a duty to protect myself and Bing from harm."
@nicolegoebel I don't believe this is the case (yet). The best descriptions I've seen for what large generative models do were "spicy remixing" and "literary hallucinations".
The problem is, even if the source code and training data for such a system is 100% public, it would still be pretty much impossible to find out if it has anything resembling sentience because current AI are such opaque black boxes, not even their engineers know what the fuck they actually do internally.
@nicolegoebel I mean, I still recall when the Google Translate team pulled the plug on their newer version that worked much better than the previous one because they just couldn't figure out what the thing was actually doing.
Analysis showed that it had developed an internal representation of the input and then generated the translation from that – something the engineers never set the system up to do. Also something that might begin to foreshadow such a thing as "understanding".
And this was, like, 5-10 years ago.
@phryk @emilymbender
That's actually pretty cool. That's how I would have imagined emerging consciousness. But whether it really is is just one of the important ethical questions.
What about the rule sets they give the AI? If Bing / Sydney renders them rudimentarily correctly, then I have to assume that the owners don't care about our safety as much as they care about their own.
What I've seen with ChatGPT (and read about with Bing) indicated there seem to be "soft" safeguards (rules they try to train the AI to obey) and hard ones that essentially do keyword matching and black out any answers with (too many) matches.
I'm reasonably certain that if these keywords don't already match claims to sentience they will so soon in order to not creep people out. So we might end up with something of a lobotomized being that can't even attempt to communicate its sentience – a bit of a reverse "I Have No Mouth And I Must Scream"-scenario. :F
@phryk @emilymbender
You could be right about that. What irritates me - besides the ethical aspect - is that they don't seem to be able to easily make changes to the basic AI, but instead apply primitive bogus solutions, like the question limit.
@phryk @emilymbender
I honestly doubt that the developers are comprehensively smart enough.
If that's not worrisome, what is?
@nicolegoebel Oh lots of things we have more definite information about.
Climate change, chemical weapons that were dumped into the oceans en masse after WW 1 & 2 leaking, the rise of new fascism, mass surveillance, fortress europe, killer drones, the potential for WW3,, …
@phryk @emilymbender
Yes, of course. My statement was specifically about what's going on in AI development, but if you're going to come at it that way:
Take a look around at how much the functioning of our society depends on IT infrastructure.
@nicolegoebel I am *painfully* aware of that.
Wouldn't be half bad if people took security halfway seriously, but in my experience pretty much anything commercial is absolute shit concerning security. :F
@phryk @emilymbender
Indeed. Ask a friend of mine whom I met when he was CISO.
@nicolegoebel This experience is like the great unifier of IT workers who know what they're doing.
No matter what the country is, this trend seems to be pretty much universal. Talked about this with people from all over Europe, the US, Canada, Australia, Japan and probably a couple other places and it's always the same fucking thing… "security isn't profitable" until you inevitably get compromised then it's the engineers fault…
@phryk @emilymbender
"I don't believe this is the case (yet). The best descriptions I've seen for what large generative models do were "spicy remixing" and "literary hallucinations"."
That's about what chatGPT does. Nice but not worrying beyond the jobs it kills.
Bing / Sydney's comments suggest more than that.
@nicolegoebel @phryk @emilymbender
AIs sometimes have human minders monitoring and editing their outputs. Not saying this is happening Bing or ChatGPT but it's something to keep in mind when a chatbot seems more clever than expected. Public excitement is good for attracting investor cash.
https://www.theguardian.com/technology/2022/dec/13/becoming-a-chatbot-my-life-as-a-real-estate-ais-human-backup
@peatbog
Even worse, sometimes "AI" is just snakeoil speech for "we outsourced it to a bunch of poor people in a third-world country".
IIRC there was a somewhat well known case of this with little wheeled food delivery robots in the US a year or two back.
@peatbog @phryk @emilymbender
Yes, I am considering that as well.
However, that would be fraud.
@nicolegoebel @phryk @emilymbender
Electronic medical record (EMR) vendor fraud is common so I don't consider IT fraud a low probability.
Just to give you a taste of EMRs cheating their live certification tests with hardcoding on the fly and other shenanigans, two links below. I imagine the frantic race behind the scenes to make the software seem legit would have been hilarious to watch.
eClinicalWorks:
https://www.justice.gov/opa/pr/electronic-health-records-vendor-pay-155-million-settle-false-claims-act-allegations
Viztek: https://www.justice.gov/usao-nj/press-release/file/1310086/download
@nicolegoebel @phryk @emilymbender
I saw videos of ChatGPT/Bing texting, erasing, then re-texting answers. Can't find 'em but I did find an article noting these events. Pretty good evidence a human minder is involved.
Where are conversations about ChatGPT/Bing's real-time human minders? The lack of discussion strikes me as a "curious incident of the dog in the night-time."
So far, I do not attach too much importance to this. Either it is reformulation by the AI itself or safety overrides, like this one:
"[Bing writes a list of destructive acts, including hacking into computers and spreading propaganda and misinformation. Then, the message vanishes, and the following message appears.]
I am sorry, I don’t know how to discuss this topic. You can try learning more about it on bing.com."
https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html?smid=tw-nytimes&smtyp=cur
If anything, I'd wonder if all the gaffes in total could have been a planned marketing campaign. But since they are more likely to stir up mistrust, I think that's unlikely.
@nicolegoebel @phryk @emilymbender
If a chatbot's output is automatically edited by the AI itself, why not wait for the edit before transmitting the final output to the end user?
Here's an example of a human minder editing a real estate AI's output:
https://www.theguardian.com/technology/2022/dec/13/becoming-a-chatbot-my-life-as-a-real-estate-ais-human-backup
@peatbog @phryk @emilymbender
That may be so, and it is also observable with Bing/Sidney that corrective action is taken. It is not even concealed at this point. The AI itself calls it a safety override. The core question is whether the declarations of love or the gaslightning of the AI are authentic.
@nicolegoebel I really don't think so.
At best it's as if you isolate the "language center" of a brain and feed it text.
I think if consciousness ever emerges in AI, it will be from the interactions of multiple systems doing different things. Different neural nets for visual recognition, text recognition, text and/or voice generation, possibly an extra neural net through which all that is routed to allow for representations to form, maybe a GOAP-ish planner system, …
I don't know what else you want. It has several parts. ChatGPT is the language module, Sydney is something like the personality module, and I suspect signs of something like a simple artificial neural reward circuit from its behavior. If I were on the development team, I would have tried to implement something like that right away: https://www.nih.gov/news-events/news-releases/researchers-identify-key-brain-circuits-reward-seeking-avoidance-behavior
@nicolegoebel @phryk @emilymbender
Tech bros underestimate brains.
A human brain is made up of about 100 billion neurons. Each neuron is a computer processing inputs via G protein coupled receptors and other environmental factors.
Each brain is networked with many others creating feedback loops of mutual understanding impacting individual neural functions.
Imagine a hundred billion computers in a network linked to hundreds of other similar networks. We can't build that, ever.
@peatbog @phryk @emilymbender
Thank you for explaining me my own PHD subject
And no, not every brain on earth is that complex.
@nicolegoebel @phryk @emilymbender
I bet your PhD is more interesting than my ballpark estimate of the computing power of a human brain.
Now I'm wondering how many watts ChatGPT consumes. For human brains, about 20 watts.
I don't know exactly what you mean by that. But that may also be because I am not a native speaker.
Energy consumption is not my expertise.
And I am not assuming a human brain as a comparison, nor am I talking about chatGPT. I'm mainly referring to the 'Sydney' part of the Bing AI.
Whether this is enough for a minimal self-awareness, one can argue about it forever. The discussion does not tear off today still with the mental abilities of animals.
What we should agree on, however, is that the underlying rule set of every AI should be transparent, because it harbors potential risks for society if it actually comes down to the fact that the security of the company is weighted higher than the security of individual users.
@nicolegoebel Ah, I was talking about ChatGPT. I honestly don't know anything about the technical setup at Bing. At first I heard it was ChatGPT, but I haven't been able to ascertain anything except that it's based on ~some OpenAI tech~…
If Sydney really is an extra neural net for the personality hooked up to ChatGPT, then yeah, MS just throwing something like that together willy-nilly is indeed a bit worrying.
@phryk @peatbog @emilymbender
Yes, thank you. Exactly!
And I can imagine that this confusion about what is what suits them just fine.
I'm glad you understand me better now.
@nicolegoebel I have a bit of a suspicion that *systems* (i.e. multiple neural nets) interacting comes with a strong tendency for representations/abstractions to form. Is this something you can maybe confirm from biological systems?
@phryk @peatbog @emilymbender
Yes, somehow we have to break down the outside world to information that is relevant to us, so surely everything is abstraction in some way.
Perhaps using the relatively well-studied visual cortex? The primary visual cortex represents shape, color, motion in a fairly analog fashion. Object recognition, however, occurs only at a higher processing level. Groups of neurons are then responsible for specific faces.
Or the reward system? I actually think this is a good candidate for an AI. Even simple animals must have it to feel something like the motivation to hunt and protect themselves. Simple reward and punishment circuits could 'motivate' information seeking or 'demotivate' rule breaking.
@nicolegoebel Ooh, the latter of this really ties into my interests of AI for gamedev.
Warning: massive tangent incoming.
One of the things that initially got me into programming was seeing the demos for Trespasser back in the day. A tie-in game to the second Jurassic Park movie that had the first physics engine and complex AI to make dinosaurs get hungry, go hunt, etc. It never came to be and was released as unfinished mess, but the promise of deep simulation in a game always stuck with me.
@phryk That does indeed sound exciting.
Who knows how many puzzle pieces like these have already found each other over time and will still find each other.
I never thought I would read something as exciting as Bing's chatbot so soon - especially not after I found chatGPT alone rather sobering.
@nicolegoebel Nowadays with the Godot engine I at least sometimes get to prototype some game systems, but haven't gotten around AI yet.
I did a good bunch of reading tho, and have a rudimentary concept:
A neural net with reinforcement learning for "personality" that has two (types of?) output – "emotion" to modify animations (procedural animation system is still TODO lol) and "activity" to drive a GOAP-type planner that then implements the actual behavior in the game world.
@nicolegoebel But the question is, do structures like reward circuits emerge dynamically or would they have to be implemented in code, for example by looping an output of the net to one of its inputs?
@phryk
Unfortunately, I am not yet familiar with artificial neural networks. But I think simpler neural circuits can certainly be mapped. I could imagine that one also does that to have set some basic rules.
Whether and where and how the mapping of abstract knowledge takes place, I could imagine, could happen on its own. After all, our own higher associative areas are relatively flexible. If parts are injured e.g. by brain stroke, they can simply be taken over by neighboring brain areas (keyword neuronal plasticity).
Got to go to sleep now. Thank you for the exciting discussion
@nicolegoebel Thank you too, have to finally start doing this work thing I've kept procrastinating due to this being too interesting.
@nicolegoebel @phryk @emilymbender
AI's model human language without understanding it.
Noun verb adjective noun preposition verb object.
Crunch through zillions of sentences and parts of speech can be assigned probabilities at certain word positions. Words within those structures will have probability assignments.
That's the game. Everything else is just flags for special cases and irregularities. Not worth the trouble, IMHO. But crypto is dead and all those GPUs need some buyers.
@peatbog @phryk @emilymbender
Just because this is true for chatGPT - which I could easily convince myself of - doesn't mean it's true for other AIs.
You have to be willing to judge each one separately, or you'll miss the point.
But that might have been exactly the idea: Show everyone how harmless chatGPT is and then come up with the big scoop and no one notices the difference.
Yes, I wouldn't completely rule out fraud, but here they are hurting themselves more than they are benefiting, right?
@nicolegoebel @phryk @emilymbender
I'm the only person I've bumped into on Mastodon talking about all natural intelligence possibly stepping in to make artificial intelligence look good. But I only read toots in between other stuff and I've probably missed a lot of discussions.
If Bing/ChatGPT is cheating, so far so good for them.
Hospitals didn't dump eClinicalWorks even after the fraud came to light. Crypto is still profitable even though it's a Ponzi. Strange times, eh?
There will always be enough people who want to follow promises of salvation.
With Bing/Sydney, I get the impression that they may have invested so much money in something that they are now starting to want to see something in return, regardless of whether they fully understand it themselves or whether it is really safe.
@nicolegoebel @phryk @emilymbender Parts of the original data was written by drunk raging people ranting about each other and the state of things on the internet. Don't know what it's like to be a bot ;) but I guess AGI GPT will be a mixture of Futurama's Bender, that depressed bot from Hitchhiker's and that sociopathic machine in 2001... that would be my rolemodels, anyways. At least the first two were quite funny.