It's not a bathtub curve. Your low-level and "high"-level tasks are the same thing: Probabilistic text generation.
It's not reasoning about your code, nor about the explanation it gives you.
> AI has no concept of "these four parts have to be closely connected, building a whole".
AI can't think. It doesn't create an internal model of the problem given, it just guesses. It fails at all these "middle" tasks because they require abstract reasoning to be correct.
It's not "whether or not it thinks" its "whether n-dimensional vector multiplication in an intricate embedding space is thinking or not".
Which on the surface is easy to knee-jerk a "no" too, but with a bit more pondering you realize that however the brain thinks must be describable by math, and now you need to carve out what math is "thinking" and what math is "computation".
Or just be a duelist and attribute it to a soul or whatever.
> however the brain thinks must be describable by math
Roger Penrose believes that some portion of the work brains are doing is making use of quantum processes. The claim isn't too far-fetched - similar claims have been made about photosynthesis.
That doesn't mean it's not possible for a classical computer, running a neural network, to get the same outcome (any more than the observation that birds have feathers means feathers are necessary to flight).
But it does mean that it could be that, yes you can describe what the brain is doing with math ... but you can't copy it with computation.
it feels self-evident that computation can mimic the brain. as a result, it's difficult to argue this line much further. to say the brain is non-computable is to assert the existence of a soul, in my opinion.
A lot of things feel self-evident then turn out to be completely wrong.
We don't understand the processes in the brain well enough to assert that they are doing computation. Or to assert that they aren't!
> say the brain is non-computable is to assert the existence of a soul, in my opinion
I don't believe in souls, but the brain might still be non-computable. There are more than two possibilities.
If it is the case that brains are doing something computable that is compatible with our Turing machines, we still have no idea what that is or how to recreate it, simulate it, or approximate it. So it's not a very helpful axiom.
> We don't understand the processes in the brain well enough to assert that they are doing computation. Or to assert that they aren't!
We absolutely do know enough about neurons to know that neural networks are doing computation. Individual neurons integrate multiple inputs and produce an output based on those inputs, which is fundamentally a computational process. They also use a binary signaling system based on threshold potentials, analogous to digital computation.
With the right experimental setup, that computation can be quantified and predicted down to the microvolt. The only reason we can't do that with a full brain is the size of the electrodes.
> I don't believe in souls, but the brain might still be non-computable. There are more than two possibilities.
The real issue is neuroplasticity which is almost certainly critical to brain development. The physical hardware the computations are running on adapts and optimizes itself to the computations, for which I'm not sure we have an equivalent.
dendrocentric compartmentalization, spike timing, bandpass in the dendrites, spike retiming etc... aren't covered in the above.
But it is probably important to define 'computable'
Typically that means being able that can take a number position as input and output the digit in that location.
So if f(x) = pi, f(3) would return 4
Even the real numbers are uncomputable 'almost everywhere', meaning choose almost any real number, and no algorithm exists to produce it as f(x)
Add in ion channels and neurotransmitters and continuous input and you run into indeterminate features like riddled basins, where even with perfect information and precision and you can't predict what exit basin it is in.
Basically look at the counterexamples to Laplace's demon.
MLPs with at least one hidden layer can approximate within an error bounds with potentially infinite neurons, but it can only produce a countable infinity of outputs, while biological neurons, being continuous input will potentially have an uncountable infinity.
Riddled basins, being sets with no open subsets is another way to think about it.
We can write code that writes code. Hell even current LLM tech can write code. It's at least conceivable that a artificial neural network could be self-modifying, if it hasn't been done already.
(b) all of classical physics is computable therefore
(c) thinking relies on non-classical physics.
(d) In addition, he speculatively proposed which brain structures might do quantum stuff.
All of the early critiques of this I saw focussed on (d), which is irrelevant. The correctness of the position hinges on (a), for which Penrose provides a rigorous argument. I haven't kept up though, so maybe there are good critiques of (a) now.
If Penrose is right then neural networks implemented on regular computers will never think. We'll need some kind of quantum computer.
> If Penrose is right then neural networks implemented on regular computers will never think.
I disagree that that is necessarily an implication, though. As I said before, all that it implies is that computational tech will think differently than humans, in the same way that airplanes fly using different mechanisms from birds.
Part of Penroses's point (a) is that our brains can solve problems that aren't computable. That's the crux of his brains-aren't-computers argument. So even if computers can in some sense think, their thinking will be strictly more limited than ours, because we can solve problems that they can't. (Assuming that Penrose is right.)
I wonder if LLM's have shaken the ground he stood on when he said that. Penrose never worked with a computer that could answer off the cuff riddles. Or anything even remotely close to it.
So the trouble with this argument is that there is no evidence whatsoever that the brain can solve problems that a turing machine can't.
There's none. No one has been able to formulate a problem in a reasonable way that a computer algorithm can't be devised to solve it that people can solve.
It is basically a bunch of handwaving nonsense like the tripartite nature of god(father, son and holy spirit...)
Searle's chinese room argument is slightly better, but is still ultimately a pile of horseshit. From an external point of view we cannot distinguish between a room full of people who do not speak chinese but can translate it following rigorous instructions and tables and a room full of qualified chinese translators. For all external purposes the black boxes are equivalent except that you can take a chinese translator out of the room and still use them to translate chinese without the rigorous instructions and reference material in the room.
There is no good philosophical argument against Strong AI. It is a bunch of quasi-religious, humans are special because we say so wishy-washy nonsense.
(a) doesn't hold up because the details of the claim necessitate that it is a property of brains that they can always perceive the truth of statements which "regular computers" cannot. However, brains frequently err.
Penrose tries to respond to this by saying that various things may affect the functioning of a brain and keep it from reliably perceiving such truths, but when brains are working properly, they can perceive the truth of things. Most people would recognize that there's a difference between an idealized version of what humans do and what humans actually do, but for Penrose, this is not an issue, because for him, this truth that humans perceive is an idealized Platonic level of reality which human mathematicians access via non-computational means:
> 6.4 Sometimes there may be errors, but the errors are correctable. What is important is the fact is that there is an impersonal (ideal) standard against which the errors can be measured. Human mathematicians have capabilities for perceiving this standard and they can normally tell, given enough time and perseverance, whether their arguments are indeed correct. How is it, if they themselves are mere computational entities, that they seem to have access to these non-computational ideal concepts? Indeed, the ultimate criterion as to mathematical correctness is measured in relation to this ideal. And it is an ideal that seems to require use of their conscious minds in order for them to relate to it.
> 6.5 However, some AI proponents seem to argue against the very existence of such an ideal . . .
Penrose is not the first person to try to use Gödel’s incompleteness theorems for this purpose, and as with the people who attempted this before him, the general consensus is that this approach doesn't work:
Not going to comment on the thinking part, because who knows what that means, but there's evidence that transformers do in fact learn predictive models of their input space. There's a cool blog post on this here: https://www.neelnanda.io/mechanistic-interpretability/othell...
I should clarify, "of the problem given" refers to the problem given in a prompt.
As you note, transformer (and indeed, most ML) models do create a "world model". They're useful for 'specific' intelligence tasks.
The problem for general tasks lies in their inability to create specific models. To stick with the board game example: The model can't handle differently shaped boards, or changes to the rules.
I could ask a human and a chess-trained AI system to, for a given chess board state and piece, what places that piece can move to. Both have their model of chess.
But if I then ask, "With the rule change that the pawn can always move two spaces", the AI cannot update their model. Where for the human this would be trivial. The human can substitue in new logic rules, the AI cannot.
And that is very core of what's required for generalized logic and "thinking" in the way most tasks require it. What's so troublesome about current generative AI is that it's trained to be extremely general (within the domain of text generation), so their internal models aren't all that good.
Ask an LLM the chess problem above and you might even get a good answer out, but it doesn't generalize to all such chess problems, especially not more complex ones.
The paper on Othello is of course a very limited model, useful because it's simple enough to study and complex enough to have interesting behaviour.
But the general takeaway is that this is evidence that large transformers like GPT, which are trained to predict text, are fully capable of developing emergent models of parts of that input space whenever it is convenient for minimising the loss function. In practice this means that GPT may have internal models of the semantics of human dialogue that are sophisticated enough for it to get by in the enormous variety of prediction tasks we throw at it.
I agree with you that it's likely these internal models aren't very detailed (for the reason you wrote - they're very general). The linked blog actually talks about this at the end - an OthelloGPT trained to be good at Othello rather than just able to play legal moves ends up with a worse board model. Presumably because it needs to "invest" more in playing better moves. But if you agree with the blog's take then this is just a matter of scale and training. And whether it's possible or not for them to develop models capable of complex tasks like strategy games with shifting rules is certainly not something you (or anyone else for that matter) can say with certainty right now.
Edit: I should clarify we're using "model" in two senses here. There's the actual transformer model, but what I and the blog are talking about is specific weights and neurons _inside_ these transformers that learn to predict complex features of the input space (like legal moves and board updates in the case of OthelloGPT). These develop spontaneously during the training process, which is why they are so interesting. And why they are not really analogous to the "ML models" you refer to in your first two paragraphs.
If you're going to suggest something you think an LLM can't do I think at the very least as a show of good faith you should try it out. I've lost count of the number of times people have told me LLMs can't do shit that they very evidently can.
I explicitly say that LLMs could do it in my response. As a show of good faith you should try reading the entire comment.
Yes, I'm using simple examples to demonstrate a particular difference, because using "real" examples makes getting the point across a lot harder.
You're also just wrong. I did in fact test, and both GPT 3.5 Turbo and 4o failed. Not only with the rule change, but with the mere task of providing possible moves. I only included the admission that they may succeed as a matter of due diligence, in that I cannot conclusively rule out they can't get the right answer because of the randomization and API-specific pre-prompting involved.
> "For chess board r1bk3r/p2pBpNp/n4n2/1p1NP2P/6P1/3P4/P1P1K3/q5b1 (FEN notation), what are the available moves for pawn B5"
I did read your entire comment, and that is what prompted my response, because from my perspective your entire premise was based on LLMs failing at simple examples, and yet despite admitting you thought there was a chance an LLM would succeed at your example, it didn't seem you'd bothered to check.
The argument you are making is based on the fact that the example is simple. If the example were not simple, you would not be able to use it to dismiss LLMs.
I am not surprised that GPT 3.5 and 4o failed, they are both terrible models. GPT4-o is multimodal, but it is far buggier than gpt-4. I tried with claude 3.5 sonnet and it got it first try. It also was able to compute the moves when told the rule change.
> It's not reasoning about your code, nor about the explanation it gives you.
We don't really know what "reasoning" is. Presumably you think humans reason about code, but humans also only have statistical models of most problems. So if humans only reason probabilistically about problems, which is why they still make mistakes, then the only difference is that AI is just worse at it. That's not an indication it isn't "reasoning".
“We don’t know how we do it, so we can’t say this isn’t how we do it” isn’t a valid argument.
We may not know exactly how we reason, but we can rule out probabilistic guessing. And even if that is a part of it, we’re capable of far more sophisticated models. We can recurse and hold links. We can also make intuitive leaps that aren’t quite built on probability.
> “We don’t know how we do it, so we can’t say this isn’t how we do it” isn’t a valid argument
Yes it is, assuming we don't know of any specific things that "this" literally can't do but that we can. Which we currently don't, we merely have suspicions.
> We may not know exactly how we reason, but we can rule out probabilistic guessing.
No we can't.
> even if that is a part of it, we’re capable of far more sophisticated models.
Yes, but that would be a difference of degree not of kind. This is what scaling proponents have been saying, eg. that scaling does not appear to have a limit.
> We can also make intuitive leaps that aren’t quite built on probability.
I don't think we have evidence of that. "Intuitive leap" could just be a link generated from sampling some random variable.
But it’s not replicating results that a human would give you.
Since it’s not giving the same type of results, then it’s not doing the same thing. If anything, LLMs have definitively ruled out probabilistic guessing as the model for human intelligence.
Even now, you’re trying to force LLMs onto human intelligence. Insisting it is despite it not delivering the results. And I’m sure you believe if we just fire up another few million gpus, we’d get there. But we’ll just get wrong faster. LLMs don’t produce new, they just remix old
> Even now, you’re trying to force LLMs onto human intelligence
I'm not forcing anything, I'm specifically refuting the claims that we know that LLMs are not how humans work, and that LLMs are not reasoning. We simply don't know either of these, and we definitely have not ruled out statistical completion wholesale.
Also, I don't even know what you mean that LLMs are not giving the same types of results as humans. An articulate human who was hired to write a short essay on given query will produce what looks like ChatGPT output, modulo some quirks that we've forced ChatGPT to produce via reinforcement learning.
> AI can't think. It doesn't create an internal model of the problem given, it just guesses.
These "AI can't think" comments pop up on every single thread about AI and they're incredibly tiresome. They never bring anything to the discussion except reminding us how inherently limited these AIs are or whatever.
Someone else already replied with the OthelloGPT counter-example that shows that, yes, they do have an internal model. To which you reply that the internal model doesn't count as thinking or abstract reasoning or something, and... like, what even is the point of bringing that up every discussion? These assertions never come with empirical predictions anyway.
GP's comment was interesting because it pointed at a specific area of what LLMs are bad at. A thousandth comment saying "LLMs can't think or do abstract things (except in all the cases where they can but those aren't really thinking)" doesn't bring any new info.
> It doesn't create an internal model of the problem given, it just guesses.
It's not entirely true. They often use some sort of memory/scratch-pad to keep a context other than previous tokens. This recent exploit lets you see claude's default prompt that have some references to this system.
It's not reasoning about your code, nor about the explanation it gives you.
> AI has no concept of "these four parts have to be closely connected, building a whole".
AI can't think. It doesn't create an internal model of the problem given, it just guesses. It fails at all these "middle" tasks because they require abstract reasoning to be correct.