The biggest giveaway is that it doesn't have an internally consistent model of what it's trying to do.
For example, I've once asked it to "prove that first order logic is not complete", a statement which is false, given that every logic textbook will give you a proof for why first order logic is complete. ChatGPT apparently knew this too, because it happily reproduced that result but somehow thought that it was a valid answer for what I asked for (the valid answer would have been "I can't prove this because it's wrong").
I can't reproduce this exact result right now (in fact when I tried, it just proved something slightly different, which was correct but not exactly what I asked for). But I was able to similarly confuse it by asking it to prove that the equation x^3+y^3=z^3 has solutions. That this is true is immediately apparent by x=y=z=0, but ChatGPT replies with:
> The equation x^3 + y^3 = z^3 represents a special case of Fermat's Last Theorem, which states that there are no solutions to the equation x^n + y^n = z^n for any integer values of n greater than 2.
> For the case of n = 3, the equation x^3 + y^3 = z^3 has infinitely many solutions. One such set of solutions is known as Pythagorean triples, where x, y, and z are positive integers.
> A Pythagorean triple is a set of three positive integers (x, y, z) that satisfy the equation x^2 + y^2 = z^2, which is equivalent to (x^2)^(3/2) + (y^2)^(3/2) = (z^2)^(3/2). By raising both sides to the power of 3/2, we get x^3 + y^3 = z^3.
> For example, the Pythagorean triple (3, 4, 5) satisfies 3^3 + 4^3 = 5^3 (27 + 64 = 125).
This answer is just confused on so many levels:
- It quotes back Fermat's Last Theorem at me (as indeed I hoped it would), but that theorem only applies to positive integer solutions and nowhere did I specify that constraint.
- If the Theorem did apply, then it would be a proof that such solutions don't exist. So ChatGPT has no internal understanding of how a theorem it quotes relates to a specific question, it just parrots off things that look vaguely similar to the input.
- Then, it just tells me what Pythagorean Triples are, which is hilarious, because those are the solutions to x^2+y^2=z^2 - and not what I asked. It then tries to somehow transform Pythagorean triples into (non-integer) solutions of my equation (which doesn't work), and then doesn't even apply the transformation to its own example (and the calculation is just... wrong).
The problem IMO is not that ChatGPT gives a wrong answer, it's that its answer isn't even internally consistent.
For example, I've once asked it to "prove that first order logic is not complete", a statement which is false, given that every logic textbook will give you a proof for why first order logic is complete. ChatGPT apparently knew this too, because it happily reproduced that result but somehow thought that it was a valid answer for what I asked for (the valid answer would have been "I can't prove this because it's wrong").
I can't reproduce this exact result right now (in fact when I tried, it just proved something slightly different, which was correct but not exactly what I asked for). But I was able to similarly confuse it by asking it to prove that the equation x^3+y^3=z^3 has solutions. That this is true is immediately apparent by x=y=z=0, but ChatGPT replies with:
> The equation x^3 + y^3 = z^3 represents a special case of Fermat's Last Theorem, which states that there are no solutions to the equation x^n + y^n = z^n for any integer values of n greater than 2.
> For the case of n = 3, the equation x^3 + y^3 = z^3 has infinitely many solutions. One such set of solutions is known as Pythagorean triples, where x, y, and z are positive integers.
> A Pythagorean triple is a set of three positive integers (x, y, z) that satisfy the equation x^2 + y^2 = z^2, which is equivalent to (x^2)^(3/2) + (y^2)^(3/2) = (z^2)^(3/2). By raising both sides to the power of 3/2, we get x^3 + y^3 = z^3.
> For example, the Pythagorean triple (3, 4, 5) satisfies 3^3 + 4^3 = 5^3 (27 + 64 = 125).
This answer is just confused on so many levels:
- It quotes back Fermat's Last Theorem at me (as indeed I hoped it would), but that theorem only applies to positive integer solutions and nowhere did I specify that constraint.
- If the Theorem did apply, then it would be a proof that such solutions don't exist. So ChatGPT has no internal understanding of how a theorem it quotes relates to a specific question, it just parrots off things that look vaguely similar to the input.
- Then, it just tells me what Pythagorean Triples are, which is hilarious, because those are the solutions to x^2+y^2=z^2 - and not what I asked. It then tries to somehow transform Pythagorean triples into (non-integer) solutions of my equation (which doesn't work), and then doesn't even apply the transformation to its own example (and the calculation is just... wrong).
The problem IMO is not that ChatGPT gives a wrong answer, it's that its answer isn't even internally consistent.