Suppose we grant the premise that human intelligence is doing something over and above computing functions in the Turing sense (I'm personally not convinced that anything in the universe can do that, but that's beside the point). That still doesn't preclude the possibility of artificial intelligence. Why? Because, you can't claim with a straight face that human brains are the only arrangements of matter that physics will grant general intelligence. That would just be too insanely arbitrary. There are a lot of possible arrangements of matter. You would expect, a priori, for a tiny proportion of those arrangements of matter to be smart. But a small fraction of an enormous, combinatorially large number is still combinatorially large.
Note that consciousness isn't necessary or relevant for the purposes of this argument. We only care about what the AI is capable of, whether it's experiencing qualia while it's taking over the world or whatever is immaterial.
Once you grant that there are other possible configurations of matter that could be generally intelligent, the only other thing you have to grant is that humanity will one day be capable of manufacturing some of them. Assuming a wide variety of possible arrangements, it would seem unlikely that there was some fundamental law of the universe that prevented us from manufacturing intelligence.
But what characterizes those other arrangements of matter?
What is their energy consumption? What are their switching speeds? Can cognition grow unbounded, and if so, why? Can any brain improve on its own process of cognition? If other arrangements of matter perform cognition equally well, why don't we see them in the independent evolution of life with wildly different structures for everything else?
There is far too much woo and not enough rigorous thinking around the notion of AI superintelligence, singularity, or whatever other incantation people care to use.
> why don't we see them in the independent evolution of life
Because evolution is a very weak optimisation process. It has very limited materials to work with - everything has to be made of meat. The only changes it can make are incredibly indirect ones - mutations in DNA base pairs, leading to different proteins being manufactured or some such (I'm not a biologist, obviously) leading to subtle changes in the makeup of the resulting organism. Can you imagine how difficult engineering would be through a medium like that? It'd be like trying to perform brain surgery with a sledgehammer. And it's not even purposefully optimising for anything, let alone intelligence. The changes all happen completely at random, and the ones that manage to reproduce stick around.
Moreover, the fact that such a weak process as natural selection can produce something as smart as us, should really make you consider that humans, with optimisation power far superior to natural selection, can produce something far smarter.
In regards to self improvement, my argument wasn't really about that. I was trying to demonstrate the viability of at least human level intelligence.
I don't know how likely intelligence explosion scenarios are. The possibility of an AI of human level intelligence being able to improve itself seems obviously plausible - if humans can engineer intelligence, then human level AI can.
Whether it can do that rapidly is far more questionable. However, even if you think the likelihood of fast improvement cycles is quite low, it still warrants some serious thought, given the size of the potential payoffs and downsides.
Note that consciousness isn't necessary or relevant for the purposes of this argument. We only care about what the AI is capable of, whether it's experiencing qualia while it's taking over the world or whatever is immaterial.
Once you grant that there are other possible configurations of matter that could be generally intelligent, the only other thing you have to grant is that humanity will one day be capable of manufacturing some of them. Assuming a wide variety of possible arrangements, it would seem unlikely that there was some fundamental law of the universe that prevented us from manufacturing intelligence.