what's to stop sentience from being a "bug?"

I knew you were going to say this. I had a paragraph written out actually, but I deleted it cause I didn't want to finish it... but here it goes!

First I'll deal with your comment about sentience spontaneously emerging...obviously this could not happen today or at any point in the near or even far future because the complexity of a system would have to be massively more complex than the 'sentient being' that emerged. I would argue that the computer would have to be doing processing on the level of simulating every atom in our galaxy + all its interactions in real time for it to even be possible given an infinite amount of time of the system running.

But thats for "spontaneous"... what I think you mean is that sentience could emerge in a system in a similar way it emerged in life. Maybe some code that starts out simply replicating itself with small changes every time and changing in a process similar to evolution. There have actually been papers written about this. What information systems are missing is an environment that selects for intelligence. Unlike life where intelligence was a massive boost to survivability... in information systems this would be a move away from standard state and the processes would be terminated (so it almost selects against it). Now maybe we would could simulate an environment that PURPOSEFULLY tries to guide algorithms into changing themselves and growing (and computer scientists have done this on a basic level, but most say we still lack the processing power to simulate a world with enough selective pressures to evolve a hard AI from scratch... and this leads to what companies are trying to do with neural nets.

Soft AI is real and exists today, but the problem is that the "thinking" part of these machines always remains unchanged... they only have areas in their scope where heuristic algorithms are allowed free rein. Google's deepmind is a good example to use to compare what this would take.... YES its incredible at learning its one task that humans have guided it to do... but not enough of it changes. What would need to happen is to create a system that allows ALL parts of the system to evolve and attempt to accomplish "Intelligence". This would also imply that we could correctly create a "game" that the machine solving would lead to the emergence of a hard AI. We could then instantiate the winner out of this system of billions or trillions of failed "minds" as a hard AI.

TLDR for that long rambling: It would take way more processing power and programming than you think.

Now for the philosophy!

Yes the only reason I know beyond a reasonable doubt that you are sentient is because I am sentient... and you as a human are very similar to me. What you're describing is the turing test and yes it has its flaws. I think my "test" is essentially a turing test with a little knowledge of what going on under the hood because I think we will very soon create a machine that can pass the turing test, but still not be sentient. It would only seem like it is and you would have to probe very deep passed its "scope" to reveal the anomalies.

I totally agree with you about animals. We really don't know... but I'm telling you that you can't lump computers in with animals. You have to remember that we design every aspect of a computers mind and even in neural nets we design the pressures and environment they learn in... and we know the full limits of they scope we've allowed them to learn in. I fully believe that some animals are sentient even if they can't express it, but for computers I don't need them to express it because I understand them much more deeply than I understand animals or even you... and maybe even how my own mind works.

We will build a hard ai at some point. But I guarantee you we will know full well what we're building and will not be surprised when it tells us it is sentient. I wanna be optimistic and say 15 years.