A good way to think about it is to consider board, card and tabletop games:
A board game has rules: who goes first, what happens when you land on a space; and state: whose turn is it, where the pieces currently are, how much each player has. So does a card game like Magic the Gathering, and a tabletop role-playing game like Dungeons & Dragons has a lot more.
Do these games have glitches too? They do, in a manner of speaking! What is a glitch? Let's say it's a vastly unexpected interaction that comes about because of some combination of states according to the rules of the game.
(Let's discount non-repeatable glitches that come about by freak specific circumstance. Knocking over the board, stealing money, pulling out the cartridge...)
If there was some quirk of the rules in Monopoly that, if strictly followed, meant that a player could take infinite turns, or collect $200 GO money every turn, that'd be a glitch. Dungeons & Dragons is vastly more complicated than Monopoly - by 'some interpretation of the rules', a super-powerful character like Pun-Pun can be developed. This and similar characters are made by looking at the rules of the game, coming up with some assumptions about how the model of the game works, and finding through trial-and-error or exhaustion the limits where the rules no longer apply. Look at the conditions you have to follow to make that character - doesn't it resemble the totally weird sets of button presses and conditions the player has to input to execute a Super Mario 64 glitch technique?
The RPG and Board & Card Games Stack Exchanges are all-but dedicated to the resolution of unusual cases and combinations of states and rules within games like this.
The 'advantage' of these non-electronic games above computer games is that the rules are being interpreted by humans. Humans have the advantage of understanding the context of the world, and can be flexible with the rules - every non-electronic game is played by mutual agreement and understanding of the players involved. If the rules listed a human player character's standard running speed in miles-per-second instead of miles-per-hour, you'd understand that it was a mistake and perhaps a misprint, unless they were The Flash. If a rule is blatantly unfair or unrealistic in context, even by design, you can strike it out.
The glitches of Super Mario 64 are more striking compared to a tabletop game because everything happens in real-time: resolution of the rules is automatic and instant, and the rest of the game continues as normal around it. There's no referee to appeal to - if 'the rules' say Mario can do it, and XYZ happens as a result, that's what happens.
How are things like this found? By accident, through observation and repetition. In video games there's also a sneaky way that we don't (yet) have in real life: examination of the code.
It's like a scientific discipline - when something unusual happens, you construct a mental model of what is happening and why, what the rules and state are and how they're represented, attempt to repeat the conditions you've laid out, and ultimately determine the conditions by which something does and doesn't happen. The game Super Mario 64 is an imperfect realisation of the intended design, either through compromise or error.
Why would 'the rules' be wrong with respect to the intent?
A perfectly stable game would have the following properties.
- The model considers every possible interaction of every possible set of objects under all circumstances.
- There's a sensible response to each of these circumstances.
- It's impossible to violate the assumptions built into the model.
- The model is an accurate description of the intended simulation.
When one of these conditions isn't met, your simulation won't match your expectations, and you'll have a glitchy game.
If you're already carrying an object, can you pick up another? What if you're sliding down a ramp while crouched and collide with a Wing Cap? Does collision still apply during the 'putting the hat on' animation? What happens if you get hit while opening a door? Is there a maximum speed if you continue move using a specific maneuver? Somebody has to think of all these things. If a limit isn't explicitly established, it doesn't exist. It might implicitly be bounded as a consequence of other rules, but if those rules can be evaded, then speed, location, score, lives, time are all up for grabs.
The N64 doesn't use floating-point, and it wouldn't be truly accurate even if it did. The fixed-point units used in the simulation represent the units of length, speed and acceleration into discrete increments. It's an imperfect representation. It looks nice to play because these increments are small enough to simulate smooth motion and friction, but the density of these increments comes at a cost of range. In the dumb, mechanical world of the CPU, large values wrap over to large negative values. These values are then combined with other values in certain circumstances, which may mean that Mario can interact with certain parts of the world and not others.
In the early 3D era, which Super Mario 64 certainly was, there's no tested engines to build off. Everything was made completely from scratch, such as the definition of the world, and everything inside it. This even includes some basic math operations, like finding the distance between two points and the direction perpendicular to a surface. These routines will also have their limits due to the ranges used. Calculating anything in 3D uses multiplications. Multiplying two numbers in fixed-point gives a result with the combined precision of both, at the cost of range.
Being complex and 3D means it's considerably more likely to have strange interactions, not less!
