Looking at mempool numbers is pointless. There is no "the mempool". Every node has a different mempool. The mempool size numbers reported by various sites are largely worthless. With default Bitcoin Core settings, a node's mempool size is almost always 200-300 MB in size.

What clearly is a problem is when transactions get stuck. However, this is mainly an issue with wallets that don't estimate fees well. People who are seeing stuck transactions are generally paying 15 sat/B or less, while my Bitcoin Core suggests paying at least 30 sat/B with current network conditions. The network is constantly changing -- you can't just pick one fee and stick with it, or even pick one fee and plan to update it every now and then.

The higher required fee right now is probably due to an attack aimed at clogging the queue with high-fee transactions. In general: To make it impossible for anyone to get a transaction through with a fee less than $x per kB, an attacker needs to pay roughly 1000 kB * $x per block. So to raise the required fee to $1/kB requires that the attacker burns about $1000 in fees per block, for example.

Ideally, wallets should notice this and say, "There's a spam attack going on. You can either pay this above-average fee to outbid the attacker or you can wait a while." Also, wallets should support RBF and/or CPFP to allow people to adjust fees after transaction creation. But in reality, most wallets just pay some constant or slowly-updating fee, which is what's causing most of the trouble.

If the max block size is doubled, then the cost to raise the required fee to a certain level is also roughly doubled. So when SegWit increases the effective max block size to ~2 MB, this attack will become more expensive. But if wallets use a fixed fee, then it still won't require all that much money to fill blocks with slightly-above-average fees. Fee estimation is required in any case.

Furthermore, I've seen a number of people saying stuff like, "If there's an attack, then the max block size should be able to increase arbitrarily to absorb it." This is a very misinformed statement. Obviously both miners (in some cases) and people who make transactions want larger blocks, but the larger blocks are on average, the more expensive it is to run a full node in terms of bandwidth, CPU usage, and disk space. And full nodes do not (and, most experts believe, cannot reasonably) get any reward for supporting increased transaction volume. So in other words, block size is a negative externality suffered by full nodes due to the actions of miners and transaction-makers, and running full nodes is a positive externality provided by full nodes to the benefit of everyone who uses Bitcoin (Bitcoin is absolutely insecure if not enough of the economy is backed by full nodes). And there's no reason to think that miners or transaction-makers would voluntarily keep blocks small enough -- since it's a common goods problem, neither has the right incentives. Bitcoin's solution to this issue is for full nodes to enforce a max block size small enough to allow full nodes to operate with a reasonably low cost. Removing the limit would make spam attacks even worse, since spammers would be able to directly attack Bitcoin's volunteer full nodes (and therefore Bitcoin's decentralization) rather than just increase fees temporarily.

Yes, the max block size limits the total number of transactions, and if everyone who is currently transacting (right now including spammers) was willing to pay an infinite amount of money to keep transacting, then fees would go to infinity. But that's not how it would actually work. In reality, rising fees make the reasonable minimum on-chain transaction value smaller and smaller. Therefore, people who are using on-chain transactions to transfer just a few dollars get pushed out in a rising fees scenario, making room for people who want to transfer larger values -- eventually, an equilibrium is reached. This pricing-out of small-value transactions has already happened: no one tries to use Bitcoin for moving a few cents of value nowadays. It is not in itself a good thing that anyone gets priced out in this way -- it'd be nice if we could costlessly do all the transactions that everyone ever wanted on-chain -- but at least some degree of this is unavoidable and survivable.