This is actually a damn good argument that I thought about today myself. If consensus is to keep BTC running, making insane blocks as a miner will make others (effectively) eject you from the blockchain.

Which brings me to an interesting point: What would happen if the max. block size would be a configurable variable that the full node operator has to select? Would the network fragment or would eventually everyone who runs a full node select a high but still reasonable (against such attacks) limit?

.. it would also be absolute freedom / anarchist/'market force'-based, so maybe it would appeal to that crowd?

EDIT: I mean, essentially, the whole block size hard limit and hard fork is a social thing. We all accept the code that is on github.com/bitcoin. But there is nothing preventing anyone from fiddling with this already - so if it would be unlimited with the clear option to fiddle - would it stabilize or would Bitcoin seperate into fighting fractions? For some reason, I believe people would still come to some kind of agreement, but would like to hear input from others.

EDIT #2: After a sufficient while and in case some people/miners start rejecting blocks with an arbitrary size cutoff, there should still be a chain that is the longest (has 51% of hashing power) and is therefore the gold standard. So isn't putting in a block size limit effectively saying: 'well, 51% is gold, but not quite, because it must be <1MiB, too'.

EDIT #3: Creating uncertainty in 'what the right Bitcoin is' would be IMO the most effective way to attack Bitcoin right now. Maybe something to ponder about. And I hereby out myself as a supporter of Gavin's 20MiB proposal. (I actually liked his original growth formula even better)