You've stumbled into a strange linguistic argument. Plants have no mammary glands, but the simple fact that English usage of milk has been applied to products that look like and are used like dairy milk would be enough to redefine the word.

For comparison, the etymology of meat comes from words used more generally to refer to food or substantial food (going back to german and middle english, i don't know when the divergence happened), which is why mincemeat still exists as a word, usually describing something mostly made of nuts (or nut meat, alternatively). In many ways, you could make the opposite argument of the milk one that you can't assume meat means animal skeletal muscle (and more so offal) and must be applied, at least, to all protein heavy foods, and potentially more.

Moral of the story, if the language is already doing it, don't fight it.