This is a quote from "The Hobbit." They are real, but rarely used words. One can say that they are "made up" words, but many words in our language began as made up words (especially slang and any new terminology, such as email or Internet). Bebother and confusticate use parts of "everyday" words, so the reader can guess their meaning, just as you did.
bebother = to bring trouble upon
http://en.wiktionary.org/wiki/bebother
confusticate = to confound or confuse
http://en.wiktionary.org/wiki/confusticate