Why did this robot ignore my /robots.txt?
It could be that it was written by an inexperienced software writer. Occasionally schools set their students "write a web robot" assignments. But, these days it's more likely that the robot is explicitly written to scan your site for information to abuse: it might be collecting email addresses to send email spam, look for forms to post links ("spamdexing"), or security holes to exploit.