Scrapy
pyspider

curl: FTP(S), GOPHER(S), HTTP(S), SCP, SFTP, TFTP, TELNET, DICT, LDAP(S), MQTT, FILE, POP3(S), IMAP(S), SMB(S), SMTP(S), RTMP, RTSP and WS(S). Wget only supports HTTP(S) and FTP
BeautifulSoup
pandas
re
lxml