This repository was archived by the owner on Jan 31, 2021. It is now read-only.
This repository was archived by the owner on Jan 31, 2021. It is now read-only.
Error API response does not contain a "content_html" field #47
Open
Description
I tried to download the comments from this video We tried Android 11!
which has 3,330 comments. and using node js LTS Version 12.18.0 running under Debian. the Scraper start fetch the comments but after some time it throws an error
API response does not contain a "content_html" field
the only solution I found is to try five or ten times in hope one time it will not throw an error
and the parameters I used are youtube-comment-scraper -f json -o comments.json 05X0RRmUtE0
and also the error happens with videos with more than 1,000 comments. videos with comments under 1,000 will not throw an error
Activity
hissatupassenger commentedon Jun 16, 2020
same.... I guess this Error raised in phil's this library https://github.com/philbot9/youtube-comments-task
He obviously recognize this issue.
maybe few people facing this error.
I wanna scrape huge amounts of comments for NLP.
who can fix these error?
Gimme mercy, phil-san.
Yakabuff commentedon Jun 21, 2020
got this error as well
maliayas commentedon Jun 30, 2020
Same error on my end. Happens randomly. I've made a feature request here, which would help overcome this issue: #48
spiralofhope commentedon Jul 1, 2020
I am getting this error consistently on all videos now, and I do not receive any data at all.
Perhaps something has changed on YouTube's end.
Example:
https://www.youtube.com/watch?v=ecIWPzGEbFc
hcanalesmx commentedon Jul 3, 2020
I am having the same problem
itzmeharsha commentedon Jul 7, 2020
me too facing same isue
bharathichezhiyan commentedon Jul 8, 2020
I am also facing the same issue.
PiyumithaNirman commentedon Jul 11, 2020
@philbot9 can you tell me what is the solution to prevent this error "API response does not contain a "content_html" field"
hissatupassenger commentedon Jul 11, 2020
my current solution
https://github.com/dddat1017/Scraping-Youtube-Comments
KamalSilava commentedon Jul 12, 2020
@philbot9, please help me. there is an error. previously it worked correctly. after the may 27 update It didn't work. always gives this error "API response does not contain a "content_html" field"
LakshanSaranga commentedon Jul 12, 2020
I want to get some youtube comments but your cli program doesn't work now. @philbot9 please give me a solution to get the data. error is "API response does not contain a "content_html" field"
PiyumithaNirman commentedon Jul 13, 2020
this is the alternative solution.it works for me.thank you @hissatupassenger
skanjINCIDE commentedon Jul 13, 2020
Hi! I have the same error in all the videos of a Youtube channel. Have somebody found a solution? Thank you!
spiralofhope commentedon Jul 13, 2020
https://github.com/dddat1017/Scraping-Youtube-Comments
I'm glad this is a solution for some of you, but to me it's next to worthless without replies.
I'd still rather continue using this program, but I created an issue for that other one:
dddat1017/Scraping-Youtube-Comments#1
skanjINCIDE commentedon Jul 13, 2020
I saw it reading the issue, but I need replies too... I hope @philbot9 can find a solution, I'm not a JS expert and I didn't find a solution yet.
andrscyv commentedon Jul 14, 2020
A couple of weeks ago Youtube's frontend was updated. It used to return plain html that already contained data such as search results and , I'm guessing, comments as well. Now it returns html with the navbar but no data on the body, I've seen people found the data on embeded script tags on the first html response but afterwards data is loaded with ajax calls. I noticed the change on my own library for scrapping search results and on other similar python packages.
It may be posible to locate the endpoints from which the ajax calls are being made and get the data using them.
barbaragribeiro commentedon Jul 14, 2020
Here is a simple and working (python) script to scrape comments AND replies, but unfortunately it has less features, e.g., no timestamp: https://github.com/egbertbouman/youtube-comment-downloader
Perhaps it can give an ideia of how to solve this current problem, since the request is apparently made to the same url, but the script above uses this url to scrape only some comments from livestreams... to download all comments and replies from regular videos, another (similar) url is used, which the author calls "old url" because it's somehow related to the old youtube API. Anyway, I'm not into js so I couldn't find out what's going with Phil's code or what has changed in the response and possible modifications to the request to solve this issue.
spiralofhope commentedon Jul 15, 2020
Thanks for this; it works for me.
It's nowhere near a replacement tool for philbot9's though, but I do hope it provides insight into any necessary changes.
FloPinguin commentedon Aug 21, 2020
Thanks, it's a good replacement for now