Skip to content
This repository was archived by the owner on Jan 31, 2021. It is now read-only.
This repository was archived by the owner on Jan 31, 2021. It is now read-only.

Error API response does not contain a "content_html" field #47

Open
@cpuuntery

Description

@cpuuntery

I tried to download the comments from this video We tried Android 11!
which has 3,330 comments. and using node js LTS Version 12.18.0 running under Debian. the Scraper start fetch the comments but after some time it throws an error
API response does not contain a "content_html" field
the only solution I found is to try five or ten times in hope one time it will not throw an error
and the parameters I used are youtube-comment-scraper -f json -o comments.json 05X0RRmUtE0
and also the error happens with videos with more than 1,000 comments. videos with comments under 1,000 will not throw an error

Activity

hissatupassenger

hissatupassenger commented on Jun 16, 2020

@hissatupassenger

same.... I guess this Error raised in phil's this library https://github.com/philbot9/youtube-comments-task
He obviously recognize this issue.
maybe few people facing this error.
I wanna scrape huge amounts of comments for NLP.
who can fix these error?
Gimme mercy, phil-san.

Yakabuff

Yakabuff commented on Jun 21, 2020

@Yakabuff

got this error as well

maliayas

maliayas commented on Jun 30, 2020

@maliayas

Same error on my end. Happens randomly. I've made a feature request here, which would help overcome this issue: #48

spiralofhope

spiralofhope commented on Jul 1, 2020

@spiralofhope

I am getting this error consistently on all videos now, and I do not receive any data at all.

Perhaps something has changed on YouTube's end.

Example:
https://www.youtube.com/watch?v=ecIWPzGEbFc

youtube-comment-scraper --stream -- ecIWPzGEbFc | tee output.json
✕ API response does not contain a "content_html" field
hcanalesmx

hcanalesmx commented on Jul 3, 2020

@hcanalesmx

I am having the same problem

itzmeharsha

itzmeharsha commented on Jul 7, 2020

@itzmeharsha

me too facing same isue

bharathichezhiyan

bharathichezhiyan commented on Jul 8, 2020

@bharathichezhiyan

I am also facing the same issue.

PiyumithaNirman

PiyumithaNirman commented on Jul 11, 2020

@PiyumithaNirman

@philbot9 can you tell me what is the solution to prevent this error "API response does not contain a "content_html" field"

hissatupassenger

hissatupassenger commented on Jul 11, 2020

@hissatupassenger
KamalSilava

KamalSilava commented on Jul 12, 2020

@KamalSilava

@philbot9, please help me. there is an error. previously it worked correctly. after the may 27 update It didn't work. always gives this error "API response does not contain a "content_html" field"

LakshanSaranga

LakshanSaranga commented on Jul 12, 2020

@LakshanSaranga

I want to get some youtube comments but your cli program doesn't work now. @philbot9 please give me a solution to get the data. error is "API response does not contain a "content_html" field"

PiyumithaNirman

PiyumithaNirman commented on Jul 13, 2020

@PiyumithaNirman

my current solution
https://github.com/dddat1017/Scraping-Youtube-Comments

this is the alternative solution.it works for me.thank you @hissatupassenger

skanjINCIDE

skanjINCIDE commented on Jul 13, 2020

@skanjINCIDE

Hi! I have the same error in all the videos of a Youtube channel. Have somebody found a solution? Thank you!

spiralofhope

spiralofhope commented on Jul 13, 2020

@spiralofhope

https://github.com/dddat1017/Scraping-Youtube-Comments

Note, only the main comments will be extracted, not any of the replies that may be under the comment.

I'm glad this is a solution for some of you, but to me it's next to worthless without replies.

I'd still rather continue using this program, but I created an issue for that other one:
dddat1017/Scraping-Youtube-Comments#1

skanjINCIDE

skanjINCIDE commented on Jul 13, 2020

@skanjINCIDE

I saw it reading the issue, but I need replies too... I hope @philbot9 can find a solution, I'm not a JS expert and I didn't find a solution yet.

andrscyv

andrscyv commented on Jul 14, 2020

@andrscyv

A couple of weeks ago Youtube's frontend was updated. It used to return plain html that already contained data such as search results and , I'm guessing, comments as well. Now it returns html with the navbar but no data on the body, I've seen people found the data on embeded script tags on the first html response but afterwards data is loaded with ajax calls. I noticed the change on my own library for scrapping search results and on other similar python packages.

It may be posible to locate the endpoints from which the ajax calls are being made and get the data using them.

barbaragribeiro

barbaragribeiro commented on Jul 14, 2020

@barbaragribeiro

Here is a simple and working (python) script to scrape comments AND replies, but unfortunately it has less features, e.g., no timestamp: https://github.com/egbertbouman/youtube-comment-downloader

Perhaps it can give an ideia of how to solve this current problem, since the request is apparently made to the same url, but the script above uses this url to scrape only some comments from livestreams... to download all comments and replies from regular videos, another (similar) url is used, which the author calls "old url" because it's somehow related to the old youtube API. Anyway, I'm not into js so I couldn't find out what's going with Phil's code or what has changed in the response and possible modifications to the request to solve this issue.

spiralofhope

spiralofhope commented on Jul 15, 2020

@spiralofhope

Here is a simple and working (python) script to scrape comments AND replies, but unfortunately it has less features, e.g., no timestamp: https://github.com/egbertbouman/youtube-comment-downloader

Thanks for this; it works for me.

It's nowhere near a replacement tool for philbot9's though, but I do hope it provides insight into any necessary changes.

FloPinguin

FloPinguin commented on Aug 21, 2020

@FloPinguin

Here is a simple and working (python) script to scrape comments AND replies, but unfortunately it has less features, e.g., no timestamp: https://github.com/egbertbouman/youtube-comment-downloader

Thanks, it's a good replacement for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @spiralofhope@maliayas@Yakabuff@hissatupassenger@bharathichezhiyan

        Issue actions

          Error API response does not contain a "content_html" field · Issue #47 · philbot9/youtube-comment-scraper-cli