0

I am trying to scrape some startups data of a site (bestsecret.at) with Puppeteer and when I try to navigate to the next page the Cloudflare waiting screen comes in and disrupts the scraper. Is there a way to bypass it with Puppeteer?

cloudflare capture page

const puppeteer = require('puppeteer-extra'); const StealthPlugin = require('puppeteer-extra-plugin-stealth'); const randomUseragent = require('random-useragent'); const userAgent = randomUseragent.getRandom(); puppeteer.use(StealthPlugin()); (async () => { var argarr = ['--no-sandbox', '--disable-setuid-sandbox']; const browser = await puppeteer.launch({args: argarr}); const page = await browser.newPage(); await page.setUserAgent(userAgent); await page.goto('https://www.bestsecret.at', {waitUntil: 'networkidle2'}); const bodyWidth = await page.evaluate(() => document.body.scrollWidth); const bodyHeight = await page.evaluate(() => document.body.scrollHeight); await page.setViewport({ width: bodyWidth, height: bodyHeight }); await page.type('#login-username', 'USARENAME'); await page.type('#j_password', 'PASSWORD'); await page.waitForTimeout(1000); await Promise.all([ page.click('#login-button'), page.waitForNavigation({waitUntil: 'networkidle2'}), ]); let bodyHTML = await page.content(); console.log(bodyHTML); await browser.close(); })();
4

3 Answers 3

3

Use puppeteer-real-browser. I tried it today and it works without any problems. Only one moment I spotted, it's when you set turnstile: true input fields will lose their focus every second. So you can set it as false.

Sign up to request clarification or add additional context in comments.

2 Comments

How to use extensions with the puppeteer-real-browser?
Use "--load-extension" argument: puppeteer.launch({ ..., args: [ --disable-extensions-except=${pathToExtension}, --load-extension=${pathToExtension}, ] })
2

Things you could try are:

  • use {headless: false} in puppeteer launch (maybe with xvfb)
  • connect through a residential connection (maybe a proxy)

Comments

2

Use puppeteer-extra-plugin-stealth. Note that you will need to use puppeteer-extra rather than just puppeteer for this to work. Found this solution through this article.

Update as of Feb 2025: It appears websites that use advanced Cloudflare protection will prevent this method from working (blocked by captcha, 403 error code).

const puppeteer = require('puppeteer-extra'); const StealthPlugin = require('puppeteer-extra-plugin-stealth'); puppeteer.use(StealthPlugin());

3 Comments

it is still work?
@PutraFajarHasanuddin As of Feb 2025, this solution doesn't appear to work with websites using advanced Cloudflare protection. I've updated my answer above.
@iefrost What do you suggest as for April 2025?

Your Answer

Draft saved
Draft discarded

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.