FAQ: Scraping automation products
Find answers to common questions about integrating, configuring, and using Bright Data’s Scraping Automation, including IP types, geotargeting, and error codes.
How can I avoid getting blocked by Cloudflare or Cloudflare Turnstile?
How can I avoid getting blocked by Cloudflare or Cloudflare Turnstile?
To scrape data from sites that use Cloudflare or Cloudflare Turnstile, we recommend using our Web Unlocker API or Browser API solutions.Both solutions use different methods such as CAPTCHA solving, custom fingerprints and headers and can easily overcome cloud flare.Web Unlocker API is recommended if you simply need to
GET HTML from a website, without doing any interactions on the page.Browser API is recommended if you need to interact with the page (for example, fill forms, click buttons et cetera)In both cases, you will be able to access the information you need on the page, even if it uses Cloudflare or Cloudflare Turnstile.How can I avoid getting blocked by Datadome?
How can I avoid getting blocked by Datadome?
To scrape data from sites that use Datadome, we recommend using our Web Unlocker API or Browser API solutions.Both solutions use different methods such as CAPTCHA solving, custom fingerprints and headers and can easily overcome cloud flare.Web Unlocker API is recommended if you simply need to
GET HTML from a website, without doing any interactions on the page.Browser API is recommended if you need to interact with the page (for example, fill forms, click buttons et cetera)In both cases, you will be able to access the information you need on the page, even if it uses Datadome.Which websites are classified as Premium Domains?
Which websites are classified as Premium Domains?
How to enable JavaScript rendering with the Web Unlocker API?
How to enable JavaScript rendering with the Web Unlocker API?
If you require the Web Unlocker to render a page using JavaScript, you should utilize the following feature: Manual Expect ElementsThis will ensure that the response that is returned by the Web Unlocker API includes the part of the page you require to be rendered.If you require for the page to be completely rendered by a browser, you should opt for using the Browser API
How do I scrape specific sites?
How do I scrape specific sites?
To scrape specific sites, follow these steps:
- Select the Right Product for you: Bright Data offers various tools like the Web Scraper IDE, Web Scraper API, and custom scrapers. Choose the one that best suits your needs for scraping the target site.
- Search for Target Site Templates: If you’re using the Web Scraper API, browse or search for pre-built scraping templates for popular websites. These templates make it easier to start scraping without building a scraper from scratch.
-
Customize or Build Your Scraper:
- If your target site isn’t available as a pre-built option, use the Web Scraper IDE to build a custom scraper tailored to the specific site or the Custom Scrapers option if you want us to build it for you
- Write your script using the provided editor and tools, or modify an existing template to match your requirements.
- Test and Execute Your Scraper: Run the scraper within the Bright Data platform to collect the desired data. Use the debugging tools available in the IDE to refine the process if needed.
- Export and Use the Data: Once the scrape is complete, download or export the collected data in your preferred format, such as JSON, CSV, or Excel, for analysis or integration into your systems.