Explore a comprehensive directory of bots and view their detailed information
2Checkout Payment Platform's webhooks integration for notifying websites about payment events. Now operated by Verifone.
VerifoneEmpowering Developers to Monitor Sites & Servers, Easily
360MonitoringThe Apple App Site Association is used to support "Universal Links" that can open in native iOS apps. The bot requests a specific path for a given hostname, which returns metadata that associates certain URL patterns with native iOS apps.
AppleAccessible Web Bot crawls customer websites to discover pages and monitor for accessibility violations on regular basis. Crawls are initiated for Accessible Web's "Page Monitoring" SaaS product.
Accessible WebAccessStatus is a bot that checks the HTTP status code of a web page. It is used to determine if a URL is active, redirected, or returning an error.
AccessStatusScanning tool that helps perfect content and improve web accessibility.
AcquiaActiveComply Bot is a crawler for a service that monitors social media for regulatory compliance. It scans for specific keywords and content for businesses in regulated industries.
ActiveComplyAdagio demand optimization solutions help publishers leverage unlimited demand sources at unprecedented revenue level, while improving user experience, SPO and carbon footprint.
AdagioAs AddSearch adds content from your site to the search, the AddSearch bot gets counted as traffic by most analytics software.
AddsearchThe AddThis bot crawls websites to gather and update content for its website marketing tools. These tools include features like social sharing buttons and content recommendation widgets.
AddThisAdIdxBot is the crawler used by Bing Ads. AdIdxBot crawls ads and follows the websites from those ads for quality control. Just like Bingbot, AdIdxBot has both “desktop” and “mobile” variants.
MicrosoftOneTag is required to crawl ads.txt files from our publishers to verify the presence of the onetag.com line
OneTag LimitedThe Adyen webhooks integration sends HTTP requests to inform web servers about payment-related events.
AdyenUse Magento 2 connector for AfterShip to track your orders and get delivery
AfterShipA web crawler by Agency Analytics that allows their clients to check their own sites for SEO
Agency AnalyticsThe AGI Agent is a productivity assistant that takes actions and makes purchases on behalf of users
AhrefsBot is a Web Crawler that powers the 12 trillion link database for Ahrefs online marketing toolset. It constantly crawls web to fill our database with new links and check the status of the previously found ones to provide the most comprehensive and up-to-the-minute data to our users.
AhrefsAhrefsSiteAudit is used by website owners (paid and free) to look for issues on their websites
AhrefsThe integration API for the Akismet spam filtering service.
AkismetAlertsite by Smartbear is the HTTP monitoring probe that monitors its customers websites for availability and performance anomalies.
SmartbearIndexes content for Algolia search engine
AlgoliaAllAfrica Global Media produces, aggregates and distributes news from across Africa, relying on agreements with more than 140 news organizations and over 500 other institutions and individuals. The AllAfrica NewsBot scrapes content from sites with whom AllAfrica has written agreements, or whose content is available without licensing restrictions or otherwise freely distributable. In all cases, the author and institution is credited in full.
AllAfrica Global MediaAgentCore Browser provides a secure, cloud-based browser that enables AI agents to interact with websites.
AmazonAgentCore Browser provides a secure, cloud-based browser that enables AI agents to interact with websites.
AmazonAgentCore Browser provides a secure, cloud-based browser that enables AI agents to interact with websites.
AmazonAgentCore Browser provides a secure, cloud-based browser that enables AI agents to interact with websites.
AmazonAgentCore Browser provides a secure, cloud-based browser that enables AI agents to interact with websites.
AmazonAgentCore Browser provides a secure, cloud-based browser that enables AI agents to interact with websites.
AmazonAgentCore Browser provides a secure, cloud-based browser that enables AI agents to interact with websites.
AmazonAgentCore Browser provides a secure, cloud-based browser that enables AI agents to interact with websites.
AmazonAgentCore Browser provides a secure, cloud-based browser that enables AI agents to interact with websites.
AmazonAmazonbot is Amazon's web crawler used to improve our services, such as enabling Alexa to answer even more questions for customers. Amazonbot is a polite crawler that respects standard robots.txt rules and robots meta tags.
AmazonBuy For Me agent places orders on e-commerce websites at the direction of customers.
AmazonAmazon uses a crawler, also known as a spider or a bot to process and index the content of webpages. The Amazon crawler visits your site to determine its content in order to provide relevant ads.
AmazonAmazon Kendra is a highly accurate intelligent search service that enables your users to search unstructured data using natural language. It returns specific answers to questions, giving users an experience that's close to interacting with a human expert. It is highly scalable and capable of meeting performance demands, tightly integrated with other AWS services such as Amazon S3 and Amazon Lex, and offers enterprise-grade security.
AmazonAmazon AdBot is a crawler used by Amazon's advertising services. It visits advertiser landing pages to ensure they are compliant with advertising policies.
AmazonThe Web Browser for AI Agents
AnchorCrawl websites and extract content to feed AI apps. Convert web data to Markdown or HTML, download files, and more.
ApifyApplebot data is used to power various features, such as the search technology that is integrated into many user experiences in Appleʼs ecosystem including Spotlight, Siri, and Safari.
AppleThe Internet Archive bot, also known as archive.org_bot, is the web crawler for the Internet Archive's Wayback Machine. It systematically crawls and preserves publicly accessible web pages for historical record.
Internet ArchiveArea 360 property search and analytics
Area360Web crawler archives the Portuguese web
ArquivoArtemis is a calm, independently-run, free web reader. The reader often runs into Cloudflare access blocks which makes it hard to follow many sites.
capjamesgatlassian-bot is a crawler for custom 3P websites that indexes data for rovo search
AtlassianThe Attracta bot is analyzes user website content as part of Attracta's SEO services
AttractaAudisto Crawler fetches all accessible URLs of a website. Audisto provides a service to audit and monitor websites for its customers. More information about the crawler is available here: https://audisto.com/bot
AudistoThe Authory bot visits websites to back up articles on behalf of journalists and other writers who use the service.
AuthoryAn end-to-end campaign and integration testing tool created to optimize your marketing, advertising and sales technology stack by ensuring setups are running as they should be.
AutomatonOn Tumblr, post authors can paste a URL in their post, and we'll "unfurl" that URL into a pretty Link "Block" for their post by making a request to the URL and parsing the response.
AutomatticAwarioSmartBot is a web crawlers sent by Awario to discover and collect new and updated web data (that is further used by Internet marketers from all over the world).
AwarioSEO web crawler to identify web page popularity and thematic
BabbarBaiduspider is the search engine crawler for the search engine Baidu.
BaiduBaidu's scrubbing proxy.
BaiduThe Bazqux Fetcher is how BazQux Reader grabs RSS/Atom feeds and comments when users choose to subscribe to your blog in BazQux Reader. Fetcher collects and periodically refreshes these user-initiated feeds.
BazquxThe BestChange bot downloads exchange rate information from 600 websites every 5 seconds.
BestChangeThe bot is used for monitoring infrastructure platforms.
Better StackBibliothèque nationale de France's mission is to collect, catalog, preserve, enrich and communicate the national documentary heritage.
Bibliothèque nationale de FranceBig Sur AI Crawler, crawlers users websites to enable AI-infused experiences
Big Sur AIBigScoots Managed Services Monitor - Uptime
BigScootsAutomates the pipeline of active product deals to advertisement
BigUpDataBinaryCanary monitors websites for availability and performance issues.
Binary CanaryBingbot crawler and handles most of Bing's crawling needs each day.
MicrosoftBingPreview generates page snapshots for Bing. Note that BingPreview has desktop and mobile variants.
MicrosoftBitbucket Webhooks for CI/CD
AtlassianFast Dynamic DAST security scanning by Black Duck Software
Black DuckSEO PowerSuite Link Explorer (webmeup.com) is the world's freshest backlink index, and the primary source of backlink-related data for the SEO PowerSuite tools. We're dedicated to providing SEOs with the most comprehensive, up-to-date backlink data on the Web.
WebMeUpBling is an online ERP system that integrates with ecommerce platforms.
BlingScanning the internet to find malicious sites that scam crypto users into draining their wallets
BlockaidBlogtrottr delivers updates from all of your favourite news, feeds, and blogs directly to your email inbox, giving you the flexibility to stay updated whilst on the go.
BlogtrottrWordPress services like backup, security, monitoring etc.
BlogVaultBluesky social pulls links in advance to render webpage previews.
BlueskyPrice comparison site for board games. Need to crawl store pages for participating stores. All stores give permission to be crawled.
KP Software Consult ApSHoneybadgerBot is the bot used by the Honeybadger error and uptime monitoring service.
Honeybadger IndustriesSiteCrawler, part of the Botify Analytics suite, gives enterprise SEO teams the power to evaluate the structure and content of their websites just like a search engine
BotifyBrave search has a crawler to discover new pages and index their content.
Brave Software, Inc.Autopilot is an SEO marketing automation tool that includes features for internal linking and image optimization. We crawl customer sites so that we can determine the best links to use on the site and to find images that need to be optimized.
BrightEdgeBrowserbase helps AI use the web. Autonomously reads, writes, and performs tasks on the web with a serverless browser.
BrowserbaseWhen Buffer users share links in their social media posts, their scraper helps create engaging previews.
BufferBugsNag integration service, now Insight Hub, is used for error and performance monitoring of web applications.
Bugs NagBushbaby is an internal bot used by Cloudflare. Its purpose is to manage and renew SSL certificates for websites that use Cloudflare's services.
CloudflareThe bot fetches RSS feeds to import into Buttondown newsletters.
ButtondownToutiao is ByteDance's automated news aggregation bot collecting content across web platforms.
BytedanceCaliperbot crawls Conductor clients' and prospects' websites for HTML feature extraction to power Content Analytics features within our Searchlight web application.
ConductorCapital One Bot crawls dealer websites for getting the usage information for Capital One lead navigator button.
Capital OneCartAI B2B rails combine execution, payments, loyalty and affiliate networks to facilitate agentic commerce
CartAIOur mission has remained the same from day one: to prioritize monitoring and observability from the end-user perspective.
CatchpointCert Chief is a certificate monitoring tool that periodically crawl web properties to check their configuration and reports problems and changes when they are detected.
Chief ToolsCloudflare Digicert DCV service.
CloudflareChannable is a tool used by some Cloudflare customers. The tool downloads data from the customers' systems for marketing automation and e-commerce automation.
ChannableChannel3Bot visits public product pages to index their content, with the aim of driving traffic back to those websites.
Channel3Chargebee provides a webhooks integration to notify web severs of payment events.
ChargeBeeAgent that can use its own browser to perform tasks for user.
OpenAIChatGPT-User is for user actions in ChatGPT and Custom GPTs. When users ask ChatGPT or a CustomGPT a question, it may visit a web page to help answer and include a link to the source in its response.
OpenAICheckly is a high-programmability active monitoring solution. We support users in monitoring their websites and APIs. Puppeteer and Playwright (both supported) are browser automation tools that can be used for a variety of tasks. For testing, they really are about E2E/component testing, not unit testing.
ChecklyChrome-Lighthouse is an automated, open-source tool for auditing web page quality and does not operate as a traditional web crawler. It runs a series of audits against a given page to generate a report on performance and accessibility.
GoogleCitibotSiteCrawler collects public data from government websites to power Citibot’s AI civic engagement tools.
CitibotCledara’s agent automates customer-approved SaaS admin tasks, including invoice collection and user management.
CledaraThe Clickagy Intelligence Bot is an ad verification bot for Clickagy.
ClickagyCloudflare AI Search is a managed service that lets you connect your data and easily build AI-powered search.
CloudflareRenders web pages in headless browsers for Cloudflare customers. Used for browser automation (screenshots, PDF generation, content extraction, etc.) and for AI agents to interact with the web. Used by Cloudflare customers via Workers bindings and REST API. Does not include the /crawl endpoint, which has a separate bot identity (Cloudflare Crawler - Signed Agent).
CloudflareThe Cloudflare Crawler is a well-behaved crawler that retrieves web content. By default, it self-identifies as a bot, honors robots.txt directives, and cannot bypass CAPTCHAs or bot protection. Used by Cloudflare customers via the Browser Rendering /crawl endpoint.
CloudflareCloudflare Custom Hostname Verification service.
Cloulflare internal service that crawls customer error pages in order to serve them directly from our edge network.
CloudflareCloudflare system bot that performs health checks and diagnostic tests
CloudflareCloudflare Healthchecks service
CloudflareSynthetic network probes for HTTP timing measurements (TCP, TLS, TTFB). Measures connection timing for customer-owned URLs.
CloudflareURL prefetching means that Cloudflare pre-populates the cache with content a visitor is likely to request next. This setting leads to a higher cache hit rate and thus a faster experience for the user. (https://developers.cloudflare.com/fundamentals/speed/prefetch-urls/)
CloudflareCloudflare Purge service.
CloudflareCloudflare Radar URL Scanner
CloudflareCloudflare SpeedTest service.
CloudflareCloudflare SSLDetector service.
CloudflareCloudflare Stream Webhook service.
CloudflareCloudflare-Traffic-Manager service.
CloudflareCloudflare Validator makes requests to verify IPs for Cloudflare Bots Directory
CloudflareCloudflare CSUP is a bot used by Cloudflare's customer support for diagnostic purposes. It is not a general web crawler and is used to investigate technical issues with customer websites.
CloudflareCloudtrellis automatically scans your entire site for broken links, accessibility issues, and potential SEO improvements
CloudtrellisCludobot crawls websites to facilitate and provide search and analytics solutions for its customers.
CludoCoccocbot scrapes websites that are request from the Vietnamese search engine Coc Coc.
CoccocCognitiveSEO is an SEO toolset that crawls the web and analyzes links.
cognitive SEO Internet Marketing Tools.Coinbase Webhooks are automated messages sent from the Coinbase platform to a user's server, used for notifying users about events such as receiving crypto payments.
CoinbaseContentKing is a cloud-based service that monitors websites from a digital marketing perspective. We monitor the websites for customers such as Netflix, Atlassian, Fedex and IBM and alert their digital marketing teams whenever a technical issue or content change is detected.
ContentKingAccesses page content to create contextual segments for targeting within several DPS & SSP platforms.
OutcomesConvermax Site Search Indexer
Convermax Corp.Used to detect cookies set by websites (for CookieHub clients) and verify if user consents are respected
CookieHubCookiebot scans the website for cookies and trackers to gather and provide the information on a cookie banner.
Cybot A/SWithin the GDPR legislation it is mandatory to ask a visitor for permission before placing so-called marketing or tracking cookies. Many websites contain a cookie notice, but what is not clear to everyone is that those cookies may not be placed before the visitor has given explicit permission. Cookie Maestro searches for all cookies that your website places in your visitors browser.
Cookie MaestroCookieYesbot scans and identify cookies and related information on websites that use CookieYes platform
CookieYesBy providing a centralized management environment for content and models, we will build a framework that promotes both data utilization and research and development of AI technology, and realize the following functions as Japan's future information infrastructure.
Research Organization of Information and SystemsCoveo provide services to website, customer service and commerce solutions so they can feature relevant experiences to their end users; said services are based on a unified index which crawls websites when configured so by our customers.
CoveoUsers subscribe to external iCalendars and sync them into their Cozi calendars. We periodically fetch these iCals to keep them up to date.
OurFamilyWizardCrawlson is a search engine crawler for the crawlson.com search engine.
CrawlsonCrazy Egg bot that takes screenshots of pages, collects assets, tests script installation.
CrazyEggCriteo Crawler is a software that visits web pages and analyzes its content to serve relevant ads on them.
CriteoWe operate a SASS for website optimisations, we have thousands of customers. Some of them use Cloudflare but I don't have access to any exact list.
Critical CSSScheduled execution of your websites and scripts.
cron-job.orgThe Cxensebot performs SEO monitoring and analysis of customer webpages.
CxensePerforms broad security research by crawling the most popular domains obtained from Crawler.Ninja.
Cloudflare AI Crawl Control bot 2
CloudflareCloudflare AI Crawl Control bot 3
CloudflareCloudflare AI Crawl Control bot 4
CloudflareDatadog Synthetics gives you a new layer of visibility on the Datadog platform. By monitoring your applications and API endpoints via simulated user requests and browser rendering.
DataDogDataForSEO Bot is a driving force of our leading product - Backlinks API, which has been developed with a single purpose: providing website owners, webmasters, and SEO professionals with opportunities to analyze the key component of website optimization – backlink analytics. You can learn more about the DataForSEO Bot on this dedicated page: https://dataforseo.com/dataforseo-bot
DataForSEODataForSEO is using RSiteAuditor to scan websites for critical on-site SEO errors and provides aggregated data in a structured form to its customer through a RESTful API.
DataForSEODataprovider.com indexes the web and structures the data.
Dataprovider.comThe NetEstate Imprint crawler crawls websites for public contact information.
netEstateKorean search engine crawler
DaumDead Link Checker is a service which crawls a customer's website reporting on any broken links (404, 500 etc) it finds
DLC WebsitesThe DeepCrawl bot crawls the websites of its customers to collect performance analytics and suggest SEO optimizations.
LumarDetectify analyses the security level of web applications. To start a scan a user has to add an access key to the domain and agree to their terms.
DetectifyThe Mediatoolkitbot is a media monitoring tool that crawls the open internet looking for phrases Determ users search for, helping marketers find relevant opportunities for advertising.
DetermDevin is a collaborative AI teammate built to help ambitious engineering teams achieve more.
Devin AIDigiCert DCV service.
DigiCertAnomura is Direqt’s search crawler, it discovers and indexes pages their customers websites.
DireqtThe discordBot scrapes URLs that are shared within the Discord chat platform. This is done to generate contextual previews of the content, including titles, descriptions, and images.
Discord, Inc.Dotbot is Moz's web crawler, it gathers web data for the Moz Link Index. This data we collect through Dotbot is available in the Links section of your Moz Pro campaign, Link Explorer, and the Moz Links API.
MozThe Doctom Monitor checks websites for uptime and performance issues.
Dotcom-MonitorLeikiBot, run by DoubleVerify, crawls webpages content for advertisers.
DoubleVerifyThe Drata Autopilot bot continuously monitors the security posture of customer domains.
DrataThe Drift Agent executes security and compliance tasks on behalf of the CISO office.
Drift SecurityDr. Link Check crawls websites to help their owners identify and fix broken links.
Dr. Link CheckDuckAssistBot is a web crawler for DuckDuckGo
DuckDuckGoDuckDuckBot is the search engine crawler for the DuckDuckGo search engine.
DuckDuckGoSearch engine providing gaming statistics and tools for the browser game "eRepublik".
Sebastian Foth - Software SolutionsEasyBill Import Manager is a tool that synchronizes order data from external systems to EasyBill.
easybill.deEasyCron is an online cron job service. Users can schedule an HTTP request to be made at a specific date and time.
EasyCronEasyDNS' uptime monitoring probe.
EasyDNSAutomated scanning service that reviews online content on behalf of end users to identify potential legal issues.
codire GmbHWe scrape full article/page content to ensure we can optimally automate the content distribution for the digital publishers we work with. Every single article a publisher releases will get scraped approx. 2-4 times by independent services.
Echoboxelmah.io Uptime Monitoring bot is a heartbeats tool monitors the availability of their users' websites.
elmah.ioCollects raw financial data that can later be used for financial planning and analysis
eMoney AdvisorNews aggregator needs to crawl news/blog articles to generate short summaries for page preview of attributed links.
TechmemeBuilt-in uptime monitoring for the EvoCommerce platform
Evo Agency Ltd.A crypto wallet application to manage cryptocurrencies like Bitcoin, Ethereum, Ripple, and more. Secure.
ExodusEzoic is a technology platform for digital publishers. You can learn more about what Ezoic does here.EzoicBot is our web crawler designed to extract valuable information about how the internet, search engines, and websites all work together. EzoicBot can helps publishers better understand how their sites work. This includes the ability for search engines, like Google, to index and rank their content.
Ezoic IncThe primary purpose of FacebookExternalHit is to crawl the content of an app or website that was shared on one of Meta’s family of apps, such as Facebook, Instagram, or Messenger. The link might have been shared by copying and pasting or by using the Facebook social plugin. This crawler gathers, caches, and displays information about the app or website such as its title, description, and thumbnail image.
MetaFactset uses a Python Selenium Crawler for web scraping to deliver reliable, current financial data.
FactsetFastmail fetch and image proxy bot
FastmailFDL Stats bot is used to generate analytical data around rocket league player information. The bot will attempt to crawl various platforms that provide player data that is publicly available to build stats about player participation.
FTW Entertainment LLCBot to download data from the ffiec, Active/Closed/Branches File, Holding Company Data, 002 Data
Fed Reporter, Inc.Feedbin's RSS reader service.
FeedbinA cloud-based RSS reader with over 500 000 users subscribed to over 3 million feed URLs
Really Simple ABFeedly RSS fetcher service.
FeedlyAccesses feed sources to ensure feed widget is up to date (crawls every 5 minutes to 5 hours depending on a user's plan).
MikleThis bot is used to fetch the RSS feed content of the websites owned by rightful publishers at follow.it
follow.itWe work with publishers/partners to obtain their content so it is formatted for the Flipboard app.
FlipboardFlipboard will use visitor's RSS feed to discover articles and generate article summaries
FlipboardForegenix perform security and risk scanning on the web sites of eCommerce merchants for a number of banks and card brands globally. The service assists these organisations in controlling and identifying fraud and financial losses, with a particular focus on trying to identify compromised merchants before they end up in the card brand's compromise investigation process. Early detection (prior to fraud losses escalating) can save the banks and merchants alike considerable sums. The solution has two primary modes of operation Scanning for active malware, this normally entails pulling a very limited number of pages within a sandboxed context for analysis at various stages of DOM initialisation. From the target sites perspective, the operation is simply another browser requesting a small number of pages as normal. Scanning for known publicly exploitable vulnerabilities and outdated software solutions as these attributes are frequently exploited by threat actors to introduce malware targeting financial information. Typically a complete scan comprises less than one hundred requests and is already rate limited on our side. Scanning is always "passive" in nature, relying on GET, HEAD and OPTIONS requests only. The scanning heads by default abide by the "robots.txt" file but this can be overridden by the scan initiator (usually one of our banking clients). This override, to force a scan/assessment is not actioned all that frequently.
Foregenix LimitedFreespoke is a search engine that believes in free speech and shows you all viewpoints.
FreespokeCheck website is online and issue an alert when its down
FreshworksFullStory is your digital experience analytics platform for on-the-fly funnels, pixel-perfect replay, custom events, heat maps, advanced search, Dev Tools, and more. FullStoryBot’s fetches and stores assets required to rebuild sites when viewing recorded sessions.
Full StoryFunnelback is an enterprise search platform, and its crawler indexes content from an organization's websites and data repositories. This powers the organization's internal search function.
Squiz - FunnelBackGhost Inspector is an automated browser testing framework.
Ghost InspectorGigablast is the only non-Big Tech search engine in the U.S. that uses its own search index and algorithms.
GigablastGooglebot is the search engine crawler for Google Search.
GoogleGoogle-AdWords-Express is a bot for a Google Ads product aimed at small businesses. It crawls advertiser websites to assist with ad creation and to verify site information.
GoogleGoogle-Adwords-Instant is a bot connected to the Google Ads platform. It visits advertiser landing pages to perform verification and quality checks.
GoogleCrawler available to site owners to request crawls of their own sites for targeted AI training
GoogleThe Digital Asset Links bot verifies statements lists made by website operators.
GoogleGoogle-InspectionTool is the crawler used by Search testing tools such as the Rich Result Test and URL inspection in Search Console. Apart from the user agent and user agent token, it mimics Googlebot.
GoogleGoogle NotebookLM bot used by notebooklm.google
GoogleGeneric crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development. https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers#googleother
GoogleTools and product functions where the end user triggers a fetch.
GoogleFetches and processes feeds that publishers explicitly supplied for use in Google News landing pages.
GoogleThe Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties
GoogleGoogle Scholar uses a bot to crawl and index scholarly literature from academic publishers, repositories, and university websites. This populates its academic search engine.
GoogleVerification is the process of proving that you own the property that you claim to own. Search Console needs to verify ownership because verified owners have access to sensitive Google Search data for a site, and can affect a site's presence and behavior on Google Search and other Google properties. A verified owner can grant full or view access to other people.
GoogleHelping build a safer Internet by providing a transparent, trusted, and reliable Certificate Authority.
GoogleThis user agent belongs to Google Web Snippet. Google Inc developed this Bot. This Bot run on Linux.
GoogleGoogle AdsBot is special-case crawler that monitors websites where Google Ads are served.
GoogleAPIs-Google is the user agent used by Google APIs to deliver push notification messages. Application developers can request these notifications to avoid the need for continually polling Google's servers to find out if the resources they are interested in have changed. To make sure nobody abuses this service, Google requires developers to prove that they own the domain before allowing them to register a URL with a domain as the location where they want to receive messages.
GoogleThe Google Images bot is the search engine crawler for Google Images Search.
GoogleThe Google Videos bot is the search engine crawler for Google Video Search.
GoogleThe Google Schema Markup Testing Tool bot, now part of the Rich Results Test, crawls pages to validate their structured data. This helps webmasters check if their schema markup is correctly implemented for Google Search.
GoogleGoogle AdSense uses a crawler called Google-Display-Ads-Bot to verify your site when you add a site to AdSense.
GoogleGoogle FeedFetcher is the RSS reader for Google.
GoogleThe Google Image Proxy bot is used to render link content sent via email in Gmail.
GoogleThe Google AdSense bot monitors the content of websites using Google AdSense.
GoogleGoogle Read Aloud service enables reading web pages using text-to-speech (TTS).
GoogleGoPay payment getaway http notification service
GoPay.czGPTBot is used to crawl content that may be used in training OpenAI's generative AI foundation models
OpenAIGrafana's Synthetic Monitoring agent provides probe functionality and executes network checks for monitoring remote targets.
Grafana LabsThe Grapeshot bot, part of Oracle Advertising, crawles web pages for content analysis and classification.
GrapeshotThe Groovinads bot gathers data from e-commerce websites to support its ad services.
GroovinadsGTmetrix is a free tool that analyzes a page's speed performance. Using PageSpeed and YSlow, GTmetrix generates scores for pages and offers actionable recommendations on how to fix them.
GTmetrixGuestpostsBot is a Web Crawler that has several functions to facilitate the website owner who has registered his site on the guestposts.com.br platform to monitor his site. The bot constantly tracks the sites registered on the platform in order to check if the partnerships made on the guestpost platform are still active, in addition to validating if the site exists to allow registration and also monitoring the status of the site from time to time to warn the website owner in case of any inoperability.
Guest PostsHaiku-SearchBot is for user actions in Haiku. When users ask Haiku a question, it may visit a web page to help answer.
CLERK SASHatena's service automatically visits web pages to get the information it needs to use in the service.
HatenaHelloWork is a French job board, and its bot aggregates job listings for its platform. It crawls company career pages and other sources to collect this information.
HelloWork GroupCrawler to integrate Helpfeel external search engine.
HelpfeelExecutes checkout via browser automation using a user's card and signed mandate
Henry LabsHetrixTools offers an Uptime Monitoring service, where our monitoring locations (bots) will check whether our users' websites are online or not. Similar to more known services such as Pingdom or UptimeRobot.
HetrixTools IncHEY email stops spy pixels and prevents user IP tracking by proxying all HTML email images, fonts, and external assets
37signalsShort Description: HIFI is a financial services company for musicians and professional creators. HIFI acts as an agent on behalf of its clients to automate the retrieval and processing of royalty earnings statements. HIFI’s clients provide access credentials for each of their portal accounts and then HIFI automates the otherwise labor intensive process of logging into each portal, downloading and then processing the relevant CSVs. HIFI analyzes and aggregates the underlying data and presents its clients with a business management comprehensive dashboard.
HIFIThe HostTracker monitor tracks website availability and performance for their customers.
HostTrackerHotjar provides user analytics and feedback for website owners.
HotjarChecks that links can be preloaded for Chromium
ChromiumHubSpot offers a full platform of marketing, sales, customer service, and CRM software — plus the methodology, resources, and support — to help businesses grow better. Get started with free tools, and upgrade as you grow.
HubSpotWhen posting to LinkedIn from Hubspot, images need to be pulled through to LinkedIn when published. The crawler performs this function
HubSpotWhen posting to LinkedIn from Hubspot, images need to be pulled through to LinkedIn when published. The crawler performs this function
HubSpotHuckabot is Huckabuy’s main crawler which is utilized by almost all of Huckabuy’s products. The primary purpose of Huckabot is to crawl and index a customer’s website, which is then rendered and optimized with our Dynamic Rendering Product. Several of the Page Speed product boosters, such as Fold Prioritization, also leverage Huckabot in order to optimize and improve a website’s performance.
HuckabuySince 2005, Hype Machine monitors music publications/blogs for posts about new artists and builds playlists using this metadata for listeners.
Hype MachineIntegral Ad Science (IAS) is the global market leader in digital ad verification, offering technologies that drive high-quality advertising media.
Integral Ad ScienceIbouBot is the crawler of the Ibou Search Engine
BabbarICC-Crawler automatically crawls the Internet and collects web pages. ICC-Crawler is operated by the Universal Communication Research Institute at the National Institute of Information and Communications Technology (NICT).
NICTRich media embed solution
IframelyRSS feed fetcher to power user-configured automations
IFTTT RSS Feed ServuceIndeed's job crawling bot that crawls job and job related information
IndeedInnguma fetcher collects and periodically refreshes these user-initiated feeds.
InngumaInoreader is one of the most popular RSS feed readers used by more than a million people.
InnologicaInstapaper is an app that lets people save articles to read later.
Instant PaperOver 50 active Cloudflare users are using Integromat to automate their workflow.
MakeInternet Archive’s Archive-It service preserves publicly accessible web pages for the historical record.
Archive-ItInternetArchiveBot looks for URL references on Wikipedia and assesses if the URL is still alive, or delivering 404s.
Internet ArchiveIsDown monitors endpoints (websites, APIs) to make sure they are up and running
IsDownBotUptime is a synthetic monitoring tool allowing Shopify merchants to validate key customer flows are not broken after making theme changes.
Jagged Pixel Inc.Uptime monitor for users of WordPress.com/Jetpack — https://jetpack.com/support/monitor/
AutomatticSimple crawler focussing on only job postings for job search site.
JobsWithGPTDigital ID Verification and Scanning Tool used by Coinbase team.
CoinbaseKagi Bot is the web crawler for the Kagi search engine. It crawls the web to build its own search index, which supports its ad-free search product.
KagiThe KakaoTalk scrap server collects and processes webpage data to create optimized previews for URLs.
KakaoKargoBot-Artemis is Kargo's autonomous content verification bot. It's a simulation of a user on an iOS device. The bot is used to scan sites for content that may be unsuitable for customers on the Kargo ad network.
KargoRoyal Danish Library collects the Danish Internet according to the Danish Legal Deposit Act for research purposes.
NetarkivetKernel is a browsers as a service platform that enables AI agents to browse the internet.
KernelOur bot makes requests towards Kinsta-hosted client sites for uptime monitoring purposes as well as to confirm successful domain pointing.
KinstaThe integration with Klaviyo will automatically capture information about who visits your site and views your products including details on what they viewed so you can send super personal follow up emails
KlaviyoKlaviyo’s web crawler for its Kai Customer Agent feature.
KlaviyoMonitor customers' sites for compliance with cookie- and other privacy-related legislation
Legal Monster ApSLet's Encrypt domain control validation service.
Let's EncryptThe Library of Congress Web Archive manages, preserves, and provides access to archived web content selected by subject experts from across the Library, so that it will be available for researchers today and in the future. More information on the programme here: https://www.loc.gov/programs/web-archiving/about-this-program/ And information about crawling policy here: https://www.loc.gov/programs/web-archiving/for-site-owners/
United States Library of CongressThe web pages crawled by Linerbot are used to find the right sources that can generate answers to your questions.
Liner BotLinespider is a Web crawler that provides a wide range of search results for LINE services while complying with the Robots Exclusion Protocol. https://help2.line.me/linesearchbot/web/?contentId=50006055&lang=en
LINE CorporationWeb Crawler that monitors the backlinks profile
LinkCheckerLinkedInBot is used by LinkedIn when rendering link preview information.
LinkedInLinksIndexerBot is an SEO bot that crawls websites to index backlinks and aggregate website summaries.
Links IndexerLinkTiger crawls customer sites and reports broken links
LinkTiger, Inc.The logicmonitor bot performs synthetic monitoring checks on websites and online services to test their availability and performance.
LogicMonitorLoomlyBot is used to extract metadata from web pages in order to show a social media post preview within Loomly so that clients can see what their social media posts will look like when published.
LoomlyMacrobond implements custom web crawlers to fetch macroeconomic data published by official data sources.
MacrobondbotMagiBot is owned by Peak Labs which focuses on the research and development of information extraction and retrieval technology to transform knowledge in natural language into immeasurable value.
Peak LabsMagnetmeBot checks the websites of our paying customers and ensures the job openings are being kept in sync
Magnet.meThe Magpie Crawler indexes content for its soical media monitoring solution.
BrandwatchThe mail.ru bot is a mail fetcher on behalf of the Mail.ru email service.
Mail RussiaWebsite Managed - MainWP Control Dashboard for accessing MainWP child sites.
Direct Support / Website ManagedMake is a no-code platform to help with task and workflow automation.
MakeThe ManageWP webhooks integration to manage mulitple Wordpress websites with a single dashboard.
ManageWPManus is the action engine that goes beyond answers to execute tasks, automate workflows, and extend your human reach.
ManusMarginalia Search is a noncommercial niche search engine focusing on old websites, personal websites, and blogs that suffer crippling discoverability problems in today's fiercely SEO-optimized lanscape.
Marginaliamarketgoo provides white label SEO tools
marketgooMars Finder is a website search service designed to utilize the maximum potential of a website. MARS FINDER has held the top share of website search service market of Japan in 2017.
Mars FlagThe Mavifinds Bot is part of a security service that monitors websites. It can automatically activate security measures, such as an under attack mode, in response to threats.
MavifindsMediaMonitoringBot crawls and indexes news and media publishers websites for a new materials and try to match it against keywords provided by our customers (subscribers) and send them updates based on that information.
MediaMonitoringBotMediaboard's web crawler (media monitoring) detects client mentions in online news and public content.
MediaboardMedialogia Bot is the web crawler for Medialogia, a Russian media monitoring company. It collects data from online news and social media for analysis.
MedialogiaOur App is called Grow and we allow publishers to enable bookmarking, social sharing, and searching on their sites.
MediavineThis bot is used to aggregate data about a popular online multiplayer game from consenting hosts who have opted-in to this collection. The data that is aggregated is reflected in a panel where players can freely search through the aggregated data. Any modification or deletion of data from the sources (consenting hosts) is reflected within the application's database within 30 minutes. The application scrapes each host for new data every five minutes, with a more thorough check for modified data every 30 minutes.
MelonMesaThe Meta-ExternalAds crawler crawls the web for use cases such as improving advertising and other business-related products and services.
MetaUse cases such as training AI models or improving products by indexing content directly.
MetaCrawler receives individual links at the user's initiative to support certain product features.
MetaAnalytics and email automation service used by eCommerce businesses. Metorik syncs data from customer sites by making API requests to their sites.
MetorikMgidBot is used for detecting context categories of the content for advertising recommendations.
MGIDMicrosoftPreview generates page snapshots for Microsoft products.
MicrosoftWe are a commercial web archiving supplier providing archival solutions for the financial and public sector.
MirrorWeb LtdMissinglettr will crawl specific blog posts on customers' sites to help turn them into social media campaigns.
MissinglettrBot for user actions in le Chat by Mistral AI, for instance when asked to open a web page.
Mistral AIMJ12bot is the web crawler for Majestic. MJ12Bot does not currently cache web content or personal data. Instead it maps the link relationships between websites to build a search engine. This data is available to technologies and the public, either by searching for a keyword or a website at Majestic.
MajesticService to manage WordPress websites via the WP JSON API.
Uniqoders Technologies SLDetails and information for webmasters regarding Mojeekbot, the web crawler for the Mojeek search engine.
MojeekMollie B.V. is a payment service provider. We use webhooks to notify our merchants about updates to their payments.
Mollie B.V.MonSpark is a website monitoring service. Its monitoring bot checks website availability, network conditions and TLS certificate validity.
MonSparkUptime monitoring bot, part ofclicky.com web analytics
MonitageThe Monitis HTTP Monitoring Probe.
MonitisThis is for the official public hosting of the open-source project https://github.com/synzen/MonitoRSS so that the bot may poll for RSS feeds of Cloudflare-protected sites to deliver news articles. Feeds are chosen by paid users, and the bot adds them to a schedule to be polled at a regular interval of 2-10 minutes.
MonitoRSSThe MontasticMonitor bot monitors website avaiability.
MontasticMotoMinerBot is MotoMiner's web crawling bot. All vehicle detail pages we index are searchable via MotoMiner's search engine.
MotominerSearch engine aimed at generating a corpus of data to be able to aggregate data in various ways.
MRG Web Services srlMSNBot was the web crawler for Microsoft's MSN Search, which has since been replaced by Bing. Its purpose was to index web pages for inclusion in the MSN search engine.
MicrosoftMuck Rack uses many approaches for source discovery such as RSS feeds, sitemaps, and other structured formats
Muck RackClearscope is an AI-driven SEO content optimization platform developed by Mushi Labs. It assists content creators, marketers, and SEO professionals in producing high-quality, search-optimized content by providing real-time keyword recommendations, content grading, and insights into search intent. By analyzing top-performing content, Clearscope offers actionable suggestions to enhance content relevance and visibility in search engine results.
Mushi LabsBot to assist social workers in navigating safety net benefit websites.
Nava LabsYeti is the web crawler for Naver, a South Korean search engine. It indexes websites to provide search results and power other services on the Naver platform.
NaverNeevabot is the web crawler for the search engine neeva.com.
NeevaExecutes secure checkout via browser agent using user card and signed mandate.
NekudaThe Netcraft Survey Agent is a bot that analyzes web server technology stacks for their Web Server Survey.
NetcraftWebsite uptime monitor service
NetumoWe checks availability and performances of our client's websites and mobile apps.
NetvigieCoders within NYT's newsroom collect public, non-copyright data, e.g. our U.S. Elections pages and Covid-19 trackers.
The New York TimesThe New Relic bot is used by New Relic's Intelligent Observability Platform to monitor customer applications for availability and performance issues
New RelicRSS News fetcher
NewsBlurThe NewsNow bot is the web crawler for the news aggregator service NewsNow.
NewsNowUKWe run a cloud based site speed optimization solution. As such, we need to make requests to our clients' sites in order to fetch the content that needs to be optimized. We have several sub systems that can fire requests and each one can be identified based on the user agent suffix.
NitroPack LtdThe NixStatsMonitoringBot is the HTTP monitoring probe for NixStats to monitor website availability and performance.
NixStatsThe NodePing HTTP Monitoring probe monitors customer websites for uptime.
NodepingOur microservice downloads js files from our users servers in order to format them and show them a human readable file. This is done to facilitate solving errors associated with said file
NoibuNoorobot is an SEO tool that periodically crawls customer websites to provide recommendations and identify potential SEO-impacting problems.
Noor Digital Agency ABRSS Reader that fetches RSS/Atom feeds
NooshubMozilla/5.0 (compatible; NostoCrawlerBot/1.0; +http://my.nosto.com/tagging)
NostoNostra accelerates site speed for managed web platforms
NostraNovellum.ai is building out tools for building agents. This MCP tool will be used by agents to crawl sites.
NovellumOAI-SearchBot is used to link to and surface websites in search results in the SearchGPT prototype
OpenAIThe Oh Dear application availability and performance monitoring checker.
Oh DearOKX Dolphin Crawler simulates visits to target web pages as part of malicious dApp scanning
OKXBot for checking omnisend integration in woocommerce shops and shopify
OmnisendEnterprise SEO platform powered by the industry-leading SEO Crawler and Log Analyzer
OncrawlIt identifies and categorizes cookies and tracking tech on customers site, pages, forms, tags, storage, and cookies
OneTrust, LLC.A bot associated with WebCEO, a company that provides SEO tools and services
Online WebceoOnlineOrNot provides website monitoring in the form of uptime checks and page speed tests.
OnlineOrNotOur API is used by mostly consumer facing products to preview links when sharing them on their platforms. For example, how when a link is shared on Facebook or Slack, those platforms provide a description/title/image to make the content more enticing.
OpengraphRSS Feed Provider / Feed Fetcher
OpenRSSThe Orlo Link Preview bot is used by the Orlo social media management platform. It fetches previews of links that are scheduled to be published in social media posts.
OrloThe Outbrain crawler analyzes content on publisher websites for the purpose of serving ads.
OutbrainValidating client website URLS to monitor for hosting/provider changes
Outsell CorporationOvercast is a podcast player application, and its bot fetches RSS feeds and audio files from podcast hosting servers. This keeps the podcast directory and episodes updated for its users.
Overcast RadioA component that serves to load previews for external and internal links. For external links, whenever possible, information from the open graph tags specified on the page (title, descr, images\video) is used, for references to internal objects, the internal representation is used (in the form of specialized blocks in the topic).
OzonWebhooks let you get notified when events happen in Paddle.
PaddleThe PaesslerCloudBot is used by Paessler PRTG to monitor websites for availability and performance.
PaesslerPanopta is an uptime monitoring service acquired by Fortinet.
PanoptaPopular content analytics tool used by many major media and content teams.
Parse.lyParticle is an AI powered aggregator that collects news from many sources
Automated browser bot that fetches invoices for users from supplier websites and attaches them to their expense records.
PayhawkThe PayPal webhooks is part of Paypal's Instant Payment Notification message service, automatically notifying merchants of events related to Paypal transactions.
PayPalpayroll-bot is an AI crawler operated by ADP, Inc. to collect publicly available legal and payroll documentation.
Qualys Web Application Scanner is a cloud-based service that provides automated crawling and testing of custom web applications to identify vulnerabilities including cross-site scripting (XSS) and SQL injection.
QualysPetalBot is to access both PC and mobile websites and establish an index database which enables users to search the content of your site in Petal search engine and present content recommendations for the user in Huawei Assistant and AI Search services, both services are powered by Petal Search engine.
HuaweiThe Pingdom bot is the HTTP monitoring probe for Pingdom's website monitoring service.
PingdomPingPing is a website monitoring service whose bots check website uptime and TLS certificate validity.
PingPingPingPing.io is a monitoring system to monitor the online status of websites and validity
PingPing.ioPinterestbot is Pinterest’s web crawler. Pinterestbot crawls, or visits public websites to index their content, with the aim of driving traffic back to those websites. It also scrapes content to make sure Pin details, like price and title, are up to date, and to detect and remove broken website links behind Pins.
PinterestFetches podcast feeds, for playback in the Pocket Casts apps
Pocket CastsThe Polar webhooks integration sends HTTP requests to inform web servers about billing events.
PolarThe Potions bot fetches product feeds and crawls data from its customers' websites, used for e-commerce related services.
PotionsIt's HTML pre rendering service for SPA(Single Page Application) Website SEO.
Prerender, LLCThe PressEngine Bot verifies coverage created by video games press as genuine and their own creation. When a member of the video games press is granted a review key for a video game they will create an article, known in the industry as "coverage". When they submit a URL to us as "coverage" we automatically verify this URL exists and is viewable. This automated code announces itself as the PressEngine Bot.
PressEngineExtract Content to Show Print Friendly version. Publishers typically embed our button - https://www.printfriendly.com/button - so that their visitors can view a Print Friendly Page and/or create a PDF
PrintFriendly.comOnline sitemap generator service.
PRO SitemapsProductsup crawls websites to import additional product data.
ProductsupWe use Project Honeypot for IP info.
Unspam Technologies, IncProject Shield, created by Google Cloud and Jigsaw and powered by Google Cloud Armor, provides free unlimited protection against DDoS attacks, a type of digital attack used to censor information by taking websites offline.
GoogleProtopage.com indexes RSS news headlines mostly from news sites
Protopage LtdThe Proximic crawler visites websites serving ads on behalf of them or their partners to determine which ads best fit a given website's content.
ComScoreWebsite monitoring service. Uptime monitoring and down alerts.
PulseticPWABuilder (pwabuilder.com) is a free, open source developer tool from Microsoft that helps developers build progressive web apps and publish them in app stores. PWABuilder tool analyzes their website for Progressive Web App capabilities, such as a web manifest or service worker
MicrosoftBot crawls customer websites to provide information to customer hosted chatbots.
Qualified.com, Inc.SSL Labs / Qualys is used to test monitor ssl rating against their site
QualysQuantcast Bot is the name of a web crawler used by Quantcast for advertisement quality assurance and to understand page content for Interest-Based Audiences.
QuantcastQuartr uses a crawler to obtain and deliver investor relations material
QuartrBased and designed in Europe, Qwant is the first search engine which protect privacy.
QwantThe Rackspace HTTP Monitor monitors customer websites for uptime and other issues.
RackspaceRakuten uses this bot to crawl product images so that we can display cashbach deals for our merchants.
RakutenRazorpay sends webhooks back to ecommerce sites.
RazorpayReadable is a collection of text analysis tools, primarily focused on clarity and plain language. We spider customers' websites, find the content of each page, analyse it, and present that to the customer.
Added Bytes LtdWe send webhooks to our customers to inform them of events occurring on own platform relevant to their site
RecurlyReelavant allows customers to dynamically update content inside their emails. Needs to fetch images of their product at runtime and send them back to the users.
ReelevantRetool platform user agent
RetoolRetroListeCOM is a service that tracks user counts on gaming-related websites, and its bot visits those sites to collect this data.
Niclas PapstOur bot crawls our customers' websites to identify SEO opportunities
RevvimWith the iboss Cloud Platform, each customer gets dedicated source cloud IP Addresses which are associated with the organization. Because of this, any data traversing the global cloud containerized gateways in the Platform will have a uniquely associated IP Address that can be mapped to the organization. This means that users always appear to be accessing the Internet from within the organization regardless of whether they’re in the office or on the road. This preserves the critical connectivity requirements that IT departments need when migrating to a cloud gateway platform.
Reward GatewaySynthetics monitoring platform used by Enterprise organizations.
SplunkRobin automatically scans school websites in the UK, providing compliance checks against statutory requirements.
Robin EducationRogerbot is Moz's site audit crawler for Moz Pro Campaigns.
MozA notification RSS bot for Telegram instant messenger
Yellow Rubber Duck ConsultingRSS API periodically requesting and parsing RSS Feeds for our customers to monitor them for any changes.
RSS API (by Tibush GmbH)Agentic checkout assistant that purchases products on behalf of end users with their explicit consent.
RyeCRM + Marketing
SalseforceSince we offer sales and marketing information we need to enrich the company information. To provide crucial company information inside our service we need to provide a preview of visitor websites. Therefore we need to visit the websites.
SalesViewer GmbHEnhances e-commerce security by monitoring stores, crucial for preventing data breaches & fighting digital skimming.
Sansec Security MonitorA personalized RSS feed reader that uses AI/ML to surface content that matches user interests.
ScourCrawler for analyzing SE Ranking clients websites for potential issues.
SE RankingBot used to evaluate customer's websites and provide SEO optimization strategy
Search AtlasProvides a free security scanning service at https://securityheaders.com
Security HeadersSeekport is an internet search engine. Originally founded in 2003, the search engine has been operated by SISTRIX, a platform intelligence provider from Bonn (Germany), since December 2014. The search engine is a public, free and independent alternative to Google. Seekport does not store user data and does not profile users. Seekport is also operated without advertising and has no conflicts of interest in the display of search results.
SISTRIXData collected by SEMrushBot is used for the Backlink Audit tool to check website backlinks
SemrushSemrushBotBacklinks is the Semrush bot collects data for Semrush's Backlink Analytics tool.
SemrushData collected by SEMrushBot is used for the Link Building tool to check website backlinks
SemrushSemrushbot crawls your website to analyze it for different SEO and technical issues.
SemrushData collected by SEMrushBot is used for the On Page SEO Checker and SEO Content template tools reports Data collected by SEMrushBot is used for the Topic Research tool reports
SemrushCheck for over 130 common website issues and get special reports about your site’s crawlability, use of markups, internal linking, speed/performance, HTTPS, and international SEO.
SemrushData collected by SEMrushBot is used for the SEO Writing Assistant tool to check if URL is accessible
SemrushSemrush is an all-in-one tool suite for improving online visibility and discovering marketing insights.
SemrushThe SendGrid Event Webhook sends email event data to customer APIs as SendGrid processes it.
SendGridSentry monitors webpages for availability and performance issues.
SentrySentry Uptime Monitoring is a feature of the Sentry platform that checks customer websites and APIs for availability.
Sentry Uptime BotSEO audit check bot is likely an automated tool within the WebCEO platform that performs comprehensive SEO audits on websites
SEO Audit CheckThe seo4ajax bot is used by a service that helps make single-page applications (SPAs) crawlable by search engines. It pre-renders JavaScript-heavy pages into static HTML so they can be indexed.
Capsule CodeThe Seobility Bot crawls websites to gather SEO Information and provide SEO analysis to its customers.
SeobilityOur monitoring agent checks website uptime on a 5 minute interval. It only checks verified customers & when the x-sequelwp header is valid.
SequelWPCrawls the Internet to assist in getting information on the link structure of sites on the web to assist SEO specialists
SE RankingSerpstatBot is the Serpstat bot collects data for Serpstat's Backlink Analysis tool
SerpstatProactive infrastructure monitoring for cloud, servers, containers & websites.
StackpathOur spider indexes the price, specifications and stock of hosting plans. We fully respect robots.txt and we have more information on https://www.serverhunter.com/spider/.
Server HunterSeznamBot is the search engine crawler for Seznam search.
SeznamShapBot helps discover and index websites for Parallel's web APIs.
ParallelShopify-Captain-Hook is a system used by Shopify to deliver webhooks. It sends automated messages to a user's server when specific events occur within their Shopify store.
ShopifyAn email client that proxies all images found in HTML emails from to protect end customer's IP address and connection private
Shortwave Communications Inc.Website accessibility, SEO, and content quality scanner
SilktideSite Search 360 is a popular Google Site Search replacement. Our crawler indexes content on our customers' sites for search.
SEMKNOX GmbHSite24x7's global website monitoring probe.
Site24x7Siteimprove content suite (i.e. Quality Assurance, Accessibility, Policy, and SEO). Crawls run on ports are 80 for HTTP and 443 for HTTPS.
SiteimproveThe Siteimprove LinkCheck crawler analyzes and monitors websites for quality assurance, SEO, and accessibility purposes, and keeps website content in line with brand guidelines and organizational policies.
SiteimproveThe SiteLock Spider is a web scanning service scans websites for malware and malicious code.
SiteLockSiteUpTimeBot is the HTTP monitoring probe for SiteUpTime.com.
SiteUpTimeSkroutz uses SkroutzBot web crawler to download XML feeds.
Skroutz S.A.Skroutz ImageBot to fetch the individual product images.
Skroutz S.A.Skype's URI Preview services fetches a page preview when someone posts a URL in a Skype message.
SkypeSlack's multi-purpose bot for service integration and webhook notifications.
SlackThis robot is used to fetch and cache images posted into Slack channels.
SlackSlickstream is a SaaS that indexes our customer's websites (with their approval) in order to provide engagement features for their site visitors, including site search, content recommendations, etc.
SlickstreamProviding archiving solutions to clients for compliance purposes
SmarshThe Smartology generates semantic vectors from domain pages in order to serve semantically-relevant ads on those pages
SmartologyCrawls partner company's websites to include them in our on-site search engine.
SMTnetSnipcart is an e-commerce solution for developers.
SnipcartDefault header that identifies SolarWinds Observability robots that are used for our synthetic monitoring.
SolarWindsPOS software connected to prestashop ecommerce website
Sora Caisse POSSpark Shipping is eCommerce automation software for retailers running WooCommerce
Spark ShippingSparkbot webhook integration is used for automating email transactions on web server events.
BirdUptime monitoring bot for healthchecks
SpectateSplunk Attack Analyzer (formerly known as TwinWave), visits URLs submitted by customers using a headless Chrome browser. DOM (Document Object Model), HAR (HTTP Archive), and other relevant data from these visits are analyzed to determine if the page is hosting malicious content.
SplunkSynthetic monitoring tool with global agents running in AWS
SplunkStape Scanner monitoring configuration of tracking tags.
Stape IncStatabot searches for stata.toc files and indexes their contents
StataCorp LLCBot to collect product prices for the official consumer price index of Austria
Statistik BotThe StatsDrone affiliate marketing statistics scraping and aggregating tool
StatsDroneThe StatusCake uptime monitor is used to monitor webpage availability and performance.
StatusCakeThe Steam Chat bot fetches previews of URLs shared within the Steam client's chat feature.
Valve SoftwareThe Google StoreBot is a search-engine-based program that automatically 'crawls' through web pages to gather and analyse data. Google uses crawlers that go through product pages and checkout processes using machine learning algorithms to fill in forms with information such as delivery addresses, and help compile other information on price, delivery, payments and more.
GoogleThe Stripe Webhooks service allows Stripe to push real-time event data to customers' application webhook endpoint when events happen in their Stripe account.
StripeStripebot is the Stripe automated web crawler that collects data from their users' websites. They use the collected data to provide services to their users and to comply with financial regulations.
StripeThe Sucuri bot is part of the Sucuri website security platform. It crawls websites to scan for malware, security risks, and blacklisting status.
SucuriScalable webhook platform featuring automatic retries, signature verification, deep observability, and a static-IP delivery bot—deploy hosted or self-hosted.
Svix Inc.Help Center Export is a Zendesk-approved app that integrates with any Zendesk help center and helps the customers with these tasks: Export all your articles and any meta-data: title, section, link, labels, updated time. Export all references to internal and external docs. Detect and export broken links and images for each article. In order to check for broken links the app is using a bot that attempts to access each link present in help center articles and check the response for errors.
Swfiteq LtdThe Taboola crawler visits websites advertiser campaign websites to audit the content of the page and gather site metadata and summary data.
TaboolaTelegramBot crawls websites to render a link preview when people send a message containing a URL in the Telegram messaging service.
TelegramTermly bot scanners for site compliance
TermlyTermlyBot is a web crawler that allows you to detect and categorize the cookies on your website automatically.
TermlyThe Terracotta bot scrapes websites for use in generating indices for serving searches using Ceramic's search product.
CeramicThousandEyes monitors network infrastructure, troubleshoots application delivery and maps Internet performance, all from a SaaS-based platform.
Thousand EyesCritical CSS Generator to Optimize Websites
MediavineTalkwalker delivers the consumer insights that help brands drive business impact. In a world full of conversations, the most successful global brands have switched to Talkwalker because we provide them with a powerful software platform to uncover, understand and derive the most valuable insights from internal and external data. Our listening and analytics platform enables more than 2,500 companies worldwide to protect their brands, measure their impact and gain the key consumer insights that drive purchase decisions.
Trendiction S.A.Trustly will send notifications / callbacks to merchant's system to provide updates on payment statuses.
Trustly Group ABThe Trade Desk crawler classifies webpage content to allow advertisers to choose where they show ads.
The Trade DeskTurnitin.com offers various services to the educational community. Most prominently, we provide a widely used and effective plagiarism detection service. Part of the plagiarism prevention service relies on comparing student papers to content found on the Internet. Since we do not know ahead of time which pages on the Internet a student will use we need to gather them all for comparison. However, we do have automated ways of throwing away content and links that would be irrelevant to our service.
TurnitinTwilio webhook requests triggered by Twilio when there are incoming SMSes, calls, etc.
Twilio, Inc.Automate complex operations end-to-end.
TwinA Twitter bot is a type of bot software that controls a Twitter account via the Twitter API. The bot software may autonomously perform actions such as tweeting, re-tweeting, liking, following, unfollowing, or direct messaging other accounts.
Twitterupday is a news aggregator app, and its bot crawls news sources. It collects and indexes articles to be recommended to users on its platform.
upday GmbH & Co. KGUpDownBot is the HTTP monitoring probe for updown.io.
UpdownUptime.com HTTP probe for website availability and performance monitoring.
UptimeUptime monitoring is a service that checks if a website is online. It will send you an alert if your website is “down”.
GoDaddyUptimeBot is an actionable website monitoring tool that works great with Slack.
UptimeBotFree Website Uptime Monitoring
UptimerobotUptimia is a website monitoring service, monitoring website performance and availability.
UptimiaGlobal load tests and synthetic monitoring
Uptrends GmbHVaultPress is a subscription service developed by Automattic, the company behind WordPress, that offers automated daily and real-time backups of WordPress websites onto WordPress.com's cloud servers. It is known for its ease of use, secure backups, and proactive security scanning.
AutomatticCrawler to extract the newest articles in the publisher's website (via feed or parsing html) to make a carrousel with images, links and text for our native ads module in order to improve recirculation in the publisher's web. Only crawls our publisher's webpages.
Digital GreenShopify theme editor alternative for live, real-time store editing via a secure iframe and controlled proxy.
Visually.ioW3C provides various free validation services that help check the conformance of Web sites against open standards.
World Wide Web Consortium (W3C)WARDBot tracks URL status codes, helping users monitor the availability of web pages they have added to the monitoring list.
WEBSPARKWebsite availability & SSL certificate expiration monitoring
WatchBotMonitoring system to check uptime on client websites.
Watchful LLCJob wrapping data processor handling jobs distribution from employer websites to multiple endpoints, like job boards, advertisement platforms, job alerts etc.
AspenTechLabs IncWebPageTest is one of the most popular and free tools for measuring webpage performance and enables you to run web performance tests on your site from a number of different locations across the world in a number of different browsers.
WebPageTestWebsitePulseBot is the HTTP monitoring probe for WebsitePulse's monitoring servince.
WebsitePulseWebStatus247 is an intelligent website monitoring bot that continuously checks the availability and uptime
WebTotem is a comprehensive security monitoring and defense platform that protects web applications and their data.
WebTotemCitoid is a Wikimedia service in VisualEditor that generates citations from URLs, DOIs, and ISBNs, relying on the Zotero Translation Server (see wikimedia-zotero) for accurate metadata, processed on demand from website visitors.
Wikimedia FoundationThe Wikimedia Foundation's Zotero Translation Server is a customized metadata extraction tool that powers Citoid (see wikimedia-citoid), retrieving citation data from URLs, DOIs, and ISBNs using Zotero translators, on demand from website visitor requests.
Wikimedia FoundationUsed by Wunderkind to perform health check on clients' domain.
WunderkindWordCountBot analyzes website word count based on public pages. All words belonging to public pages and included in HTML source code
WeglotThe Worldline Bot is associated with Worldline, a payment and transactional services company. It handles notifications and callbacks related to payment processing.
WorldlineThis is the major egress IP for our containerised WordPress platform so it is likely to be many flavours of WordPress and the potential to be any domain.
NamecheapPayment confirmation callbacks to ecommerce backends
WorldPayWormlyBot is HTTP monitoring probe for Wormly's uptime monitoring service.
WormlyWe offer WOVN.io, a service for localization websites. We run a crawler to get source language of our clients' websites
Wovn Technologies, Inc.Our plugin WPTimeCapsule is installed in more 30000 WordPress sites. When our backup servers send requests to trigger the backup on the WordPress sites, it is being blocked
WP Time CapsuleFetches data from Wordpress enabled sites for Umbrella plugin users.
WP UmbrellaWP Umbrella is the ultimate all-in-one solution to manage, maintain and monitor one, or multiple WordPress websites.
WP UmbrellaRuns a full scan of a site to find any broken links
WPMUDEVWPMUDEV Uptime Monitor 5.0 (https://wpmudev.com)
WPMUDEVWebsite archiver for our customers who have archive compliance requirements to fulfill them.
XY Archive ComplianceYahoo Japan Advertising Bot
Yahoo Japan CorporationYahoo Japan search engine crawler for SEO analysis
YahooYahoo Mail Proxy is a content fetch proxy that retrieves the page content of URLs that are embedded within emails sent to Yahoo Mail users. Having the content displayed through the proxy improves the security for email users while reducing overall network usage.
YahooYahoo Ad Monitor monitors the contents of webpages where Yahoo! ads are served.
YahooYahooCacheSystem caches website contents as part of the Yahoo! Search Service.
YahooYahoo! JAPAN manages and operates a system that accesses web pages published on the Internet for the purpose of providing services, research, development, maintenance, etc.
Yahoo! JAPANYahoo Link Preview's bot fetches data from URLs shared on Yahoo platforms.
YahooThe Yahoo Mail proxy fetches link content rendered in Yahoo's webmail service.
YahooYahoo! Slurp was the search engine crawler for Yahoo's search engine.
YahooThe main indexing robot for Yandex search.
YandexThe Yext Crawler provides Yext customers with a tool to retrieve data from their own websites.
YextA content based scraper only for partners we collaborate with who have given permission to have their website scraped.
Yokoy is a spend management SAAS solution. Webhooks generate requests to book expenses or invoices to customer's ERP system whenever the processing and approval process has been completed in Yokoy.
Yokoy Group AGYou.com Search Engine Crawler
You.comEasy automation for busy people. Zapier moves info between your web apps automatically, so you can focus on your most important work.
Zapier Inc.Webhooks for development of Zendesk ticketing system and apps.
ZendeskAI assistant for e-commerce stores. Only crawls sites upon request from the site owner.
Zipchat IncZoominfobot is an indexing robot for a web search engine, similar to Google. Created by Zoom Information Inc.(www.zoominfo.com), Zoominfobot’s patented technology continually scans millions of corporate websites, press releases, electronic news services, SEC filings and other online sources. Using advanced natural language processing algorithms, ZoomInfo has created a next generation search engine focused on finding pages with information about businesses and business professionals.
ZoomInfoZumBot is a web crawler that indexes webpages for Zum Open Internet Search.
Zum Internet Corpzvelo fetches content for web categorization.
zvelo