Crawler

To build a powerful CMS, you need data from all over. One great source of content is your own website. The Crawler scrapes the underlying content on your website, which our Connectors can then turn into structured data for your CMS in just a few clicks. With highly customizable configuration options, the crawler will pull the exact information you want from your website — even text from PDF files you store online — saving you time and expanding your data source options.

Scrape Data from Your Website

The Yext Crawler can scrape your HTML and/or PDF file content from a specified set of domains, pages, or sub-pages under a domain, giving you control over exactly which content is brought into your CMS. If your website data changes consistently, configure crawls to run on schedule, or, if you just need a one time backfill, crawl once. Don't want to crawl a specific set of pages? Easily blacklist unwanted URLs for the crawler to skip.

Configure a Crawler

Once your crawler scrapes your website, a Connector converts and structures the raw HTML into content for entities. Highly customizable configuration allows you to extract exactly the data you need. You can specify a target path based on CSS or XPath selectors or use built-in selectors to capture commonly extracted data types, like Page Title and Body Content. Connectors can extract Text, HTML, URLs, Images, and more. Learn more about Connectors and The Crawler working together here.

Transform Your Data

The data on your website might not be formatted exactly how you want it in your CMS. Use transforms in Connectors to manipulate data scraped by the Crawler before it enters Content. Connectors allow you to preview any changes to your data in real time to ensure maximum accuracy. With transforms, you can remove unwanted characters, fix capitalization, find and replace text, format dates, and more.

Want to become a Yext Expert?

Join Hitchhikers, the new Yext training platform and community, to test your knowledge, earn badges, and engage with the experts.

Create your free account

One Platform. Unlimited Solutions.

Drive revenue and delight customers

Transform site visitors into customers and learn more about what matters most to them with a dynamic AI site search on your website.

Learn More

Capture and convert more local business with location listings. Turn maps, GPS apps, and local directories into your best marketing channel.

Learn More

Your online reputation can win or lose you business. Generate, manage, and respond to reviews at scale with Reputation Management from Yext.

Learn More
Learn More

Empower customers and agents to self-serve

Enable customers to find instant answers to their support questions via FAQs, guides, tutorials, videos, ebooks, manuals, and more

Learn More

Decrease case creation by recommending related content based on the issue being raised - right within the case submission form.

Learn More

Resolve cases in record time with agent desktop search. Empower agents to drive faster case resolution as they troubleshoot customer issues at scale.

Learn More

Answer your customers’ questions before they even reach your help site by improving your ranking and presence in search.

Learn More

Deliver a seamless experience with in-app support. Allowing customers to seek help right within your app makes for a superior user experience.

Learn More
Learn More

Increase sales

Connect customers with relevant products, services and experiences from search to checkout.

Learn More

Yext Pages has powerful tools to let you confidently launch, manage, and scale your business, by building stunning detailed product pages that capture high-intent traffic.

Learn More

Boost your local online presence with Yext's local landing pages. Easily create and manage customized pages for each of your business locations, complete with accurate information, photos, and calls to action. Attract more customers and drive more sales with Yext's local landing pages.

Learn More

Create an in-person sales experience online that available 24/7. Guide your customers towards the right products or services that align with their individual needs.

Learn More

Make it easier for customers to find your products by connecting your physical and digital storefront.

Learn More

Tailor product data in bulk to easily update information wherever customers shop your inventory.

Learn More

Effectively manage all of your product data in one place and easily share details to third-party marketplaces.

Learn More

Build shopping excitement and trust that fosters more purchases.

Learn More

Turn each interaction into a long-lasting relationship by connecting better with shoppers at every step of their customer journey

Learn More
Learn More

Improve employee productivity and satisfaction

Employees need company information to do their jobs. Make it easy, fast, and fun to find with AI-powered workplace search solutions from Yext.

Learn More

Build custom experiences fast

Build on the Yext platform for a fully custom AI search experience — fast. With SDKs, APIs, and robust documentation, the Yext Answers Platform provides the building blocks to create a bespoke search experience.

Learn More

Explore Related Features

Yext is the leader in managing your brand’s digital presence. Learn why.