A site crawl is an automated pass through a website in which a program, called a crawler or spider, starts from one or more pages and follows links to discover the rest, listing the pages it finds with details such as status codes, titles and links. Search engines, SEO audits and price research rely on it.
How a crawler moves through a site
A crawl begins with seed addresses, usually the home page or a sitemap. The crawler requests a page, records what it returns, collects every link on it and adds new addresses to a queue. It then takes the next address from the queue and repeats, skipping pages it has already seen. The process continues until the queue is empty or a limit is reached.
- Seeds: the starting addresses, often supplemented by an XML sitemap.
- Queue: the list of discovered addresses still waiting to be visited.
- Scope rules: which domains, folders or page types the crawl may enter.
- Limits: maximum depth, number of pages and request rate.
The order of the queue shapes the result. A breadth-first crawl visits every page one click from the start before going deeper, which quickly maps the main sections. A depth-first crawl follows one branch to its end before backing up. For stores, breadth-first is usually preferred, because category and listing pages near the top lead efficiently to products.
Crawling compared with scraping
Crawling is about discovery and structure; scraping is about extracting specific values. A crawler answers the question of which pages exist and how they connect. A scraper answers what a given page says, such as its price. In a competitor price project, the crawl finds product pages within a rival’s category, and the scraper then reads the price from each page found.
The two steps also run on different schedules. Product lists change slowly compared with prices, so a team might crawl a competitor’s categories once a week to refresh the page list, while reading prices from those known pages daily. Separating the jobs keeps the frequent task light and avoids rediscovering the same pages again and again.
Uses in SEO and site health
Site owners crawl their own stores to see them as a search engine would. A crawl reveals broken links, redirect chains, duplicate titles, missing descriptions, pages that are orphaned with no internal links pointing to them, and pages blocked from indexing by mistake.
For an online store, crawl results often show product pages that are hard to reach. If a product sits five or six clicks from the home page, or only appears through a filter, search engines may visit it rarely. Fixing internal links can bring such pages back into view.
Crawls are also a useful check after a platform migration or a large catalogue import. Comparing the page list before and after shows which old addresses now return errors and need redirects, protecting both search visibility and customers who arrive from saved links or comparison sites.
Uses in price and assortment research
Crawling a competitor’s public category pages shows what they sell, not only what they charge. Comparing the discovered products with your own catalogue can reveal gaps, items they carry that you do not. BlackAha’s product gap finder is built around this kind of comparison.
Discovery also keeps price comparisons current. Competitors add new products, retire old ones and change page addresses. A periodic crawl of the relevant sections updates the list of pages to watch, so price checks do not keep reading removed pages or miss new ones.
Sizing a crawl: an example
Suppose, as an example, a garden supplies retailer wants to map a competitor’s outdoor furniture section. The section has 12 listing pages with 40 products each, so about 480 product pages plus the listing pages themselves. At a polite rate of one request every two seconds, a full pass takes roughly 16 minutes. Crawling the whole store, perhaps 30,000 pages, at the same rate would take more than 16 hours, which shows why scope rules matter.
Narrow scope saves time on both sides. A crawl limited to the sections you actually compete in gives a faster, cleaner result and puts less load on the site being read.
Crawling responsibly
- Respect the robots rules file and the site’s terms of service.
- Keep request rates low and avoid busy periods where possible.
- Stay out of logged-in areas, carts and account pages.
- Identify the crawler honestly rather than imitating a browser user.
- Stop or slow down if the site starts returning errors.
Rules on automated access differ between countries and depend on how collected data is used. When crawling sites you do not own, it is wise to take legal advice on your specific plans.
Common crawl problems
- Infinite spaces, such as calendars or filter combinations that generate endless new addresses
- Duplicate pages reached through tracking parameters or sort options
- Content that only appears after scripts run, which simple crawlers miss
- Pages that look different by region, language or currency
- Crawls that run so long the earliest pages are out of date when the last ones finish
Normalising addresses, removing unneeded parameters and setting a firm page limit solve most of these. For price work, it also helps to record the crawl date on each discovered page, so stale entries can be rechecked first.
Finally, check the output for gaps. If a section that should hold hundreds of products returns only a handful, the crawler probably missed a pagination pattern or a script-loaded list, and the scope rules need adjusting before the data is used.
Key takeaways
- A site crawl follows links automatically to discover the pages of a website.
- Crawling finds pages; scraping reads specific values such as prices from them.
- Store owners crawl their own sites to find broken links and hard-to-reach products.
- Crawling competitor sections reveals assortment gaps and keeps price comparisons current.
- Tight scope, low request rates and respect for site rules make crawls faster and fairer.
Frequently asked questions
What is the difference between a site crawl and indexing?
Crawling is the discovery step, where pages are visited and links followed. Indexing is a later step, where a search engine stores and organises page content so it can appear in results. A page can be crawled and still not be indexed.
How often should I crawl my own store?
Many stores run a full crawl monthly and a lighter check after major changes, such as a theme update or a catalogue import. Large stores that change daily may crawl key sections more often.
Does a sitemap replace crawling?
No, but it helps. A sitemap lists pages the owner wants found, which speeds discovery. Crawling still shows how pages link together and finds pages missing from the sitemap.
Related terms
In BlackAha, Store Platform Integration helps you connect your store to sync product prices automatically.