<% user.full_name %>
<% util.level ? util.level.title : '' %>
Explorer Pioneer Apollo Viking Voyager Pathfinder Orion
Points
<% user.points ? user.points.toLocaleString() : 0 %>
Badges
<% util.badgeCount %>
Streaks
Congratulations You've <% util.newestBadge.isCurriculum ? 'completed' : 'earned' %> the
<% util.newestBadge.title %> <% util.newestBadge.isCurriculum ? 'curriculum' : 'badge' %>!
previous
next
Share to Social Media
Linked In Facebook
Status
<% util.level ? util.level.title : '' %>
<% util.level ? util.level.points : '' %>
<% util.nextLevel ? util.nextLevel.title : '' %>
<% util.nextLevel ? util.nextLevel.points : '' %>
Level up! <% util.levelPointsRemaining %> points to go
Level up! <% util.levelPointsRemaining %> points to go
Badges
Completed Hitchhikers Track Completed Knowledge Graph Track Completed Pages Track Completed Search Backend Track Completed Search Frontend Theme Track Completed Search UI React Track Completed Solution Templates and the Admin Console Track Completed Technical Skills Track Completed Platform Basics Track Completed Listings Track Completed Reviews Track Completed Industry Overview Track Completed Analytics Track Completed Industry Overview & Product Basics Track Completed Multi-Language Experiences Track Completed Chat Track
<% util.expandBadges ? 'Less' : 'More' %>
Playground Accounts
New +
No Playground Accounts
Create Playground +
View Profile
Log Out loading
Mission Complete! 🚀
Congratulations, you’ve completed your first module!

We hope you enjoy exploring the rest of the site. Please don’t forget to leave us feedback - we are constantly looking to improve. Happy Hitchhiking!
Home
Welcome to Hitchhikers! 👋
To learn more about the platform we encourage you to get started by completing the Hitchhikers Program Track and earning your first badge!
Start Here
Close notification
Guide
Create a Crawler Connector
Arrow Back
Completed
Introduction
Introduction
completed
Completed 1
Step 1
Whitelist User Agent and IP Addresses
completed
Completed 2
Step 2
Create a Crawler
completed
Completed 3
Step 3
Domains Source Type Settings
completed
Completed 4
Step 4
Sitemap Source Type Settings
completed
Completed 5
Step 5
Build Your Connector
completed
home

Step 4: Sitemap Source Type Settings

If you’ve chosen a Sitemap as your source type, enter your sitemap URL and choose whether you want to reference the lastmod tag on your sitemap when crawling pages.

Which Pages or Domains Would You Like to Crawl?

Enter a sitemap URL. To get your sitemap URL, refer to the documentation for your content management system, or ask your web team. Below are instructions for some content management systems:

You’ll also choose whether the crawler should look at the lastmod tag on URLs in your sitemap when performing a crawl. Selecting this option means that only URLs that have been modified since the last crawl took place will be crawled again.

Enter the sitemap details

Which URLs Should Be Omitted from the Crawl?

This is optional: specify any URLs you want to blacklist.

Enter blacklisted URLs

Exclude specific URLs from a crawl, even if they match your chosen crawl strategy and other settings. Enter each URL to blacklist on a separate line. You can also use wildcard notation here.

Advanced Crawler Settings

The rate limit determines how many tasks the crawler can execute on a site at one time, without impacting site performance.

By default, this is set to 100. You may need to consult your web team in order to determine an ideal rate limit for your site.

Set the rate limit

Save Crawler

Once you’ve configured your crawler settings, click Save Crawler at the bottom of the screen.

<% elem.innerText %>