An XML sitemap becomes even more important when: 

• A site doesn’t have strong internal linking.
• There are no strong backlinks for your site.
• When you add new pages regularly and quite frequently. 

It is designed for search engine bots to guide them easily navigate through your website. 

On the other hand, an HTML sitemap is designed for humans. 

An HTML sitemap has links that can direct users to various pages on your website.

If you use an HTML sitemap, then why exactly would you need an XML sitemap? 

The thing is you need both the sitemaps! An XML sitemap is designed for your search engines while an HTML sitemap is designed for your human readers. 

Your XML sitemap is like a blueprint of your website directing the search engine to all the important pages. 

It can also let your site be discovered when there is not enough internal linking. Not only does it  

Tell Google about the content on your site, but also help the crawlers to better crawl them. 

Is it important to implement it on every website?

Yes! Having an XML sitemap is always a big advantage to a website, new or old. It allows even a messed-up site to mention the URL of its pages in an organized and systematic manner. 

Always include the pages that are relevant for your website’s traffic and have great content! Do not include URLs of the sites that you don’t want to be registered with the search engines. 

But, how do you exactly do this? 

Well, it is done with the help of meta robots!

Robot.txt file: this file is used when we don’t want the crawlers/spiders to crawl through certain pages/posts. 

This instructs the crawlers what you want to be crawled and what not. This file guides the bots when they navigate through the site. 

However, these instructions will only be followed by the web crawlers and not the bots that invade your site. Before understanding this, let us know what the crawling budget is? 

All the crawlers have a certain limit to which they can crawl any website. This is what we call a crawling budget. The number of pages a crawler can crawl within a fixed time window defines their crawling budget. 

If your budget gets exhausted but the site is not fully crawled, the leftover pages will not get indexed by Google. 

Though these bots are so smart at their work that most of the time, such situations don’t arise. 

Not only does it tell the bots which page to crawl, but also stops them from crawling the pages that you don’t want to index. 

Situations where you need to be concerned about your crawling budget: 

• When you have a big site: a large site will have a large number of pages that you would want to be crawled. Due to this big network of pages, Google might not be able to find them all easily. 
• Extra budget for new pages: if you have lately added new pages to your site, you would also need a crawling budget for getting them indexed too. In order to achieve this, it is important that your site doesn’t waste any budget. 
• Redirects: if you have a lot many redirects on your site, you will waste a lot of your crawling budget stuck in the redirect chain. 

As we just learned that due to a definite amount of crawling budget, you just can not waste it. 

So, it becomes important to only crawl pages that are important for your website and not waste the budget on not-so-important pages.