{"id":2419,"date":"2022-08-26T09:33:08","date_gmt":"2022-08-26T09:33:08","guid":{"rendered":"https:\/\/www.adlift.com\/in\/?post_type=blog_post&#038;p=2419"},"modified":"2023-08-02T10:50:29","modified_gmt":"2023-08-02T10:50:29","slug":"the-beginners-handbook-to-robot-txt","status":"publish","type":"blog_post","link":"https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/","title":{"rendered":"The Beginner&#8217;s Handbook to Robot txt"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_66_1 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title \" >Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-1\" href=https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/#Key_Takeaways title=\"Key Takeaways\">Key Takeaways<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-2\" href=https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/#What_Exactly_is_a_Robotstxt_file title=\"What Exactly is a Robots.txt file?\">What Exactly is a Robots.txt file?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-3\" href=https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/#But_is_Robotstxt_Required title=\"But, is Robots.txt Required?\">But, is Robots.txt Required?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-4\" href=https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/#Finding_the_Robottxt_File title=\"Finding the Robot.txt File\">Finding the Robot.txt File<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-5\" href=https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/#What_Does_a_Robottxt_File_Look_Like title=\"What Does a Robot.txt File Look Like?\">What Does a Robot.txt File Look Like?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-6\" href=https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/#What_Issues_Can_Robotstxt_Cause title=\"What Issues Can Robots.txt Cause?\">What Issues Can Robots.txt Cause?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-7\" href=https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/#Conclusion title=\"Conclusion\">Conclusion<\/a><\/li><\/ul><\/nav><\/div>\n<p>The robots.txt is the only file where its size doesn&#8217;t matter! It may be tiny, but it has big implications for your website and can impact your ranking considerably. Understanding what this file stands for and why you need to update it properly is a crucial aspect of technical SEO, so don&#8217;t miss out on this one! In this blog, you will learn all there is to know about this amazing file that lives (rent-free) on your site.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Key_Takeaways\"><\/span><strong>Key Takeaways<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-full wp-image-2421\" src=\"https:\/\/www.adlift.com\/in\/wp-content\/uploads\/sites\/2\/2022\/08\/1.png\" alt=\"\" width=\"768\" height=\"424\" srcset=\"https:\/\/www.adlift.com\/in\/wp-content\/uploads\/sites\/2\/2022\/08\/1.png 768w, https:\/\/www.adlift.com\/in\/wp-content\/uploads\/sites\/2\/2022\/08\/1-300x166.png 300w\" sizes=\"(max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/p>\n<ul>\n<li>txt files live at the root of your domain name.<\/li>\n<li>txt files allow you to restrict search engine crawlers&#8217; reach on your website.<\/li>\n<li>This file can be used per the requirement to give directives of partial or full access.<\/li>\n<li>txt file can assist in managing your &#8216;crawl budget&#8217;.<\/li>\n<\/ul>\n<h4><span class=\"ez-toc-section\" id=\"What_Exactly_is_a_Robotstxt_file\"><\/span><strong>What Exactly is a Robots.txt file?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>Robots.txt file actually lives on your website, you can easily find it at: &#8216;example.com\/robot.txt&#8217;.<br \/>\nSo, if you&#8217;re curious about whether or not you have a <a href=https:\/\/www.adlift.com\/in\/seo-tools\/robots-txt-generator>robots.txt<\/a> file, just go over to your site, add \/robots.txt to your domain name and voila! There it is.<\/p>\n<p>What exactly does the robot txt file do? Well, it tells Googlebot robots whether or not to crawl your website. It makes recommendations to the search bots who are crawling your webpage. This file tells the search engine Googlebot robots where they can crawl and where they can&#8217;t. It allows you to block all portions of your website or index the website. With this file, you can even block certain pages from being crawled.<\/p>\n<p>However, a robots.txt file cannot absolutely guarantee that Googlebot robots won&#8217;t crawl an excluded page because it is a voluntary system. Though it is rare for major search engine bots to disobey directives, bad crawl robots, such as spambots, malware etc., are not exactly famous for being obedient. These bots can still ignore the directives and crawl the restricted page.<\/p>\n<p>Another interesting thing about a robots.txt file is that it is publicly available, meaning anyone can access it. We already specified its exact location, but to refresh your memory: Adding a \/robots.txt can lead you right to it. Since it is publicly accessible, we recommend not including any files or folders which contain business-critical information.<\/p>\n<p>When Googlebot robots txt interpret directions in the robots.txt file, they receive one of three instructions:<\/p>\n<ul>\n<li>Partial access: Individual elements of the site can be crawled for partial access.<\/li>\n<li>Full access: Crawling everything is possible.<\/li>\n<li>Full denial: Robots are not allowed to crawl anything.<\/li>\n<\/ul>\n<h4><span class=\"ez-toc-section\" id=\"But_is_Robotstxt_Required\"><\/span><strong>But, is Robots.txt Required?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>Now that we are familiar with the concept of a robot txt file let us tell you the importance of having one. It is recommended by Google to ensure that your website has a robots.txt file. Therefore, if Google and other crawlers can&#8217;t find it, there is a chance that they might not crawl your website at all.<\/p>\n<p>They are important because they can help manage crawler activities on your website. This is done so that they don&#8217;t overwork your index pages or websites, which are restricted for public viewing. For your better understanding, we have made a list of the reasons why you need to have a robots.txt file.<\/p>\n<p><strong>1) Hide Duplicate and Non-Public Pages:<\/strong> One of the primary benefits of a robots.txt file is that it helps in blocking certain pages or files from being crawled. You won&#8217;t need every page on your website to rank; this is where a robots txt file can help you block them from <a href=https:\/\/www.adlift.com\/in\/seo-tools\/robots-txt-generator\/>Googlebot robots crawler<\/a>. Some examples of these pages are login pages, duplicate pages, internal search results pages etc.<\/p>\n<p><strong>2) Hide Miscellaneous Resources:<\/strong> In some cases, website owners might want to hide certain resources, like PDFs, videos etc., from search results. In such a case, using a robots.txt file is one of the best to prevent them from being indexed.<\/p>\n<p><strong>3) Utilize Crawl Budget:<\/strong> Crawl budget can be described as the number of pages a search engine will crawl at any given time. A crawl budget is important because it helps you ensure that your number of index pages does not exceed your crawl budget. A robot txt file can help you optimize your crawl budget by blocking unwanted pages from being indexed.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Finding_the_Robottxt_File\"><\/span><strong>Finding the Robot.txt File<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>As we said before, the Robot txt file lives on your website itself. Its size has been specified at 500 KB by Google. Hence it does not take up a lot of space. You can check out this file for any website by adding: &#8216;xyz.com\/robots.txt &#8216;.<\/p>\n<p>One important thing to know about this file is that it should always live at the root of your domain. If, in case, it lives anywhere else, then the search engine crawlers will not find it and automatically assume you don&#8217;t have one.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"What_Does_a_Robottxt_File_Look_Like\"><\/span><strong>What <\/strong><strong>Does a Robot.txt File Look Like?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>A robot txt file consists of one or multiple blocks of directives, where each one is specified as a &#8216;user-agent&#8217;. It also consists of a simple allow or disallow button. This is what it looks like:<\/p>\n<p>There are two common directives in the robots.txt file. They are user agents and disallowed.<\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li><strong>User-agent:<\/strong> Every block of directive starts with a user agent which directs the crawler being addressed. For instance, if you want to tell a Googlebot not to crawl a certain page, then your directive will begin like this:<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>User-agent:*<br \/>\nDisallow: \/directory-name\/<\/p>\n<ul>\n<li><strong>Disallow: <\/strong>The second most common directive in a robot txt file is disallow rule. It specifies the folder or sometimes the entire directory which is to be excluded from crawling.<\/li>\n<\/ul>\n<h4><span class=\"ez-toc-section\" id=\"What_Issues_Can_Robotstxt_Cause\"><\/span><strong>What Issues Can Robots.txt Cause?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>A small mistake in a robot txt file can have some detrimental effects on your website. But it&#8217;s not the end of the world! There&#8217;s nothing a little bit of &#8216;attention to detail\u2019 can&#8217;t fix. Here are some mistakes that should be avoided with robotstxt files<\/p>\n<ul>\n<li><strong>Blocking the Entire Site:<\/strong> It sounds silly, but it does happen. Web developers block one section of a site while working on it but then forget to unblock it once finished. This affects the website&#8217;s rankings considerably. Therefore, the next time you block one section, ensure you unblock it after the site goes live.<\/li>\n<li><strong>Omitting Previously Indexed Pages: <\/strong>A word of caution: don&#8217;t block pages which are already indexed. This is because the indexed page will get stuck in Google&#8217;s index. To remove them from the index, add a meta robots &#8220;noindex&#8221; tag to the sites themselves and allow Google to crawl and analyze that.<\/li>\n<\/ul>\n<h4><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span><strong>Conclusion<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>Understanding robots txt files is not an easy feat. Whatever we have covered in this blog is just the tip of the iceberg. To fully comprehend how important this seemingly small file is, we recommend getting in touch with us at AdLift. We have years of experience when it comes to digital marketing, and we never miss out on small details like the correct updating of robots.txt files.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The robots.txt is the only file where its size doesn&#8217;t matter! It may be tiny, but it has big implications for your website and can impact your ranking considerably. Understanding what this file stands for and why you need to update it properly is a crucial aspect of technical SEO, so don&#8217;t miss out on &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;The Beginner&#8217;s Handbook to Robot txt&#8221;<\/span><\/a><\/p>\n","protected":false},"author":98,"featured_media":2420,"parent":0,"menu_order":0,"template":"","format":"standard","meta":[],"post-tag":[],"blog-category":[17],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v17.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>The Beginner&#039;s Handbook to Robot txt - AdLift India<\/title>\n<meta name=\"description\" content=\"The robots.txt is the only file where its size doesn&#039;t matter! It may be tiny, but it has big implications for your website and can impact your ranking\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The Beginner&#039;s Handbook to Robot txt - AdLift India\" \/>\n<meta property=\"og:description\" content=\"The robots.txt is the only file where its size doesn&#039;t matter! It may be tiny, but it has big implications for your website and can impact your ranking\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/\" \/>\n<meta property=\"og:site_name\" content=\"AdLift India\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/AdLiftMarketingPrivateLimited\/\" \/>\n<meta property=\"article:modified_time\" content=\"2023-08-02T10:50:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.adlift.com\/in\/wp-content\/uploads\/sites\/2\/2022\/12\/Artboard-4.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1431\" \/>\n\t<meta property=\"og:image:height\" content=\"655\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@adliftindia\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"6 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The Beginner's Handbook to Robot txt - AdLift India","description":"The robots.txt is the only file where its size doesn't matter! It may be tiny, but it has big implications for your website and can impact your ranking","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/","og_locale":"en_US","og_type":"article","og_title":"The Beginner's Handbook to Robot txt - AdLift India","og_description":"The robots.txt is the only file where its size doesn't matter! It may be tiny, but it has big implications for your website and can impact your ranking","og_url":"https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/","og_site_name":"AdLift India","article_publisher":"https:\/\/www.facebook.com\/AdLiftMarketingPrivateLimited\/","article_modified_time":"2023-08-02T10:50:29+00:00","og_image":[{"width":1431,"height":655,"url":"https:\/\/www.adlift.com\/in\/wp-content\/uploads\/sites\/2\/2022\/12\/Artboard-4.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_site":"@adliftindia","twitter_misc":{"Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Organization","@id":"https:\/\/www.adlift.com\/in\/#organization","name":"Adlift","url":"https:\/\/www.adlift.com\/in\/","sameAs":["https:\/\/www.facebook.com\/AdLiftMarketingPrivateLimited\/","https:\/\/www.instagram.com\/adliftindia\/","https:\/\/www.linkedin.com\/company\/adlift-marketing-pvt-ltd-\/","https:\/\/youtube.com\/channel\/UCmkSk7fwZboGVIEPH5gsxxA","https:\/\/twitter.com\/adliftindia"],"logo":{"@type":"ImageObject","@id":"https:\/\/www.adlift.com\/in\/#logo","inLanguage":"en-US","url":"https:\/\/www.adlift.com\/in\/wp-content\/uploads\/sites\/2\/2023\/02\/adlift-logo.png","contentUrl":"https:\/\/www.adlift.com\/in\/wp-content\/uploads\/sites\/2\/2023\/02\/adlift-logo.png","width":340,"height":220,"caption":"Adlift"},"image":{"@id":"https:\/\/www.adlift.com\/in\/#logo"}},{"@type":"WebSite","@id":"https:\/\/www.adlift.com\/in\/#website","url":"https:\/\/www.adlift.com\/in\/","name":"AdLift India","description":"","publisher":{"@id":"https:\/\/www.adlift.com\/in\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.adlift.com\/in\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"ImageObject","@id":"https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/#primaryimage","inLanguage":"en-US","url":"https:\/\/www.adlift.com\/in\/wp-content\/uploads\/sites\/2\/2022\/12\/Artboard-4.jpg","contentUrl":"https:\/\/www.adlift.com\/in\/wp-content\/uploads\/sites\/2\/2022\/12\/Artboard-4.jpg","width":1431,"height":655},{"@type":"WebPage","@id":"https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/#webpage","url":"https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/","name":"The Beginner's Handbook to Robot txt - AdLift India","isPartOf":{"@id":"https:\/\/www.adlift.com\/in\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/#primaryimage"},"datePublished":"2022-08-26T09:33:08+00:00","dateModified":"2023-08-02T10:50:29+00:00","description":"The robots.txt is the only file where its size doesn't matter! It may be tiny, but it has big implications for your website and can impact your ranking","breadcrumb":{"@id":"https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.adlift.com\/in\/blog\/the-beginners-handbook-to-robot-txt\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.adlift.com\/in\/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https:\/\/www.adlift.com\/in\/blog_post\/"},{"@type":"ListItem","position":3,"name":"The Beginner&#8217;s Handbook to Robot txt"}]}]}},"_links":{"self":[{"href":"https:\/\/www.adlift.com\/in\/wp-json\/wp\/v2\/blog_post\/2419"}],"collection":[{"href":"https:\/\/www.adlift.com\/in\/wp-json\/wp\/v2\/blog_post"}],"about":[{"href":"https:\/\/www.adlift.com\/in\/wp-json\/wp\/v2\/types\/blog_post"}],"author":[{"embeddable":true,"href":"https:\/\/www.adlift.com\/in\/wp-json\/wp\/v2\/users\/98"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.adlift.com\/in\/wp-json\/wp\/v2\/media\/2420"}],"wp:attachment":[{"href":"https:\/\/www.adlift.com\/in\/wp-json\/wp\/v2\/media?parent=2419"}],"wp:term":[{"taxonomy":"post-tag","embeddable":true,"href":"https:\/\/www.adlift.com\/in\/wp-json\/wp\/v2\/post-tag?post=2419"},{"taxonomy":"blog-category","embeddable":true,"href":"https:\/\/www.adlift.com\/in\/wp-json\/wp\/v2\/blog-category?post=2419"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}