{"version":"1.0","provider_name":"HostGator","provider_url":"https:\/\/www.hostgator.com\/blog","author_name":"Amy Green","author_url":"https:\/\/www.hostgator.com\/blog\/author\/amygreen\/","title":"How to Optimize Your WordPress Robots.txt - HostGator","type":"rich","width":600,"height":338,"html":"<blockquote class=\"wp-embedded-content\" data-secret=\"mdUx8xkWVn\"><a href=\"https:\/\/www.hostgator.com\/blog\/how-to-optimize-your-wordpress-robots-txt\/\">How to Optimize Your WordPress Robots.txt<\/a><\/blockquote><iframe sandbox=\"allow-scripts\" security=\"restricted\" src=\"https:\/\/www.hostgator.com\/blog\/how-to-optimize-your-wordpress-robots-txt\/embed\/#?secret=mdUx8xkWVn\" width=\"600\" height=\"338\" title=\"&#8220;How to Optimize Your WordPress Robots.txt&#8221; &#8212; HostGator\" data-secret=\"mdUx8xkWVn\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\" class=\"wp-embedded-content\"><\/iframe><script>\n\/*! This file is auto-generated *\/\n!function(d,l){\"use strict\";l.querySelector&&d.addEventListener&&\"undefined\"!=typeof URL&&(d.wp=d.wp||{},d.wp.receiveEmbedMessage||(d.wp.receiveEmbedMessage=function(e){var t=e.data;if((t||t.secret||t.message||t.value)&&!\/[^a-zA-Z0-9]\/.test(t.secret)){for(var s,r,n,a=l.querySelectorAll('iframe[data-secret=\"'+t.secret+'\"]'),o=l.querySelectorAll('blockquote[data-secret=\"'+t.secret+'\"]'),c=new RegExp(\"^https?:$\",\"i\"),i=0;i<o.length;i++)o[i].style.display=\"none\";for(i=0;i<a.length;i++)s=a[i],e.source===s.contentWindow&&(s.removeAttribute(\"style\"),\"height\"===t.message?(1e3<(r=parseInt(t.value,10))?r=1e3:~~r<200&&(r=200),s.height=r):\"link\"===t.message&&(r=new URL(s.getAttribute(\"src\")),n=new URL(t.value),c.test(n.protocol))&&n.host===r.host&&l.activeElement===s&&(d.top.location.href=t.value))}},d.addEventListener(\"message\",d.wp.receiveEmbedMessage,!1),l.addEventListener(\"DOMContentLoaded\",function(){for(var e,t,s=l.querySelectorAll(\"iframe.wp-embedded-content\"),r=0;r<s.length;r++)(t=(e=s[r]).getAttribute(\"data-secret\"))||(t=Math.random().toString(36).substring(2,12),e.src+=\"#?secret=\"+t,e.setAttribute(\"data-secret\",t)),e.contentWindow.postMessage({message:\"ready\",secret:t},\"*\")},!1)))}(window,document);\n\/\/# sourceURL=https:\/\/www.hostgator.com\/blog\/wp-includes\/js\/wp-embed.min.js\n<\/script>\n","thumbnail_url":"https:\/\/www.hostgator.com\/blog\/wp-content\/\/uploads\/2015\/07\/wordpress-plugins.png","thumbnail_width":610,"thumbnail_height":417,"description":"What is a Robots.txt File? The robots.txt is a very small but important file located in the root directory of your website. It tells web crawlers (robots) which pages or directories can or cannot be crawled. The robots.txt file can be used to block search engine crawlers entirely or just restrict their access to certain [&hellip;]"}