What is robots.txt? Examine why it's critical to your site's SEO architecture. Robots.txt blocks search spiders from accessing unimportant areas of your site. It's an important tool, because you want to avoid indexing low-value areas of your site. This lesson teaches you what directories to block with Robots.txt, how to make the file, and what CNN's and Google's robot.txt files look like.
Length: 6 min.
You will learn:
Why robots.txt is critically important to your site's SEO
How to construct and proofread your robots.txt file
Why you should block indexing of pages that dilute your website's link popularity and use tools to find these
How to download other sites' robots.txt files to see what directories they choose to block indexing
Make sure search engines, and your audience, can find your site by
learning today's most effective SEO best practices from recognized
authorities, led by respected speaker and consultant Todd Malicoat.