Software, your way.
How To Get Good Custom Software
(Download)
(PDF)
burger menu icon
WillMaster

WillMaster > LibraryWebsite Development and Maintenance

FREE! Coding tips, tricks, and treasures.

Possibilities weekly ezine

Get the weekly email website developers read:

 

Your email address

name@example.com
YES! Send Possibilities every week!

Keeping Test Pages Out Of Search Engine Indexes

If you develop web pages for clients or friends, you'll want to keep the development and test pages out of search engine indexes. There are a couple reasons to keep the pages out of indexes.

  1. The search engine's algorithms may get the idea your website is about, or at least partly about, the focus of the pages you are developing. If your website is about developing websites and you are creating pages for a site related to flying squirrels, getting the flying squirrel pages indexed could well dilute your website's focus (from a search engine's point of view).

  2. If the development pages on your website are indexed, then later moved to the client or friend's website, the client or friend's site may be perceived as containing duplicate content.

There is an easy solution. Two steps:

  1. Create a directory wherein all your development and testing will take place. (Let's assume that directory is named "/dev/")

  2. Put 2 lines into the robot's text file. (Example assumes the development director is "/dev/")

    User-Agent: *
    Disallow: /dev/
    

A robots.txt file is used to control web robot and spider access to certain directories and files. It is a plain text file that is publicly available. (Ours is here.) Web robot and spider compliance with robots.txt directives is voluntary. I believe all popular search engine spiders do comply. See About /robots.txt for more information.

Do all development and testing in the robots.txt disallowed directory or its subdirectories. Search engines that respect robots.txt (all popular search engines) will not index files in the disallowed directory or subdirectories.

Will Bontrager

Was this article helpful to you?
(anonymous form)

Support This Website

Some of our support is from people like you who see the value of all that's offered for FREE at this website.

"Yes, let me contribute."

Amount (USD):

Tap to Choose
Contribution
Method

All information in WillMaster Library articles is presented AS-IS.

We only suggest and recommend what we believe is of value. As remuneration for the time and research involved to provide quality links, we generally use affiliate links when we can. Whenever we link to something not our own, you should assume they are affiliate links or that we benefit in some way.

How Can We Help You? balloons
How Can We Help You?
bullet Custom Programming
bullet Ready-Made Software
bullet Technical Support
bullet Possibilities Newsletter
bullet Website "How-To" Info
bullet Useful Information List

© 1998-2001 William and Mari Bontrager
© 2001-2011 Bontrager Connection, LLC
© 2011-2024 Will Bontrager Software LLC