|
Software
to help you create a robots.txt file
By Sumantra Roy
Robogen
is a software for Windows 95, Windows 98 and
Windows NT based computers which can help
you create robots.txt files without having
to worry about the correct syntax of a robots.txt
file. It comes in two editions: the free Limited
edition and the Standard edition which costs
$12.95. You can get Robogen from http://www.rietta.com/robogen/.
In
this article, I'll tell how you can use Robogen
to simplify the process of creating a robots.txt
file. This article assumes that you have read
my article on what a robots.txt file is and
how you can manually create it. If you missed
the article, it is available at http://www.1stSearchRanking.com/t.cgi?3761&robots.htm
I
would recommend that you download the trial
version of the Standard Edition. The Limited
edition does not support as many search engines
as the Standard Edition.
Once
you have downloaded Robogen and installed
it in your hard drive, start the program.
Here are the steps that you should follow
to create a robots.txt file:
Step
1: Click on the Logon FTP Server button. In
the dialog box that appears, type in the address
of the FTP server for your web site, make
sure that the "This server requires me to
log in" option is checked, and then type in
the username and the password which you use
to access the server in the appropriate boxes.
Now,
depending on how your web host has set up
the server, the files for your web site may
be present in the root directory of the server
itself. Or, it may be present in a sub-directory
of the root directory. In the latter case,
click on the "Use specified root" option and
type the relevant sub-directory in the text
box besides the option. If you don't know
in which sub-directory the files for your
web site are present, ask your web host to
give it to you.
Done?
Now click on OK. After a while, Robogen will
display the directories and files which are
present in your site in the right pane of
the main Robogen program window. If you find
that Robogen has not displayed the files in
your site, it means that you did not enter
the correct sub-directory in which the files
of your web site are present. In this case,
ask you web host to give the correct sub-directory
to you.
Step
2: In the left pane of the main Robogen window,
click on the small button present to the right
of the drop-down combo box at the top. In
the menu that appears, select "Popular Search
Engines".
Step
3: Now, click on the drop-down combo box and
choose the search engine that you are targeting
(say, AltaVista). In the right pane, select a page
which is NOT meant for AltaVista. Now click
on the "<<<" button. You will find that the
file which you had selected in the right pane
is now present in the left pane. In this way,
click on each of the pages which are not meant
for AltaVista and press the "<<<" button.
Repeat
step 3 for each of the search engines that
you are targeting. Just select the search
engine that you are targeting from the drop-down
combo box, select the pages which are not
meant for the search engine one by one from
the right pane, and click on the "<<<" button
after clicking on each age. Repeat this process
for all the search engines that you are targeting.
Step
4: Upload the file to your server using the
"Upload Robots.Txt" button. If you want, you
can also use the Save menu item from the File
menu to save a copy of the robots.txt file
in your hard drive.
Article by Sumantra
Roy. Sumantra is one of the most respected
search engine positioning specialists on the
Internet. To have Sumantra's company place
your site at the top of the search engines,
go to http://www.1stSearchRanking.com/t.cgi?3761
For more advice on how you can take your web
site to the top of the search engines, subscribe
to his FREE newsletter by going to http://www.1stSearchRanking.com/t.cgi?3761&newsletter.htm
|
