robot txt file

[needhelp]

I was checking over my adsense account and noticed that a couple of pages had been blocked for adsense,
The reason given is something to do with a robot txt file on the site not allowing adsense on it.
Is this something i should be concerned about?

[cameraplus]

You have a robots.txt file at http://www.candleliteonline.com/robots.txt which contains the following:


# robots.txt for www.candleliteonline.com

User-agent: *
Disallow: /account/


That will block all spiders/bots from accessing the /accounts/ directory on your site, though I got a "404 Not Found" when I tried to access that, so not sure if there's anything there anyway. Google uses a separate spider/bot for spidering pages for Adsense, but that would be blocked too with the robots.txt file you have in place.

It only really matters if you want to have Adsense on pages in the /accounts/ directory, in which case you'll need to change the robots.txt file or remove/delete it completely to allow the the Google Adsense bot to access those pages.

[needhelp]

You have a robots.txt file at http://www.candleliteonline.com/robots.txt which contains the following:


# robots.txt for www.candleliteonline.com

User-agent: *
Disallow: /account/


That will block all spiders/bots from accessing the /accounts/ directory on your site, though I got a "404 Not Found" when I tried to access that, so not sure if there's anything there anyway. Google uses a separate spider/bot for spidering pages for Adsense, but that would be blocked too with the robots.txt file you have in place.

It only really matters if you want to have Adsense on pages in the /accounts/ directory, in which case you'll need to change the robots.txt file or remove/delete it completely to allow the the Google Adsense bot to access those pages.
i would just have adsense on the pages that visitors would see so mabe it doenst matter,'I thought though that the code was to allow the robots to access it, That is what i got from adsense.

[cameraplus]

I thought though that the code was to allow the robots to access it, That is what i got from adsense.

It is, but if the Adsense bot can't access those pages then Google won't be able to determine the best ads to display. They may end up displaying no ads, or just the generic Public ads.

[needhelp]

Should i just leave that file out of my site then or what can i do to fix that, I have already done what adsense said to do.

[cameraplus]

What did Adsense say to do?

It may be OK to just delete the robots.txt file. I couldn't access anything under the /accounts/ directory and I've never seen that used in a URL path in any SOL sites.

That doesn't necessarily mean the /accounts/ directory isn't used though. It could be a sort of hidden directory, not available publicly on your site, containing stuff about your SOL account, which would explain why SOL don't want that directory spidered. I don't know enough about the SOL server setup to know that for sure though, so you're probably best off asking SOL about that and whether it's OK to remove the robots.txt file, or what the implications of doing so would be.

If it is some sort of private directory on the SOL server, then you'd be best off leaving the robots.txt file as it is - you wouldn't want that directory to be spidered for Adsense or any other reason in that case.

[needhelp]

I didnt know that it was a directory, It wasnt in there until i asked SOL to put it on there with the note from ADSENSE,

[cameraplus]

I'm seeing something at http://www.candleliteonline.com/accounts/ now - looks like just an empty template page with a the menu and other navigation links but no content in the middle part, and no Adsense ads. That wasn't there before when I checked.

I don't know what Google/Adsense have told you, nor what you've arranged with SOL, so don't think I can advise much further on this. Looks like it's probably OK from what you've done though.

[needhelp]

Well, im glad its ok then. I was wondering why that page look empty too. i WILL probably just leave it if its just a directory.