Donna said:
Yes it will for bots that respect the robots.txt. Ask seao if their plugin does. If not, it should and should not take a customization as this is a very basic thing.
As for the rest of the URLs, I am not sure. I suggest testing with google once the robots.txt is on your server. You can do a test in the google tools that has it fetch the file after it's on the server.
Hi Donna,
We do provide the File Editor option to edit the Robots.txt file and allow/disallow URLs in the Admin Panel of our Ultimate SEO/Sitemaps Plugin as shown in the attached screenshot.
As mentioned above, we are suggesting customization only for making Private Pages[403 Forbidden Page] accessible for SEO via bots, as we will need to invest efforts to modify our plugin as explained by our team to Yungsun.
Regards,
Team SocialEngineAddOns
Yungsun said:
Ok so my plugin is not certified, I bought it on Socialengineaddons.com
I told them that I don't want the plugin to index private pages, I am waiting for their response now.
I have regenerated sitemaps, deleted those on Google Webmaster Tools and sent new sitemaps, and I have now this :
0 Soft 404
11 Access denied
413 Not found
This is better but the Not Found Pages are still increasing regularly.
@Yungsun : We already have a feature in our Advanced Error Pages Plugin which you can purchase here https://www.socialengine.com/marketplace/app/advanced-error-pages-plugin to handle the 403 redirections for Page Not Found issues.
You can check the working of the plugin by entering any garbage value in the site and it will redirect you to the Home Page of the demo website like this link:
https://demo.socialenginesolutions.com/sajhusgdf
We provide free installation for the plugin, if you need our assistance for installation.
Regards
Team SocialEngineSolutions (SES)
thanks for the info SEAO and SEShopefully it will help @Yungsun
Yes it will help without any doubt, thanks a lot for the informations
Hey Yungsun,
I had the exact same issue, I too have the Ultimate SEO.
It took me a month by trial and error to get it working right.
- Because there is no proper documentation on how to set this up.
it requires a little patience as google takes 4 days to index etc sometimes
and you have to keep checking every 3 or 4 days if the update is done by google.
Heres how I did it.
You have to configure and use Google Webmaster Tools AND Ultimate SEO plugin setup to make it all work.
Google Webmaster Tools
--------------------------------
https://www.google.com/webmasters/tools/home?pli=1
Login in with your account, add your domain/property etc
Validate your domain etc
Added 2 sitemaps
1) /index.php/sitemap?format=xml&rewrite=1
2) /sitemap.xml
Use the
a) fetch As Google to fetch and index
b)robots.txt tester
c) sitemaps
Ultimate SEO
---------------------------
The hardest and trickiest part is on this page :
https://www.yourdomain.com/admin/siteseo/settings/support/target/3
The below is where you can easily forget to do.
Make sure you do the above step by step and very carefully.
In Schema Markup Tab, make sure you dont fill ANYTHING for "Organization" drop down.
SEAO has a demo site you can reference as to how to fill-in the Ultimate SEO Info
http://admindemo.socialengineaddons.com/admin/siteseo/settings
When your finally done regenerate and submit from here
https://www.yourdomain.com/admin/siteseo/sitemap
Robots.txt
--------------
User-agent: *
Disallow: /application/
Disallow: /development/
Disallow: /externals/
Disallow: /install/
Disallow: /phpmyadmin/
Disallow: /temporary/
#Disallow: /public/
Disallow: /siteminify/
Disallow: /vendor/
#OLD
#sitemap: /index.php/sitemap?format=xml&rewrite=1
#ULTIMATE SEO PLUGIN
#Sitemap: https://www.yourdomain.com/sitemap.xml
#CURRENT
Sitemap : https://yourdomain.com/index.php/sitemap?format=xml&rewrite=1
It is tedious and tricky, but once you complete it
you will show up right on top in google
Hope this helps.
It is frustrating cause there is no detailed/accurate step-by-step documentation to accomplish this on the SEAO site.
Also make sure you disable content types you dont need on this page,
or you will get tons of 404 and not found errors
Only enable limited things and where you see "sitemap" in the right corner
Keep it to a minimum
https://www.yourdomain.com/admin/siteseo/sitemap
Hello Playmusician,
Thank you for all these details, I will check on my end how are my settings. And well I think this is already much better than a few months ago.