Donna said:

Yes it will for bots that respect the robots.txt. Ask seao if their plugin does. If not, it should and should not take a customization as this is a very basic thing.

As for the rest of the URLs, I am not sure. I suggest testing with google once the robots.txt is on your server. You can do a test in the google tools that has it fetch the file after it's on the server.

Hi Donna,

We do provide the File Editor option to edit the Robots.txt file and allow/disallow URLs in the Admin Panel of our Ultimate SEO/Sitemaps Plugin as shown in the attached screenshot.

As mentioned above, we are suggesting customization only for making Private Pages[403 Forbidden Page] accessible for SEO via bots, as we will need to invest efforts to modify our plugin as explained by our team to Yungsun.

Regards,

Team SocialEngineAddOns

Yungsun said:

Ok so my plugin is not certified, I bought it on Socialengineaddons.com

 

I told them that I don't want the plugin to index private pages, I am waiting for their response now.

 

I have regenerated sitemaps, deleted those on Google Webmaster Tools and sent new sitemaps, and I have now this :

 

0 Soft 404
11 Access denied
413 Not found

 

This is better but the Not Found Pages are still increasing regularly.

@Yungsun : We already have a feature in our Advanced Error Pages Plugin which you can purchase here https://www.socialengine.com/marketplace/app/advanced-error-pages-plugin to handle the 403 redirections for Page Not Found issues.

You can check the working of the plugin by entering any garbage value in the site and it will redirect you to the Home Page of the demo website like this link:

https://demo.socialenginesolutions.com/sajhusgdf

 

We provide free installation for the plugin, if you need our assistance for installation.

 

Regards

Team SocialEngineSolutions (SES)

 

Yes it will help without any doubt, thanks a lot for the informations

Hey Yungsun,

I had the exact same issue, I too have the Ultimate SEO.

It took me a month by trial and error to get it working right.

- Because there is no proper documentation on how to set this up.

it requires a little patience as google takes 4 days to index etc sometimes

and you have to keep checking every 3 or 4 days if the update is done by google.

 

 

Heres how I did it.

You have to configure and use Google Webmaster Tools AND Ultimate SEO plugin setup to make it all work.

Google Webmaster Tools

--------------------------------

https://www.google.com/webmasters/tools/home?pli=1

Login in with your account, add your domain/property etc

Validate your domain etc

 

Added 2  sitemaps

1)  /index.php/sitemap?format=xml&rewrite=1

2) /sitemap.xml

 

Use the 

a) fetch As Google to fetch and index

b)robots.txt tester

c) sitemaps

 

Ultimate SEO

---------------------------

The hardest and trickiest part is on this page : 

https://www.yourdomain.com/admin/siteseo/settings/support/target/3

The below is where you can easily forget to do.

• Add and Verify your Website in your Google search console if you have not added or verified.
• Click on your website link in Google search console. Click on the setting icon on the top right corner and select Users and property owners. Add a new user with email you copied from credentials page and choose permissions to full. You have to provide full access to each website where you want to integrate google search console.

Make sure you do the above step by step and very carefully.

In Schema Markup Tab, make sure you dont fill ANYTHING for "Organization" drop down.

 

 

SEAO has a demo site you can reference as to how to fill-in the Ultimate SEO Info

http://admindemo.socialengineaddons.com/admin/siteseo/settings

When your finally done regenerate and submit from here

https://www.yourdomain.com/admin/siteseo/sitemap

 

 

Robots.txt

--------------

User-agent: *
Disallow: /application/
Disallow: /development/
Disallow: /externals/
Disallow: /install/
Disallow: /phpmyadmin/
Disallow: /temporary/
#Disallow: /public/
Disallow: /siteminify/
Disallow: /vendor/

#OLD
#sitemap: /index.php/sitemap?format=xml&rewrite=1

#ULTIMATE SEO PLUGIN
#Sitemap: https://www.yourdomain.com/sitemap.xml

#CURRENT
Sitemap : https://yourdomain.com/index.php/sitemap?format=xml&rewrite=1

 

 

It is tedious and tricky, but once you complete it

you will show up right on top in google

 

Hope this helps.

It is frustrating cause there is no  detailed/accurate step-by-step documentation to accomplish this on the SEAO site.

 

 

 

This post was edited by playmusician at October 8, 2018 12:54 PM EDT

Also make sure you disable content types you dont need on this page,

or you will get tons of 404 and not found errors

Only enable limited things and where you see "sitemap" in the right corner

Keep it to a minimum

https://www.yourdomain.com/admin/siteseo/sitemap

 

Hello Playmusician,

 

Thank you for all these details, I will check on my end how are my settings. And well I think this is already much better than a few months ago.