[Rank Math SEO Plugin] Author Sitemap blocked by robots.txt

[Guest designweblog]

Hi,

I got an error in the sitemap and that is author blocked in robots.txt but the author is not blocked in robots txt. I add a site URL in a sensitive area, you can check this.

So, why I get this error 

[Montgomerie]

Hello,

There is no definite answer to what you are asking. Whether or not you want to index the categories and tags depends on you.

If you would ask me (assuming I run that website), I would not index those archive pages as they would not add any real value to my users and would simply waste my crawl budget.

Also, that would serve as a duplicate content and my inner pages would compete with each other for the same keywords.

For the traffic generation, you should find the following articles helpful:
https://mythemeshop.com/blog/blogging-seo/
https://mythemeshop.com/blog/how-to-do-seo-optimizing-your-site-for-google/
https://mythemeshop.com/blog/how-to-do-seo-off-page-optimization/
https://mythemeshop.com/blog/keyword-research/
https://mythemeshop.com/blog/wordpress-speed-optimization/
and
https://mythemeshop.com/blog/increase-website-traffic/

Hope that helps. If you have any further question(s), please let us know. Thank you.

[Montgomerie]

Hello,

Thank you for contacting MyThemeShop and sorry for any inconvenience that might have been caused due to that.

Can you please upload a screenshot using some free service and add that in the Sensitive data section so we can check further?

That error essentially means that there are a few URLs in your sitemap which are blocked by your Robots.txt file.

Unless we know the URLs, we are limited in helping you.

Looking forward to helping you. Thank you.

[Guest designweblog]

Hello,

I have updated the sensitive data as requested. Can you please check further?

Thank you.

[Pratik.d]

Hello,

There are no blocked URLs in your robot.txt file. Also, I see the URLs which are blocked by Google Search console, on your_site_url/author-sitemap.xml, so there shouldn't be any problem.

Please clear the cache, delete the Sitemap in search console and add afresh.

If that doesn't help, please let us know.

Thank you.

[Guest designweblog]

Update the sitemap. let's see what happen.

I found problem in sitemap tag. tag page is open in a blank page in sitemap. I test this on all theme and showing same.

[Montgomerie]

Hello,

Can you please confirm if you are using 0.9.15 version of the plugin?

Also, please try editing your Robots.txt file to include the following rules and nothing else:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

I would not recommend indexing your attachment, category, tags & author archives on Google. Those add hardly any value to the users and simply cannibalize your own site's content.

You would be better off by simply getting your posts and images indexed.

Finally, after whatever you decide, please get the Google crawl your website again and wait for a couple of weeks to see the changes.

Hope that helps. If you have any further question(s), please let us know. Thank you.

[Guest designweblog]

Should I remove all robots.txt lines and only add these

User-agent: *

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

I'm using updated version which is 0.9.15

[Montgomerie]

Hello,

Yes. Please do that and once you decide on the above recommendations, please submit the Sitemap manually on Search Console:
https://search.google.com/search-console/sitemaps

and follow this image tutorial:
http://take.ms/IniBD

Hope that helps. Thank you.

[Guest designweblog]

But I checked Rank Math tutorial where they recommend to enable category, tags, and author archive. But you said disable all these.

Okay, Now I disable category, tags and archives in sitemap and change the robots.txt with default one you provided.

Edited October 3, 2018 by designweblog
after changing

[Montgomerie]

Hello,

Can you please link me to the tutorial where you read that?

Also, it depends. As rule of thumb, you should index your archive pages if you have many posts on your website and finding them related content through your category/archive pages would be helpful.

Otherwise, you would cannibalize your own site's content.

Again, it's just an opinion and you are free not to follow it.

Hope that helps. If you have any further question(s), please let us know. Thank you.

[Guest designweblog]

here is the screen shots. 

 

 

 

[Guest designweblog]

Hello,

I have updated the sensitive data as requested. Can you please check further?

Thank you.

[Montgomerie]

Hello,

Thank you for pointing that out.

That is certainly not correct. We will get that fixed soon.

I'm very sorry for the confusion and any inconvenience that might have been caused due to that.

If you have any further question(s), please let us know. Thank you.

[Guest designweblog]

kindly check the sensitive area where I ask question. 

It's mean I need to disable tags, category and author archive.

[Montgomerie]

Hello,

There is no definite answer to what you are asking. Whether or not you want to index the categories and tags depends on you.

If you would ask me (assuming I run that website), I would not index those archive pages as they would not add any real value to my users and would simply waste my crawl budget.

Also, that would serve as a duplicate content and my inner pages would compete with each other for the same keywords.

For the traffic generation, you should find the following articles helpful:
https://mythemeshop.com/blog/blogging-seo/
https://mythemeshop.com/blog/how-to-do-seo-optimizing-your-site-for-google/
https://mythemeshop.com/blog/how-to-do-seo-off-page-optimization/
https://mythemeshop.com/blog/keyword-research/
https://mythemeshop.com/blog/wordpress-speed-optimization/
and
https://mythemeshop.com/blog/increase-website-traffic/

Hope that helps. If you have any further question(s), please let us know. Thank you.

[Guest designweblog]

Thank you for kind support, will defiantly check this.

[Montgomerie]

Hello,

It was a pleasure helping you. We are glad you found our help useful. If you have any other questions in the future, do let us know. We are here to help you.

Could you please leave us a testimonial (if you haven't already done that) on about your overall experience with MyThemeShop? This will help us serve you, our members, better. We appreciate your time and patience.
https://community.mythemeshop.com/forum/6-testimonials/

Thank you in anticipation.