Index Now

[Shawn Gossman]

XenForo offers an option in the SEO settings to enable IndexNow.

How do you feel about this option? Do you enable it? Do you think it's helpful for a forum?

Home | IndexNow.org

IndexNow enables webmasters, website owners to inform search engines about the latest content changes on their website. Helps search engines to maintain latest data about a website and redirect searchers to the fresh content. Webmasters/Website owners should use IndexNow to overcome crawl delays...

www.indexnow.org

 

[Tracy]

Bing prefers it now. So it's not really a bad idea. If you remember, in the older versions of XF you were able to submit updated sitemaps to Google and Bing automatically. They did away with allowing that though.

 

[Shawn Gossman]

Bing prefers it now. So it's not really a bad idea. If you remember, in the older versions of XF you were able to submit updated sitemaps to Google and Bing automatically. They did away with allowing that though.

Okay. So, it's more of a BING thing then?

I need to touch back up on my SEO knowledge. There are so many changes that I get behind, LOL.

 

[Tracy]

Yeah, I think the primary pusher of it right now is Bing. I haven't really delved into Google or the others.
I know that most of the major search engines did away with automated submission of sitemaps to them.

 

[Shawn Gossman]

Yeah, I think the primary pusher of it right now is Bing. I haven't really delved into Google or the others.
I know that most of the major search engines did away with automated submission of sitemaps to them.

I know I was getting server errors on XF once about the Bing sitemap feature. I just disabled it.

 

[Tracy]

I know I was getting server errors on XF once about the Bing sitemap feature. I just disabled it.

That was the official XF recommendation. First Bing refused submissions and there were lots of posts on XF about it and the answer always was disable it and then Google got in on it.

 

[Shawn Gossman]

That was the official XF recommendation. First Bing refused submissions and there were lots of posts on XF about it and the answer always was disable it and then Google got in on it.

They need to take those features out.

If it doesn't work, it's nothing but software bloat. It's a waste of space.

 

[Tracy]

They need to take those features out.

Yes and no. Some search engines still may allow you to submit them. But AFAIK, all the major ones have gotten away from automatic submission of sitemaps. They expect to see it in the robots.txt.

 

[Cpvr]

I think it’s helpful as the search engines can crawl and index your content a lot faster. Just like if you have your sitemap enabled and listed on the search engines.

Also, if you have your rss feed enabled it helps as well.

They do crawl your content from both the rss feed and sitemap.

 

[Heatman]

Also, if you have your rss feed enabled it helps as well.

They do crawl your content from both the rss feed and sitemap.

No wonder a friend of mine recently had his own rss feed enabled and also asked for more replies to be focused on those rss contents.

 

[Shawn Gossman]

I think it’s helpful as the search engines can crawl and index your content a lot faster. Just like if you have your sitemap enabled and listed on the search engines.

Also, if you have your rss feed enabled it helps as well.

They do crawl your content from both the rss feed and sitemap.

I do have RSS available. I've always tried to keep them enabled.

 

[frm]

I'm starting to see that IndexNow on a forum with more active traffic is probably doing more harm than good. I frequently get these warnings:

ErrorException: IndexNow error: [500] An unexpected error occurred. Please try again later. src/XF/Error.php:82

At a certain point, you're sending too many requests as each thing gets replied to, blocking the feature from working as intended.

I'm still uncertain whether to turn this off or not because it'll be blocked for some time, and then a flood of new replies kicks the block off again when they finally get through.

 

[Tracy]

At a certain point, you're sending too many requests as each thing gets replied to, blocking the feature from working as intended.

No big surprise. Xenforo just recently implemented it and they probably have not fully tested it out, instead relying on their current license holders to report the bugs and help trouble shoot it for them. Seems that is their current MO. It was one of the reasons I put off so long upgrading my 2.2.16 site that was working fine.
I regularly am seeing ACP errors shown due to bugs in their releases.
The Xenforo of today is not the Xenforo of 5 years ago.

 

[frm]

No big surprise. Xenforo just recently implemented it and they probably have not fully tested it out, instead relying on their current license holders to report the bugs and help trouble shoot it for them. Seems that is their current MO. It was one of the reasons I put off so long upgrading my 2.2.16 site that was working fine.
I regularly am seeing ACP errors shown due to bugs in their releases.
The Xenforo of today is not the Xenforo of 5 years ago.

It's probably still something that XF would need to edit from time to time. Even if they made it a job to IndexNow 5 posts every hour, you could still go over the daily quota (that could be subject to change at any time). It'd be a constant battle of trying to figure out the rate limit and updating it.

 

[Tracy]

With (currently) a 10,000 URL a day limit (100 at a time), submitting every 15 minutes. That's 400 an hour (more than most sites get posts of that number in an hour)
and around 8800 a day, still under their max URL's. And it should only be hitting for the updated pages.
And as for "constant battle of trying to figure out"... nope, just make it admin selectable for the times and number submitted in one run. If the admin over-submits (in other words, abuses the feature) it will be blocked on the other end and the admin will have to adjust his/her settings appropriately.

 

[frm]

With (currently) a 10,000 URL a day limit (100 at a time), submitting every 15 minutes. That's 400 an hour (more than most sites get posts of that number in an hour)
and around 8800 a day, still under their max URL's. And it should only be hitting for the updated pages.

There's no way that's the limit as I've hit the error after submitting 2-6 posts per day on a small forum (in admin error log).

I mean, that may be the limit of posts acceptable, but they have to be viewing the post content too to determine if it's "spam". Updating a resource, for instance, creates a new reply submitted to IndexNow. The only thing in that resource-associated post is "X has been updated. View details".

Speculation, but there's more to it than just 10,000 URLs per day.

 

[Tracy]

There's no way that's the limit as I've hit the error after submitting 2-6 posts per day on a small forum (in admin error log).

Last I checked, that was what Bing (Microsoft) was using. Now, other IndexNow related search engines...thats up to them.

As far back as 2019 they had upped it to 10,000 a day.

Bing Allows Site Owners to Submit Up to 10,000 URLs Per Day

Bing has significantly increased the number of URLs site owners can submit for indexing per day.

www.searchenginejournal.com

And this from Bing in 2023.

Bing Webmaster Tools

www.bing.com

Speculation, but there's more to it than just 10,000 URLs per day.

Oh, I imagine they will throttle you if you are hitting it every time you get new content created on your site.
And yes, there is the ability of the search engines to monitor content and filter/deny based upon that from what I read.
I doubt they are going to be any different than any major search engine that won't index low quality content.

The only thing in that resource-associated post is "X has been updated. View details".

And therein lay the issue. If it is sending it as it's being updated, I can easily see them applying throttling to a sender. That's why I commented about batch submissions done on a timed basis.

 

[frm]

batch submissions done on a timed basis.

Batch submissions (jobs) still wouldn't account for quality though, if that is an underlying metric used for IndexNow.

 

[Tracy]

Batch submissions (jobs) still wouldn't account for quality though, if that is an underlying metric used for IndexNow.

But it would for hitting their servers too often. Like all providers, they only have so much bandwidth. Once the item is submitted, then their filtering routine (which they love as is apparent with the issue with Hotmail/Live/Outlook mail delivery) will process through those submitted. I have a feeling it's more a rate limited throttle than a quality issue as if it was not something they wanted to index, they simply will not index it. They aren't going to throttle your site based upon quality but on quantity of traffic. If they were going to do anything with quality, they'd simply do a total block of the site submissions.
In fact, your original comment was about the number of requests sent... which ends up with getting blocked apparently. By a batch send, that will not happen as you aren't hitting their servers every time there is new content.

 

[frm]

I have a feeling it's more a rate limited throttle than a quality issue as if it was not something they wanted to index, they simply will not index it.

I have never even come close to 10,000 submissions in one day but still received this error.
ErrorException: IndexNow error: [500] An unexpected error occurred. Please try again later. src/XF/Error.php:82

As it doesn't explicitly say that I'm above the limit, an error code 500 indicates the server was unable to process a request. One request, cool, two, okay, ten spread out over a day (far from the 10,000)? They have to be denying them based on other criteria set and possibly known over just the 10,000 limits.