Google gives websites more control over indexing with the new Robots tag

A new Robots tag called indexifembedded, websites can give Google more instructions about what content to index in search results.

You can use this tag to instruct Google to only index content on a page if it is embedded through iframes and similar HTML tags.

The indexifembedded tag overrides the noindex tag.

That is, you can use noindex to exclude an entire URL from search results, and apply the indexifembedded tag to make specific content indexable when embedded in another webpage.

Google claims to have created this tag to address an issue affecting media publishers:

“…although they want their content to be indexed when embedded on third-party sites, they don’t necessarily want their media sites to be indexed themselves.”

When should the Indexifembedded tag be used?

This new robots tag doesn’t apply to many publishers as it’s intended for content that has a separate URL to embed.

For example, a podcast publisher may have web pages dedicated to each podcast episode, each with their own URLs.

Then there would be URLs pointing directly to the media that other sites could use to embed the podcast on one of their pages.

Such a URL can be used when including a podcast episode as a reference source, as I did in a recent article on Googlebot crawling.

The podcast creator may not want the media URLs to be indexed in search results. Until now, the only way to keep them out of Google searches was with a noindex tag.

However, the noindex tag prevents embedding of the content in other pages during indexing. So if the publisher wanted to allow embedding, they had to index the media URL as well.

With the indexifembedded tag, publishers now have more control over what gets indexed.

The indexifembedded tag can be used with the noindex tag and will override it if the url with noindex is embedded in another page via an iframe or similar HTML tag.

Google provides the following example:

“For example, if podcast.host.example/playpage?podcast=12345 contains both noindex and indexifembedded tags, it means that Google is indexing the content hosted on that page in Recipe.site.example/my-recipes .html can embed. ”

Related: When to use Rel Canonical or Noindex…or both

How to use the Indexifembedded tag

There are two ways to use this new Robots tag.

To have your content indexed only when embedded on other pages, add the indexifembedded tag in combination with the noindex tag.

See an example of what the code would look like in the image below:

Alternatively, you can specify the tag in the HTTP header.

Check out the image below for an example of what that would look like.

Currently only Google supports the indexifembedded tag.

source: Google Search Central blog

---

Featured image: IgorGolovniov/Shutterstock