Google's indexing algorithm


Google uses algorithms to analyze each page of a website to determine if it meets its own proprietary standards for being included in its index.  While tools can be used to force Google to analyze a page, unfortunately there is no way to force Google to include a specific page in its index.


Google takes into consideration several factors when determining if a page will be included in its index or not:


  1. The page cannot be subject to any manual actions or legal issues.
  2. The page cannot be a duplicate of another indexed page; it must either be unique, or selected as the canonical version of a set of similar pages.
  3. The page quality must be high enough to warrant indexing.



Google provides information about how to make sure URLs on a website can be found by Google: https://developers.google.com/search/docs/beginner/intro-indexing


The Brilliant Directories software has tools available to make pages discoverable and indexable by Google.  However, even when a website owner leverages these tools to take a proactive approach to making the URLs of their site discoverable by Google, their documentation also mentions that "While these approaches help Google find the URLs on your site, it doesn't guarantee that your content will be indexed and served in Search results."


 

 


This can be due to a number of factors that their algorithm takes into account, including content not being unique (the information can also be found elsewhere online) or robust enough (according to Google's algorithm) to be included in their index.



Ensuring a page is discoverable and able to be indexed


Unfortunately, no website owner can control which website pages Google chooses to include in its index. However, a website owner can make sure the page exists, is discoverable, and is available to be indexed (the page has the correct meta tags, the page loads correctly, the robots.txt file allows the page to be crawled, etc).

If a page was recently created or the sitemap was recently submitted with a new page, it can take time for Google to index the page.  Please allow at least a week after submitting a sitemap or individual page index request before assuming there is an issue that must be addressed. If the page or site change is recent, check back in a week or two to see if it is still missing.


If the Sitemap or an individual page index request was submitted to Google several weeks ago and some pages are not being indexed, the URL Inspection tool in the Google Search Console can be used to check if there is a technical issue preventing it from being indexed, or if Google has simply chosen not to index the page.

When using this tool, does Google show that there is a technical problem preventing it from being indexed (something we can look into and fix on our end), or just that it is not currently indexed (which is at Google's discretion and we have no control over)?


If there is a technical problem, please create a new ticket by clicking here and add the screenshot of the error Google is showing, so our development team can take a look.


If there isn't an error but Google is not indexing the page / site, please continue with the steps below.


Step 1. Verify that the page or site is missing


It sounds obvious, but first, verify that the page or site is actually missing from Google's index. Many people assume that they are not on Google, when in fact their page simply appears very low in Search results.


To verify the page/site presence on Google:


  1. Turn off safe search, which might be filtering the results.

  2. Search Google for the site or page:
    - For a missing site: Do a site search with the syntax site:your_domain_name
    Examples: site:example.com or site:example.com/petstore
    - For a missing page: Search Google for the full URL of the page.

  3. If there are results, then the site or page is in the index:
    - For a site: It is possible that not every page on the site is indexed, but the site itself is in our index. Consider adding a sitemap to help Google discover all the pages on the site.
    - For a page: If a page is in the index, but not performing as well as it should, check our webmaster guidelines for tips on improving search performance.
    If the page has suffered a recent ranking drop, there is some troubleshooting needed.
    If there are multiple versions of a page (for example a mobile and a desktop version, or two URLs that point to the same page), Google will consider one to be canonical (authoritative) and all others to be duplicates, and Search results will point only to the canonical page.
    The URL Inspection tool can be used on a page to see if it is considered a duplicate.

  4. If the site or page cannot be found in search results, please go to Step 2: Fix the problem.



Step 2. Fix the problem


Please make sure a Search Console account was created before following the steps below because it is much easier to diagnose indexing problems using Search Console.


Was the website bought or inherited from someone else? It's possible that the site has existing manual actions filed against it. The history pages in the Manual Actions report and Security Issues report will show any outstanding actions filed against it.


  1. If the site or page is new, it might not be in our index because we haven't had a chance to crawl or index it yet. It takes some time after a new page is posted before Google crawls it, and more time after that to index it. The total time can be anywhere from a day or two to a few weeks, typically, depending on many factors.
    Learn how Google crawls the web.

  2. If recently restructured the site, or moved to a new domain or to https, pages that previously performed well may now rank poorly if mistakes were made when executing the move.
    To fix: Use 301 redirects ("RedirectPermanent") to redirect users, Googlebot, and other crawlers. (In Apache, it can be done with an .htaccess file)
    If the site was recently moved to https, check for the presence of both http and https URLs in Google.
    Read more about how to move the site with minimal effect on Search results. 

  3. Check to see if any manual actions have been applied to the page.
    Manual actions will lower the page ranking or remove it entirely from Search results. The Manual Actions report should provide guidance on how to fix the manual action. For legal removals, learn more about legal removal requests and removal policies.

  4. Check to see if any security issues have been reported on the site.
    Security issues can lower the page ranking, or display a warning in the browser or in search results. The Security Issues report should provide guidance on how to fix the manual action.

  5. Inspect the page using the URL Inspectioxn Tool:

    - OPTION 1: If the report says that the page has not been indexed
    Here are the most common reasons:
        1. The page is being blocked with a robots.txt file, a noindex directive, or some other mechanism, such as password protection. In any case, use the appropriate means to unblock it.
        2. If the report describes other technical issues, read the documentation to learn why else the page might be blocked.
        3. If there are no errors, and the page is not blocked to Google, there might be a problem with findability.
    Request indexing of the page using the URL Inspection tool.
        4.Request indexing of the page using the URL Inspection tool.

    - OPTION 2: If the report says that the page has been indexed
        1. Check whether someone successfully requested that the site or URL be removed from the index. Open the URL removal tool to look for approved requests of URL or site removal. If so, the reuqest can be revoked.
        2. The page may have been dropped or omitted from the index for totally innocuous reasons. (The web is immense, and Google doesn't get to every page, though we try to!) Ask Google to recrawl the page.
        3. Still having problems? Visit the Webmaster Forum and explain the problem (please be sure to describe the issue completely, and include links to the site).