What’s indexation ? #
Today, if you want your blog/portfolio to be visible to others it is almost mandatory that it be listed on google. For your site to appear on the search engine, Google needs information!
This information is usually collected automatically by a robot that Google has developed. This “Googlebot” wanders from site to site and from page to page to make a “map” of the web. This allows then to make the search engine of Google very relevant during your research.
If you want more details on how the Googlebot works, I let you see the documentation here!
As I said above this process is automatic, but we can help it to do a better job and a more accurate scan of our site. This will allow us to be better referenced and potentially visible by more people!
Register your domain for indexation #
First we will have to add our domain to the Google Search Console site. To access the site you must be logged in with a Google account.
https://search.google.com/u/0/search-console/welcome
You should land on the following page which gives us the choice between two options.
Domain : this option will be used only if you use your domain name only for your website and you don’t have any subdomain. For example in my case I have several subdomains (ex. status.d3vyce.fr) but I don’t want them to be indexed.
URL prefix : this second option allows to declare a precise URL and not an entire domain. This is the option that I choose for my blog.
You can then enter your domain. In my case it is https://www.d3vyce.fr. If all goes well the ownership verification of your domain is automatic. But if it’s not the case, don’t panic, an error message will tell you how to solve the problem. Globally, Google will provide you with a file that you should host on your site, this will verify that you have control of the site and therefore the domain.
From this moment, the google robot will have to visit your site soon to do a scan. We could stop here but we will provide additional information to help the robot !
To do so, we will provide google with a map of our site. These are usually self-managed files called “sitemap” in xml format.
The majority of CMS (Content management system) have this functionality integrated. In my case I use Ghost, so I will continue this tutorial with this example. If you use another CMS (eg Wordpress, Shopify, Hugo, …) you should have quite a lot of resources on the subject with a google search.
In my case the sitemaps files are located at the following address: https://www.d3vyce.fr/sitemap.xml
This link leads to a sitemap index which is itself composed of several sitemap files. We have the choice to add file by file or directly the sitemap index.
To add our sitemap, we must go to Index > Sitemaps. Then we add the link to our index file.
After a few minutes we notice that our sitemap index has been detected and that our 5 sitemaps have been imported!
After this step, there is nothing left to do but wait. This can take from a few hours to several weeks in some cases.—
Update #
After about 36 hours, my blog has been indexed on google and is now accessible with a simple search!
I then went to see the access logs of the site and we can observe the passages of the Googlebot which scans the site:
[02/Feb/2022:03:11:17 +0100] - 200 200 - GET https www.d3vyce.fr "/favicon.ico" [Client 66.249.69.27] [Length 3798] [Gzip -] [Sent-to X.X.X.X] "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
[02/Feb/2022:03:26:17 +0100] - 404 404 - GET https www.d3vyce.fr "/contact/" [Client 66.249.69.29] [Length 660] [Gzip -] [Sent-to X.X.X.X] "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
[02/Feb/2022:02:16:53 +0100] - 200 200 - GET https www.d3vyce.fr "/author/nicolas/" [Client 66.249.69.27] [Length 5923] [Gzip -] [Sent-to X.X.X.X] "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
[02/Feb/2022:01:52:01 +0100] - 200 200 - GET https www.d3vyce.fr "/sitemap.xml" [Client 66.249.69.27] [Length 637] [Gzip -] [Sent-to X.X.X.X] "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
I could also see the passage of other bot like Bing :
[02/Feb/2022:00:34:43 +0100] - 200 200 - GET https www.d3vyce.fr "/robots.txt" [Client 40.77.167.104] [Length 107] [Gzip -] [Sent-to X.X.X.X] "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"
[02/Feb/2022:00:34:51 +0100] - 200 200 - GET https www.d3vyce.fr "/" [Client 40.77.167.40] [Length 6253] [Gzip -] [Sent-to X.X.X.X] "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"