Practical guides
Practical guides
The first guide in this section explains how to check a website before launch so that search engines can find and index it. The result: pages load without errors, robots.txt doesn't block anything it shouldn't, live pages carry no leftover noindex directives from testing, a sitemap is in place, important pages are linked to each other, and site ownership is verified in webmaster tools. The examples in the guide are hypothetical and are included for illustration only.
A pre-launch check won't magically make your site appear in search faster. It simply removes the obstacles you put there yourself.
Before you start
- The site address exactly as it will work after launch
- Access to the site files or control panel so you can edit robots.txt and the sitemap
- A list of your most important pages: home page, sections, contacts
- Google and Yandex accounts to set up webmaster tools
Check that pages load
Go through the main pages as an ordinary visitor, without logging in. Pages should load without server errors, and addresses that don't exist should honestly report that the page is missing rather than show an empty placeholder.
Hypothetical example: any mistyped address on the site shows the home page. To a search engine this looks like a large number of identical pages.
Check robots.txt
The robots.txt file sits at the root of the site and tells crawlers which sections they don't need to crawl. The protocol appeared in 1994 and was standardised as RFC 9309 in 2022. Make sure the file doesn't block the entire site, and remember that it controls crawling but does not guarantee a page stays out of the index.
Hypothetical example: the whole staging site was blocked from crawling, and the rule was never removed after the move to production. Search engines see nothing but the block itself.
Remove leftover noindex directives
Besides robots.txt, an indexing block can sit in the code of individual pages or in your content management system settings. Check that live pages have no such blocks, and that utility pages do have them where they are genuinely needed.
Hypothetical example: the CMS still has the "discourage search engines from indexing this site" box ticked from the development stage.
Prepare a sitemap
A sitemap in the Sitemaps protocol format (sitemaps.org) is a list of the addresses you want to show to search engines. The protocol has existed since 2006 and is supported by the major search engines. Include only live pages that should appear in search.
Hypothetical example: the sitemap includes the shopping cart and account pages. They are better removed, as they don't belong in search.
Link your pages together
Search engines discover pages mainly through links. Every important page should be reachable from the menu, section pages or other content on the site, not only through site search or a form.
Hypothetical example: articles can only be opened through the internal search box. The crawler has nothing to follow, and some content goes unnoticed.
Write clear page titles
Each page should have its own title and short description that reflect its content. Identical titles on dozens of pages make it hard for both visitors and search engines to tell the pages apart.
Hypothetical example: every product page has the title "Online store". In search results these pages are impossible to tell apart.
Mark paid and user-generated links
If your site carries advertising with links, mark those links with rel="sponsored" or rel="nofollow", as search engine policies require. Links in comments and forums can be marked rel="ugc". Buying and selling links to manipulate rankings is prohibited by these policies.
Hypothetical example: an affiliate banner links to an advertiser's site. The link in it is marked as sponsored.
Set up webmaster tools
Verify ownership of your site in Google Search Console (called Google Webmaster Tools until 2015) and in Yandex Webmaster, and submit your sitemap. These tools show data from the search engines themselves: which pages they know about and which errors occur during crawling.
Hypothetical example: once connected, the tool reports that some pages are inaccessible to the crawler. That is a reason to go back to steps 1–3.
Common mistakes
- Leaving a crawling or indexing block from the staging site in place
- Assuming robots.txt reliably removes a page from search
- Adding utility and non-existent addresses to the sitemap
- Making important pages reachable only through a form or internal search
- Using the same title on every page
- Buying links or taking part in link schemes to improve rankings
Alternatives
- If you don't have access to the site files, ask your developer or hosting provider to check robots.txt and the sitemap against this list.
- If your site runs on a website builder, look in its help pages for search engine settings: many options there are simple checkboxes.
- If you aren't ready to set up webmaster tools, at least check whether your site shows up in search for its exact name, bearing in mind that this is only a rough check.
Final checklist
- Main pages load without logging in and without errors
- Non-existent addresses are not replaced by the home page
- robots.txt does not block the entire site
- Live pages carry no leftover noindex directives from testing
- A sitemap is in place and lists only live pages
- Important pages are reachable through links
- Paid links are marked sponsored or nofollow
- Site ownership is verified in Search Console and Yandex Webmaster
Article catalogue for English-speaking readers
This section will feature standalone articles written for English-speaking readers. They take into account the search engines and tools that site owners in English-speaking markets actually use, chiefly Google and Bing, and they are not translations of articles from other language versions of the site.
- GTA 6 Story Mode Progression Guide: What Players Should Know
What is actually known about GTA 6 story progression before launch, what GTA V suggests, and how to separate official information from guesswork.
- Destiny 2: Dungeons — What To Know Before Starting
A practical primer on Destiny 2 dungeons: what they are, how to prepare, finding a fireteam through LFG, and how to judge outside advice and services.
- Aion 2 abyss guide: What New Players Should Know
A plain-language introduction to the Abyss in Aion 2: what the faction PvP zone is, how to prepare, and how to judge advice while the global version is new.
- Planning for gaming news and guide directories: Gaming Technology
How to read gaming community trends through news feeds and guide directories, separate real signals from noise, and plan around game technology changes.
Topics the editors are planning
- Reading Google Search Console's Page indexing report: a beginner's walkthrough
- Bing Webmaster Tools: what it shows and how it differs from Search Console
- Canonical URLs and duplicate content: how search engines choose which page to show
- Moving a site to a new domain: redirects, robots.txt and sitemaps to check
Editorial rules
- Articles in this catalogue are written separately for each language version and are not translations.
- Some articles link to third-party websites, including gaming services; we do not control their content, prices or terms.
- We do not publish unverifiable ratings of websites or services, ranking promises or reviews.
Related sections
- Questions and answers
Short answers about PageRank, indexing, links and website metrics.
- Glossary
Glossary of terms: search and indexing, links, metrics.
- Reading lists
Reading paths: your first website, the history of search metrics, links without spam.
Sources
- Google Search Central — documentation and spam policies
- Yandex Help for webmasters
- RFC 9309 (Robots Exclusion Protocol)
- sitemaps.org
Information is current as of the date of preparation. Search engine documentation, policies and webmaster tools change over time, so always check the latest details in the original sources.