Skip to content
PageOneBoost
Guide

Robots.txt: how to configure it properly without blocking your climb on Google?

Short answer: yes, it's within your reach — provided you go about it methodically. Here's how, point by point.

TL;DR

Robots.txt is a tiny file with outsized power: a few lines are enough to forbid Google from crawling your entire site. Configured well, it goes unnoticed; configured badly, it cancels all your SEO efforts — it's one of the first checks in any audit. PageOneBoost applies this method for its clients — one-time yearly payment from €300, no monthly subscription, free audit.

01

What you need to understand

Understand what it controls: crawling, not indexing : Robots.txt tells crawlers which areas they may visit. An important subtlety: a page blocked from crawling can still appear in the index if links point to it — to deindex, you need the noindex tag, not robots.txt.

Hunt down the killer Disallow: / : The line "Disallow: /" forbids the whole site: it's the default setting of many development environments, sometimes forgotten at launch. Site invisible, traffic at zero — check this file after every deployment to production.

Block what has no business in Google : Admin areas, shopping carts, internal search results, sorting parameters: these URLs, worthless to searchers, waste crawl budget. Robots.txt keeps them out cleanly.

02

The method, point by point

Never block CSS and JavaScript : Google needs to load your styles and scripts to render the page like a visitor would. Blocking them stops it from properly evaluating the mobile rendering and the experience — a legacy of old practices to abandon.

Declare the sitemap and test : Add the Sitemap: line with the full URL of your sitemap: crawlers find it on their first visit. Then test the file from Search Console to confirm your important pages are crawlable.

  • Understand what it controls: crawling, not indexing
  • Hunt down the killer Disallow: /
  • Block what has no business in Google
  • Never block CSS and JavaScript
  • Declare the sitemap and test
03

What PageOneBoost does for you

Everything above takes time, method and experience. That's exactly what PageOneBoost does: a free audit to measure your potential, then the complete foundation built — technical, content, Google Business Profile, reviews, authority — to target the first page for the long run.

Our model is simple: a one-time yearly payment, from €300, with no monthly subscription. The service covers 12 months and renews by tacit renewal. 100% white-hat method, measurable results. To talk it through: +33 1 84 80 13 42.

Frequently asked questions

Where is my site's robots.txt file?

Always at the root of the domain: yourdomain.com/robots.txt. Type that address into your browser — the file is public, including your competitors'.

My robots.txt blocks a page I want deindexed: is that right?

No, that's the classic misconception: blocked from crawling, the page can no longer show its noindex tag to Google and can stay in the index. Allow crawling, set the noindex, and let Google see it.

Can a site do without robots.txt?

Yes — with no file, crawlers explore everything, which suits a small site with no sensitive areas. But a minimal file declaring the sitemap remains good practice that costs a minute.

How much does serious SEO support cost?

At PageOneBoost, it's a one-time yearly payment from €300, with no monthly subscription: the service covers 12 months and renews by tacit renewal. The initial audit is free.

How long before you see results?

The first effects often appear within a few weeks on local or low-competition queries; rankings consolidate over three to six months. Your competition and your site's starting point make this timeline vary.

Get onto the first page of Google

Free audit, one-time yearly payment from €300, no monthly subscription. PageOneBoost builds your visibility to last.

Request my free audit

Also worth reading

Robots.txt: how to configure it properly without blocking your climb on Google? · PageOneBoost