Robots.txt on Custom / Headless - Technical SEO Playbook

Beginner Recommended

Static File (Universal)

Standard.

1

1

Works for Static HTML, React, Vue, Svelte, Laravel, Django.

2

Place robots.txt in the Public/Static Root folder.

3

Content:

User-agent: *
Allow: /
Sitemap: https://site.com/sitemap.xml

Intermediate Recommended

Dynamic.

1

1

Create app/robots.ts.

2

export default function robots() {
  return {
    rules: { userAgent: '*', allow: '/' },
    sitemap: '...'
  }
}

Easy Recommended

Extension.

1

1

Use astro-robots-txt integration.

2

Generates file at build time based on config.

Intermediate

Dynamic.

1

1

If you need environment-specific rules (e.g., disallow staging):

2

Route get '/robots.txt' => 'seo#robots'.

3

Render text response based on Rails.env.