Allow custom root robots.txt, sitemap.xml exclusions, and noindex for auth-gated routes

Title: Allow custom root robots.txt, sitemap.xml exclusions, and noindex for auth-gated routes

Base44 currently auto-generates root robots.txt and sitemap.xml with no override or exclusion controls.

Problem:

• I cannot edit root /robots.txt

• I cannot exclude private or non-canonical routes from root /sitemap.xml

• Auth-gated private URLs can return a 200 login page with no noindex/X-Robots-Tag control

Why this matters:
Private routes and unwanted URLs can remain indexable in search, and app-level workarounds are not enough because the root files and auth response behavior are platform-controlled.

Requested features:

1. Ability to customize or override root /robots.txt

2. Ability to exclude routes from root /sitemap.xml

3. Ability to mark routes/pages as noindex

4. Ability to apply X-Robots-Tag or equivalent noindex behavior to auth/login responses for private routes

This is important for SEO hygiene, privacy, and proper search index control on production apps.