Insights
Mage-OS 4.0's built-in SEO and llms.txt: setting up AI discoverability
Mage-OS 4.0 bundles structured data, canonical and hreflang control, and AI discoverability documents as part of the distribution, where most Magento stores reach for a paid extension. Two modules do the work: module-seo handles JSON-LD, Open Graph, robots and hreflang, and module-aeo publishes /llms.txt, /llms-full.txt and /llms.jsonl along with AI crawler directives. Almost none of it is on when you upgrade. The documents and the crawler rules are off by default, and everything depends on one admin record that is empty until you fill it in.
What module-seo gives you
module-seo outputs JSON-LD for products, Organization and Breadcrumbs, with 16 product-type templates, configurable products expressed as ProductGroup, and merchant-listing fields. It also handles Open Graph and Twitter cards, robots and canonical control, and multistore hreflang.
It replaces two modules that Mage-OS shipped separately before: module-meta-robots-tag and module-hreflang. If you are coming from 3.6, that replacement is the one upgrade step most likely to stop Composer, and we cover it in upgrading to Mage-OS 4.0.
A third module, module-faq, provides an FAQ manager, a theme-agnostic widget and a Page Builder content type, and outputs FAQPage JSON-LD that matches the questions actually present on the page. That last detail matters more than it sounds: FAQ schema that does not reflect visible page content is exactly what search engines penalise.
Both modules require PHP 8.3 to 8.5 and Magento Open Source or Mage-OS 2.4.7 or newer. They are not Adobe Commerce modules, so do not assume a Commerce store can install them.
What module-aeo publishes
module-aeo adds three documents and a set of crawler directives. All four are off by default, switched on per store view.
| URL | What it contains | Default |
|---|---|---|
/llms.txt | Organisation name, description, base URL, locale, schema types, the first 5 FAQs of the selected groups, AI contact | Off |
/llms-full.txt | The above plus social profiles, the full category tree and every FAQ of the selected groups | Off |
/llms.jsonl | NDJSON, one compact JSON-LD Product node per line | Off |
robots.txt additions | Disallow groups for the AI crawlers you choose, from 14 known ones | Off |
Each document is written in the store view's own language, from the same Organization, locale and FAQ sources the structured data uses. /llms.jsonl reads product availability from Magento's MSI stock index.
How the documents are generated
This is the part that changes how you operate the store, and it is the part most likely to look broken if you miss it.
The documents are pre-generated to files and are never built by a web request. By default they are written to var/mageos_aeo/ and served from there. A change that affects them queues a rebuild on the SEO module's rebuild queue, and a nightly cron acts as a full-rebuild safety net.
That means the mageosSeoFeedRegenerate consumer has to be running, under cron or a supervisor, or the documents simply go stale. If consumers are not running reliably on your server, the cause usually sits upstream in cron: see Magento cron not running.
To build them immediately rather than waiting for the queue:
bin/magento seo:rebuild -g llmsThe documents are cacheable for 24 hours in shared caches and 5 minutes in browsers, and are served without a session so shared caches can store them. A feed over 0.5 MiB is streamed from its file rather than read into memory.
Allowing or blocking AI crawlers
module-aeo can write Disallow groups into robots.txt for the AI crawlers you choose, from 14 known ones, with CCBot and Bytespider selected by default once you switch the feature on.
Publishing /llms.txt and blocking crawlers are separate decisions, and it is worth being deliberate about both rather than accepting the defaults. If you want the reasoning on which crawlers to allow, which to refuse, and what robots.txt does and does not achieve, we cover it in how to block AI crawlers on Magento.
Setup checklist
- Fill in Marketing > SEO > Organization first, including the description, or
/llms.txtand the structured data stay incomplete - Decide per store view which of the three documents you actually want published
- Decide which of the 14 AI crawlers to allow or disallow, rather than leaving the defaults unexamined
- Confirm
mageosSeoFeedRegenerateruns under cron or a supervisor - Run
bin/magento seo:rebuild -g llmsand fetch each document to confirm it is complete - Check the FAQ groups feeding
/llms.txt, since only the first 5 appear in the concise version - Review the JSON-LD the SEO module emits, which is live from the moment it is installed
- Confirm FAQPage schema matches the questions visible on each page
The hosting side
Two requirements here are infrastructure rather than content, and both are easy to get wrong on a multi-server setup.
First, the consumer. AI discoverability output is only as fresh as the queue consumer that builds it, so consumers need supervision that restarts them rather than a cron entry that assumes they are alive.
Second, where the files live. On a single server, var/mageos_aeo/ is fine. Across multiple web servers the feeds need to sit in Magento's remote storage, or the web servers and the cron and consumer host have to share a mount, which takes an entry in each machine's app/etc/env.php as well as the admin setting. Without one of those, a visitor can hit a web server that has no copy of the document the consumer just rebuilt elsewhere.
EveryHost can host Mage-OS stores on our managed plans, with consumers supervised and cron monitored as standard, and Redis or Valkey set to match your Magento version. See managed Mage-OS hosting for what that covers.
If you want to know whether any of this is actually working, that is a measurement question rather than a configuration one. Our AI visibility service reports whether AI assistants mention your store, and Magento search visibility tools covers the wider tooling. For the distribution itself, see what Mage-OS is.
Sources
All accessed 3 November 2026.
- Mage-OS 4.0.0 Feature Release, 30 October 2026
- mage-os/module-seo on Packagist, with its README
- mage-os/module-aeo on Packagist, version 1.1.2, published 29 October 2026, with its README
Frequently asked questions
Want AI discoverability set up properly?
We can host Mage-OS on our managed plans, with queue consumers supervised and cron monitored, so the documents that feed AI assistants stay current instead of quietly going stale.