Doc crawl pages that no longer exist

Hey guys, first-time poster – and I have a head scratcher for you :slight_smile:

I’m indexing Shopify Polaris docs. Cursor is indexing pages that no longer exist, and not including some recent additions. It’s almost as if it’s using a cached version. Shopify Polaris v12 was released early October, but the indexed docs all refer to Polaris v11 (and Polaris v12 beta).

Some of the scraped/indexed docs no longer exist in the sitemap, or on the website (I looked through the HTML). Yet they appear as pages indexed by Cursor. And some new sections/pages added since the release of v12 isn’t among the indexed docs.

If I ask the docs “what’s the latest version of Polaris?” I get the answer:

The latest version of Polaris is version 12, which is currently in beta and scheduled for a stable release at the end of September.

Similarly, if I ask “What stack component exists?” I get:

  1. VerticalStack
  2. HorizontalStack
  3. LegacyStack

Both 1 & 2 above were replaced in Polaris 12, and are no longer in the sitemap or on the website. The list of indexed docs include links to components/layout-and-structure/horizontal-stack and …/vertical-stack, which now are 308 redirects to inline-stack and block-stack.

Where does the scraper/indexer find these? Can I do something to work around this?

Steps to reproduce:

Try to find e.g. inline-stack or block-stack components, or anything about

Version: 0.13.4
VSCode Version: 1.83.1
Commit: fe924b35718993dc0821b4583f718e4cbe79b330
Date: 2023-10-29T00:38:06.904Z
Electron: 25.8.4
ElectronBuildId: undefined
Chromium: 114.0.5735.289
Node.js: 18.15.0
OS: Darwin arm64 22.4.0

Wanted to include a few more links and screenshots, but as a new user I was told that I could have at most two links and one screenshot :confused: