Syncing a website

When you add a website to Knowledge, hiy reads the pages you picked and nothing else. A site changes after that — prices move, a guide is rewritten, a page is retired — so every site you have added gets its own screen where you can re-read it, choose how often, and decide what a re-read is allowed to look at.

Open it from Knowledge → Sources: find the site in the list and press Open site on its row.

Nothing is fetched again unless you ask#

Auto-sync is off for every site, including the ones you added before this existed. A site is read when you add it, and after that only when you press Re-sync now — unless you turn a schedule on yourself.

That is deliberate rather than cautious. A re-read costs you nothing, but a site that changes under your agent without you knowing is an agent answering from material you have not seen. So the schedule is a decision you make per site, and there are two of them:

There is no hourly or daily option. A website that changes that often is a feed, and re-reading one every hour would spend your allowance to find that nothing had moved.

New pages are offered, never added#

A re-read does two things. First it re-reads the pages you already hold. Then it asks the site what else it has now.

Anything new shows up on the site's screen as “12 new pages found” with a list you tick through. Nothing is imported until you press the button. If a site publishes forty posts in a week, you will be told — you will not wake up to forty pages you did not choose and an allowance spent on them.

What a re-read actually costs#

Almost nothing, most of the time. hiy sends the page's ETag from the last read, and a server that answers 304 Not Modified ends the exchange without sending the page at all. Where a site sends no ETag we compare a fingerprint of the text instead, so a page whose layout changed but whose words did not counts as unchanged.

A page that changed is stored immediately and joins what your agent answers from at the next train — your own, or the overnight one if you have that switched on. Until then the agent keeps answering from the text it already had, which is why a sync can never leave you with a gap.

What gets read#

The first group in Sync settings, and the one worth setting for a big site.

Use the site's sitemapon by default#

How hiy finds pages. Turn it off and only the addresses you list yourself are read.

Respect robots.txton by default#

Honours the Disallow rules in the site's robots.txt. Leave it on for a site you do not own — we ask other crawlers to stay out of our own app, so we extend the same courtesy. You can switch it off for a site that is yours.

Skip addresses with a query stringoff by default#

Filters what gets offered. It never changes which page is which: ?page=2 is its own page either way.

Never read theseone pattern per line#

Patterns, re-applied on every sync. https://example.com/tag/** excludes a whole section; https://{store,docs}.example.com/** excludes two subdomains.

Also read theseone address per line#

Pages the sitemap misses. They must be on the same site, and they are offered even when the sitemap answers.

If you list a page under Also read these and a pattern under Never read these that covers it, the exclusion wins — it is the more specific instruction.

What gets kept#

What survives from a page once it has been fetched.

Strip these elementsCSS selectors, one per line#

Removed before the text is read. Use it for a cookie banner or a navigation bar that keeps ending up in your agent's answers — .cookie-banner, #site-nav. A selector that does not match anything is ignored.

Ignore canonical linksoff by default#

Records whether a page declares a canonical address. Nothing is collapsed by it yet — two addresses the site calls the same page are still two pages — so this only changes what we store, not what gets read. Turn it on if you want the tag ignored outright.

How pages are read#

Most pages are plain HTML and are read directly. Some sites paint their text with JavaScript, and those come back empty.

FastHTML only#

Never renders. The cheapest option, and right for a blog or a documentation site.

Autothe default#

Reads the HTML first and only renders a page that came back with nothing useful.

Fullrenders every page#

The expensive option. Worth it only for a site where every page needs JavaScript.

When a page cannot be read#

A page that fails is kept, not dropped. It stays in your agent's knowledge with the text it had from the last successful read, and the site's screen shows it under Couldn't be read with the reason — a timeout, a 404, a sign-in wall.

Press Try again on the row to re-read that one page. A failure is recorded against the page itself, so it is still there tomorrow and still retryable; it does not disappear when the sync that found it finishes.

Stopping, and deleting#

Two different things, and the site's Manage menu keeps them apart on purpose.

Stop syncing switches the schedule off. Every page stays, your agents keep answering from them, and your settings and sync history are kept too — so turning it back on later is one tick, not a rebuild. Nothing is reindexed, because nothing changed.

Delete site and its pages removes the pages too, and everything your agents learned from them. It asks you to type the site's address back because there is no undo — re-importing the pages is the only way back. Any agent using that material stops being able to answer from it at its next train.

Sync history#

Every completed run is listed under Manage → Sync history: when it ran, whether you started it or the schedule did, and what it found. That is the place to look when an answer starts citing something you did not expect — the history says what changed and when.

If a site stops being useful#

When nothing from a site has been cited in an answer for ninety days, its screen says so. It is a note, not an action: the schedule keeps running and nothing is paused on your behalf. It is worth a look — either the material has stopped being relevant, or it is the wrong material — but the decision is yours.