---
title: "Syncing a website"
description: "Re-read a site on a schedule you choose, or when you ask. Auto-sync is off until you turn it on, and new pages are offered rather than added."
source: https://hiy.ai/docs/website-sync
---

# Syncing a website

When you add a website to Knowledge, hiy reads the pages you picked and nothing else. A site changes after that — prices move, a guide is rewritten, a page is retired — so every site you have added gets its own screen where you can re-read it, choose how often, and decide what a re-read is allowed to look at.

Open it from **Knowledge → Sources**: find the site in the list and press **Open site** on its row.

## Nothing is fetched again unless you ask

Auto-sync is **off** for every site, including the ones you added before this existed. A site is read when you add it, and after that only when you press **Re-sync now** — unless you turn a schedule on yourself.

That is deliberate rather than cautious. A re-read costs you nothing, but a site that changes under your agent without you knowing is an agent answering from material you have not seen. So the schedule is a decision you make per site, and there are two of them:

| Cadence | What it does |
| --- | --- |
| **Only when I ask** | The default. Nothing happens on a timer. |
| **Weekly** | Re-reads every seven days. |
| **Monthly** | Re-reads every thirty days. |

There is no hourly or daily option. A website that changes that often is a feed, and re-reading one every hour would spend your allowance to find that nothing had moved.

## New pages are offered, never added

A re-read does two things. First it re-reads the pages you already hold. Then it asks the site what else it has now.

Anything new shows up on the site's screen as **“12 new pages found”** with a list you tick through. Nothing is imported until you press the button. If a site publishes forty posts in a week, you will be told — you will not wake up to forty pages you did not choose and an allowance spent on them.

## What a re-read actually costs

Almost nothing, most of the time. hiy sends the page's `ETag` from the last read, and a server that answers **`304 Not Modified`** ends the exchange without sending the page at all. Where a site sends no `ETag` we compare a fingerprint of the text instead, so a page whose layout changed but whose words did not counts as unchanged.

A page that changed is stored immediately and joins what your agent answers from at the next train — your own, or the overnight one if you have that switched on. Until then the agent keeps answering from the text it already had, which is why a sync can never leave you with a gap.

## What gets read

The first group in **Sync settings**, and the one worth setting for a big site.

- **Use the site's sitemap** (`on by default`) — How hiy finds pages. Turn it off and only the addresses you list yourself are read.
- **Respect robots.txt** (`on by default`) — Honours the `Disallow` rules in the site's `robots.txt`. Leave it on for a site you do not own — we ask other crawlers to stay out of our own app, so we extend the same courtesy. You can switch it off for a site that is yours.
- **Skip addresses with a query string** (`off by default`) — Filters what gets offered. It never changes which page is which: `?page=2` is its own page either way.
- **Never read these** (`one pattern per line`) — Patterns, re-applied on every sync. `https://example.com/tag/**` excludes a whole section; `https://{store,docs}.example.com/**` excludes two subdomains.
- **Also read these** (`one address per line`) — Pages the sitemap misses. They must be on the same site, and they are offered even when the sitemap answers.

If you list a page under **Also read these** and a pattern under **Never read these** that covers it, the exclusion wins — it is the more specific instruction.

## What gets kept

What survives from a page once it has been fetched.

- **Strip these elements** (`CSS selectors, one per line`) — Removed before the text is read. Use it for a cookie banner or a navigation bar that keeps ending up in your agent's answers — `.cookie-banner`, `#site-nav`. A selector that does not match anything is ignored.
- **Ignore canonical links** (`off by default`) — Records whether a page declares a canonical address. Nothing is collapsed by it yet — two addresses the site calls the same page are still two pages — so this only changes what we store, not what gets read. Turn it on if you want the tag ignored outright.

## How pages are read

Most pages are plain HTML and are read directly. Some sites paint their text with JavaScript, and those come back empty.

- **Fast** (`HTML only`) — Never renders. The cheapest option, and right for a blog or a documentation site.
- **Auto** (`the default`) — Reads the HTML first and only renders a page that came back with nothing useful.
- **Full** (`renders every page`) — The expensive option. Worth it only for a site where every page needs JavaScript.

**Warning:** Rendering is not available yet, so **Auto** and **Full** currently read a page the same way **Fast** does. The **Wait for this element** and **Click this element first** selectors are saved against your site and will be used the moment it arrives.

## When a page cannot be read

A page that fails is kept, not dropped. It stays in your agent's knowledge with the text it had from the last successful read, and the site's screen shows it under **Couldn't be read** with the reason — a timeout, a 404, a sign-in wall.

Press **Try again** on the row to re-read that one page. A failure is recorded against the page itself, so it is still there tomorrow and still retryable; it does not disappear when the sync that found it finishes.

## Stopping, and deleting

Two different things, and the site's **Manage** menu keeps them apart on purpose.

**Stop syncing** switches the schedule off. Every page stays, your agents keep answering from them, and your settings and sync history are kept too — so turning it back on later is one tick, not a rebuild. Nothing is reindexed, because nothing changed.

**Delete site and its pages** removes the pages too, and everything your agents learned from them. It asks you to type the site's address back because there is no undo — re-importing the pages is the only way back. Any agent using that material stops being able to answer from it at its next train.

## Sync history

Every completed run is listed under **Manage → Sync history**: when it ran, whether you started it or the schedule did, and what it found. That is the place to look when an answer starts citing something you did not expect — the history says what changed and when.

## If a site stops being useful

When nothing from a site has been cited in an answer for ninety days, its screen says so. It is a note, not an action: the schedule keeps running and nothing is paused on your behalf. It is worth a look — either the material has stopped being relevant, or it is the wrong material — but the decision is yours.