Syncing a website
When you add a website to Knowledge, hiy reads the pages you picked and nothing else. A site changes after that — prices move, a guide is rewritten, a page is retired — so every site you have added gets its own screen where you can re-read it, choose how often, and decide what a re-read is allowed to look at.
Open it from Knowledge → Sources: find the site in the list and press Open site on its row.
Nothing is fetched again unless you ask
Auto-sync is off for every site, including the ones you added before this existed. A site is read when you add it, and after that only when you press Re-sync now — unless you turn a schedule on yourself.
That is deliberate rather than cautious. A re-read costs you nothing, but a site that changes under your agent without you knowing is an agent answering from material you have not seen. So the schedule is a decision you make per site, and there are two of them:
| Cadence | What it does |
|---|---|
| Only when I ask | The default. Nothing happens on a timer. |
| Weekly | Re-reads every seven days. |
| Monthly | Re-reads every thirty days. |
There is no hourly or daily option. A website that changes that often is a feed, and re-reading one every hour would spend your allowance to find that nothing had moved.
Early Access re-reads
On Early Access, a site you re-read yourself can be re-read once every 7 days.
| Rule | What it means |
|---|---|
| One clock per site | Re-sync now here, and Sync and Re-read in the Knowledge Hub, all count against it. |
| Too soon | The button says when the next re-read is available. |
| Scheduled re-reads | A weekly or monthly one doesn’t count. While one is running, a press waits: the site is being re-read on its schedule right now. |
| A page that failed | Try again always works. |
| Sooner than that | Ask us — we can allow an early re-read. |
New pages are offered, never added
A re-read does two things. First it re-reads the pages you already hold. Then it asks the site what else it has now.
Anything new shows up on the site’s screen as “12 new pages found” with a list you tick through. Nothing is imported until you press the button. If a site publishes forty posts in a week, you will be told — you will not wake up to forty pages you did not choose and an allowance spent on them.
What a re-read actually costs
Almost nothing, most of the time. hiy sends the page’s ETag from the last read, and a server that answers 304 Not Modified ends the exchange without sending the page at all. Where a site sends no ETag we compare a fingerprint of the text instead, so a page whose layout changed but whose words did not counts as unchanged.
A page that changed is stored immediately and joins what your agent answers from at the next train — your own, or the overnight one if you have that switched on. Until then the agent keeps answering from the text it already had, which is why a sync can never leave you with a gap.
What gets read
The first group in Sync settings, and the one worth setting for a big site.
Use the site's sitemapon by defaultHow hiy finds pages. Turn it off and only the addresses you list yourself are read.
Respect robots.txton by defaultHonours the
Disallowrules in the site’srobots.txt. Leave it on for a site you do not own — we ask other crawlers to stay out of our own app, so we extend the same courtesy. You can switch it off for a site that is yours.Skip addresses with a query stringoff by defaultFilters what gets offered. It never changes which page is which:
?page=2is its own page either way.
If you list a page under Also read these and a pattern under Never read these that covers it, the exclusion wins — it is the more specific instruction.
What gets kept
What survives from a page once it has been fetched.
How pages are read
Most pages are plain HTML and are read directly. Some sites paint their text with JavaScript, and those come back empty.
Auto renders a page with a browser only when the plain read comes back with almost nothing, so most pages never cost a render. Full renders every page and uses your daily allowance fast. Each workspace gets a limited number of rendered pages a day, and all workspaces share one overall limit; a page over the limit keeps the plain read and can be tried again tomorrow.
Rendering respects your site’s robots.txt while Respect robots.txt is on: a page it disallows is not rendered. Pages are rendered by Cloudflare’s browser service, which identifies itself as HiyBot. Wait for this element is passed to the browser. Click this element first is saved but not used yet.
When a page cannot be read
A page that fails is kept, not dropped. It stays in your agent’s knowledge with the text it had from the last successful read, and the site’s screen shows it under Couldn’t be read with the reason — a timeout, a 404, a sign-in wall.
Press Try again on the row to re-read that one page. A failure is recorded against the page itself, so it is still there tomorrow and still retryable; it does not disappear when the sync that found it finishes.
A page that builds its text with JavaScript and could not be rendered shows needs JavaScript to read: either the day’s allowance is used up (try again tomorrow) or rendering isn’t available for it, and pasting the page’s text works instead.
Stopping, and deleting
Two different things, and the site’s Manage menu keeps them apart on purpose.
Stop syncing switches the schedule off. Every page stays, your agents keep answering from them, and your settings and sync history are kept too — so turning it back on later is one tick, not a rebuild. Nothing is reindexed, because nothing changed.
Delete site and its pages removes the pages too, and everything your agents learned from them. It asks you to type the site’s address back because there is no undo — re-importing the pages is the only way back. Any agent using that material stops being able to answer from it at its next train.
Sync history
Every completed run is listed under Manage → Sync history: when it ran, whether you started it or the schedule did, and what it found. That is the place to look when an answer starts citing something you did not expect — the history says what changed and when.
If a site stops being useful
When nothing from a site has been cited in an answer for ninety days, its screen says so. It is a note, not an action: the schedule keeps running and nothing is paused on your behalf. It is worth a look — either the material has stopped being relevant, or it is the wrong material — but the decision is yours.