HttpSettings is an XML configuration that you use to control how SeoTools behaves when fetching content.

You can use HttpSetting in the following ways:

  • As a parameter to functions like XPathOnUrl, JsonPathOnUrl and RegexpFindOnUrl.
  • Edit HttpSettings in SeoTools.config.xml to affect all functions in SeoTools (Settings > Global http settings).
  • In the Spider to control the crawling.


Get all links from a mobile page.

This can easily be done for desktop pages using XPathOnUrl:


But to get the mobile page we need to change the UserAgent SeoTools uses when fetching the page.

Open the HttpSettings editor: HTTP > HttpSettings and add Request header User-Agent to: (iPhone)


Insert renders the following formula:


Whose result becomes:

    <Header Name="User-Agent">Mozilla/5.0(iPhone;U;CPUiPhoneOS4_0likeMacOSX;en-us)AppleWebKit/532.9(KHTML,likeGecko)Version/4.0.5Mobile/8A293Safari/6531.22.7</Header>

Combine this with XPathOnUrl and we get the completed formula:


Interval between request

This will ensure that requests to the same host will be executed with a delay of random ms between 1000 and 1500. If same can be "Host" (default), "Domain" or "Url". This is used in some Scrapers as a strategy to not get blocked too quickly.


You can supply SeoTools with one or more proxy to run requests through. If you enter several proxies, SeoTools with distribute requests between them.

Fail on HTTP error

By default, SeoTools doesn't fail on HTTP response codes like 404 and 500. You can set "Fail on HTTP error" in HttpSettings. See Error management.

Collect cookies

Enables cookies collection between HTTP requests. Button Clear cache & cookies in About resets the global cookie container.

Get help with this function in the community →