HttpSettings

HttpSettings is an XML configuration that you use to control how SeoTools behaves when fetching content.

How to use HTTP Settings window

Purpose

You can use HttpSetting in the following ways:

Parameter

For functions like:

Global Settings

Change the HttpSettings in to affect all functions in SeoTools from the menu:
Enter global settings manu

Spider

Control the crawling in the Spider.

Example

Get all links from a mobile page.

This can easily be done for desktop pages using XPathOnUrl:

=XPathOnUrl("https://webpage.com","//a","href")

But to get the mobile page we need to change the UserAgent SeoTools uses when fetching the page.

Open the HttpSettings editor: HTTP > HttpSettings and add Request header User-Agent to: (iPhone)
Mozilla/5.0(iPhone;U;CPUiPhoneOS4_0likeMacOSX;en-us)AppleWebKit/532.9(KHTML,likeGecko)Version/4.0.5Mobile/8A293Safari/6531.22.7

Insert renders the following formula:

=HttpSettings(TRUE;;;;;;;{"User Agent"\"Mozilla/5.0(iPhone;U;
 CPUiPhoneOS4_0likeMacOSX;en-us)AppleWebKit/532.9(KHTML,
 likeGecko)Version/4.0.5Mobile/8A293Safari/6531.22.7"})

The final result:

<HttpSettings>
  <RequestHeaders>
    <Header Name="User-Agent">Mozilla/5.0(iPhone;
 U;CPUiPhoneOS4_0likeMacOSX;en-us)AppleWebKit/532.9(KHTML,
 likeGecko)Version/4.0.5Mobile/8A293Safari/6531.22.7</Header>
  </RequestHeaders>
</HttpSettings>

Combine this with XPathOnUrl and we get the completed formula:

=Dump(XPathOnUrl("http://webpage.com","//a",
"href",HttpSettings(TRUE;;;;;;;{"User-Agent"\"Mozilla/5.0(iPhone
;U;CPUiPhoneOS4_0likeMacOSX;en-us)AppleWebKit/532.9
(KHTML,likeGecko)Version/4.0.5Mobile/8A293Safari/6531.22.7"})))

Interval between request

This will ensure that requests to the same host will be executed with a delay. The user can change the minimum and maximum delay in ms (milliseconds):

Setting intervals between requests

If same can be "Host" (default), "Domain" or "Url".

HTTP Timeout

You can configure the timeout period by changing the following value, located in the SeoTools.config.xml file in the installation directory. Standard time is set to 120 seconds (time below is in micro seconds):

<Timeout>120000</Timeout>
</GlobalSettings>
</HttpDownloader>

Proxies

You can supply SeoTools with one or more proxy to run requests through. If you enter several proxies, SeoTools with distribute requests between them.

Fail on HTTP error

By default, SeoTools doesn't fail on HTTP response codes like 404 and 500. You can set "Fail on HTTP error" in HttpSettings. See Error management.

Collect cookies

Enables cookies collection between HTTP requests. Button Clear cache & cookies in About resets the global cookie container.

See

Guides

Caching