Spider

Purpose

With the Spider you can combine functions in SeoTools into a powerful webpage crawler.

URL Columns

  • Url
  • Url.AnchorText
  • Url.DiscoveredOn
  • Url.Error
  • Url.Internal/External
  • Url.Level
  • Url.Rel
  • Url.Type
  • UrlProperty

HTTP Columns

  • CsQueryOnUrl
  • HttpHeader
  • HttpStatus
  • IsFoundOnPage
  • JsonPathOnUrl
  • RegexpFindOnUrl
  • WordCount
  • XPathOnUrl

SEO Columns

  • HtmlCanonical
  • HtmlH1
  • HtmlH2
  • HtmlH3
  • HtmlMeta
  • HtmlMetaDescription
  • HtmlMetaKeywords
  • HtmlMetaRefresh
  • HtmlTitle
  • LinkCount
  • PageCodeSize
  • PageCodeToTextRatio
  • PageEncoding
  • PageSize
  • PageTextSize
  • ResponseTime

Connector Columns

  • GoogleCacheDate
  • GooglePageSpeed.Score
  • GooglePageSpeed.GoogleMobileReady.Mobile Ready Pass
  • Majestic.IndexData.Item
  • Moz.Metrics.Canonical URL
  • OnPageScrapers.Email
  • OnPageScrapers.Google AdSense Id
  • OnPageScrapers.GoogleAnalyticsId
  • OnPageScrapers.InstagramAccount
  • OnPageScrapers.LinkedInAccount
  • OnPageScrapers.SocialMetaTags.OGTitle
  • OnPageScrapers.TwitterAccount
  • SocialUrlShares.BufferShares
  • SocialUrlShares.FacebookComments
  • SocialUrlShares.FacebookLikes
  • SocialUrlShares.FacebookShares
  • SocialUrlShares.GooglePlusOnes
  • SocialUrlShares.LinkedInShares
  • SocialUrlShares.PinterestPinned
  • SocialUrlShares.StumbleUponStumbles
  • SocialUrlShares.VKShares

Settings

  • Number of Hits - Decide how many pages you want to crawl.

  • Search Depth - Decide how "deep" you want o crawl. "2" means two layers of Sub Pages.

  • External Links - Decide if you want to include pages from external domains

  • Mode - Decide if you want to crawl web pages (Spider) or local files (List)

See

Guides

Related Functions


Get help with this function in the community →