Bulk check Google Mobile Test in Excel

So with mobilegeddon raging around us we need a good way to bulk check all of our websites for mobile friendliness. A perfect job for Excel and SeoTools!

Google has released a tool that help test whether a website is considered "mobile friendly" or not. But only for one website at a time. What if we need to bulk check lots of websites to spot the ones we need to fix?

In this post I'll show you how we can create a SeoTools Scraper that check this for us in Excel.

The Google tool internally uses a JSON REST API whose result can look like this:

We're after value with the JSON path $.ruleGroups.USABILITY.pass.

To build a new scraper, create a file myscrapers.xml under /scrapers/ in your SeoTools installation path with the following contents:

<?xml version="1.0" encoding="utf-8" ?>
<Scrapers>
  <ValueScraper Identifier="Google.MobileFriendly" Title="Google.MobileFriendly" Category="Onpage" SpiderScraper="true">
    <Parameters>
      <TextBoxParameter Identifier="Url" Title="Url" DefaultValue="" Required="true"/>
    </Parameters>
    <Fetch Url="https://www.googleapis.com/pagespeedonline/v3beta1/mobileReady?url={Url.UrlProperty.Absolute}">
      <HttpSettings>
        <IntervalBetweenRequests RandomFrom="1000" RandomTo="1000" IfSame="Host"/>
        <Cache>false</Cache>
      </HttpSettings>
    </Fetch>
    <Match>
      <JsonPathMatcher Expr="$.ruleGroups.USABILITY.pass"/>
    </Match>
    <Parse>
      <StringParser/>
    </Parse>
  </ValueScraper>
</Scrapers>

I've added 1000ms interval between requests to not piss Google off (so they won't block my IP).

If you have Excel open you need to restart Excel. We now have a new scraper:

Since the Google API is quite slow you will need need to wait a while for the results. Excel might seem to freeze, but that's just SeoTools working in the background.

Update: As of SeoTools 4.3.5 the scraper created in this post is included in the release.

Do you have other ideas for scrapers? Let me know in the comments!