Screaming Frog and the Search Console URL Inspection API

Send us a text

Today is February 10th and I wanted to talk about Screaming Frog as well as the new Search Console URL Inspection API. 

Full Blog Post / Show Notes: https://opinionatedseo.com/s2e20

Let’s jump right in, what is screaming frog and what can you do with it?

Let’s start at the beginning, it’s free but limited, and you don’t run it online. The paid version is 149 pounds / year, or about $200 US dollars. So you are talking about $16 a month if you wanted to break it down that way. Buy it, write it off, and call it a day, it will save you that money in the first hour you use it.

So what can it do?

Screaming frog is an SEO spider, think GoogleBot but you get to save the data it captures.

Here’s a list of the top features for it:

• Find broken links, errors, and redirects
• Page title and meta data analysis
• Met robots and directives review
• Audit hreflang
• Discover exact duplicate pages
• Generate xml sitemaps
• Create site visualizations.
• You can do up to 500 URLs in the free version, unlimited to your computer’s capacity on the paid version.

But the real power comes in when you use the paid version, and I’m going to just touch on a few of the features.

• Javascript rendering: Actually being able to view the page as a fully rendered version. This includes a screenshot as well as being able to review source code for the rendered version.
  • We’re using this to better understand how the rendering affects crawlability, as well as what’s on the site and what isn’t depending on JS.
  • If you do any kind of client side rendering, using react, or just not seeing Google crawling your site like you think it should, this is a great way to better understand.
• URL Inspection API Integration - I was going to save this for last, but it’s just too good to not put at the top of the list. There are a few integrations you get with screaming frog out of the box. 
  • 1 - Google Analytics 
  • 2 - Google Search Console - this includes the inspection API integration. 
  • 3 - Page speed insights 
  • And non Google related companies like Majestic for backlinks, ahrefs for things like backlinks and URL rating, and Moz for data like page authority and backlink info from their open site explorer metrics.
• Site Visualizations
• Custom Extraction
• Meta Tags - Titles, descriptions, etc 
• You can even generate XML sitemaps
• Find out all pages that have broken links both internally and externally. 
• 301s - 

Integration with the new Search Console URL Inspection API which was officially launched on January 31st, 2022 by Google - and it allows you to see indexation status of your URLs via an API. This was integrated in the latest release in the last few weeks and you’ll see a good share of tutorials out there. 

You probably want to know - what am I doing with this new data, and what data do we get that we can use?

First, the data is used to understand indexation status in Google, errors that are coming up, and last crawl date.

And what am I doing with that data? Let’s start with indexation status - you have a few variations of the summary of indexation, things like index and submitted, indexed not submitted in sitemap, crawled not indexed, discovered but not crawled, soft 404, error, and probably a few more I’m forgetting. By pulling this data on pages, you can get a better understanding of how Google sees your s