What is Screaming Frog?

Screaming frog is one of the many SEO tools available with plenty of great features for SEOs. This is one of our tools of choice. After scraping your website, it can help you to visualise the structure of the website, finds broken links and is full of useful information for anything from H1s, Metas, Canonicals. It can even create sitemaps.

In context of todays topic, we can also use it to find specific text on a website. This can be so useful in so many scenarios. For example, a company changes phone number or email address and we need to find any page that this information exists.

Case Study Scenario why we used Screaming Frog to find specific content

The example today where we use Screaming Frog to make what could be an arduous task easy is this. A website we work with once advertised and offered a ‘lifetime warranty’ on its products. We now offer a very lengthy warranty period but no longer advertise ‘lifetime’. All references to ‘lifetime warranty’ had previously been removed but our clients were concerned that there may have ben an instance of ‘lifetime’ lingering around on the website.

Solution: This is how to use Screaming Frog to find that Specific Text

Load your Screaming Frog application

Enter in URL details as you normally would. On the menu, go to Configuration > Custom > Search 

The Custom Search Screen

You’ll then get to the ‘Custom Search’ screen. This is where you input the search terms you are looking for. You can enter multiple search terms or ‘filters’ so you only need to perform one crawl to get data for multiple searches. Great if you need to change an email address and a phone number. In the screenshot below, you can see we searched ‘lifetime’, ‘lifetime warranty’ and ‘warranty’.

Pro Tip: ‘Does Not Contain’ is a search option. When would you use this? if you have a piece of code that needs to appear on many pages or all pages. Often this might be something like your chat tool or a tool like Hotjar, Google Tag Manager or Google Analytics. It would be much easier filtering through the pages where the code doesn’t appear rather than potentially hundreds of pages where the code does appear.

Note: As you can see in the screenshot, this searches the source code of the internal HTML page. What this means is it searches the coding of the page and not necessarily what you see as a user. I’ll explain later why this is important to know.

The Search Result

You will find your search results under the ‘custom’ tab. The overview will show your search results and you can also select which filter you want to look at on the left hand side. From there, you can check through the resulting URLs to then go through and update the relevant pages on your live website.

Note: Back on the ‘source code’ topic. You will see in the above screenshot that there are 27 instances of ‘warranty’. The reality of this particular website is that ‘warranty’ does not appear on the webpage when you land. The reason being is that there is a piece of code which noted something like ‘id_warranty’ in the HTML so this crawler has picked up on it.

Pro Tip: Robots.txt – This is an important one

As a good crawler should, Screaming Frog is set to obey robots.txt as a good robot should. However, this does not mean there isn’t a page linked somewhere on the site which might still contain the text you are looking for. To go that one step further, we make our robot a little naughty and tell it to ignore robots.txt. What this does is makes the crawler effectively crawl everything it can and not stop even if robots.txt tells it not to.

Step 1 – Go To robots.txt settings

Go to robots.txt settings by clicking on the menu and going to Configuration > robots.txt > Settings

Step 2 – Ignore robots.txt

Select the dropdown and ‘Ignore robots.txt’

Note from DiGiHUB

As with any tool, it’s not always going to be 100% so it always pays to double check. Screaming frog is a super powerful tool and for those who work on campaigns with small websites can take advantage of the fact that it will crawl 500 URLs for free. What we also did in this exercise is to check with a google search with an advanced operator. In this case we can search with the following ‘lifetime warranty site:www…..’ which will result based on googles indexed pages for pages that are relevant to this search term. In this case that resulted in nil pages. Another check is also on the backend of your website if your CMS allows for you to perform a search.

Download Screaming Frog SEO Spider Tool

Note we are not affiliated in any way but this is a well known and accepted tool for SEOs in the industry. Downloading is free and you can get up to 500 URL crawls under the free version with lots of functionality. We use a number of different SEO tools that crawl sites and all have their subtle differences but this is definitely one of the better ones out there which also has Windows and Mac versions.

This is the link to the official site, not an affiliate link. If you’re not familiar with the tool or want to check their pricing options, you can view it from here: https://www.screamingfrog.co.uk/seo-spider/

Hope this article helps and happy searching!