Alexander Ramadan

Where I Sometimes Write Stuff


Headless Browsing for SEOs
2016-03-16

I was totally inspired by Mike King's talk about technical SEO.

I've always had most of my interest in SEO around the technical aspects, which have spoken to me more than the content/outreach side of the field. So this talk discussed a lot of topics that were very interesting to me, one of them being headless browsing.

Headless Browsing for SEO

I had always heard of headless browsers, but ignored them for the most part, relying on other tools to do my audits. But I decided it was time to put my fear of the unknown behind me and give using one a shot. Checkout this article for a good run down on what Headless browsing is.

I went with PhantomJS as it seems to be the de facto headless browsing implementation. The PhantomJS website walks you through installing the library, and gives you some example code to start with. Having some knowledge of JS helps getting all this working quickly.

Here are some of the most useful sections of the site or tutorials I found for using it:

Below is a small code sample that opens a webpage in a headless browser, renders it, then takes a snapshot and saves that picture to your harddrive.

A lot of this code is from the example code on the site, but I made some changes so that it was more useful to me. The main change is that I leveraged command line arguments so that you can pass the URL, device size (mobile or desktop) and a file name to save the image with as arguments rather than having to edit the source code each time you want to crawl a new page. I also included the viewportSize for a mobile device and a desktop screen so that you can choose which you want to render. It seems like the script originally wants to default to mobile, so by passing either "desktop" or "mobile" as an arugment will let you render in the different sizes.

To use the script do this:

  1. Download and install PhantomJS
  2. Copy the code from the window above
  3. Paste into a text editor (not Microsoft Word)
  4. Save the file as example.js to your desktop (or where ever you want)
  5. Go to the terminal or command line
  6. Navigate to where you saved the file via the command line
  7. Run the code like explained below

Once you have the script saved, to run it you'll need to pass 3 arguments to the script:

To run the code, use the command line, navigate to where the script is saved and use this to execute the script,

phantomjs example.js url device filename

So, if I wanted to run it for this site's homepage, I would type

phantomjs example.js http://alexanderramadan.com mobile homepage

This will download and render http://alexanderramadan.com with the viewPort set to a mobile device size. It will also save the image of the page we rendered as "homepage.png".

It's been very interesting playing around with this and I can definitely see the benefits of pragmatically browsing a page. It's also been interesting to see what renders and what doesn't.

Definitely a nice tool to have handy if needed. Next, I'll be looking at HorsemanJS which seems to give users a simipler way to interact with PhantomJS.