All Hail The SpaceNode Spider!

All Hail The SpaceNode Spider!

One of the many benefits of owning a SpaceNode is that your visitors are able to search your content, news and updates directly from your own website without resorting to using search engines. This means that fewer visitors are leaked to your competitors.

SpaceNode Developer Alex Kent discusses how our own success has meant a radical rethink on how we index your SpaceNode websites and provide accurate search results in a fraction of a second…

Hi everyone,

I’m pleased to announce that SpaceNode has just given birth to the SpaceNode Spider whose job it is to visit your sites and index your content.

When we launched SpaceNode earlier this year, we used a third-party solution called Sphider. However, with our rapid expansion we quickly outgrew what it was capable of doing for us.

In the beginning, we could index all the SpaceNode sites daily but with over one hundred websites and 23,000 pages of content (and counting) this quickly became a mammoth task and we slipped to indexing every three days to minimise disruption.

When the Sphider Spider was running for upto 12 hours at a time, there was a noticeable slowdown in how quickly search results were returned to the user.

With SpaceNode websites having a reputation for their fast page load times, this was a problem that deserved our utmost attention.

So – we began looking around for a replacement solution but we soon discovered that we’d have to make one ourselves.

Main features of the new SpaceNode Spider are:

  • Visits rapidly changing pages up to once per hour
  • Visits pages that rarely change daily, weekly or monthly
  • Understands SpaceNodes so that search results do not contain blocks, headers or footers. Only the ‘meat’ of the webpage is indexed so results are more accurate
  • Multiple SpaceNode Spiders can index different SpaceNode websites at the same time
  • The SpaceNode Spider/s (which visits sites) and the SpaceNode SearchEngine (which returns search results to the user) need not be on the same physical server meaning we can take advantage of cloud computing

We have launched the SpaceNode Spider this week and engineers are currently testing its performance. Already we are mightily pleased with our new addition to the SpaceNode team and hope he has plenty of happy spidering adventures ahead of him!


Alex Kent
SpaceNode Developer


Related Pages:

Posted Jul 21, 2010