Gearbot: Now with 1.5L more helper-ness per square mile!

March 10th, 2012

Hi Guys!

I have been working hard since the last update and thought I would point out the bits and bobs that have been added to the site! Only 3 months into the year and already 2 blog posts!

Stock Levels

For a number of retailers we now list the stock level indicated on their site on all product listing pages! When calculating the lowest price, we exclude any products which are not in stock. This should make finding what you want a lot easier and faster! This has been requested a few times, and took a LOT of work to get done.

4 New Categories

Since our last blog post, we have added the following new categories:

Don’t worry – we are committed to only listing IT/Computer stuff, so our menu bar wont be getting any larger for a while!

Existing Category Updates

Our existing categories have new products added to them all the time. This is a manual process, but is so stream lined that it happens to any category, multiple times a day, so we don’t really post about it much. When we find a product which is listed in multiple stores and not categorized, we add it to the category – simple as that.

The product groupings on top level categories are now limited to 40 products per group, as they were starting to get a bit big. This helps keep the pages smaller/more readable, and makes them load a bit faster.

Listed Product Changes

We have made a few fundamental changes to the live site recently, mostly to speed up site access, but one of the larger changes has been how we list products on the site.

  • Category and Search pages only list products that have been found within the last week.
  • Any products that we haven’t found on a retailers site for over a week are removed from our index and listed as archived pages.

We are also listing breadcrumbs on all indexed product pages, which should make finding your way around quicker.

Feedback page

We now have a feedback page up and running, so you can leave us some anonymous feedback. We want to hear anything you have to say! Good or bad! We are quite happy with where the site is currently at, so if you have any ideas for new features, please let us know!

As ever, check our Facebook page for the latest updates.

Cheers

-Michael

Posted in Gearbot Updates | No Comments »

Gearbot update

February 5th, 2012

Hi everyone! Happy 2012! After 2 days of downtime this week, I thought I had better give a few updates on what has been going on! Where to start?

New Mobile site!

Our fancy new mobile site is up and running! built on the Jquery mobile platform, it is built for todays touch screen smart phones, not the old WAP interface we had.  It also looks great on tablets! give it a whirl!

Gearbot mobile website >>

The Down time :(

Site growth is always great! unfortunately it also can cause headaches if the site is not set up to handle the extra load. The extra growth, and a change to the way URLs are generated, caused Gearbot to fail constantly over the last week.

The main issue being that the caching framework the site was using was built for content which is added and hardly ever updated. Re-caching ~300,000 products every day, and re-caching ~10,000 categories  daily was really beginning to take its toll. As a result we had to bin the framework and rebuild it from scratch! We are now experiencing super fast page loads!

Product listing

One part of the site that we felt really needed work was the categorization of products. Today we have rolled out a new categorization method which solves the 2 problems we were having. The first issue was products getting into categories they were not supposed to. The new system means unless it is a human error, products will not be turning up into incorrect categories (if they do, please let us know!). The second issue was speed:  Pages with a lot of products were loading slow. The new setup means they will load really really fast!

Site speed

One of the great things about moving the scraper to a separate server is that the average page load time went from about 7 seconds, down to about 4 seconds. With the new framework and product listing changes, we now hope to get the average page load time down to under 2.5 seconds per page load! Saving you even more time!

Site usage

Since October, the average amount of daily visits has increased by about 25%, which is great news! It is really motivating to know that people are using our site. Expect a lot more great stuff from us in the future!

That’s about all for now. If you have any online retailers you would like added, give us a yell on our facebook page!

Until next time!

-Michael

Posted in General News | No Comments »

New site up and running

October 10th, 2011

Hi Guys,

It has been a while since my last post, but I have been hard at work, and have finally pushed out some big updates which are now live on Gearbot!

New Website Scraper

Gearbot has recently been experiencing a lot of downtime, which can be attributed to 2 things, the first of which is the scraper.

The scraper has been completely re-written, for 3 reasons:

  • Speed – The (now old) scraper was written in CakePHP and hitting all kinds of performance issues, the largest of which was MySQL bottle necks. These have all been resolved by writing the scraper in bare-bone Python/MySQL, and is now multi-threaded, improving the bots speed by a factor of 10 over the previous scraper.
  • Adding sites to the scraper was a rather time consuming task – Even through our best efforts it took between 15 – 45 minutes to add and test a new site. The new scraper has had a lot of work done to improve the speed at which we can add sites, and a bunch of other features to cut down on the time it takes to test new sites and get them live.
  • Separation from the live site – The scraper now sits on a completely separate server, so if it does run into issues, it wont take the live site with it. This also gives the site an all-round speed increase due to it not having to share resources.

New Host

I have pushed this update out a little sooner than I would have liked. This was due to the second reason for all the downtime: Our web host began to suck. Our new host will hopefully fare a bit better!

Wider Layout

With a majority of our user base using 1366px or wider screens, we have moved to a wider ‘liquid’ style layout. The width of the page will automatically fit the width of your browser, from 950px to a max width of 1300px. This should make it a little easier to view category pages and search results. Let me know what you think!

The search bar is back!

We had a few comments regarding the removal of the search box on the home page when we did the last redesign. I have put it back, just for you!

What the future holds:

Pushing up the update this early means that there are a few fewer stores than in the old site, so I will be adding a few more stores before going back to keeping the product page categories tidy and up to date.

Unfortunately with every new release, come unavoidable bugs. Please bear with us while we sort out any unexpected errors. If you find any issues, or have any suggestions, flick me an email. I always love to hear from you guys!

Onwards and Upwards

There will be a few more features I hope to sneak in before the Christmas rush so stay tuned! As always, the latest updates will be on Facebook.

Thanks

-Michael

Posted in General News | No Comments »

Yearly blog update!

April 1st, 2011

Hi guys,

Michael here, the creator of gearbot. Unfortunately gearbot has not had the best maintenance plan going on over the last few months. Categories are out of order, no new parts listed, and even a few retailers have been removed due to incorrect scrapping (Ascent is currently on hold :( , one of my favourite retailers!).

All this is due to a major life-changing move: At the start of the year I sold all my possessions, moved to another country, got a new job, and have a new life routine (I have lost 15kg in the last 2 months!).

While gearbot has been neglected, I am ready to get it back up to scratch. I would like to start by thanking the ~3000 monthly visitors to the site who are generating around ~10,000 page views. I love viewing the stats and it feels great that I am helping you find the cheapest IT prices.

If you are seeing this blog post, then the DNS has transferred over and gearbot is on its shiny new VPS. From here I will be (in rough order):

  • Finish up testing all the parts of gearbot to make sure they work on the new server (if you find any bugs, please email me!)
  • Update the About us pages
  • Re-add the stores that have been removed
  • Add new products to the product listing pages
  • Fix current product listing categories
  • Add a bunch more features
  • Pizza and beer

For the latest news, I suggest joining our Facebook page (link in the top right!) for updates as they happen. Once again, I would like to thank you for using gearbot and hope you will continue to do so!

- Michael

Posted in General News | 3 Comments »

Happy New Year!

January 8th, 2010

Merry Christmas and happy new year everyone!

Here at Gearbot we had a great Christmas and new years break, and had a great start to the first week of the new year, so without any further ado, here are the updates since Christmas:

  • 23 More retailers added! Bringing the total to 93 retailers and around 250,000 products!
  • Just before Christmas, we added an Audio, Laptops and Cellphones category, bringing Gearbot to a total of 18 Categories.
  • This shiny new blog has been added, giving you an easy, anonymous, way to comment about any news and changes we make!
  • The site styles have changed a little, removing the Christmas buzz it had.
  • And the biggest change in a while: A new, light weight Mobile website: http://m.gearbot.co.nz !

With most of the updates being the same as previous ones, the only one I really want to talk about a bit on the blog is the Mobile website addition:

The mobile site was added after talks with a retailer about what they would like added to the site, and this really stuck out as an obvious one that we felt would really benefit gearbots users. If you are in a store and think you see a good deal, what better way to confirm your suspicions than to straight away find the product on our new mobile site. A brief overview of its features:

  • To find products you can either search through all 250,000 products on gearbot from a search menu, or navigate to any of the page categories via the navigation interface.
  • The mobile site doesn’t use any images, and minimal css/html to give super small pages between 0.6 – 0.9KB for category pages, and just over 2KB for a page listing 50 products
  • Being unable to get the company to pay for a iPhone for testing (You better be reading this Nick!), it has only been tested on my LG phone. It probably will have bugs. If you find any, please email me!

We hope you find these updates useful. As ever please join our facebook group for updates as they happen, and remember to spread the word about gearbot! The more people we get the more I can justify spending time on making it better!

All the best for the new year guys, until the next shocking update!

Thanks,

The Gearbot team.

Posted in Gearbot Mobile Site, Gearbot Updates, General News | 5 Comments »

Weekend Updates!

December 20th, 2009

Another week hard at work. Big updates over the last few days have been:

  • 10 new retailers added over the last 2 days, bringing Gearbot to 70 stores, and about 180,000 products.
  • A Cameras Section has been added
  • A Apple Hardware Section has been added, listing all apple computers, iPods, iPhone etc..
  • Template fixes: Lots of em. Finally got rid of the ugly search buttons too!

Feedback on these updates, and any bugs you may find, is always appreciated!

Unfortunately some site progress has been a little slow, as the big stores seem to have a lot of requirements that need to be met before we can list thier store products. I hope to have it sorted over the next week, otherwise after Christmas. Phone, Sound and Car audio sections will be coming up in the next week.

To keep up to date with what’s happening on Gearbot, join our Gearbot facebook page and get all the updates as they are individually rolled out.

Many thanks, and Merry Christmas!

The Gearbot team.

Posted in Gearbot Updates, General News | No Comments »

More Stores and New Layout!

December 18th, 2009

Over the last 2 days we have rolled out 2 Major updates to gearbot:

  • New Site template – Finally Gearbot has its own look.
  • 27 new retailers added over the last 2 days, bringing Gearbot to 60 stores, which is twice as much as 2 weeks ago.

Feedback on these updates, and any bugs you may find, is always appreciated!

Over the next week we will be adding a few more categories, and as even more stores to bring you the cheapest prices. To keep up to date with what’s happening on Gearbot, join our Gearbot facebook page and get all the updates as they are individually rolled out.

Many thanks, and Merry Christmas!

The Gearbot team.

Posted in Gearbot Updates, General News | No Comments »

Upgrades Upgrades Upgrades!

December 13th, 2009

After a busy day we have managed to roll out 6 updates for you. These include:

  • A Software Category listing various Operating systems, Work software, and gaming software.
  • A Console Category for all your gaming needs, just in time for the holidays.
  • Completed requested tweaks to the store listing on product pages, including City & a smaller scrape time listing
  • A “Report an incorrect product listing” link at the top of all product listing pages, so you can help keep Gearbots product listings in shape if we have incorrectly listed products.
  • Brand sub categories are now listed on Optical drive, Graphics card, Monitor, and RAM pages. Just click on the small triangle for some AJAXy goodness
  • A couple more stores added

Feed back on these updates, and any bugs you may find, is always appreciated!

We will hopefully be rolling out some more updates before Christmas, to keep up to date with what’s happening on Gearbot, join our Gearbot facebook page and get all the updates as they are individually rolled out.

Many thanks, and Merry Christmas!

The Gearbot team.

Posted in Gearbot Updates, General News | No Comments »

100,000 Products listed!

December 2nd, 2009

Today (Wednesday), at 11.25pm, Gearbot scraped & inserted its 100,000th product. Yes thats right: 100,000 products!

These products are all searchable from the search menu, and viewable in the page categories that they fit into. Reaching this milestone has been a difficult task, as it has been a bumpy ride along the way due to DB scalability problems, server crashes, script timeouts, and mysql transaction errors, and slow page displays.

Reaching this milestone has meant a lot of fine tuning to get the sites scraped, into the database, and indexed with room for improvement.

We are very happy to have reached it for 2 important reasons:

1) Our HTML/Feed reading bot has been heavily optimized and is completing all 30 stores in about 8 hours.
2) The Database, Scraper and Indexer all work together & have been tested with ~350,000 products Without any speed issues.

The 100,000 products come from 30 stores.

A big thanks to everyone who has found bugs and suggested retailers for the site. We will now endeavour to add more of the requested functionality & fixes that you have asked for. Anything you would like to see on the site? send me an email, and we will do our best to get it done.

For now why not join our shiny new Gearbot facebook page, so you can hear about fixes & updates as they happen.

Many thanks

The Gearbot team.

Posted in Gearbot Updates, General News | No Comments »

1000 Site visits!

November 21st, 2009

Today (Saturday), in only 20 days, gearbot reached 1000 visits! Thanks to everyone for checking out our site. The 1000 visits came from 500 unique users, and has totalled to about 6000 page views. This has been a great milestone to hit today, 1 day before gearbot turns 3 weeks old.

We will continue to work on the site to provide you with the cheapest price listings in the simplest format. There have been a few bumps on the road, but everything has come together and in under three weeks we have a brilliant site listing 262696 products from 24 retailers, with more retailers added every day.

A extra big thanks to all the people who took the time to send us feedback on the site. It has not only helped make the site better for everyone else, but also helped motivate us to spend more time working on the site, and getting it finished faster.

Many thanks

The Gearbot team.

Posted in Gearbot Updates, General News | No Comments »