by toobsock » Mon Jun 16, 2014 9:13 am
We are hosting the web site on our own server. All of the "software" to gather the stats and display them has been developed by us.
The unique thing about this is that we need to access the Social Club thousands of times per day to retrieve all of the information for the players we are tracking.
As a typical security measure that 99% of the web servers out there use, they will block an IP address after so many requests to avoid being "flooded". So to get around that, we have incorporated "Proxy Servers" which allow us to make requests from several different IP addresses while gathering the data.
If we had another site or anything that utilized "Proxy Servers" we would just utilize them for this site as well, just like we do with the web server.
The cost of 5 proxy servers per month to rent is around $88. As the number of players we track grows, the scans take longer and longer. Right now we scan 6 players at a time every 15-20 seconds. Right now our scan time is nearing 3 hours.
A couple of options that we could pursue would be:
1) Cut the number of users we track by 90%, then we could eliminate the need for the proxy servers.
2) Eliminate the proxy servers and scan about 3 players per minute, which would put the scan time up at around 15 hours (at our current user base). Right now, after our initial scan, we are continually scanning the Social Club when a player's stats don't seem to make a logical progression (seems about 50-75 players per day have stats that don't progress properly) and when new players sign up to the site. That goes on for about 16 hours after the initial scan is done per day (rescanning about 3 players per minute).
The "rescanner" logic we use is pretty simple, we test several of the stats for a "natural progression", so when we see a field (for example) like "Car Crashes" go down instead of up or equal to previous scans, we keep on scanning until the Social Club serves up correct numbers.
So, the donations are strictly for us to rent the proxy servers.