Where’s all the time going – CPU time that is?
Posted on 9 Oct ’12 by Earl
This website/blog is possessed!
That’s about the only answer I have left after trying for five days to discover why it’s using so much CPU time on my shared host. And of course, my hosting provider, rightly so, is wanting me to find that answer or else they will have to limit resources or pull the plug on this site.
So, Meandering Passage is running bare bones just now. There’s not even a contact form available at the moment.
I’m down to only two WordPress plugins, Akismet Anti-SPAM & WP Super Cache;
I’ve stopped the internal WP-CRON from running on each page load and now it’s manually scheduled to run once every six hours; and
I’ve changed to the previous theme which I kn0w ran before with no problem.
The only thing I’ve been able to find suspicious after reviewing the site stats was a web crawler “80legs” generating a disproportionate amount of hits and bandwidth. I’ve blocked it in my robot.txt file but it may still be crawling the site. I’ve exhausted what little knowledge I know of this “stuff.” I’m out of ideas.
My host, ICDSoft, and their support staff have been top notch in supporting and working with me on this problem.
ICDSoft Support just notified me they made a change in my .htaccess file to deny/block the “80legs” bot from my site.
Now we wait a couple and see what happens — to see if Meandering Passage remains “among the rocks” as well.
After two days it’s clear the “80legs” web crawler was the culprit for the high (3x over the limit) CPU usage of my site. I don’t know why “80legs” was crawling my site at such a high rate. I don’t believe it’s anything I did knowingly and if that’s the case this could happen to almost any site. In my opinion, even if “80legs” is distributive web crawling as they claim it doesn’t react in a timely and responsible manner to instructions in the “robots.txt” file. Here’s someone else talking about this issue.
Below is a screenshot of the CPU time usage chart. The web-crawler was blocked on Oct 9th resulting in a marked difference.