Forums

I have a script that scrapes a website by making parallel HTTP requests.

I find that when I run the script, the cpu time usage goes up by different amounts seemingly not related to how long the script took to run.

For example, it ran for 50 seconds and only used 40 seconds of cpu time. Then I ran it again and it took 65 seconds and the cpu time usage increased by 135 seconds.

The script was doing the exact same requests both times. Is there a reason the cpu usage increase is so variable and is there anything I can do to make it more predictable?

Even adjusting the script to use one thread, it still has massively variable cpu time usage compared to actual script run time, sometimes massively over, sometimes a bit under.

Is it the only thing you run on PA? You could check running processes on the Tasks and Consoles page and see if there's something else that uses CPU as well, maybe?

Just checked tasks and consoles and there's nothing else running. Apart from my site, the only thing I run on there is another scraper that runs for about a minute once a day. Other than that there's nothing that would increase my cpu usage. My website's general use doesn't impact my cpu usage at all.

If that's the case, maybe add more logging to see if the script is not performing more tasks/computations when you see larger amount of CPU used.