Forums

Curious, I saw this in another thread about scheduled tasks. I print a TON of information to console, sometimes the entire source code of a web page, then that source code broken down. I've always just left it there as a way for me to monitor what is going on, but I have noticed my consoles getting stuck here.

Does mass printing cause this sort of problem? If so, could someone maybe explain why this is? They would sometimes, much more rarely than they do here, but sometimes do this on my personal server too. How come?

Does the problem get fixed if you refresh the console page? In general, printing lots of stuff should be OK on the server side, but the JavaScript terminal emulator we use can sometimes get stuck if it's sent too much data too quickly. Normally just refreshing the console page will fix that, though.

Yeah, whenever I refresh the page that fixes it, but I also don't leave the console page up. I just have something that checks to make sure the database is being updated, and I get alerted if not.

Does the process carry on running OK?

I guess what I'm asking is, is this

• a minor irritation with consoles hanging but work-aroundable by refreshing the page (which would be hard for us to fix because it's a JavaScript performance thing), or
• are your processes being blocked by it (which is obviously much more serious, and is also something we might be able to fix on the server side)?

Well, when I refresh the console page, yes they will carry on running just fine. If I do not refresh them though, they will just continue hanging until I refresh the page or restart the script.

It's still not clear though. Are you saying the process is hanging I.E. not updating the database you mentioned above, or just the display in the console is hanging?

It is not updating the database. The hang is more than just visual.

It also seems to come in waves. Lately, it has not been happening, but a couple days ago it was a constant battle for a good 24 hours heh...then, before that, it was solid again.

That's really strange. Is there any chance you're using lots of CPU and getting tarpitted?

Nope, definitely not that. I currently have a 40K/day limit even. Used to be coming very close the 20K limit, but I haven't passed it, even since I upgraded to the 40K (was a custom thing).

At the moment:

14% used (5896.89s of your 40,000 second CPU allowance) Allowance resets in 6 hours, 15 minutes

Curiouser and curiouser. And the processes just hang, they don't exit? Is it possible that it's not related to the printing, but something on the database side? Are you writing really large amounts of data there?

It's totally possible it is not related to printing. Don't think it is database-side, data sent is pretty small, about 6 characters or so.

Some of the scripts will store a lot in RAM. Sometimes upwards of 7,000 URLs for the big ones, but the smaller guys that only store a few hundred also get hung up with no noticeable decrease in likelihood.

Hmm. Hard to diagnose. Perhaps adding more logging to print out what's happening when they hang would help? For example, they could be being rate-limited by the sites you're looking at and hanging on requests to them. (I suspect that you're already considered that specific case and rejected it, but perhaps logging could unearth something less obvious but of the same kind.)