Forums

Hello. Im trying to run some custom action by shedule, and meet some strange behavior. In my shedule ive got a record: /usr/local/bin/python2.7 /home/xux/<project dir>/manage.py scanner

Code of scanner.py is very simple: class Command(BaseCommand): def handle(self, args, *options): sys.exit('rty')

As i see my script runs well, but when it hangs, and execution time could be infinite. I have tryied different variations of command in shedule (including and excluding absolute paths), try to set enviroment variables, and others..

As soon as script hangs i did not see anything in the log, there is only "Task has not been run yet". Also, i try to raise exception from different places of my scanner.py, and get that this module loads and class initialized, but functon handle hangs..

Could anybody help me to run my custom command by shedule?

I don't see any way that the code you posted could hang. The "Task has not been run yet" message is only present until a task has started not when it's finished. I see 2 scheduled tasks owned by you that are running. One has been running for a about 1 hour 45 min and the other for about 45min. If your tasks take a long time to run (i.e. much longer than the gap between tasks) you could end up in a state where you're not allowed to start any more tasks. Could that be it?

Does manage.py fork a new process or use subprocess or similar to execute scanner.py? Could it be that you're not calling join() on the subprocess to reap its return code?

glenn, as you see, this simple script runs for a long time, but it should do nothing. manage.py is a django admin manager, maybe problem in it? Does anybody runs django custom commands from shedule? Also, if I run the same command from console it works fine, and it takes less then a second to perfom it.

Thanks for answers, I have found some bugs in my manage.py file, now I am using manage.py from default project on PA and everything is OK.

EDIT: Glad you got it working, I hadn't seen your latest post when I posted this.

Ahha. I really should read up on Django one of these days - I'm not likely to use it myself, but it's sufficiently popular that I really should be familiar with it.

This is a total guess but could it be that your Django environment variables aren't fully set up when executing scheduled tasks, as per this SO question? That question talks about cron jobs, but PA's scheduled tasks are conceptually very similar.

@xux -- glad to hear it's all working! Just for our reference, were the bugs you found something that it would be worth sharing in case anyone else is seeing the same issue?

It is strange but bug again reproduced. All I have done previous - is to use manage.py from PA default django project. But today after one of my sheduled script worked more than 5 hours, new sheduled task start to run in a strange way. What I`ve got:

1. I run script by shdule each 10 minutes (6 runs each hour)

2. At the begining of the script i check for a locking of a file.

3. Now, I see when script starts it does not create lock file, may be something wrong in env, but listing of enviroment from script run by sheduler shows that all is ok.

@Cartroo My situation is very similar to those problem on SO, but i checked for env variables, they are OK.

Could somebody answer, what is the difference between running script from console and from shedule? Maybe some enviroment variables, user or path? How does shedule implemented? Is it a cron, or it`s a custom script?

I suspect the scheduled scripts are run via a custom framework rather than plain old cron, but that's something the PA boffins would need to clarify. However, I suspect it doesn't matter all that much as the issues are likely to be similar (although there is a chance that you're hitting a problem which wouldn't happen with cron).

To clarify, your specific problem right now is that your script isn't creating a lock file when run from a schedule, but it is when run from the console - right?

Would you be willing to share the bit of code where you're attempting to take out the lock? There could be a number of issues and it would be an awful lot easier to diagnose with a bit of code.

Not exactly.

The problem as a previously, that i have got a custom django command, which should make some actions(first action is to create lock file). The simplified code of my custom action, placed in <app folder>/management/commands is:

class Command(BaseCommand):
    def handle(self, *args, **options):
        make_some_action()

make_some_action could be as simple as you want, I tried also make_some_action: print 'hello' If I run it from console everything is perfect, make_some_action runs, but from schedule it hangs(i do not actually know at which step), but for the test, i try to put raise exception in each line:

_raise Exception('1')
class Command(BaseCommand):
    _raise Exception('2')
    def handle(self, *args, **options):
        _raise Exception('3')
        make_some_action()

And after running it from schedule I could get exceptions 1 and 2, but if I try to obtain exception 3, this script hangs.

p.s.: sorry for code inserting, I could not understand how to use :::python

[edited by admin to fix code formatting]

An important difference between running a script from a console and from the schedule that may affect the creation of your lock file is that the working directory is likely to be different. Do you specify a full path to the lock file?

I tried to use full path, and even to change current dir to a root directory of my project. But the problem that as I see, my script do not reach that line.

I'll try some experiments tomorrow. It sounds like django needs some bit of the enviironment that we're not providing.

Yes, if handle() isn't even being called, that's a bit of a problem, I guess.

To me the odd thing is why your script was working and then suddenly stopped again - I wonder what changed? Maybe a recent upgrade has changed the environment in some subtle way and introduced an issue?

One aside - according to this page it sounds like there are self.stdout and self.stderr which are supposed to be used instead of the standard ones, I assume so Django can log it properly. Probably best to make sure you use these when adding debugging to track down the problem (though I don't know enough about Django to know how important it is).

I don't think it could have been an upgrade -- we haven't changed anything on the site since March 12.

However, we did discover an out-of-control process on one of the task servers this morning. @xux -- could you check if it's working now?

Yes, Everything is working now! Thanks.

It was caused by me or not? What could I do when it will happen next time?

Thanks for confirming that it works now!

The process that went out of control is a database-related one, so unless your process is doing tons of DB queries then I don't think it's you. Investigations are underway. We'll keep an eye on the server until we know more, but if we miss something and your task stops working again, send us a message using the "Send feedback" link (top right-ish of any page) and we'll get onto it ASAP.