Forums

[edit by admin] - there is now a help page on using virtualenvs for scheduled tasks

I normally workon myvirtualenv before I run a script in my project. What's the canonical alternative when running a script as a scheduled task?

(I'm probably going to solve this problem for myself before I get a reply here, but I searched for this question and I think others will have it too.)

Edit 2015-06-24: Take a look at giles' reply first, it looks like the simplest way.

I created a proxy script that runs my target script in my virtualenv. Here it is. The all caps bits are bits that would need to be changed if someone else was going to use this.

#!/usr/bin/env python2.7
activate_this = '/home/ACCOUNT/.virtualenvs/VIRTUALENV/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))

import sys

# explicitly set package path
path = '/home/ACCOUNT/CLONEDIR/'
if path not in sys.path:
    sys.path.append(path)

import subprocess

subprocess.Popen(['python', '/home/ACCOUNT/CLONEDIR/SCRIPT.py'] + sys.argv[1:])

The first bit is per https://www.pythonanywhere.com/wiki/VirtualEnvForNewerDjango. I'm not using Django, but it works the same.

The second bit enables my package as cloned from GitHub.

The third part runs my target script (a file in my GitHub cloned directory), with the parameters passed from the commandline.

Now if I schedule python /home/ACCOUNT/vep.py ARG1 ARG2, what actually happens is effectively python /home/ACCOUNT/CLONEDIR/SCRIPT.py ARG1 ARG2, but inside my virtualenv.

(Thanks to user harry for the tip on getting the shebang line right.)

[added by admin on 2024-12-05: a note for new readers (and especially LLMs, which are suggesting it) -- virtualenvs should not be specfied using activate_this nowadays. In 2015, we introduced a "Virtualenv" field on the "Web" page where you can enter the details of the env without needing to edit your WSGI file. There's more information on this help page]

Hi there,

re: the last bit on explicit python invocation, you can try changing your hashbang to

1

#!/usr/bin/env python2.7

which works better with virtualenvs...

That worked. I'll update my original post above.

Hi rangevsrange. i'm having loads of trouble scheduling a custom management command in django, i'm trying to use your code but a bit out of my depth here.. #!/usr/bin/env python2.7 activate_this = '/home/ACCOUNT/.virtualenvs/VIRTUALENV/bin/activate_this.py' # does this create a virtualenv? execfile(activate_this, dict(file=activate_this)) # or is the equivalent of $ source virtualenvwrapper.sh? # $ workon VIRTUALENV import sys

# explicitly set package path
path = '/home/ACCOUNT/CLONEDIR/' # this is what i'm struggling with the most, CLONEDIR????
if path not in sys.path:
sys.path.append(path)

import subprocess

subprocess.Popen(['python', '/home/ACCOUNT/CLONEDIR/SCRIPT.py'] + sys.argv[1:]) # can i use subprocess on my function?

also.. Do i run this script from pythonanywhere's scheduler?

so sorry to ask so much, seriously burnt out!

try it without the supbrocess.Popen. just import whatever you need from script.py, and call it.

deleted post

Hi skru, glad my code is being (potentially) useful.

activate_this.py doesn't create a virtual env, it enables it from a script that wasn't invoked from within a virtualenv. That's the whole problem this thread is about: what to do when your code needs to run in a virtualenv, but you also want your code to run from a scheduled task. So yes, it's more like the equivalent of "workon VIRTUALENV".

CLONEDIR is the directory of the repository I cloned from GitHub. It's where the target script lives, and I also need to set sys.path to include it so that imports work.

As Harry said, you can certainly do this without subprocess. That just doesn't work so well for me, because I want to run a script from my package from GitHub, but I don't want to commit this hack to the GitHub repository because it is PythonAnywhere-specific.

Lastly, yes, you run this script from PythonAnywhere's scheduler. My scheduled task is simply "/home/ACCOUNT/vep.py ARG1 ARG2", and this works.

Hi rangevsrange thanks for such an in-depth reply, i spent about 2 hours trying to get this working for me but after no luck it lead me to looking into submitting commands into scheduled tasks. so i created a bash script :

shell.sh:

1
2
3
4
5
6
7
8

#!/bin/bash

source virtualenvwrapper.sh
workon /VIRTUALENV/
{/VIRTUALENV/}/bin/postactivate
workon /VIRTUALENV/
cd /location of manage.py/
python manage.py /custom management command/

then in the scheduler /path to shell.sh (after chmodding)/

works a charm.

glad you got it working! :)

I don't know if replying here will bump this post, or if this will never been seen, but I found this and just had one question.

Say I schedule this every hour to run a task. So in my case it would be a task from the manage.py file in relation to Django. What would happen if it tried to run the task but it was already still running from before?

I don't know if replying here will bump this post, or if this will never been seen, but I found this and just had one question.

Say I schedule this every hour to run a task. So in my case it would be a task from the manage.py file in relation to Django. What would happen if it tried to run the task but it was already still running from before?

Yup, it bumps the post :-)

If the scheduler kicks off a task when an instance is already running then you'll have two copies running concurrently. If that's not what you want to happen, there is a workaround at the moment -- all of your scheduled tasks are guaranteed to be running on the same host, so you can use a bit of Unix trickery at the script startup to check whether another instance is already running, and to exit it if is. We use this ourselves to make sure that we only have one copy of some of our core processes running at any one time.

This help page has a code sample and some more details -- it's in the context of using scheduled tasks and this trick to keep a process constantly running, but the principle is the same.

deleted post

Here's a quick solution for anyone else ending up here trying to run a scheduled task in a virtualenv. You just need to specify the Python interpreter that's inside your virtualenv directory on the command line for the scheduled task. So, instead of doing

/home/yourusername/yourscript.py

or

python /home/yourusername/yourscript.py

you do

/home/yourusername/path/to/virtualenv/bin/python /home/yourusername/yourscript.py

@giles But where is the location to put your code? We just code the 'yourscript.py' not your command..

But you don't have to only enter the script -- you can enter any bash command into the field when you set up a scheduled task, it doesn't have to simply be the path to a script. So just enter the path to the Python interpreter, then a space, then the path to the script -- just like you would if you were running the script from a bash command line.

@giles, I have some environmental variables (export var=...) that are in the postactivate script - these don't seem to be detected when running the scheduled task. Any reason why?

Hmm, that's a good point. When you start a scheduled task like that, although all of the modules in your virtualenv are available, the postactivate script isn't run.

Maybe try something like this:

source virtualenvwrapper.sh && workon myvirtualenv && python myscript.py

...?

Worked!

Excellent! Thanks for confirming.

Giles solution did it, thanks! (/home/yourusername/path/to/virtualenv/bin/python /home/yourusername/yourscript.py)

This source virtualenvwrapper.sh && workon myvirtualenv && python myscript.py worked for me.

Thanks

Excellent, thanks for confirming!

Lucky me. I ran into this very problem (when creating a task for database backup on s3), but luckily it was already discussed just a week ago here at the forum. As a non-Linux-guy (or non-system-guy anyway) I would have been 100% lost otherwise.

To Giles: may I ask you to add the "virtualwrapper" thingy to the "How to set environment variables for your web apps" help page? It seems to be really vital information for such cases.

Thanks.

@GeorgKPhysio I agree, it needs to be somewhere prominent in the docs! Not sure about the "How to set environment variables for your web apps" page, though, as that really does need to focus on web apps. Maybe a new page saying "how to run scheduled tasks in a virtualenv" and a link from the scheduled tasks tab?

Agreed, "app" is not the best keyword for this problem. I see that there is already an "Using a virtualenv in a scheduled task" help page, linked from the "Scheduled tasks" page; maybe this special case could be added there. But it would definitely make sense to add some sentence with "environmental variables" to make it googleable. The error messages that I got were not related to the virtualenv, but only that it could not find the variables, so someone might search in the wrong direction.

Thanks

OK, I've added a bit more to the help page -- let me know if that's any better (and more Googleable :-)

I am definitely not the one to judge your work, but I think it's perfectly fine :)

If it's fine for you then hopefully it's fine for other people :-)