Forums

I just upgraded my system image (to haggis) and I created a new virtualenv per the instructions but whenever I try to access it on the webapps page ("Start a console in this virtualenv") I get the error listed in my title. If I open a bash console and use the "workon <env_name>" command, it activates just fine. I feel like I've done all the things to get this working. I cleared the cache before making the virtualenv, I deleted the old virtualenv and started again many many times now. I created the virtualenv using mkvirtualenv and virtualenv commands. Nothing is working and my website is down until I can get this sorted. Please help.

Ok...odd. I was reading on this page https://help.pythonanywhere.com/pages/DebuggingImportError about "Debugging sys.path issues in web apps" and using those instructions, I was able to determine that I needed to install mysql-connector-python. My website it now working again but the link from the webapp page to my virtualenv still says "python: No such file or directory". It not the end of the world as I can still get to it from a bash console. It's just weird.

That does sound very strange! Could you try running this from a freshly-created Bash console?

/bin/launch_bash_in_virtualenv.py lkasdfjlsadkjal <path-to-your-virtualenv> /home/drewxcom

The <path-to-your-virtualenv> should be replaced with the full path to your env, eg. /home/drewxcom/.virtualenvs/envname, but the rest should be as-is, including the keyboard-mashed first parameter (which just needs to be something unique).

Well, that worked. It opened my virtual environment in the bash shell. What would be the next thing to try for the link on the webapp page? my "/home/drewxcom/.virtualenvs/<my env name>" still takes me to the same error.

Hmm, very odd! What do you see if you run

/usr/bin/env python

...in a fresh Bash console?

Thought I responded to this earlier, sorry.

17:45 ~ $ /usr/bin/env python Python 3.10.5 (main, Jul 22 2022, 17:09:35) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>>

My site has been a mess since this update. Constant db errors which I did not have before and none of my scheduled tasks will work anymore. I have been rebuilding the entire site locally and am thinking of wiping everything and starting over. How would I go about getting my mysql db updated to the latest version you offer?

I am also not able to issue commands in a mysql console: mysql> SELECT User, Host FROM mysql.user; ERROR 1142 (42000): SELECT command denied to user 'drewxcom'@'10.0.0.43' for table 'user'

Is this related to my other issues? I have no idea whats gone wrong :(

For database update contact us on support@pythonanywhere.com.

For problems with your website, could we take a look at your files? We can do it but we ask first.

That would be great, and thanks for asking. My apologies in advanced. I created the site five years ago and it's a bit of an amateur mess.

I see your site seems to work now -- were yo able to fix database connections? As for the issue with mysql console -- I think you shouldn't use mysql.user as the table name -- if you have table user, use select ... from user;. You may also run describe user; to make sure you're selecting right columns.

The site is sort of working but randomly throws up "Something went wrong" screens. My error log has a bunch of errors like this:

"2024-01-05 19:27:10,169:   File "/home/drewxcom/.virtualenvs/rhenv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 655, in _invalid_transaction
2024-01-05 19:27:10,169:     raise exc.PendingRollbackError("

Most of the time a refresh of the page will get past the error page but some users are reporting that they are not able to fill out one of our application forms because the site is erroring out.

None of my schedule tasks will run any more either so my database isn't getting backed up and the content on my home page wont update.

The link to my virtual environment console on my "webapps" page still gives me the

/usr/bin/env: ‘python’: No such file or directory
Console closed.

error.

I'm just really confused by the randomness of the errors on the site. There doesn't seem to be any rhyme or reason to it. I can't get it to trigger when I click through a bunch of pages but if I've been away for a while, I see it when I come back. This has been really frustrating.

Running commands on my mysql console seem to be working now which is a relief.

The saga continues but i have more information to share.
There are several issues going on, I'll break them down.

(1) "Something went wrong error page"
I can now replicate this error. If I am on the site and am inactive for a while (5-10 minutes), then the next page load gives me the error page and I get a wall of output in the error log. The most interesting part of the log for me is these lines:

File "/home/drewxcom/.virtualenvs/rhenv2/lib/python3.10/site-packages/mysql/connector/connection_cext.py", line 706, in cursor
        raise OperationalError("MySQL Connection not available.")
sqlalchemy.exc.OperationalError: (mysql.connector.errors.OperationalError) MySQL Connection not available.

I've tried lower the recycle time and added SQLALCHEMY_POOL_PRE_PING = True but it doesn't seem to be helping. Again, this was not an issue before the upgrade to haggis.

As a troubleshooting step I opened a separate account with you guys, installed an upgraded version of my site (current packages, changed to app factory pattern and reworked my imports) but I am still experiencing the same issues on both sites. If I can just get this part fixed I could just use my new account (user: hyraxmax) instead but I'm not making any progress after days and days of work. Any help with this would be great.

(2) I seem to have lost privileges to run a dump of my database. When I try to run a dump command from the bash shell I get errors.

(rhenv2) 11:19 ~ $ mysqldump -u drewxcom --set-gtid-purged=OFF -h drewxcom.mysql.pythonanywhere-services.com 'drewxcom$rh'  > db-backup.sql                                                                     
mysqldump: Error: 'Access denied; you need (at least one of) the PROCESS privilege(s) for this operation' when trying to dump tablespaces
mysqldump: Couldn't execute 'SELECT COLUMN_NAME, JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"') FROM information_schema.COLUMN_STATISTICS WHERE SCHEMA_NAME = 'drewxcom$rh' AND TABLE_NAME = 'announcement';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109).

I tried this with my virtual environment active and also with it deactivated. When I try to grant privileges I get another error:

mysql> GRANT ALL PRIVILEGES on rh.* TO 'drewxcom'@'%' WITH GRANT OPTION;
ERROR 1044 (42000): Access denied for user 'drewxcom'@'%' to database 'rh'
mysql> use rh;
ERROR 1044 (42000): Access denied for user 'drewxcom'@'%' to database 'rh'

I don't know whats going on with this but I don't have the same issues with my second account. Again, if I could just get a current dump of the database I could update my second account site and use that but I'm stuck.

(3) Scheduled tasks won't run. I've tried all the things I can think to do here, and nothing is working. I am not having the same issues on my second account. Everything there works as expected. This isn't a huge deal if I can get the other two issues sorted, then I'll just move over to my second account.

[edited by admin]

Thanks for reporting!

Regarding the usr/bin/env: ‘python’: No such file or directory error, we'll need to do some more investigating. (EDIT: see below)

Ad (1) How are you managing the db connection? Is it being created and closed each time your code is doing a transaction, or do you rather create one instance of the connection when the app loads?

Ad (2) You don't have access to granting privileges on MySQL servers, as it's a shared infrastructure. In this case, however, adding --column-statistics=0 option to the mysqldump command should fix it.

Ad (3) How are scheduled tasks failing? What do you see in the logs?

EDITED: Regarding /usr/bin/env: ‘python’: No such file or directory error: is it possible that you added a custom .bash_profile file or changed/removed the default .profile file? If so, you need either to restore .profile with this line:

source ~/.bashrc

or add this line to your custom .bash_profile/.profile.

Brilliant! I'm finally making some progress (with your valued help).
Regarding the usr/bin/env: ‘python’: No such file or directory error....I did modify my bash_profile at some point.
Adding the source ~/.bashrc fixed the issue and now the link to the venv from the webapp page works again.
Coming back to (1)
(2) Adding the --column-statistics=0 option did the trick. Backup tasks are working again.
(3) I was getting 127 errors in my task logs but they seem to be working now (bashrc solution maybe?). I did find one of my other tasks was not working because I had changed the name of my virtual environment (a few times) and it was set wrong. All three of my task are now running correctly again.

Back to (1) In my currently version of the flask app, I create the SQLAlchemy db object in my flask_app/init.py file and then import that object in other files as needed. I'm not calling close on that object anywhere in my code. I've also tried using the app factory pattern to create the db object in a create_app function but that seems to be having the same issues so I'm guessing that's unrelated. The thing with this issue is that none of my code has changed as far as the database creation and calls. This site was working fine before the upgrade.

Thanks for your ongoing help with this. I created this site years ago and almost never touched it since. I was dreading this update but people are relying on me to get this working again so I'll keep at it until we can get this sorted.

Holy heck, I think I've got it figured out.
I switched: app.config["SQLALCHEMY_POOL_RECYCLE"] = 280
over to: app.config['SQLALCHEMY_ENGINE_OPTIONS'] = {'pool_recycle' : 280}
and my site has been running correctly ever since.

I found this on SQLAlchemy's website:
SQLALCHEMY_POOL_RECYCLE
Number of seconds after which a connection is automatically recycled. This is required for MySQL, which removes connections after 8 hours idle by default. Note that Flask-SQLAlchemy automatically sets this to 2 hours if MySQL is used. Some backends may use a different default timeout value. For more information about timeouts see Timeouts. Deprecated as of v2.4 and will be removed in v3.0.
And also:
Changed in version 2.4: * SQLALCHEMY_ENGINE_OPTIONS configuration key was added. * Deprecated keys SQLALCHEMY_NATIVE_UNICODE
SQLALCHEMY_POOL_SIZE
SQLALCHEMY_POOL_TIMEOUT
SQLALCHEMY_POOL_RECYCLE
SQLALCHEMY_MAX_OVERFLOW

So, what happened was that when I created my new virtual environment, I grabbed that latest version of SQLAchemy which is now 3.1.1 and that was where the issues began.

I really hope this helps others facing the same issue.

Excellent, glad to hear you could solve the issues, and thanks for letting us know!