Forums

Repeating network issues in always-on tasks

I am experiencing intermittent network issues when deploying services via always-on tasks, particularly the following:

  1. (2003, "Can't connect to MySQL server on '[username].mysql.pythonanywhere-services.com'")
  2. Request timeouts. My services heavily rely on the Telegram Bot API, which typically handles high-volume requests (hundreds per second). Therefore, it's unlikely that the issue stems from an API-side rate limiter. However, I suspect that multiple bots making numerous requests from the same always-on task IP might be contributing to the problem.
  3. Error: socket.gaierror: [Errno -3] Temporary failure in name resolution when trying to connect to the PythonAnywhere MySQL database. This issue, although persistent in the past, seems to have been resolved for now, as it hasn't appeared recently.

These issues tend to occur in periodic intervals, typically lasting 30-60 minutes, affecting all always-on tasks and causing service downtime. Interestingly, launching the apps from regular consoles resolves the problem, as these issues are isolated to the always-on tasks. The most recent outage began about 20-30 minutes ago, starting with timeouts and then followed by MySQL error 2003, prompting me to switch the apps to regular console deployments to maintain service continuity.

We have added additional capacity to the always on cluster to reduce the incidence of this in future.