Forums

Hi I created a Python script that scrapes data and the endpoint provided by Pythonanywhere I use to fetch that data and use it in my backend, It was working good, however I was able to fetch only around 50 data before my endpoint gave me error 504 loadbalancer error, I was using access_token, I thought maybe by increasing access_token I could fetch more data So I created 2 or maybe 3 total access tokens (1 initial that I was using initially), however what happened was that the data reduced to only 10. And not only that, If I use one access_token, I won't get data on other access_tokenn, its not like my complete limit is still around 50. Basically my work flow is like -> Scrap data, Save it in sql lite -> Send it to python webpage provided, -> Scrap that webpage for download. However the webpage won't work after a few fetches, what Can I do to better optimize it, any helps?

Sorry, could you clarify which endpoint provided by PythonAnywhere you are using?

alendis.pythonanywhere.com/..../....

alendis.pythonanywhere.com is your site, not the endpoint provided by Pythonanywhere

And that site is having load balancer 504, and 502 issue, how can I solve that. Can I even solve those.

A 504 error means that it's timing out -- that is, your code is taking more than 5 minutes to respond to an incoming request. A 502 error is a more general error -- it can happen if you get lots of incoming requests and your site isn't handling them fast enough. That is because if your code is busy handling requests, new ones get put into a queue to wait for processing, and if the queue fills us, it will start returning a 502.

So my guess is that your code is taking a long time to handle requests, will will lead to a 504 after five minutes if the site's queue is empty, or a 502 almost immediately if the queue is full.

The first thing to do would be to look at your views' code and try to see if there is anything obvious there that would take a very long time to return a response.