Forums

Hi there,

After making the user pass a test on our Flask web app with run some analysis on data collected. This analysis takes a few minutes and is executed by sending requests to an API. To prevent server freezing during this lengthy process, we start a subprocess, redirect the user to a loading screen, and have the frontend check whether the analysis is complete.

We observed a surprising behavior: our logs indicate that several analyses are initiated simultaneously after starting the subprocess. These multiple instances of our code run to completion, resulting in the results screen displaying the same analysis multiple times and generating multiple PDFs (from another API).

We suspect this might be due to a worker issue, with multiple workers executing the subprocess task, but we are not certain.

We attempted a flagging method, marking when an analysis has started, but this was unsuccessful. This leads us to believe that the subprocess might directly cause 3 or 4 workers to undertake the analysis.

We also tried using threads, but encountered the same issue.

Here are some examples from our logs:

2024-02-07 16:25:12,772 - INFO - Found the file!
2024-02-07 16:25:12,830 - INFO - Loading screen exited
2024-02-07 16:25:12,830 - INFO - checking if analysis file exists
2024-02-07 16:25:12,832 - INFO - text processing
2024-02-07 16:25:12,833 - INFO - reading url
2024-02-07 16:25:12,837 - INFO - Loading screen exited
2024-02-07 16:25:12,840 - INFO - checking if analysis file exists
2024-02-07 16:25:12,841 - INFO - text processing
2024-02-07 16:25:12,844 - INFO - checked if file existed
2024-02-07 16:25:12,845 - INFO - Found the file!
2024-02-07 16:25:12,848 - INFO - checked if file existed
2024-02-07 16:25:12,848 - INFO - Found the file!
2024-02-07 16:25:12,850 - INFO - reading url
2024-02-07 16:25:12,878 - INFO - checked if file existed
2024-02-07 16:25:12,878 - INFO - Found the file!
2024-02-07 16:25:12,910 - INFO - Loading screen exited

Here is how we star the analysis (with the flag system):

if is_analysis_running(first_name, last_name):
    logger.info(f"Analysis already running for {first_name} {last_name}")
else:
    logger.info(f"Starting to run the analysis for {first_name} {last_name}")

    create_lock_file(first_name, last_name)

    # Start the analysis subprocess
    analysis_thread = subprocess.Popen(['python', '/home/FutureAi/mysite/analysis.py', first_name, last_name])

logger.info("Now showing the loading screen")

Would someone know if this is indeed a worker problem, and how to prevent multiple workers from taking on the task?

We have a help page on how to do that sort of thing in web apps: https://help.pythonanywhere.com/pages/AsyncInWebApps/