Forums

Server issues - DAMN ! worker 1 died, killed by signal 9

Hey, web server has been working fine for a long time until this morning. Nothing got changed in the last couple of days. Today I see this, over and over again:

2024-11-22 08:41:05 Fri Nov 22 08:41:04 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 1) ***
2024-11-22 08:41:06 Fri Nov 22 08:41:04 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:07 Fri Nov 22 08:41:04 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:08 Fri Nov 22 08:41:04 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:09 Fri Nov 22 08:41:05 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 2) ***
2024-11-22 08:41:10 Fri Nov 22 08:41:05 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:11 Fri Nov 22 08:41:05 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:12 Fri Nov 22 08:41:05 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:13 Fri Nov 22 08:41:06 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 3) ***
2024-11-22 08:41:14 Fri Nov 22 08:41:06 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:15 Fri Nov 22 08:41:06 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:16 Fri Nov 22 08:41:06 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:17 Fri Nov 22 08:41:07 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 4) ***
2024-11-22 08:41:18 Fri Nov 22 08:41:07 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:19 Fri Nov 22 08:41:07 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:20 Fri Nov 22 08:41:07 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:21 Fri Nov 22 08:41:08 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 5) ***
2024-11-22 08:41:22 Fri Nov 22 08:41:08 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:23 Fri Nov 22 08:41:08 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:08 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:09 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 6) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:09 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:09 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:09 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:10 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 7) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:10 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:10 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:10 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:11 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 8) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:11 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:11 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:11 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:12 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 9) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:12 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:12 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:12 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:13 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 10) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:13 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:13 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:13 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:14 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 11) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:14 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:14 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:14 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:15 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 12) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:15 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:15 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:15 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:16 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 13) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:16 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:16 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:16 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:17 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 14) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:17 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:17 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:17 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:18 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 15) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:18 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:18 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:18 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:19 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 16) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:19 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:19 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:19 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:20 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 17) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:20 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:20 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:20 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:21 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 18) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:21 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:21 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:21 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:22 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 19) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:22 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:22 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:22 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:23 2024 - *** HARAKIRI ON WORKER 1 (pid: 38, try: 20) ***
2024-11-22 08:41:24 Fri Nov 22 08:41:23 2024 - HARAKIRI !!! worker 1 status !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:23 2024 - HARAKIRI [core 0] 10.0.5.105 - POST /download since 1732264263
2024-11-22 08:41:24 Fri Nov 22 08:41:23 2024 - HARAKIRI !!! end of worker 1 status !!!
2024-11-22 08:41:24 DAMN ! worker 1 (pid: 38) died, killed by signal 9 :( trying respawn ...
2024-11-22 08:41:24 Respawned uWSGI worker 1 (new pid: 43)
2024-11-22 08:41:24 spawned 2 offload threads for uWSGI worker 1
2024-11-22 08:41:24 Fri Nov 22 08:41:24 2024 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /download_full (ip 10.0.5.105) !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:24 2024 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET /download_full (10.0.5.105)
2024-11-22 08:41:24 Fri Nov 22 08:41:24 2024 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /upload_full (ip 10.0.5.105) !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:24 2024 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during POST /upload_full (10.0.5.105)
2024-11-22 08:41:24 Fri Nov 22 08:41:24 2024 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /download_full (ip 10.0.5.105) !!!
2024-11-22 08:41:24 Fri Nov 22 08:41:24 2024 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET /download_full (10.0.5.105)

Reloading webapp doesn't help. It says: "Your webapp took a long time to reload. It probably reloaded, but we were unable to check it."

When I check access.log, calls that took 0.5s previously started taking 200-300 seconds today:

What do i do?

AscensionTSM | 3 posts | Nov. 22, 2024, 8:50 a.m. | permalink

What are those calls doing? Perhaps you could share your code?

Staff giles | 12640 posts | PythonAnywhere staff | Nov. 22, 2024, 6:10 p.m. | permalink

Well, considering I didn't change anything and it practically stopped working one would expect it's not a "me" issue. And considering the issues went away on their own in a couple of hours after reporting it, I believe we can safely say it was a hiccup on your guys' side ;).

AscensionTSM | 3 posts | Nov. 22, 2024, 7:57 p.m. | permalink

What time did it start working again?

Staff nkahr | 610 posts | PythonAnywhere staff | Nov. 23, 2024, 8:34 a.m. | permalink

Can't pinpoint it exactly since the logs have moved on since then. I noticed something's not right at 8:45 and at 12:31 that it started working fine (CET).

AscensionTSM | 3 posts | Nov. 25, 2024, 8:52 a.m. | permalink

Thanks for reporting this and please let us know if it happens again.

Staff nkahr | 610 posts | PythonAnywhere staff | Nov. 25, 2024, 9:26 a.m. | permalink