Forums

I have been developing a Facebook app with Django on PA. Everything went fine until yesterday. Now I keep getting "error 111, connection refused". From what I read, Facebook is whitelisted. SO WHAT IS GOING ON?

OK OK OK, Django has been upgraded since some imports did not work anymore. See http://stackoverflow.com/questions/8074955/cannot-import-name-patterns

@pynchia: Welcome to our PA community.

I'm glad you already solved your first problem...☺

IT IS NOT WORKING. AGAIN. I HAVE WASTED MY DAY ON SOMETHING WHIC HWAS FULLY WORKING YESTERDAY MORNING. I CAN'T BELIEVE IT. MY app is just a few lines of code!

Is there anything wrong with the imports in the urls.py? IS THIS OK? from django.conf.urls.defaults import *

if I use:
from django.conf.urls.defaults import patterns, url

it does not find the name "patterns" !!

urlpatterns = patterns('', url(r'^$', 'mysite.sputafrasi.views.home', name='home'), url(r'^update_status$', 'mysite.sputafrasi.views.update_status', name="update_status"), url(r'^editpref$', 'mysite.sputafrasi.views.editpref', name="editpref"), url(r'^setpref$', 'mysite.sputafrasi.views.setpref', name="setpref"), )

from django.contrib.staticfiles.urls import staticfiles_urlpatterns urlpatterns += staticfiles_urlpatterns()

Reposting code from above so it can be read:

Is there anything wrong with the imports in the urls.py? IS THIS OK?

 from django.conf.urls.defaults import *

if I use:

from django.conf.urls.defaults import patterns, url

it does not find the name "patterns" !!

urlpatterns = patterns('', url(r'^$', 'mysite.sputafrasi.views.home', name='home'), url(r'^update_status$', 'mysite.sputafrasi.views.update_status', name="update_status"), url(r'^editpref$', 'mysite.sputafrasi.views.editpref', name="editpref"), url(r'^setpref$', 'mysite.sputafrasi.views.setpref', name="setpref"), )

from django.contrib.staticfiles.urls import staticfiles_urlpatterns
urlpatterns += staticfiles_urlpatterns()

thank you. The problem seems to be (just seems to be) here:

@canvas_only
def home(request):
me = request.facebook.graph.get_object('me')

i.e. the call to the facebook library.
It worked yesterday. It has worked again today, briefly.
Now I am back to square one.
Is PA doing anything weird today? Installing new things? Today I had to replace imports so I guess they are tweaking things. I am trying the service and so far I have liked it. But if they change settings like this, removing the earth under the feet of your apps.... then I am having second thoughts about buying anything here.

We definitely haven't changed anything related to this recently -- in fact, our last upgrade was more than a week ago. Let me take a look and see if I can work out what's causing this.

OK, my best guess is that this is actually all being caused by the problem we had earlier with the proxy server. Someone is making ~100 requests/second to a site that's blocking them, which is having knock-on effects on everyone else.

I'm going to track down the offending process and stop it.

Got it. Someone was trying to script an online game that is not on our whitelist, and was making over 500 requests per second to it. The proxy was basically being DoSed. I've killed the offending process and everything seems to be running fine now.

Just so you know, this will not have affected anyone with a paid account -- paid accounts have direct internet access so they don't go through the proxy.

Tomorrow we'll try to find out why we weren't alerted when the proxy server went down, anyway -- I should have got a message on my phone as soon as it started misbehaving.

If it's useful it's quite possible to use iptables to rate-limit incoming connections. I've used this before to prevent large numbers of incoming SSH connections (mainly for the trivial reason that when people try to brute-force account names/passwords it doesn't fill the logs with junk). However, I can't see why you couldn't use the same technique with a higher threshold over a shorter period to protect the proxy server - I'd hope that iptables was a lot more efficient at dealing with large connection rates than Squid.

Essentially you just temporarily blacklist an IP address which makes more than X connections in Y seconds. Of course the values would need to be chosen to not interfere with legitimate use, but something like more than 400 connections in 10 seconds causing the address to be blocked for 60 seconds might work.

EDIT

I just noticed that page uses the state module, whereas I suppose new code these days should use conntrack (though I'm not aware of any plans to deprecate the former). Either way there will be a performance impact if the machine doesn't already use stateful firewall rules, but I'd expect this to be negligible compared to userland stuff. It's been relevant to me in the past on machines tracking millions of concurrent connections (with the aid of a kernel module) but that's something of a special case!

Thanks, Cartroo -- that makes sense. I guess the trick would be in making sure that ten users on a given console server doing something high-bandwidth but legitimate were not regarded as a problem, but one user doing ten times as much was. Perhaps there's something we could do on the console server side instead to track that... hmm.

Hm, that's a very good point - I'd completely failed to consider the IP-address-shariness on PA. D'ohballs.

I suppose in principle you could use MPLS to tag the originating user of traffic and use that for accounting and limiting, but that's getting into fairly deep magic (and relies on there being an external router which could strip off the MPLS framing prior to sending on).

I suppose that it might still be useful to stop really excessive abuse, but the potential for impact on other users is definitely a big downside (although I suppose that flooding it has an impact anyway).

An alternative would be to start using per-user network namespacing and then limiting stuff coming out of a specific user's space there. That, of course, would play well with the IPython notebook/allowing arbitrary HTTP/WebSocket server and so on... I reckon we're probably going to have to do it sooner or later.

IT IS HAPPENING AGAIN.
luckily, last night I tarred the entire tree structure of my site. It was working fine.
So now I HAVE JUST NUKED THE SITE FROM ORBIT AGAIN.
I then recreated the django app.
Then I extracted the entire site form the tar.
AND IT DOES NOT BL**DY WORK.
What am I supposed to do?
Is this sort of thing going to happen to me even if I upgraded to a paid plan?
PLEASE ANSWER THE QUESTION.

It's hard to say whether you'll suffer the same issues on a paid plan because you haven't given any details about the problem you're seeing at present, whether it's one of the ones from earlier in this thread or something new.

If the problem is the proxy server again then this wouldn't affect you on a paid plan, since there is no proxy server for paid users - they have unfiltered internet access. The proxy is only there to protect the service from abuse of free accounts. Giles has already explained all of this earlier in the thread.

Also, while everyone appreciates that these sorts of issues can be frustrating, you're likely to get a solution more promptly if you spent less time seething and more time explaining the exact nature of your problem.

@Giles: On the subject of connection rate limiting, I was wondering if a combination of the rules I linked earlier with the --uid-owner constraint from the owner module might allow per-user connection limits. This would have to be on the outgoing box, though, as --uid-owner is only valid for the OUTPUT chain (for obvious reasons).

@pynchia, Nuking your code is just a waste of time. If your code works and then stops working it's much more likely to be a problem with us. In this case, it was another user that hammered our proxy server until it fell over. I'm going to add some monitoring so we get alerted if it happens again.

I have restarted the proxy and cleaned it up so it's working now. If your code works now, then it was the proxy that was causing the trouble and upgrading will ensure that you are not affected by proxy server outages.

@Glenn: thank you, it now works like a charm again.
The problem is "error 111, connection refused", as described at the very beginning of this thread.
Thanks again for allowing me to keep testing my app.

AHEM.... with the very same site (I have overwritten it yet again, without nuking it)
I now get:
JSONDecodeError at / Expecting value: line 1 column 1 (char 0) Request Method: POST Request URL: http://pynchia.pythonanywhere.com/ Django Version: 1.3.5 Exception Type: JSONDecodeError Exception Value:
Expecting value: line 1 column 1 (char 0) Exception Location: /usr/local/lib/python2.7/site-packages/simplejson/decoder.py in raw_decode, line 393 Python Executable: /usr/local/bin/uwsgi Python Version: 2.7.3 Python Path:
['/var/www', ...

Looks like you're trying to parse something that's not JSON or invalid JSON. What is it that you're trying to parse?

And I still don't understand what your obsession with nuking/overwriting is.

I am not trying to parse anything. The very same site I just restored 30mins ago is not working anymore.
It could be Facebook is just now giving me silly responses authenticating the app, but I doubt it.

So I can see from the requests going through the proxy that there were a few requests around 14:00 UTC and then another few at around 14:45 UTC. I don't know the Facebook API very well, but it could be that you're creating an access token and then it's expiring after not being used for a while. You can test this by reloading your webapp and trying again. If it starts working then it's probably an expiring token and you need to work out how to get a new one.

I've been having issues with this as well. I've discovered that there may be an issue with the django-facebook api. It makes a call to facebook without validating whether the data present in the cookie is correct. After that facebook simply returns an empty json I believe refusing multiple connections in such a short span of time?

If you could clear this issue up, it would be really helpful.

That sounds like an issue to raise with the django-facebook project.