Forums

Hello, Do you have any plans to make the ipython notebook interface available to pythonanywhere users?

Hey Nicholas,

it's on our list, but it's not a straightforward fit with our infrastructure, so it's not easy. I'd say it'll be a couple of months before we can look at it...

I'll +1 the ticket on your behalf though!

Could you +1 it again! I hear local installation can be a bear.

Sure! I've done that.

In all fairness to the IPython Notebook dev team, though, we've never had problems installing it on our own machines while testing. That said, we're on Debian/Ubuntu, and it's perfectly possible that it's less fun to get working on other OSes.

On other OSs, we generally recommend downloading EPD Free, which has all the bits needed to run the notebook, along with useful modules. It's a big download, but it saves installing several pieces individually.

Been reading good comments regarding ipython notebook.

Would be a great addition to PA if it was available, any chance?

Cheers!

We're definitely sensing the demand from you all :-) But yeah, it is going to require quite a lot of wrangling to fit it onto our infrastructure. Nothing impossible but certainly time consuming.

I run it on Win 7 and had no issues getting it going w/o EPD. Not saying you shouldn't go with the recommendation, just saying I didn't have any issues. And pip upgrade from .12 to .12.1 and .13 went without any issues.

Yup, we've been playing with it quite a bit here, it installs like a dream and works really well. It's a bit harder to work out how we can get it working on a system with large numbers of users, though. It tends to grab system-wide resources like port numbers...

I can't wait until IPv6 is widely implemented and we can do things like giving every instance it's own public IP. Then port use won't be a problem...:)

Very true... might be a while, though!

Yes, and sadly now that we've gone through the IPv4 shortage, ISP's have gotten used to charging for IP's...I'm not so sure it'll be easy to get them to let go of that revenue stream!!

It'll be interesting to see. They might find it a bit harder to get away with... or at least we can hope so.

The good news is that it isn't much revenue, so not much reason for them to fight.

i would love to have the ability of ipython notebook within pythonanywhere! Please :-)

I've upvoted the ticket for you.

The blog post linked from a recent re-tweet from the PA account seems to imply that iPython notebooks are now supported. Is this something I've managed to miss?

Nope. I think it was just an over-excited blogger.

Ah well, I thought it was worth asking. As an aside, having had the briefest of peeks at the code, I can see why it's quite a pain to implement - it's not really set up for shared hosting, is it? Best I could figure was to have a single dedicated pre-configured instance per user on its own port somewhere, and set up nginx as a reverse-proxy under some portion of their URL space. It's such a pain to have to use a dedicated port, however.

I suppose another approach would be some sort of custom hosting which just uses the portions of iPython which deal with parsing notebook files and converting them to HTML, but it's starting to become a maintenance headache to keep up with new releases at that point.

EDIT: In case it contains anything useful, I did find this post a little while ago which mentions some of the quirks of iPython's notebook mode.

EDIT: Also, I've only just become aware of this IPEP which looks like it might make life an awful lot easier for hosting notebooks. Perhaps it's worth having a peek through and submitting comments at the design stage which might make life easier to host the stuff on PAW?

IPEP3 is interesting. If that's the route they end up going down then our implementation will be a breeze.

+1 on this from me as well. Having such a system available would be great, especially for workshops.

+2 just for you ebressert!

I had ignored ipython notebook somewhat as most demos I saw did not make full use of it, but I saw a talk last night by Thomas Wiecki of Quantopian and I'd really like to use it in my own classes. Can we stick another +1 on this? Or maybe some kind of integration between a local notebook and PA?

Yes of course. Though knowing that there is the IPEP about supporting multi users in the works means that I think we should wait for that to be finished before we start work on it. Once that IPEP is completed it would turn a very difficult (and very hacky) piece of work into something both possible and elegant.

If it plots right into the browser, here's another plus one.

Cool. I've recorded that.

Yup, definitely of interest here too. I'd like to give Internet classes with IPython Notebook from Python Anywhere!

Unfortunately I don't think we should hold our breath - I can't find any more discussion of IPEP 3 on the mailing list after the original thread.

Yup, it definitely looks like we can't rely on IPEP3 to save our bacon on this one. @holdenweb, I've added an upvote for this one on your behalf.

Technically, getting it done looks doable but hard. IPython notebooks need to listen on a particular port -- they're not WSGI-based. So we'd need to provide network namespaces so that each user could run their own socket-based servers (which we're sure is doable but haven't investigated in depth yet) and then set up our front-end proxy so that it could route through to the socket servers based on the hostname (which we have achieved in a testing environment). Usefully, getting both of those done would provide support for non-WSGI web apps, which is also a very popular feature request, so that pushes the priority up -- we're always keen on two-fers!

The only other approach, I suppose, would be to take the existing notebook stuff as a base and wrap some WSGI layer on top of it. You've still got the issue of running iPython kernels persistently, but I suppose that's not much harder than running uWSGI instances - not sure if you could spin then up and down dynamically in the same way.

Of course, this might well end up being more work and is less generally applicable so it's not necessarily feasible - I just wonder if it's something that might be partially supported by the iPython community.

Giles pointed me here and figured I'd just +1 for the record. Totally understand the challenges, but it definitely would be most helpful with the kids @CoderDojoDC. Would also be helpful to get a unique URL to share a page out of notebook, so the kids can easily share their work - something like https://thimble.webmaker.org/p/f1mq/edit, http://jsfiddle.net/ or http://jsbin.com/welcome/1/edit

Already though this is a fantastic service to the community and is much appreciated!

That sounds like an excellent use for IPython notebooks. I've upvoted the ticket for you.

I've already started using ip notebooks. The advantages kick in instantly while presenting scientific content and coding. Google understands filetype:ipynb and displays loads of use cases. Both wakario and picloud offer online notebooks, the latter in beta with autoshutdown, the former with 20 free hours wall clock time. There is also something at appspot where you enter your amazon machine credentials and get billed accordingly.

Since NBs have shell integration files = !ls they cannibalize bash as admin tool. You prefer R, ruby, Javascript? Just write it. Actually I think QA sites like stackoverflow should relaunch with NBs, so you could evaluate any answer right below the question.

What looks like a major roadblock: Every user needs his own virtual machine plus file system. Hopefully someone can abstract that at reasonable cost, sounds like a game changer. Can't imagine everybody and his brother with his own vm in the Internet.

Good article about Ipython note book here: http://about.wise.io/2013/03/23/fernando-perez.html

Thanks, that's really useful! We think that our own Python execution sandboxes can work as minimum viable "VM"s for the purposes of IPython notebooks, so we'll have to put some more time into this.

BTW on the subject of executable answers to Stack Overflow questions, we just deployed something which we're doing some initial live testing on, and hope to be able to announce it officially shortly...

Giles, you are such a tease. I certainly look forward to the announcement!

Plus one for ipython notebooks :)

@giles: I think I get the GIST of what you were referring to...☺

It might help with port allocation, if you only allow a specific amount of notebooks for each user. I don't think there is a use case for more than ten open NBs at any time.

Do you mean individual notebooks (i.e. documents) or notebook servers? The notebook server is what takes up the port, and I don't think there's a use case for more than one of those per user since (as far as I'm aware) you can have as many open notebooks concurrently as you like in a single server. I'd assume that there would be simple start/stop options for the user's single notebook server, and everyone would be allocated a random port for their server to use while it was running.

It's also conceivable to have a bunch of VMs with private addresses from the RFC-1918 range fronted by an nginx instance, or possibly a WSGI application if vanilla nginx isn't up to it, which reverse-proxies subsets of the URL space to the running notebook instances. So, users could specify a URL path to represent their iPython notebook server which would be inserted into the auto-generated WSGI application that PA already creates (or competent users could always edit it in by hand, like they currently can). This was what I was trying to suggest above, although I don't think I was particularly clear. This reduces the pressure on port allocations because there are a large number of available addresses, but is rather trickier to implement. Also, I'm not sure if the VMs PA uses are able to use RFC-1918 addresses.

Sure, multiple NB servers per user is overkill, I meant active notebooks, each needing its own port. On the other hand it would be very comfortable, to access NBs via URLs, so server start and stop happens in the background, although that implies to rewrite the servers dash.

Just a little update - I found a little time today to try and set up my own notebook server behind an nginx proxy and it's actually a little trickier than I first suspected... The main issue was that I had ipython running on localhost and that meant it was convinced that it should connect to the web socket running on localhost, which didn't work too well. You can configure it to place things on a portion of the URL, but it's clearly not expecting to be running behind something which changes its externally-facing hostname.

Tricky! ('_')

Someone should make an IPython Notebook containing instructions on how to enable/use it on PythonAnywhere! Oh, wait...since they aren't working yet none of us could use the instructions...☺

Sorry, I'm in a silly mood this Friday afternoon...☺

Another update - I finally got ipython notebooks proxied through nginx working. Woot!

There were multiple issues - turns out that you need a relatively recent nginx version to support websocket proxying (1.3.13 or later, I think - I'm using 1.4.1 which I believe is the latest stable). Also, you need to get the configuration right, although that's pretty simple (didn't stop me screwing it up the first time, though - messing with stuff before my morning cup of tea is never a good idea). This page has the info for proxying websockets in general.

The bulk of the issues was sorting out the ipython notebook configuration required to have it listening on an arbitrary port on localhost, yet have it send all the right URLs to the browser remapped to go through the primary domain name. Once I got that working I decided to make more pain for myself by making the front-end interface HTTPS but ipython notebook only HTTP.

Still, I eventually got it working, and in the end I only had to patch a couple of lines in ipython to add a new configurable - it assumes that the web socket scheme (i.e. ws or wss) should always match the interface on which the notebook server itself is listening (i.e. http or https). In my case, however, I want the notebook server listening via plain HTTP on an arbitrary port on localhost but the browser is connecting via TLS to nginx, which then makes a reverse-proxy connection over plain HTTP to localhost. So, I needed a configurable to override the websocket protocol to always use wss.

If it's at all useful to anybody (PA devs or otherwise) I'll be happy to share both my ipython notebook and nginx configurations, although be aware that the HTTPS websocket fix won't be in mainline unless they accept the pull request I need to get around to submitting with my teensy little change.

I must admit I did underestimate the complexity of it and I hadn't realised websockets spoke quite such a different protocol to standard HTTP. Still, it's been an education!

EDIT

For anybody who's interested, here's my pull request. It also fixes a few issues which probably only apply to tip, but it's a pretty small change so it should be pretty easy to back-port to one of the stable releases.

@Cartroo - uber cool! I'm sure the devs will be in contact to find out more details

@Cartroo -- wow, thanks! That's fantastically useful. I [did some experiments] with reverse-proxying websockets](http://www.gilesthomas.com/?p=559) a while back, so we had that bit covered, but I'm glad you've done the experiments necessary for IPython notebooks specifically -- I don't think we're realised there were pitfalls there.

I think that for this to work, we have four steps we need to do for PythonAnywhere:

• Implement support for long-running server tasks. This would be generally useful to support (eg.) Twitter stream readers or other things people want running all the time.
• Implement network namespaces so that people can bind to a port without interfering with other people. This requires a more recent kernel than we currently have, and we're working on that right now -- we're switching the execution servers over to Ubuntu Raring Ringtail (alongside PostgreSQL work, which is almost ready for a very limited alpha release [I think I know who we're going to invite for that])
• Add the reverse proxying for people's webapps. This will mean we can support Tornado and other non-WSGI servers, and is basically a solved problem.
• Do the IPython notebook-specific work, where I'm sure your research will be invaluable!

Well, at least it's served as a proof-of-concept. Really the pitfalls are probably fairly clear to someone quite familiar with how ipython notebooks work under the hood, but since I was coming at it fresh I had to poke around a bit to find which settings I needed to change. Still, at least I know my way around the ipython code slightly now!

I guess I was expecting support for long-running tasks would be a generalisation of how the uWSGI instances are spun up and down on demand, but I guess less of the pre-existing machinery to do it exists than did for uWSGI which probably makes it trickier. The ports issue is always going to be a slightly fiddly one, I'd imagine, but I have great confidence in the PA devs. (^_^)

With regards to my changes, one of the ipython devs came up with a client-side approach they preferred which achieves the same end result. Hopefully it won't be too long before it makes it into a stable release.

EDIT

On a related note, writing an init script for it was also a bit fiddly because it's a bit fussy about permissions (it appears to refuse to run unless it has write permission to the profiles directory, for example) and also because it interacts with stdout and stdin and there don't appear to be any options for stopping it (at least that worked for me). In the end some hacky shell script with a combination of sudo and nohup seemed to work, but it's a bit of a shame they consider a public server from the start and write it to optionally fork into the background and detach from the controlling terminal.

Still, it's open source so I suppose I can't really complain unless I submit a patch! (^_^)

Here is a great article how IPNs simplify a teachers life: http://peak5390.wordpress.com/2013/09/22/how-ipython-notebook-and-github-have-changed-the-way-i-teach-python/

The author linked the notebooks used in classes to teach Python. It starts here: http://nbviewer.ipython.org/urls/raw.github.com/ehmatthes/intro_programming/master/notebooks/syllabus.ipynb

IIRC there is a new version, does it fit better into PA's infrastructure?

They haven't changed the user model yet... though I think we'll be making some changes to the underlying infrastructure of PythonAnywhere soon that will make it easier.

And another plus one for ipython notebooks. This is a tricky/hacky feature - but I reckon it would be worth it for the extra interest it could generate.

iPython is amazing and it is especially nice to see the codebase getting cleaned up...͡° ͜ʖ ͡°

NOTE: Above is my first attempt at a new ☺I hope you like it.

I wish I could do this at PA:

http://www.reddit.com/r/Python/comments/1vrekn/economics_simulation_by_peter_norvig/cevbb59

This would be the icing on the cake in an educational context, for sure. We like how you makes it easy for students to prepare their own dynamic web-pages - for various reasons a good educational target for an introduction to information systems course. But we also love the ipython notebooks because of the fast and excellent visual feedback. Has there been any movement on this issue?

We're working towards it, but it's a slow process, there are quite a lot of other things we need to get done first. We'll keep people updated!

+10 for notebooks. That and pandas are the two tipping points which greatly favor Python over R for scientific work. Notebooks natively support markdown and LaTeX which can be converted to other formats, e.g. html, but also rst.

once the notebooks are up ...

and Julia makes the image,

then IJulia notebooks will be

an easy, but powerful addition.

https://www.pythonanywhere.com/forums/topic/1420/#id_post_10468

carry on with the fine work !!!

IPython Notebooks are getting closer to become main stream:

http://www.nature.com/news/ipython-interactive-demo-7.21492

Technology: http://lambdaops.com/ipythonjupyter-tmpnb-debuts

Hi there, we now have an early beta of the IPython notebooks feature which we're ready to start inviting people to. If you're on this thread, you should find you now have the feature.

We're still working on the UI, but if you go to the "Files" tab, you'll find the option to start new notebooks. You'll also be able to view any existing notebooks you have or upload.

Let us know what you think!

Posting, looking for option in Files tab. Nope, not working yet. :-)

Please? :-)

We'll sort you out greg. Anyone else?

If it could be possible I would also like to test the notebook. Thanks

Sure! That's done.

I am interested to try the beta for IPython notebook. Thank you so much for your work :)

OK! Remember folks, we need your feedback, as lengthy and as detailed as possible please :)

Hi there, the ipython notebook implementation was just the drop needed to fill my cup of reasons to start using python anywhere. may I be invited to test it out as well? In which format would you like the eventual feedback?

That's switched on for you! If you want to send feedback, just drop us a line at support@pythonanywhere.com. Or you can post here in the forums if you prefer, though if you find a security problem we'd definitely prefer email :-)

Hi there, could you please allow me to try the beta for the ipython notebook, too? Thank you!

Sure! That's switched on for you -- just go to the "Files" tab and you can create new notebooks and also view any notebooks that you upload. Feedback gratefully appreciated!

Could you switch it on for me as well please? Thanks!

Sure! That's done.

I'm also a volunteer for it!!! I'm really looking forward for this functionality!

We've just (literally, a few minutes ago) switched it on for all paid accounts :-)

Hi! Can I also try it out? I'm on a free account for now but promise I will send feedback :)

Hi I just switch to the $5/mo account to use the ipython notebook feature and every time I want to open or create one it gives me a "The operation timed out." error. any ideas? I'm on chrome, OSX 10.11

Thanks.

I tried to repro on a macbook 10.11.2 with google chrome but I am able to open a notebook. My current guess would be that maybe a lot of people decided to try out jupyterhub notebooks on the server that you were on and so the notebooks took too long to start, resulting in that error? If you try again now it should work? Let us know otherwise

It works just fine now. thanks!

Run into an issue with export as HTML and PDF, both gave a 500 and HTML said:

nbconvert failed: markdown2html requires mistune: No module named 'mistune'

An empty NB works with both formats.

Thanks for reporting that! Will try and get it fixed for the next release.

The workaround is probably to do a

pip3.4 install --user mistune

Thx, the resulting Zip for PDF is not really needed, the graphics are inlined anyway.

I'm having a similar issue where every time I try to open a new ipython notebook, all I get is a timeout. (Also a Mac, used chrome and safari.) Switched to a paid account primarily to use this... :(

Are you behind a firewall or some other kind of proxy server?

This is heavy incentive to switch to a paid account. Looking forward to trying it out :)

:-)

BTW the print preview feature should work fine as of this morning's system update.