Forums

Hello,

I am trying to integrate haystack (using the xapian backend) on my web application. I have been able to get this work on my local version of the app but there seems to be an issue when moving my code to PA. I am currently running Django 1.6 in a virtualenv and I installed xapian core and python bindings from source. When using a console with my virtualenv activated, I can import xapian without issue by using the command:

python -c "import xapian"

However, when attempting to access my site, I get an "Unhandled Exception" error. Opening my site's error log, I see the following error:

File "/home/Beison/.virtualenvs/django16/lib/python2.7/site-packages/xapian_backend.py", line 26, in <module> raise MissingDependency("The 'xapian' backend requires the installation of 'Xapian'. " 2014-07-19 20:50:07,585 :haystack.exceptions.MissingDependency: The 'xapian' backend requires the installation of 'Xapian'. Please refer to the documentation.

Has anyone else dealt with this issue before? I'm wondering if this has something to do with using xapian within the wsgi environment (which is not replicated in my own setup).

Any help that someone can provide would be appreciated!

Hmm, not sure what it might be, but googling the error message turned up this page -- perhaps there were some subtly different commands to build it there than the ones you used?

Unfortunately that doesn't seem to have fixed the issue. Is there any chance that the xapian bindings can be installed in the system-wide python library location to see if the library can be found under that configuration?

Does the exception in your first post come from the line where you're importing xapian, or from a line where you're trying to use xapian. I suspect that the Xapian that needs to be installed in the exception is not the Python library, but the C++ library. It looks like you can install the Xapian libaries into a location that you have access to. Have a look at this page and look for "If you don't have root access to install Xapian"

I have installed the C++ library as well as the Python bindings. I've tried two different locations to see if there is simply a problem with the install locations in the WSGI environment. The C++ library is located here:

/home/Beison/lib

and here

/home/Beison/.virtualenvs/django16/lib

The Python bindings are here:

/home/Beison/lib/python2.7/site-packages

and here

/home/Beison/.virtualenvs/django16/lib/python2.7/site-packages

I'm almost certain that haystack and xapian are installed correctly because I was able to issue the django-haystack "rebuild_index" command within my virtual environment to build my search index without any issue. When a WSGI application is run, is there a way to set LD_LIBRARY_PATH to include a local directory? I'm wondering if the library simply cannot be found when requests come into the app.

Hmmm. Perhaps if you added your local directory to LD_LIBRARY_PATH via os.environ at the start of your WSGI file it would pick it up?

Thank you. Unfortunately, that didn't fix the issue either. For now, I am using the whoosh search backend for django-haystack which gets around this issue but I still would like to use xapian.

I got a more detailed output of the error with django's debug mode turned on. It lists the python executable as "/usr/local/bin/uwsgi" with a version number of 2.7.5. This is not the python version that I run within the console as I'm using 2.7.8.

Is it possible for me to set the python executable that is used within the WSGI environment?

We'll have a look at the version difference, but the web server has its own Python interpreter built in, so using the same interpreter doesn't really make sense.

I have some additional information on this issue.

This MissingDependency exception is raised within the xapian_backend module. I went into the source code of this module to try importing xapian while letting the original error propagate. What I got was the original ImportError which is causing the problem:

ImportError: /home/Beison/.virtualenvs/django16/lib/python2.7/site-packages/xapian/_xapian.so: undefined symbol: PyUnicodeUCS2_EncodeUTF8

By searching for this type of error on Google, I found the following page:

http://effbot.org/pyfaq/when-importing-module-x-why-do-i-get-undefined-symbol-pyunicodeucs2.htm

Using the test suggested in the article:

import sys
if sys.maxunicode > 65535:
print >> sys.stderr, 'UCS4 build' else:
print >> sys.stderr, 'UCS2 build'

and running this within the Django shell and from the web server, I discovered different responses. My Python (installed using pythonz) prints 'UCS2 build'. The Python interpreter built into the web server prints 'UCS4' build. So, it appears that the issue does stem from using two different versions of Python for building the xapian bindings and running the code on the web server.

Is there a way that I can have the web server use the Python binary installed in my virtualenv when running my site on the web server?

Unfortunately there's not :-( The web server runs the Python interpreter via a specially-compiled plugin.

Might it be possible to compile xapian to use UCS4?

Yup. That did the trick. I installed with the Python installed in my virtualenv using the mkvirtualenv command. I'm all set.

Thanks!

Excellent, thanks for confirming!