imclab / dxr

An intelligent source code browser

DXR, A Code Search and Cross-Reference Tool

Introduction

DXR is a source code browser and search engine, building on the work of LXR and MXR. It supports full-text and regex searches as well as structural queries like "Find all the callers of this function." Behind the scenes, it uses trigram indices, the re2 library, and static analysis data collected by instrumented compilers. DXR runs only on Linux for the moment.

Documentation

DXR is fairly well documented, so whether you're looking to deploy DXR, write plugins or templates for DXR have a look at the docs/ folder. The documents in this folder is a great starting point for anybody who wants to deploy and/or develop DXR.

Quick Start

To try out DXR on a HelloWorld-size project...

git clone --recursive git://github.com/mozilla/dxr.git
cd dxr  # (the top level of the repository, not the dxr folder within it)

# If you want to customize VM configuration for your environment, run the
# line below, and change the settings you need, e.g. turning off NFS:
cp vagrantconfig_local.yaml-dist vagrantconfig_local.yaml

vagrant up
vagrant ssh

On the VM...

cd ~/dxr
make
cd ~/dxr/tests/test_basic
make
dxr-serve.py --all target

You need to bind to all interfaces (--all) or else you will be unable to get to the Vagrant box's test server from the host machine. Then, on your host box, surf to http://33.33.33.77:8000/, and poke around to your heart's content. You may have to substitute the IP address of your Vagrant machine.

Please note that the vagrant image is built against virtualbox 4.2.0. If your version is older, the image might not work as expected.

Troubleshooting

If, after trying a search, you see the error...

Server Error
.....
Database error: no such module: trilite

SSH into the VM and, run ldconfig as root to sort out the shared library linking problem. Then restart dxr-serve.py, and all should work as expected.

Installing on a Real Machine

dxr requires exactly llvm-3.2 to build. Anything newer or older will break the build.

To install DXR on a machine of your choice, do the following

1. Install the dependencies in docs/deployment.mkd.

2. You can either install dxr at the system level (requires root) or inside a virtualenv which is the recommended way.

   • For a system-level install, run python2.6 setup develop (to hack on DXR itself) or python2.6 setup.py install to install it.

   • The recommended way is to install it as an unprivileged (non-root) user. For this, use virtualenv:

         virtualenv my-dxr-env
         source my-dxr-env/bin/activate  # You'll have to repeat this line each time
                                         # you want to use DXR in a new shell.
         pip install -r requirements.txt
         python setup.py develop         # Magic path munging will make this the
                                         # copy of Python in the virtualenv.
     

3. Next, you need to build the various binaries that dxr needs. Running make at the top level should do the needful.

   • It should build the libtrilite.so library inside the trilite directory.

   • It should build the libclang-index-plugin.so inside the dxr/plugins/clang directory. You might have to edit the makefile in that directory to make the locations of llvm-config, clang and clang++ accurate. As mentioned earlier, the compile will succeed only with exactly llvm/clang-3.2. Anything else will fail with this message.

        dxr-index.cpp: In function 'std::string {anonymous}::getQualifiedName(const clang::NamedDecl&)':
        dxr-index.cpp:93:13: error: 'const class clang::FunctionType' has no member named 'isConst'
     

4. The tests/test_basic directory contains a small tree which you can build an index for to test your build. cd into this directory and run make. If it successfully runs, your build is fine.

5. Then run LD_LIBRARY_PATH=/path/to/dxr/trilite dxr-serve.py tests/test_basic/target to start the server and browse the code. Without the LD_LIBRARY_PATH, it will complain about not being able to find the trilite module when you do a search on the web interface. You can dispense with this requirement by installing the library globally on your machine: on Linux, move libtrilite.so to /usr/local/lib, then run ldconfig.

Things To Do

This is a short list of major things that could be done to improve DXR.

• Refactor the database schema
  • Remove plugin specific tables
  • Provide data collection API for plugins (ie. no direct database access for plugins)
  • Optimize schema for search queries
    • Store extents as a blob in *_refs, types, functions, etc.
    • Use trilite for functions, types, macros, etc.
    • Store the transitive closure of indirect calls
  • Port search logic to new schema
• Write proper search query parser
  • Space is a delimiter unless in dice [ ], quotes " " or escaped \
  • Assume grep syntax for terms, path:, type:, etc. (ie. * is literal, \* is kleene star)
  • Let quotes "text" be literal terms, path:, function:, etc.
  • egrep syntax for regexp:, regex: and egrep: ie. * is kleene star, \* is literal)
  • Prefix + implies case sensitivity (and fully qualified type:, function:, etc.)
  • Prefix - implies negation (prefix -+ and +- is case sensitive negation) (Prefix +, -, +- and -+ can be applied to terms, path:, type:, egrep:, etc.)
• Minor refactoring of template interface to minimize size
  • Reuse annotations (i.e., don't store the same annotation for each line)
  • Reuse menus (i.e., don't store the same menu for each occurrence)
• Minor UI fixes and cleanup
  • Colorize highlighted matches (highlight with classes, .m1, .m2, ..., use merge-extents)
  • Make the UI pretty
  • Add option in "Advanced" search to disable redirect (use cookies to remember setting)
  • Minimize template size (at 20 MiB for 20k lines, this is a real issue)
• Add tree format version number to generated trees (Don't merge generated output with different formats, when configuring update cron job)