How are include paths determined about edts

[tjarvstrand]

Hi!

I try to make edts "do what you mean", so it's pretty liberal about what paths are added (so you should never assume you have a working project just because it compiles in edts :P ). In addition to ".", which is added by default, edts adds ./ebin, ./test and the ebin- and test-directories of every subfolder of a project's lib-dirs (setting lib-dirs to ("lib" "deps") would add lib//ebin, lib//test, deps//ebin and deps//test to the code-path. Additionally, when edts compiles a module, the out-directory of the beam-file is also added to the code-path if it's not on there already.

The code for deciding what to add is in edts_server:expand_code_paths/2, by all means dig in :) I've previously run into problem's with symbolic links and paths with ".." in them because the otp doesn't really handle that. I've tried to work around the ".." thing myself, but the symlink issues are much harder to account for.

[srstrong]

Having had a dig, it looks like I'm interested in edts_code:compile_and_load/2. It looks like it discovers the include directories for compilation from module_info on a pre-existing beam. However, it's getting the "wrong" answer from module_info. To explain, my project structure is:

proj/deps
/dependency1 etc
/apps
/app1
/src
/include

which is basically a standard rebar setup. For an ebin that was built from a file in apps/app1/src, it has an include path of {i, "include"), rather than {i, "apps/app1/include"}. Hence the compile in EDTS is failing to find the required hrl files.

I'm guessing, but not validated, that rebar is running in the context of the apps/app1 directory when it compiles stuff in apps/app1/src, hence the include path. The EDTS node has it's working directory in "proj" however, which means the include path isn't pointing at the correct place.

Have I interpreted this correctly? If so, do you see any quick fixes? I could imagine recursively finding the include directories and adding them to the compile options, but clearly there's an issue here around getting the ordering correct. An alternative would be to simply specify the include paths in the configuration for a project, which would be very simple and less prone to error, although it's a manual step which is always a pain :)

[tjarvstrand]

Yes, IIRC rebar will compile with the application directory as the cwd.

tl;dr Move from include-directives to include_lib and you will no longer be (as) dependent on the high-level structure of your project.

Longer answer:
Relative includes are very brittle and prone to bad practices. IMO they should be shunned except perhaps when including "application-private" header-files (header-files located in the src- or priv-directories of your application). include_lib is much more robust and there are very few cases where it can't be used.

When you have to use an include-directive you should always:

• Use a relative path. Using an absolute path is like hardcoding, it's bad for your karma (but you know this:)).
• OTP tries to find include-files both relative to "." and the directory where the module's erl-file is located, but since you have no control of where the module will be compiled from (eg. the erlang shell etc), use a path relative to the module's location.
• Do not include anything outside the module's own application because this introduces a dependency on the high-level structure of your project. Suppose for example that you use a relative include for webmachine.hrl. After some time the project gets unwieldy and at one point you decide you want to move all external dependencies to another directory. Now you can't do that without also updating you code-base. What's worse, if you would like to use your application in another system you're forcing that system to use the same structure.

IMHO it's a design flaw that otp looks for include-files in cwd at compile, it makes for difficult-to-understand build-systems that work by coincidence. You could argue that this is a bug, since it's very easy to end up outside your project without realizing it. This link is about a different bug, but appeared because of this very problem:
http://comments.gmane.org/gmane.comp.lang.erlang.bugs/3187

Sorry for the rant, but I'm a bit opinionated on the matter. Anyway, in summary I will most likely not add support for adding include-directories to projects unless there's an overwhelming demand for it :)

[srstrong]

Another long comment - I'll give the summary if you don't want to read it all:

• I think my directory setup and use of include vs include_lib is exactly correct, and is the canonical setup for rebar-enabled projects. So I don't think I'll be alone here.
• I have a simple workaround that just involves configuring each app as a separate project in my .emacs file - it all works just fine then :)

And here's the fuller version to back up my claim of using include / include_lib correctly :) ...

I completely agree with your thoughts on include_lib, but the issue I'm getting is only files that are "local" to that application - what you call "application-private" files, although quite often they act as in interface for other application to consume services offered by my application (webmachine.hrl being a good example).

So for headers in my dependencies, I certainly use include_lib; anything else would be a source of continual pain :) But for my "own" headers, the include directive is the correct one since it's indicating that I'm including a header that's part of the current project. Since these headers are often acting as record and type definitions for other users, I have them in the include directory rather than the src directory - makes packaging much easier.

Now, if I just have a project structured thus:

/proj
     /src
          *.erl
    /include
          *.hrl

then all is good - both rebar and EDTS play quite nicely in this environment. However, quite often we are working on large projects that themselves are divided into separate applications:

/proj
     /app1
        /src
             *.erl
       /include
             *.hrl
     /app2
        /src
             *.erl
       /include
             *.hrl

If app1 needs a header from app2, then it most certainly uses include_lib. If it wants its own header though, it just uses include. rebar compiles everything just fine, but dues to its "cwd" for each application, the include paths that module_info return are incorrect for EDTS, which is based at the root of the tree. This directory setup is, I think, correct behaviour, and is a documented in a number of places as the canonical way to setup rebar configurations. Hence, I don't think I'll be alone in this problem :)

The good news is I have thought of a simple work-around. Rather than listing a single project, "proj" in my .emacs file, I can list a separate project for each app, e.g., "proj/app1", "proj/app2". It's a little more manual configuration but once done it won't change often, so it's no big deal.

[tjarvstrand]

Yes, sorry for rambling a bit. My issues are not with the way your project is structured (edts should be able to support most structures) nor with your choice of include vs include_lib, I got a bit sidetracked.

I do however think that it is unwise to make assumptions about what the cwd will be at compile-time, since this introduces unnecessary dependencies to the build tools/config you're using, as is apparent, since you're getting this problem. What I wanted to say was that I believe you can easily solve these issues by using a path that is relative to your module's location instead of the application directory, so
-include("../include/header.hrl").
instead of
-include("include/header.hrl").

[srstrong]

I take your point on using relative paths for "local" includes - I wasn't aware that the compiler would look for includes relative to the .erl directory as well as "."; I assumed it was just ".". So experimenting with that as well - so far, all is working as expected :)

[tjarvstrand]

Great! Let me know if you run into any more problems

[tjarvstrand]

Is it ok if I close this issue?