julialang / julia Goto Github PK

Our current test suite-in-a-file approach is starting to be a little unwieldy and we haven't kept up with writing tests very well. We should have a more systematic test suite with different files containing tests of different varieties so that various subsets can be run independently. A really systematic test suite for all numerical functions should exist too, to verify that we meet the best standards that Alan can hopefully help us determine.

We need a framework to express certain mathematical optimizations in julia itself. These may be expressed as rules that are run after types have been inferred. Examples are:

1. A' * B, A' \ B: Can be computed without computing the transpose
2. A - B + C .* D: Can be computed without temporaries
3. A[m:n, p:q] + B, A[m:n, p:q] * B: Avoid computing the subsref. This may require implementation of views.

Maybe should have made this one issue with #42:

julia> try; catch; end
syntax error: expected variable in catch

This should be an implementation of much of Alan's knowledge — an executable encoding of how various math functions should behave, ensuring that the Julia implementation does the right thing and continues to do so. Should also help ensure portability, since the nasty corner cases are where different implementations tend to disagree.

We need a general purpose FFI that can call any C library, using either tcc or clang.

Suggested interface:

loadc("libpcre") # loads the C library, reading headers and all
/* the namespace libpcre could be used to avoid naming clashes _/
pcre_compile("foo\s_bar", PCRE_CASELESS, ...)

Generate literal byte arrays wherein \x and \u are just shorthand for sequences of bytes.

In an email discussion we came to the conclusion that it made sense to have multiple inheritance in Julia with one fairly simple restriction:

If two abstract types are are used for dispatch in the same "slot" of the same generic function object then they cannot share a common, concrete descendant (all types share None as a common abstract descendant).

This restriction, together with Julia not allowing inheritance from non-abstract types, seems to address all the practical issues one typically encounters with multiple inheritance. The following, for example, would be disallowed:

abstract A
abstract B

type C <: A, B
end

f(A) = 1
f(B) = 2 # ERROR: A and B share a common descendant

Note that a generic function is an object external to all types, not a name inside of a type as it would be in a traditional object-orientation language. Thus, one can have f(a::A) in one namespace and f(b::B) in another namespace without problems, so long as the fs in these two namespaces are distinct generic function objects.

Goals:

• printing circular structures
• printing objects with type annotations
• printing objects structurally instead of "pretty printing" them

It should probably be possible to switch between different printing modes in the repl. It's especially often quite handy to have type annotations in the repl.

The current implementation was never meant to be more than a very temporary stop-gap and is frankly embarrassing.

Ctrl-C currently kills the repl, which is very annoying. It should abort the current computation as quickly as possible.

I suspect many of our critical array functions like ref, assign, cat, and transpose are highly suboptimal in many cases. For example ref/assign should be able to use memcpy (or equivalent) for the inner loop even for N-d.
We should time each of these functions in various cases and deal with any problems.

This was seen on a 32-bit Xeon system:

julia> ndigf(n)=float64(log(float32(n)))

julia> ndigf(256)
5.5451774444736657

julia> float64(log(float32(256)))
5.5451774597167969

julia> log(256)
5.5451774444795623

The last answer is correct, the middle one is the float64 conversion of the correctly-rounded float32 answer, and the first one is something strange.

julia> num2hex(5.5451774444795623)
"40162e42fefa39ef"

julia> num2hex(5.5451774444736657)
"40162e42fefa2000"

Looks like the float64 answer with the last 13 bits zero'd.

Happens with some functions, e.g. log and sin, but not with sqrt.

This may be a mac specific problem with finding libraries.

$ ./julia/julia
dyld: Library not loaded: libjulia-release.dylib
Referenced from: /Users/viral/./julia/julia
Reason: image not found
Trace/BPT trap (core dumped)

$ otool -L julia
julia:
libjulia-release.dylib (compatibility version 0.0.0, current version 0.0.0)
/Users/viral/julia/external/../external/root/lib/libreadline.6.2.dylib (compatibility version 6.0.0, current version 6.2.0)
/usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.10)
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 38.0.0)
/System/Library/Frameworks/vecLib.framework/Versions/A/vecLib (compatibility version 1.0.0, current version 268.0.1)

The base fdlibm does not have arch specific assembly code. The freebsd lib/msun is a derivative of fdlibm, which does seem to have these. We need to check if speed is indeed better, and if accuracy remains the same, and if so, use freebsd msun instead of fdlibm.

http://svnweb.freebsd.org/base/release/8.2.0/lib/msun/

Currently we need to do ugly things to make circular references:

type MyType
  self::Union((),MyType)
  function MyType()
    v = new(())
    v.self = v
  end
end

That is really bad, both ugly and compiler-unfriendly. With the change, we'd have

type MyType
  self::MyType
  MyType() = (this.self = this)
end

For constructors inside the type block, this is supplied, and the type's static parameters will be visible as well. this will also be returned automatically, and using return in a constructor will be an error.
Here's what Rational would look like:

 type Rational{T<:Int} <: Real
     num::T
     den::T

     function Rational(n::Int, d::Int)
         # can use T here
         g = gcd(n,d)
         this.num = div(n,g)
         this.den = div(d,g)
         # this is returned implicitly
     end
 end

 Rational{T}(n::T, d::T) = Rational{T}(n,d)
 Rational(n::Int, d::Int) = Rational(promote(n,d)...)
 Rational(n::Int) = Rational(n,1)

Notice the constructor inside the type block is only usable given an instantiated version of the type, Rational{T}. An outside generic function Rational() is defined to do this for you. If there are no user-defined constructors, we can still provide the same default constructors we have now.

One downside to this is that we'll have to check each field access for uninitialized references, as we currently do for elements of pointer arrays.

You should be able to do make install and have a compiled copy of all of julia installed on a system. The default location should be under /usr/local but should be configurable. It would be nice even for us developers to have both an installed (unbroken) copy of julia and have the development copy still live in the repo directory. That way when the development version is broken, you can still use julia for other stuff.

Doc stuff should perhaps move to the wiki, wherever it makes sense. Some of the stuff is notes, and then there is the manual as well.

When trying to make julia, test/unittests.j fails.

with the following error: Assertion failed: isinf(-(Inf))==true ./test/unittests.j:87

error in context:
make[1]: Leaving directory `/home/g3/julia/ui'
ln -f julia-release-readline julia
./julia ./test/unittests.j
Assertion failed: isinf(-(Inf))==true ./test/unittests.j:87
make: *** [release] Error 1

system info:
g3@ubuntu:~/julia$ uname -a
Linux ubuntu 2.6.38-8-generic 42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

This non-default test has been failing for quite some time, I think.

Our array printing is currently quite embarrassing. It needs to be fixed.

See this thread. This changes the behavior for bare string literals previously described in #4 to be the following:

• all bytes < 0x80: ASCIIString
• has bytes ≥ 0x80 and is valid UTF-8: UTF8String
• invalid UTF-8: throws an error

The b"..." string form (see #11) will let you use string syntax with \x and \u to make byte arrays. If you want to make a UTF-8 string that contains invalid UTF-8, you can do something this:

UTF8String(b"\xff\xff")

Now that we have sufficient external process functionality, this should be easy.

Simple linear algebra is likely to be one of the first things people try, and it needs to perform reasonably.
We could use ATLAS, or gotoblas:
http://www.tacc.utexas.edu/tacc-projects/gotoblas2/downloads/
gotoblas is not maintained, but it is already very fast, BSD licensed, and runs everywhere julia runs.

The current approach uses polymorphism to make RopeString objects. This is pretty inefficient for the typical small string use-case. To efficiently construct a C-style string in the current framework, one makes the current output stream a memio object and then prints to it. The general pattern I've used is to write a print_whatever function and then wrap it in a whatever function that returns a string using print_to_string. Should we stick with this pattern? It has the advantage of allowing the printing version to be very efficient, but it's kind of awkward to write. Should we figure out a different pattern? Something like C#'s StringBuilder pattern?

Perhaps it suffices to make strcat check the size and encodings of its arguments and use print_to_string approach to concatenate them into a copied string where appropriate — namely when the arguments are of compatible encodings (e.g. any mixture of ASCIIString and UTF8String), and if concatenated they would be below some size threshold. For larger strings, we should continue to use the RopeString approach. Also, string slices should copy their contents as well unless the resulting string is above the "large string" threshold, in which case, they can continue to use the current SubString with the known issue that this pins the superstring in memory.

Currently, we do not get any line numbers for errors:

julia> hpl(rand(5,5), rand(5))
no promotion exists for Array{Int32,1} and Int32

Why don't we disallow combining comma and semicolon, and comma and space? In other words, the only allowed syntaxes are

[1,2,3,4] # 1d array
[1 2;
3 4] # matrix

Use the first one when you are thinking of arrays or lists, and the second one when you are thinking of matrices.

See this email discussion.

Currently we have a half-baked, implicit load path for .j files loaded via the builtin load() function. It should be possible to add a list of directories to look in for .j files. The rule should probably be that names without / in them are looked up via the load path, while names beginning with / are considered to be absolute and names with / anywhere else are relative to the current directory.

See the discussion here. The salient conclusion is this:

Escapes continue to work the way they do now: \x always inserts a single byte and \u always inserts a sequence of bytes encoding a unicode character. Literals are turned into String objects according to the following simple check:

• ASCIIString if all bytes are < 0x80;
• UTF8String if any bytes are ≥ 0x80.

If you want to use \x escapes with values at or above 0x80 to generate invalid UTF-8, that's your business. We can also introduce an Latin1"..." form that uses the Latin-1 encoding to store code points up to U+FF in an efficient character-per-byte form. Finally, the b"..." macro-defined string form can let you use characters and escapes (both \x and \u) to generate byte arrays.

We can safely and quickly concatenate ASCIIStrings with each other, with UTF8Strings, or with Latin1Strings. Mixing UTF8Strings and Latin1Strings, however, requires transcoding the Latin1Strings to UTF-8. This, however, will not occur with string literals since they will always be ASCIIStrings or UTF8Strings.

After building julia, its directory uses around 300MB. The vast majority of this space is intermediate files from the build process that are not needed to run julia. We should add a target like make compact that runs make clean in the subdirectories of external/, and removes our object files. All the llvm libraries can be removed from external/root/lib since they are statically linked.

General pattern matching a la Magpie (discussion here) used for function dispatch, destructuring, switch case dispatch, and exception handling. See the brief email thread on the subject. We could still really use some pattern matching. The hardest part would likely be integrating it with multiple dispatch.

Example:

julia> read(open("/dev/null"), Uint8, 1)
^C

Worse still, it busy-waits. This bug kicks in when make test-utf8 is run without run make in the test/unicode directory first.

Method signatures should look into Union types to find a more specific match.

The recent checkins in sparse.j result in a less specific version being called:

Warning: new definition +(SparseArray2d{T},Union(Array{T,N},Number)) is ambiguous with +(Tensor{S,N},Tensor{T,N}). Make sure +(SparseArray2d{S},Array{T,N}) is also defined.
Warning: new definition +(SparseArray2d{T},Union(Array{T,N},Number)) is ambiguous with +(Tensor{T,N},Number). Make sure +(SparseArray2d{T},Number) is also defined.

Mac already provides a libBLAS and libLAPACK. At the very least, we want the ability to use the libBLAS on the mac. Basically, a flexible framework is required to load BLAS and LAPACK.

fdlibm seems to be missing a few functions here and there. fdlibmf has log2f, but fdlibm does not have log2:

julia> log2(2.)
dlsym: /home/jeff/src/julia2/julia/lib/libfdm.so: undefined symbol: log2

On the other hand, fdlibm has lgamma_r but fdlibmf does not.
I found implementations of these from BSD sources by searching for e_log2.c and e_lgammaf_r.c.

• We should check in fdlibm like we do for fdlibmf, and add the missing files
• We only use platform libm for 5 functions, so we might as well stop linking against it and use fdlibm for everything

Like it says, every time I run make in external/ it causes ARPACK to get recompiled.

julia> a.
(type-error string.find string #)
unexpected error: #0 (char-numeric? #)
#1 (next-token/lambda/lambda

#. #)
#2 (next-token/lambda #.)
#3 (peek-token [#f # #f #f])
#4 (parse-call/lambda/lambda/lambda

a)

In some builds, the SEGV handler does not work until SIGFPE has been raised. We used to address this with the infamous "awful_sigfpe_hack", then at some later point testing showed it was no longer necessary. However, for me the problem is back:

julia> f(x)=f(x)
Methods for generic function f
f(Any,)

julia> f(2)
Segmentation fault (core dumped)

VS.

julia> div(1,0)
error: integer divide by zero

julia> f(x)=f(x)
Methods for generic function f
f(Any,)

julia> f(2)
error: stack overflow

By frobbing it a bit I found that setting up the signal handlers after loading the system image (in init.c) "fixes" the problem again. I don't understand this at all. It might have made sense that something was different after entering a signal handler (e.g. the signal mask, but I tried frobbing that too), but I don't know what is different this time. This is a mystery.

We need a whos command, which shows all global variables.

We discussed this once a long time ago and decided that it would be nice to have a Nothing object that prints as nothing. Then when entering code into the repl, one doesn't have to worry about putting an unseemly ; at the end of a for loop to suppress the annoying (). Also it would make some potential errors more sensible: if you try to do anything with the Nothing value it can throw a fairly specific error, whereas () is a perfectly legitimate value for many expressions to produce.

Shouldn't this not create the file by default?

The current interface to things like hashes and sets is pretty thrown together and dodgy. For example, you can't put a set into a set at the moment because add(s::Set, s2::Set) adds the contents of the second set to the first, rather than adding the second set to the first as an item.

[edit: see Stefan's list below for bulleted list]

This includes:

• JuliaLang.org website
• up-to-date README on GitHub
• complete wiki documentation of all v1.0 language features

These are all necessary for the v1.0 release.

The current implementation is frankly embarrassing.

Enough said.

The conclusion of our last discussion on immutability versus mutability was that it is largely a psychological distinction:

• when an object is identified by its value, then it should be immutable: integers, floats, complexes, strings;
• when an object is a container, whose identity can remain the same while its contents change, then it should be mutable: arrays, linked list nodes, tree nodes.

To declare that a new concrete type is immutable, prefix its declaration with immutable:

immutable type Complex{T<:Real} <: Number
  re::T
  im::T
end

Alternately, only the immutable keyword could be used:

immutable Complex{T<:Real} <: Number
  re::T
  im::T
end

This says it all:

julia> tic();println(remote_call_fetch(2,+,1,1));toc()
2
elapsed time: 0.0006420612335205 sec
julia> tic();println(fetch(remote_call(2,+,1,1)));toc()
2
elapsed time: 0.0400099754333496 sec

This is all between two processes on the same machine. A single message roundtrip (request, response) is fast. But if we do the same thing in two messages (send call, send fetch, get response) it takes 60x longer. This is crazy. And there is still only one round trip, just a single extra one-way message from p1 to p2.
We already tried setting TCP_NODELAY and it did not make any difference. So far I'm stumped.

It would be great if all the individual .j files are compiled separately, and then somehow combined into sys.ji. Even a one line modification triggers sys.ji to be built from scratch. Would be nice if it was faster.

Depends on issue #28.