csyonghe / spire Goto Github PK

pack all components into vec4s

Xor operator ^ is not processed in ParseOperators(). (https://github.com/csyonghe/Spire/blob/master/Source/SpireCore/Lexer.cpp#L55)

There appear to be two identical files in the repository:

Source/SpireLib/include/Spire.h
LibraryRelease/Spire.h

It isn't immediately clear to me as somebody looking at the repository which of these would be the one I'm supposed to include in a user project, and which I'm supposed to edit if I want to make changes.

It would be a lot more obvious to just have include/spire.h, and no additional copies. The "packaging" step makes some amount of sense for the .cpp distribution (although we still need to address the issues of people editing generated code), but it makes almost no sense for the .h.

To compile GLSL / HLSL code directly, spire compiler need to parse and understand global variables and constant definitions.

It seems like Spire supports M * v, but not v * M.

This is a bit annoying for me because my mini app is using a row-major layout for its matrices, and rather than transposing them when filling in a uniform buffer, it is just reversing the order of multiplication in the GLSL shaders.

I can work around this (especially if transpose() is supported in Spire), but it would be nice to have this just work out of the box.

(I guess another issue along those lines is whether I can affect the default matrix layout convention for the generated GLSL...).

I've been using require statements to work around the lack of explicit inter-module interfaces (see #19), but now found that if I try to use:

require sampler2d someSampler;

I get an error:

error 33035: 'someSampler': sampler, struct and array types only allowed in input worlds

This seems like an oversight in the case of a require, since this simply results in a placement constraint such that the component will only ever land in an input world.

I'm also not sure why there would be a ban on struct or array types in non-input worlds, but I figure I can file a bug against those if and when they become a problem.

As far as I can tell, there is nothing essential in the Spire implementation that requires VS2015. It would be good to revise it to work with VS2013, since that would widen the range of applications that could make use of Spire.

Ideally, I'd want to have the whole suite of matNxM types, where 2 <= N, M <= 4, and be able to index into them etc.

I started trying to add this myself, but the number of places I had to touch was daunting. The main issue is that each matrix type is reflected direclty in an enum (with hard-coded values, which seem to need to match another enum of IL-level types (?)). As a result, there are long conditional chains all throughout the code that need to handle all the cases (I would have preferred switch statements, since they are easier to add a case to on the fly), and all of those places need to be fixed up to handle a new type. One particularly worrying spot is trying to guess the IL type to use based on the size of a constant (so 9 floats -> mat3 and 16 floats -> mat4), but this breaks down for non-square matrices.

I'd really like to see a pretty fundamental change to the type representation in Spire, so that is has a general thing like a struct MatrixType { Type* elementType; int rows; int cols; } and then Type is either an inheritance hierarchy or a tagged union...

It is a pretty common idiom in HLSL/GLSL code to have functions that receive some inputs through their ordinary parameter list, but that also depend on shader inputs (uniforms, textures, etc.) that were declared at the global scope.

When porting to Spire, we can either translate these functions into a module (which has its own issues), or keep them a function, but in the latter case it seems like we have to make all the parameters explicit for now, because we can't nest a function inside a module or shader (at least it didn't work when I tried).

This seems like an important feature to have, although it could have thorny interactions with placement constraints.

If I declare a simple vertex + fragment shader, then I find that the projCoord computed in my vertex shader is being declared as a VS output and an FS input, even though the value is never used in the FS.

It seems like the logic for determining which components are "live" coming out of a stage needs to understand that system-consumed "outputs" in a given stage do not actual make the corresponding component live-out (although they do make it "live" from a DCE perspective).

It looks like the problem is that you are declaring and int:

int bufferSize;

and then passing a pointer to it where a size_t* is expected:

mbstowcs_s((size_t*)&bufferSize, ...);

This pattern is repeated for the variable pos later in the same function.

Changing both of these to size_t seems to eliminate the crash for me, but leaves me wondering whether I should be trying to use Spire in a 64-bit project...

The D3D10+ HLSL Texture2D and related resource types use C#-ish method-call syntax, e.g.:

int3 pixelCoord = int3(x,y, 0);
float4 val = someTexture.Load( pixelCoord );

This syntax fails to parse in Spire. This could be worked around if Spire decides to support the same basic operations as ordinary functions, rather than members.

Repro attached: test.spire.txt

When I run the Spire compiler on this file, I get an output .cse file. That file doesn't appear to contain any GLSL code to compute the rs_Position property mentioned in the source Spire, but it does contain a line of code to copy that component over to gl_Position.

It is possible that I'm not setting up my pipeline definition correctly, but I don't know the rules well enough to fix the issue myself.

HLSL allows functions to be declared with default parameter values:

float myFunc(float a, bool doThing = true) { ... }

Syntax like this currently leads to parse errors in Spire. This can be worked around by declaring extra overloads:

float myFunc(float a, bool doThing) { ... }
float myFunc(float a) { return myFunc(a, true); }

Addressing thing might not be a priority in the near term, but this feature will definitely be a sticking point if we need/want to accept general HLSL input.

Even with the deletion of history, a fresh clone of the project takes a really long time on my slow-ish connection and occupies ~500MB on disk.

This might be due to the spire1 branch, but I honestly don't know how git works well enough to make any kind of definitive statement.

An extra null pointer check is not needed in functions like the following.

• Dictionary::Free
• StringBuilder

Current compiler implementation performs dependency sorting on the components, giving shader authors the flexibility to define components out-of-order. However this feature is rarely used in practice and the compiler sorting process some times introduces surprises as the component definition order may differ from source code after the sorting. This becomes an issue when the engine assumes the order of input/output components in shader source code, and forces the user to specify layout(location=x) attributes.

If I am defining a module that needs access to component that will be provided by some other module (perhaps one of several possible implementations), the current Spire compiler forces me to require every individual component. This is tedious, and also seems like it will give a bad user experience when adding/removing/renaming components that are shared in this way.

Ideally, I'd like to be able to declare an interface or signature that lists a set of related components:

interface TangentFrame
{
    // names in an `interface` act as if declared `public` in a module
    vec3 normal;
    vec3 tangentU;
    vec3 tangentV;
}

and then in a module that wants to depend on these components, I do so through a require:

module Lighting
{
    // require some implementation of the interface, and import its components into explicit namespace
    require tangentFrame : TangentFrame;

    // implicit namespace variation also allowed, as for `using`
    // require TangentFrame;

    vec3 nDotL = dot(tangentFrame.normal, ...);
}

A module can declare that it implements the interface using ordinary object-oriented syntax:

module NormalMapping : TangentFrame
{
    vec3 normal = ...;
    // ...
}

In the simple case, a module that implements an interface will need to declare components with matching names/types (and for simplicity one might require that they be public, but that shouldn't really matter). More explicit syntax could be adopted if the object-oriented case doesn't feel right.

When using a module like Lighting above, the interface requirement can be filled in just as for an ordinary component require; the only difference is that it gets filled in with a module instead of a component.

shader Simple
{
     // completely implicit case:
    using NormalMapping; // use a module that implements interface
    using Lighting; // module that requires interface automatically gets it as input

    // completely explicit case:
    using nm = NormalMapping(...);
    using lighting = Lighting(tangentFrame: nm);
}

The implicit case should probably be limited so that we can only implicitly fill in a required module when there is exactly one module in the current scope that implements it.

It should be clear that everything above would extend to having a require on module "classes" rather than just interfaces, and it would also seem to generalize to supporting function or module members in an interface. Those generalizations aren't part of the initial feature request, though.

I tried to name my frament world fragment and so forth, but when I try to name a world uniform, the Spire front-end generates GLSL code with a declaration like:

uniform uniform { ... }

which is obviously not valid GLSL. I haven't done any digging into whether using GLSL keywords for other sorts of names causes problems, but it would be nice if the output GLSL went to some effort to "mangle" names to avoid collision with GLSL keywords.

A simple strategy would be to prefix every name with something like SPIRE_*, but this would result in less immediately readable GLSL output.

support Texture and Sampler types
support built-in HLSL functions
add ++/-- operator
add c-style for loop

This might be closed as "by design," but I was a bit surprised when I had the equivalent of:

vec3 P = ...;
vec4 P_homogeneous = vec4(P, 1);

The compiler informed me that there was no overload for (vec3, int), and of course I fixed the issue by changing the 1 to 1.0.

Note that I'd imported the expression in question by copy-pasting the GLSL I started with, so small issues like this will probably irk new users more than people who write in Spire from scratch.

// shaders.spire

pipeline StandardPipeline
{
world w2;
stage s2 : FragmentShader
{
World: w2;
}
}

typedef A B;
typedef B A;

shader Test
{
out @w2 A a = 5;
}

A lot of existing GLSL/HLSL shaders make use of mutable variables in ways that are convenient, but maybe not essential to their semantics. In broad strokes, there are two cases:

1. "Local" mutation used to compute a value incrementally from sub-terms.
2. "Global" mutation used to modify a logical quantity like "position" or "normal" over the course of a shader.

The "local" case is stuff like:

float4 sum = 0.0;
sum += firstThing();
sum += secondThing();

and:

float4 foo = someFunction();
foo.xyz *= foo.a;

In general, we can re-write such component declarations into bigger expressions, or so that they use Spire's support for statement initialization. This is usually mechanical, so I expect most programmers could be trained to write in the Spire-like style.

On the flip side, this kind of local mutation seems really easy for a system like Spire to support (it is basically just what Shade Trees had). If we think of sum or foo above like a variable, that points to a specific shader-graph node, then the subsequent assignment operations can be seen as just creating new shader graph nodes and then pointing the variable at a new node. This same basic logic could be extended to support simple cases of conditional assignment.

The "global" case is more interesting, and probably also more challenging. A lot of large GLSL/HLSL shaders work along the lines of collecting most/all of the shading state into a single struct, and then passing that struct as an inout parameter to one or more functions that will read-modify-write it along the chain.

Within the Spire formulation, these steps along the chain might best be implemented as modules. We already have support for the output of one module to become the input to another module (so if A outputs normal and B expects normal as input, it can get it implicitly). The missing thing right now is that if we want a module that takes normal as input and produces normal as output, we have to carefully manage the names, and do the module wiring more explicitly.

I suspect (but am not sure) that it is reasonable to extend the intuition of the "local" case to the global one, and allow for modules that conceptually have inout parameters. The order of using declarations for modules represents a logical "order of execution" (here referring to the order of execution of shader-graph-building code, and not the runtime code), so that the shader author would still be in control of the final result (just as if they had called a bunch of functions that perform read-modify-write operations in order).

I'm a bit nervous about all of the above, but I also know that porting existing shader code that makes heavy use of mutation tends to result in ugly idioms (lots of distinct names for the same concept), if the target language doesn't support some kind of idiomatic mutation construct.

When GLSL added support for non-square matrices (e.g., mat3x4) it also added alias of the existing square matrix types, so you can use both mat4 and mat4x4.

Spire seems to only support mat4, which could inhibit easy porting of existing GLSL code.

This is a builtin function in older HLSL (and I believe also in GLSL). This is probably just a matter of adding it as a builtin, but we probably need to take a pass and either systematically add all the texture-fetch operations we need, or start working on using unparsed HLSL/GLSL strings.

Low priority for now, but would eventually be needed if broad compatibility with GLSL/HLSL shaders is a goal.

I seem to recall that we'd gotten this working at some point, but it appears that the concatenated Spire.cpp now doesn't build with VS2013 due to use of constexpr in IsBaseOf and IsConvertible.

I will try to switch these to enums and see if that is the only workaround required.

The readme for the Vulkan engine refers to a GameEngine.sln which is not present in the repo.

Can this be added to the repository? Without it one has to guess what libraries to build in which order and where to put them to get this up and running. The GameEngienCore project also won't compile due to VK_CPP_TYPESAFE_CONVERSION not being set which I guess is done in the missing .sln.

In current implementation, pipeline-specific semantics cannot be checked per module because a module does not specify which pipeline it targets. It is probably necessary to add require world declarations in a module to allow such consistency checks within modules.

Currently, the Basic.h header in the library release makes (very limited) use of C++ binary literals (the 0b prefix), which causes it to fail to parse in Visual Studio 2013.

Ideally, Spire should not require C++ features that aren't broadly supported unless absolutely necessary.

A recent change added a bool type, but the parser still doesn't recognize true or false as keywords.

This looks like it is a one-line fix, but the TypeSyntaxNode::ToExpressionType function doesn't seem to handle bool, even though later stages of the compiler support BaseType::Bool.

This code:

float a, b, c;
a = true ? b : c;

Yields the error message:

error 30080: the two value expressions in a select clause must evaluate to same type.

I had code inside a component like:

vec4 foo = ...;
foo.rgb *= foo.a;

and I got an error on the *= operator, about there being no available overload.

I worked around the issue by changing the code to not use a compound assignment operator:

foo.rgb = foo.rgb * foo.a;

Thus is appears that the issue may be as simple as compound assignment operators being unimplemented (?).

When trying to use Spire in a VS2013 app, this first arises in the SortInterfaceBlock function.

Use of auto like this is a just a convenience feature, so the code could be made portable to more compilers by using an explicit type.

If I invoke the Spire compiler as:

SpireCompiler.exe .\foo\bar\baz.spire

and I get an error message, it is formatted without the path:

baz.spire(99): error 33023: argument ...

This makes it so that I can't double-click on the error message in my editor and go to the line, because the error message seems to refer to a non-existent file.

Is future development to remain at this repo, or will it be moving to spire-lang?

This line should be modified as:
if (op0->Type->IsVector() && !op0->Type->IsFloatMatrix())

Otherwise, given mat3 a; float b=a[1][1]; Spire will generate code like b=a.x.x;.

If I write some code that uses an undefined variable (or I just spell it wrong) I get:

error 30017: component 'foo' is not accessible from shader 'Test'

Similarly, if I try to call a function that is undefined I get:

error 30021: Foo: no overload takes arguments (int)

It would seem a bit nicer to have clear errors for the case where there is nothing of the given name in scope, and reserve the errors above for cases where there is something in-scope but it is not accessible or applicable in the current context for one reason or another.

It seems like the rules for what components are visible to a using are "recursive" so that if I have:

module Nested { public float foo = 1.0; }
module Wrapper { using Nested; }

module UsesFoo { require float foo; }

then the following works:

module Works
{
    // `Wrapper` doesn't export anything, but `Nested` does
    using Wrapper;

    // Automatically satisifies the `Foo` requirement using the `foo` in `Nested`
    using UsesFoo;
}

However, the same "recursive" rules don't seem to apply when I try to just use foo directly:

module DoesntWork
{
    using Wrapper;

    // I get an error on this line, because `foo` isn't defined here
    float bar = foo + 1.0;
}

This is a little bit confusing on the face of it, and I also can't really see a workaround, that would allow the Wrapper module to somehow "re-export" a component output by something nested in it.

I mean, I guess that I could make Wrapper use an explicit using and then do a named export:

module Wrapper
{
     using n : Nested;
    public float foo = n.foo;
}

But that seems like a really kludgy workaround. Maybe a nice syntax would be to allow public using to automatically re-export everything that was public in the sub-module?

module Wrapper { public using Nested; }

Any sort of guidelines here (even just "don't do that") would be appreciated.