dotnet / corefxlab

This repo is for experimentation and exploring new ideas that may or may not make it into the main corefx repo.

Hello, this repository is now archived.

Going forward, the .NET team is using https://github.com/dotnet/machinelearning for Microsoft.Data.* and https://github.com/dotnet/runtimelab for experiments.

corefxlab's Issues

Composite format should support escaping "{"

The following test will fail:

        var formatter = new StringFormatter();
        formatter.Format("{{{0}", 1);
        Assert.Equal("{1", formatter.ToString());

Add license text for FreeBSD Daemon

The FreeBSD Daemon has a license requirement that we are not yet following. We need to fix that.

This is a follow-on to #83.

/cc @josteink

Add memory allocation tests

Should Json really be under System.Text?

There isn't a System.Text.Xml, and I would argue that while json, like xml, is represented by text, its implementation details shouldn't be part of Text.

Create core geometry, drawing and imaging libraries

This issue follows from the discussion at https://github.com/dotnet/corefx/issues/1563.

It is proposed to create a legacy-free library or set of libraries that would eventually be incorporated into corefx. The libraries would define primitive types and interfaces in the 2D and/or 3D geometry, drawing and image processing areas that are complementary to corefx. Such definitions would encourage compatibility of use and implementation across platforms and between libraries that target corefx in these areas.

The coreclr and corefx newly define SIMD-optimized types in System.Numerics that have special processing by the JIT compiler. These types include Vector2/3/4 and related Matrix and geometry types. The new primitive types defined for geometry, drawing and image processing could make use of these optimized types where appropriate.

Some of the use cases that would fall under this topic are:

• Image processing libraries for asp.net and other server scenarios
• Cross-platform 2D drawing libraries similar in scope to the Win2D library for Universal Windows Apps
• 2D and 3D geometry libraries that perform computational geometry or mesh processing

The proposal is not to provide comprehensive cross-platform implementations to cover these areas. Rather, the intention is to define common types and API patterns (perhaps in the style of OWIN for http handlers) that various implementations could align with. These libraries should consolidate work done for corefx that fall within the geometry, drawing and image processing scope.

There are many questions:

1. What are the use cases for such libraries?
2. What is an appropriate breakdown of the areas covered? How do the bitmap and vector worlds interact?
3. How wide should the scope be? Should it include color, fonts, text layout, UI? Now or later?
4. How should the new libraries relate to existing open-source implementations (single platform and cross-platform) in these areas (both pure .NET and managed wrapper libraries)?
5. What are the first steps and how should the effort proceed and be managed?

Can we make a list of existing .NET (open-source or otherwise) libraries in this space:

1. How popular and active is the library?
2. Is it likely to target corefx in the future?
3. What primitive types and interfaces would be relevant to that library?

Dispose method on CilAssembly needs to be updated on System.Reflection.Metadata.Cil

The dispose method on CilAssembly needs to be updated to keep track of disposed state in order to avoid disposing the readers twice and get an exception.

See:

https://github.com/dotnet/corefxlab/blob/master/src/System.Reflection.Metadata.Cil/src/CilAssembly.cs#L300-L303

cc @AlexGhiondea

It's unfortunate that formatters need to be classes (hard problem)

Composite format should support alignment

The following test will fail:
var formatter = new StringFormatter();
formatter.Format("{0,2:D}{1,-2}#", 5, 6));
Assert.Equal(" 56 #", formatter.ToString());

dotnet.exe should support building dlls

System.Reflection.Metadata.Cil does not emit accessibility on types

It looks like Visit(CilTypeDefinition typeDefinition) is missing handling for Attributes.

System.Numerics.Matrices does not build

The project does not build currently as it's using C# 6 features (string interpolation) which is not supported by the current engineering system in corefxlab.

Implement numeric 'R' format

Repro:

1. Uncomment //GenerateTest(writer, type, 'R', maxPrecision); in src\System.Text.Formatting\GenerateCompatibilityTests\Program.cs
2. Run the program. It will generate additional format tests
3. Build the formatting library and run the test (run build.cmd from command line)
   Result:
   Some formatting tests will fail.

Number formatting "N' format does output the first decimal separator

Repro:

1. Uncomment line //GenerateTest(writer, type, 'N', maxPrecision); // TODO: this is implemented, but has bugs in src\System.Text.Formatting\GenerateCompatibilityTests\Program.cs
2. Run the program. It will generate additional 'N' format tests
3. Build the formatting library and run the test (run build.cmd from command line)
   Result:
   Some 'N' formatting tests will fail.

Add DateTime compatibility tests

Should CilDecoder decode tokens for instructions as CilEntity on System.Reflection.Metadata.Cil?

Should decoded tokens be referenced as CilEntity objects instead of strings on CilStringInstruction?

In order to do this there should be an implementation of CilMemberReference (FieldRef, MethodRef) and CilMethodSpecification.

cc @AlexGhiondea

Add non-invariant culture compatibility tests

Measure and improve code coverage

Marshal descriptors on CilFields and CilParameters and on System.Reflection.Metadata.Cil

Need to decode marshal descriptors on CilFields and on CilParameters.

See:

https://github.com/dotnet/corefxlab/blob/master/src/System.Reflection.Metadata.Cil/src/CilField.cs#L144-L158

https://github.com/dotnet/corefxlab/blob/master/src/System.Reflection.Metadata.Cil/src/CilParameter.cs#L121-L135

In order to be able to do this it is needed to have a Marshal Decoder on System.Reflection.Metadata.Decoding, issue #2688

cc @AlexGhiondea

Alternative string formatting design

I'd be interested in hearing your thoughts on a library I've been working on that's similar in spirit to what you're doing here: https://github.com/MikePopoloski/StringFormatter

Mostly I needed a way to do formatting without allocations during my game loop. I think what I came up with is a pretty nice API.

x86

I have a Windows 10 x64 Preview machine running Visual Studio 2015. When I git clone the corefxlab project, I am able to compile a 64bit version of the console application and it works perfectly including nuget grabbing all necessary files. If I set application to compile as x86, the application compiles but close immediately. Would you expect that this would work with x86?

Type Implementations on CilTypeDefinitions: System.Reflection.Metadata.Cil

Missing type implementations IEnumerable Property on CilTypeDefinitions.

Probably a CilTypeImplementation class should be create them to represent them and visit them with CilToStringVisitor.

implements Interface on type definition on CilToStringVisitor missing when type signature being dumped to writer.

cc @AlexGhiondea

CilFile missing: System.Reflection.Metadata.Cil

File declaration missing to implement and dump on CilToStringVisitor.

Example:

.file /*26000005*/ nometadata normnfkd.nlp
    .hash = (5D C0 2A 17 57 BB CC BF 38 84 66 BF 65 80 86 4D   // ].*.W...8.f.e..M
             99 13 6E 9C ) 

cc @AlexGhiondea

Composite format should support precision

The following test will fail:

        var formatter = new StringFormatter();
        formatter.Format("{0:D2}", 1);
        Assert.Equal("01", formatter.ToString());

Add better documentation

This includes better wiki

Consider porting the codebase to ByteSpan

The whole codebase needs to be ported to ByteSpan. Otherwise it's not possible to make UTF8 encoding of small strings competitive with native code implementation, as there are too many calls to pin and unpin the byte[] buffers.

For this to happen, ByteSpan needs to be made public and BufferPool needs to probably use native memory instead of pinning. I am not sure if these tradeoffs are worth it.

Alternatively, I would need to wait for a real GC tracked Span before seeing great runtime perf.

Add support for dates and times to SymbolTable

Single quoted names on System.Reflection.Metadata.Cil

When an identifier (type, method, file, module, field, property, etc) name is not a valid msil name.

i.e:

<>_System

It needs to be single quoted in order to be valid and not have a syntax error.

i.e:

'<>_System'

cc @AlexGhiondea

Simplify BufferPool

The pool is quite complex and I don't think the complexity is worth it. Interlocked operations are very expensive, and I think performance sensitive scenarios should use a pool per thread. Also, the interlocked synchronization is not much faster than locks but it causes some buffers to be dropped on the heap.

System.Reflection.Metadata.Cil shouldn't use yield return on IEnumerators.

This shouldn't be used because this is inefficient an sometimes unpredicatble.

See:
https://github.com/dotnet/corefxlab/blob/master/src/System.Reflection.Metadata.Cil/src/CilAssembly.cs#L324-L332

There is a lot of places where this is used.

cc @AlexGhiondea

Can BufferFormatter and StringFormatter be combined?

Add support for surrogates

One of the tests has the following line commented:
//utf8Writer.Append("\uD950\uDF21"); // TODO: surrogate pair; this is NIY

Uncommenting this and running tests will cause test failures. This is because the library does not support surrogates. The fix should be moderately difficult.

Add float and double compatibility tests

Resolution Scope on CilTypeReference: System.Reflection.Metadata.Cil

The resolution scope on CilTypeReference is not yet decoded. This could be one of the following options:

-String (resolution scope name)

-CilEntity (CilModuleDefinition, CilModuleReference, CilTypeReference, CilAssemblyReference)

See https://github.com/dotnet/corefxlab/blob/master/src/System.Reflection.Metadata.Cil/src/CilEventDefinition.cs#L80-L91

cc @AlexGhiondea

What is the criteria of graduation from corfxlab?

Microsoft.Experimental.Collections has a useful implementation of MultiValueDictionary, but it now appears abandoned in alpha state. A discussion on corefx indicates corefxlab is a good place for it, but what is the process to bring it to release from here?

Implement numeric "F" format

Repro:

1. Uncomment //GenerateTest(writer, type, 'F', maxPrecision); in src\System.Text.Formatting\GenerateCompatibilityTests\Program.cs
2. Run the program. It will generate additional format tests
3. Build the formatting library and run the test (run build.cmd from command line)
   Result:
   Some formatting tests will fail.

HTTP headers are ASCII not UTF8

https://tools.ietf.org/html/rfc7230#section-3.2.4

Historically, HTTP has allowed field content with text in the
ISO-8859-1 charset [ISO-8859-1], supporting other charsets only
through use of [RFC2047] encoding. In practice, most HTTP header
field values use only a subset of the US-ASCII charset [USASCII].
Newly defined header fields SHOULD limit their field values to
US-ASCII octets. A recipient SHOULD treat other octets in field
content (obs-text) as opaque data.

So https://github.com/dotnet/corefxlab/blob/master/demos/LowAllocationWebServer/HttpRequestParser.cs
should be parsing bytes based on US-ASCII rather than UTF8 data

Can the formatting library support URL, XML, and HTML encodings?

It would be great if we allowed text generators to compose, i.e. if the output needs to be encoded, formatting DateTime (for example) would result in already encoded output.

Add TimeSpan compatibility tests

Run HelloWorld on the CoreCLR my own build

I did the following steps described at http://blogs.msdn.com/b/dotnet/archive/2015/02/03/coreclr-is-now-open-source.aspx?Redirected=true&PageIndex=2

1. Modify CoreCLR to your heart’s content
2. Build CoreCLR via build.cmd x64 release
3. Copy the following files from coreclr\binaries\x64\release to corfxlab\demos\CoreClrConsoleApplications\HelloWorld\NotYetPackages\CoreCLR:
   coreclr.dll
   CoreConsole.exe
   mscorlib.dll
4. Rebuild HelloWorld.sln (either using the command line or in VS)
   It seems nothing happen. Why should I copy files to the folder 'NotYetPackages\CoreCLR', and how the HelloWorld.exe find the newly added dlls/exes?

Argument Exception when trying to run dnu restore

Hi I am trying to run the demo Hello World program on Centos7 but getting the error. It isn't able to install the dependencies.

⇒ dnu install System.Console
Microsoft .NET Development Utility CoreCLR-x64-1.0.0-beta7-12264

GET https://www.nuget.org/api/v2/
GET https://www.nuget.org/api/v2/FindPackagesById()?id='System.Console'
Warning: FindPackagesById: System.Console
The AddressFamily 117 is not valid for the System.Net.IPEndPoint end point, use InterNetwork instead.
Parameter name: socketAddress
GET https://www.nuget.org/api/v2/FindPackagesById()?id='System.Console'
Warning: FindPackagesById: System.Console
The AddressFamily 117 is not valid for the System.Net.IPEndPoint end point, use InterNetwork instead.
Parameter name: socketAddress
GET https://www.nuget.org/api/v2/FindPackagesById()?id='System.Console'
Error: FindPackagesById: System.Console
The AddressFamily 117 is not valid for the System.Net.IPEndPoint end point, use InterNetwork instead.
Parameter name: socketAddress
System.ArgumentException: The AddressFamily 117 is not valid for the System.Net.IPEndPoint end point, use InterNetwork instead.
Parameter name: socketAddress

Implement numeric 'E' format

Repro:

1. Uncomment //GenerateTest(writer, type, 'E', maxPrecision); in src\System.Text.Formatting\GenerateCompatibilityTests\Program.cs
2. Run the program. It will generate additional format tests
3. Build the formatting library and run the test (run build.cmd from command line)
   Result:
   Some formatting tests will fail.

Move off hacky parsers

The formatting library uses several small hacky parsers (mainly to parse composite formatting strings). We should create a real non-allocating parsing library and use it instead.

Implement non-allocating float and double formatting

Currently, the floating point formatter has a temporary implementation that simply calls ToString.
This needs to be implemented properly, but such implementation is very tricky.

System.Numerics.Matrices projects don't build

When I try to build the main and test projects, I get the following error:
.\corefxlab\src\System.Numerics.Matrices\test\System.Numerics.Matrices.Tests.csproj : error MSB4057: The target "BuildAndTest" does not exist in the project.
.\corefxlab\src\System.Numerics.Matrices\src\System.Numerics.Matrices.csproj : error MSB4057: The target "BuildAndTest" does not exist in the project.

Write more performance tests and optimize runtime performance

1. Formatting UTF8 strings does not have performance tests
2. Number formatter is calling ToUpper on the format character. This regressed perf a bit in the past.
3. Math.DivRem (used in hot formatting loop) generates two idiv instructions. it should be just one.
4. It would be worth to again profile the perf tests. There were significant changes since last time.
5. Composite formatting does not have perf or memory tests.

Add DateTimeOffset compatibility tests

Should parameter and local types be represented as a CilEntity on System.Reflection.Metadata.Cil?

Should parameter and local types be represented as CilEntity to have either a CilTypeReference, CilTypeDefinition, CilTypeSpecification?

This would be such as in CilEventDefinitions.

See:
https://github.com/dotnet/corefxlab/blob/master/src/System.Reflection.Metadata.Cil/src/CilEventDefinition.cs#L80-L91

Add support for non-invariant cultures

There needs to be a way to get FormattingData instances for other cultures.
There are two ways to do it: either get the data from .NET CultureInfo or write a parser for the standard Unicode database.

Precondition APIs need to be better

The should allow verifying ranges.
Also, when a failure happens, better diagnostics information should be easily available.

System.Reflection.Metadata.Cil does not emit interface implementations on types

Looks like a known issue based on the NotImplementedException, opening this issue to track it.