aumcode / nfx

C# Server UNISTACK framework [MOVED]

The NFX project has been superseded by Azos Project https://github.com/azist/azos

NFX - .NET Unified Software Stack + Big Memory

Azos ANNOUNCEMENT

Azos is based on .Net Standard 2, having all cli tools targeting both .NET FX 4.7.1+ and .NET Core 2+. Azos does not support fx < 4.7.1 (though possible).

NFXv3* (this repository) will continue to target .NET 4.5/Mono and may get some support for major bug fixes however all new feature development is going to happen in the NEW REPO for Azos.

Server UNISTACK (unified full software stack, no 3rd paty libs) framework.

Documentation http://nfxlib.com

nfx's Issues

Samples?

Your stuff looks interesting, would be good to have a few sample projects to play with.

Create TypedRow serializer "A(daptive)Row" which is version-change tolerant and faster than Slim

• Uses var bit encoding
• Versioned stream
• Dont use string prop names, use 8 byte long monikers (8 char ASCII)
• If cant match field write to Amorphous data
• Compile into C# for embedding into pre-compiled Core.CLR

BigMemory - add support for memory-mapped Cache (including the keys)

Add trim to Slim serializer TS_refPool, as it gets stuck after large object graph processing

Need to add auto-trim ability to cap the large graph.
The case when a customer serializes a huge graph - leaves the refPool with
100s of thousands of instances dangling on the heap.

Need to introduce MAX_REF_POOL_CACHE_ENTRIES and say cap at 1000 or something like this

Add Pile.Put(PilePointer) to overwrite exisitng pointer, prealloc buffer + linking

Something akin to:

    /// <summary>
    /// Tries to put the new object over an existing one at the pre-define position.
    /// The pointer has to reference a valid allocated block.
    /// If object fits in the allocated block returns true, otherwise tries to create an internal link
    /// to the new pointer which is completely transparent to the caller. The linking may be exlicitly disabled
    /// in which case the method returns false when the new object does not fit into the existing block
    /// </summary>
    /// <param name="obj">A new/modified CLR object to put into the Pile over an existing one</param>
    /// <param name="lifeSpanSec">
    ///  Optional lifeSpanSec will auto-delete object after the interval elapses if
    ///  the pile SupportsObjectExpiration and SweepExpiredObjects is set to true
    /// </param>
    /// <param name="ptr">The pointer to the existing valid allocated block</param>
    /// <param name="link">False to prohibit internal pointer linking</param>
    /// <returns>True if object was inserted, false otherwise (i.e. when linking is false)</returns>
    bool Put(PilePointer ptr, object obj, uint lifeSpanSec = 0, bool link = true);

NamedInterlocked need to take and zero-out longs atomically

87b39a1 @itadapter fixed with snapshots

Text logs dont respawn on date change

Erlang receive RPC

Thank you for this great library!
Just got one question, is there a method in Erlang module that can receive RPC calls from Erlang node? Thx.

Decorate IArchiveLoggable with IBsonSer/Deser to ensure serializability

Add support for IBSONSer/Deser

Reconstruct type based on BSON doc _t which will map to GuiodTypeResolver

Have you already added PDF generation stuff?

NFX.Security.Password manager - pwd token parsing leaks

The exception leaks from invalid pwd token when it can not be parsed.

Maybes it is a problem in configsec manager?

Casting issues in NFX.Erlang

The implicit cast from ErlLong to int is very dangerous and indeed led to errors in the library code. Here is a snippet from the ErlOutputStream.cs file which contains a bug that we discovered recently:

public void WriteLong(ErlLong l) { writeLong(l, l < 0); }
private void writeLong(long l, bool isNegative);

If l is positive but greater than int.MaxValue then the condition will evaluate to true meaning l will be serialized as a negative number (because l is cast to an int for the less-than operator). The solution is simple:

public void WriteLong(ErlLong l) { writeLong(l, l < 0L); }

I suggest you make the ErlLong to int cast explicit to avoid these subtle bugs in the library code and make clear for the users where data loss might happen on their part.

BigMemory Pile - add Memory Mapped File support

NLSMap - get rid of Dictionary<> for speed

Use linear search instead of Dictionary<>

Command-line options parsing

Implement case-sensitive command-line options parsing

Mongo lost TCP connection

Hey dude! I have this issue now: Mongo server gets turned off. I get a message from the closed socket and then it gets stuck as I see in log "Invalid operation" on closed socket or smth like this. What it looks like to me - something gets stuck, like a socket. I did not have time to investigate. How to reproduce: connect to mongo using your nfx driver then tear the server down, then bring it back alive - my client app is lost now. Any help will be highly appreciated!!!! I use Mongo for high-speed queue in my project and your driver showed substantial speed-up over the "official"one provided by them

thanks,
Chad

Create CMS concept for WAVE

Need to have CMS functionality for portals.
Base on virtual file system - revisions - approve/unapprove by content admin.
Basically a catalog of structured resources: files, configs, NLSmaps, menus
and page templates . tbd email templates

Add primary X color extractor function

Add PileCache.TableMode{Speculative = 0, DURABLE} mode per table to PileCache

In this mode perform rehashing into a chunk block stored in Pile,
this way no override will ever happen and all the the data will be guaranteed to be preserved

Move Erlang to NFX.Erlang assembly

This is part of restructuring.

JS Compiler: Return text representation

Please, add to JS Compiler an ability to get a text representation of created DOM element object.

Packaging

Perhaps packaging things like the erlang connectivity outside of the nfx package would be reasonable?

There is a whole ton of stuff in nfx and I imagine the set of people who want to use everything in nfx does not overlap well with people who want to use erlang connectivity.

Also I noticed lack of things like SecureString is this because these are bloated or?

Need Nuget Update

Need latest Nuget package

JS Compiler: current element pointer

Please, add to JS Compiler source laconf code an ability to get a pointer to current DOM element.

Pile: Improve String and byte[] performance

Directly write byte[] and String bypassing Slim layer.
This will improve speed for storing pre-rendered JSON and other textual data

Get rid of NUnit altogether

Can't change value in input with type='date'

Can't change value in input with type='date' in WAVE.RecordModel

Refactor WV.js using LJS compiler and includes

Erlang list parsing is incomplete

According to the Erlang external term format documentation the list term has a Tail that usually is a Nil (or any other type, for that matter). Current implementation of the list parser misses a Nil for the tail, therefore parsing nested lists incorrectly.

Here is a fix:

public ErlList ReadList()
{
    int arity = readListHead();
    if (arity == 0)
        return new ErlList();

    var list = new List<IErlObject>(arity);
    for (int i=0; i < arity; i++)
        list.Add(Read());

    // Read list tail
    if ((ErlExternalTag)m_Buffer[m_Position] == ErlExternalTag.Nil) {
        Read();
    } else {
        list.Add(Read());
    }

    return new ErlList(list, false);
}

Also, the ErlExternalTag.String case is superfluous in the readListHead() method in ErlInputStream.

JS Compiler - add includes

JS Compiler: direct insertions into js code

Please, extend syntax of JS Compiler to make insertions of source laconf into js code directly, like:

var dlg = new WAVE.GUI.Dialog({
  header: "Some Infromation",
  body: /*** div { h2="Header" {} p="Paragraph" {}  }  ***/
  ...
});

Configuration thread safety - review

Do we need loc{} on attributes and section gets?
Better yet create new instance on mutations, not gets as confs are read-only primarily.

On the other hand this micro optimization is not that important as config is mostly string based and "heavy" anyway

The BSON serializer WriteCString() bug with wide unicode chars

 [System.ArgumentException] The output byte buffer is too small to contain the encoded data, encoding 'Unicode (UTF-8)' fallback 'System.Text.EncoderReplacementFallback'.

from BSON.BinUtils.WriteCSTring(stream, value)

Get rid of NUnit Assert.AreEqual()

The
Assert.AreEqual() is improperly implemented, i.e. it gives precedence to IEnumerable<>
which is used along with SequenceEquals() which is wrong.

That makes testing unpredictable in many places where some class may change interface and
that would completely silently change all of the unit tests meanings.
ArEqual() must use object.Equal only.

JS Compiler: add support of NFX markup

Please, add to JS Compiler support of NFX markup.

How to remove old log files

How I can remove old log files? Is there some attribute for it?

JS Compiler: fix interaction with NHT code

Please, fix JS Compiler to correctly parse NHT code insertions like as:

div {
  span="?[:SomeVariable]" {}
}

GLUE: Refactor to not use message instances and have ability to execute sync-only calls with custom bindings

Need to not require Message instance at all (it is an extra allocation).
Is it possible to bypass CallSlot for sync-only bindings?

Do we need to constrain binding with Contact?

Create TypedRow serializer which is version-change tolerant and more efficent than Slim

It may be more efficient because it does not need to store reference cycles etc...

ErlMap causes unit test NFX.NUnit.Erlang.ErlEncodingTest.ErlTermSerializeTest failure

@filmor @saleyn
The ErlEncodingTest.ErlTermSerializeTest fails with:

NFX.NUnit.Erlang.ErlEncodingTest.ErlTermSerializeTest:
NFX.Erlang.ErlException : Unsupported type of element ErlMap: ErlMap

See:

AROW Serializer: Add interface to perform custom serialization of unsupported types

IArowSupporter / IArowSerializable

How do I run all integ tests?

I want to run all integrations but apparently there are configs missing or smth. Please help as I want to check out your stripe thing

Contributing

Are there any particular things you would like followed on contributions?

One thing I may see get some pushback in the style is the use of m_ as most people will have to reconfigure their tooling for it.

JSONReader.ToRow() - need to respect Backend Name per target

Use Bloom Filter in /NFX/ApplicationModel/Pile/LocalCache

Optimize caching table by providing flag to enable Bloom filter.
This can significantly improve performance by filtering out one-time hits that are rarely used.
Need to design multi hashing strategy for Table

.NET Standard

After reading https://www.infoq.com/articles/Big-Memory-Part-2 I want to try NFX Pile in the ClosedXML project to improve memory and performance issues. However, we have a separate goal of support .NET Standard at some stage. Do you have any idea of the effort involved in making NFX Pile .NET Standard compliant?

Locale cache overwrites entries when table key is not an <int>. See test modified from [PutGetWithoutMaxCap]

[Test]
public void T080_PutGetWithoutMaxCap_key_as_string()
{
using (var cache = makeCache())
{
var tA = cache.GetOrCreateTable("A");

            const int CNT = 80000000;
            for (var i = 0; i < CNT; i++)
            {
                var pr = tA.Put(i.ToString(), "value" + i.ToString(), priority: 10);
                // Console.WriteLine("{0} -> {1}", i, pr);
                Assert.IsTrue(pr == PutResult.Inserted);//fails here even though key values are unique
            }
        }
    }

wv.all.min.js is missing in NFX.Wave - build error

Templatization\StockContent\Embedded\script\wv.all.min.js file is missing in Source code. It was probably included by mistake or may be some pre-build commands missing?

Revise DisposableObject thread safety

use Interlocked for reference comparison.
Also revise DisposeAndNull()