GithubHelp home page GithubHelp logo

developmentalmadness / sqlstream Goto Github PK

View Code? Open in Web Editor NEW
6.0 6.0 1.0 7.65 MB

The most performant way to send arrays or bulk-import (streaming) from .NET to Sql Server via Table-Valued Parameters (TVP). For use with Sql 2008 or later.

C# 100.00%
c-sharp sql-server tvp

sqlstream's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

stjordanis

sqlstream's Issues

Write a better README

The readme is currently too minimal. Include some sample code as well as why anyone would want to use TVPs. Include some stats.

Also, include the fact that the stream is buffered and the actual output to Sql Server is asyncronous and uses lock-free concurrency to share the data between threads.

Fix Log4Net

I don't like the current defaut log4net output when running the test in NUnit. Find out why the configuration in the setup is being ignored.

Use ring buffer to reduce memory used

The current implementation using queues of buffers/chunks doesn't enforce any memory limits. The ring buffer a la LMAX would prevent running out of memory with large files sending data to a slow/overloaded Sql Server.

Support breaking each buffer into separate calls to Execute()

Sql Server doesn't start the stored procedure until the stream has completed (IEnumerator.MoveNext() == false). In cases where the full dataset may be too much for Sql Server to process at once. The client would prefer to call ExecuteXXX() each time the buffer flushes instead of buffering to the same stream. This should all work internally w/o any changes to the interface other than adding a new value to the SqlStreamBehavior enum.

NOTES:

  • This option should be ignored by ExecuteReader and ExecuteScalar, or we can offer a callback for these so the client can handle what they want to do with each response.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.