GithubHelp home page GithubHelp logo

http-multipart-data-parser / http-multipart-data-parser Goto Github PK

View Code? Open in Web Editor NEW
294.0 30.0 112.0 751 KB

A C# Http Multipart/form-data parser that works correctly on binary data and very large files.

C# 98.32% PowerShell 1.60% Shell 0.08%

http-multipart-data-parser's Introduction

Http Multipart Parser

License Sourcelink Build status tests Coverage Status CodeFactor

Release Notes NuGet (stable) MyGet (prerelease)
GitHub release Nuget MyGet Pre Release

About

The Http Multipart Parser does it exactly what it claims on the tin: parses multipart/form-data. This particular parser is well suited to parsing large data from streams as it doesn't attempt to read the entire stream at once and procudes a set of streams for file data.

Installation

The easiest way to include HttpMultipartParser in your project is by adding the nuget package to your project:

PM> Install-Package HttpMultipartParser

.NET framework suport

  • The parser was built for and tested on .NET 4.8, .NET standard 2.1, .NET 5.0 and .NET 6.0.
  • Version 5.1.0 was the last version that supported .NET 4.6.1, NET 4.7.2 and .NET standard 2.0.
  • Version 2.2.4 was the last version that supported older .NET platforms such as .NET 4.5 and .NET standard 1.3.

Usage

Non-Streaming (Simple, don't use on very large files)

  1. Parse the stream containing the multipart/form-data by invoking MultipartFormDataParser.Parse (or it's asynchronous counterpart MultipartFormDataParser.ParseAsync).
  2. Access the data through the parser.

Streaming (Handles large files)

  1. Create a new StreamingMultipartFormDataParser with the stream containing the multipart/form-data
  2. Set up the ParameterHandler and FileHandler delegates
  3. Call parser.Run() (or it's asynchronous counterpart parser.RunAsync())
  4. The delegates will be called as data streams in.

Examples

Single file

// stream:
-----------------------------41952539122868
Content-Disposition: form-data; name="username"

example
-----------------------------41952539122868
Content-Disposition: form-data; name="email"

[email protected]
-----------------------------41952539122868
Content-Disposition: form-data; name="files[]"; filename="photo1.jpg"
Content-Type: image/jpeg

ExampleBinaryData012031203
-----------------------------41952539122868--
// ===== Simple Parsing ====
// You can parse synchronously:
var parser = MultipartFormDataParser.Parse(stream);

// Or you can parse asynchronously:
var parser = await MultipartFormDataParser.ParseAsync(stream).ConfigureAwait(false);

// From this point the data is parsed, we can retrieve the
// form data using the GetParameterValue method.
var username = parser.GetParameterValue("username");
var email = parser.GetParameterValue("email");

// Files are stored in a list:
var file = parser.Files.First();
string filename = file.FileName;
Stream data = file.Data;

// ==== Advanced Parsing ====
var parser = new StreamingMultipartFormDataParser(stream);
parser.ParameterHandler += parameter => DoSomethingWithParameter(parameter);
parser.FileHandler += (name, fileName, type, disposition, buffer, bytes, partNumber, additionalProperties) =>
{
    // Write the part of the file we've received to a file stream. (Or do something else)
    filestream.Write(buffer, 0, bytes);
}

// You can parse synchronously:
parser.Run();

// Or you can parse asynchronously:
await parser.RunAsync().ConfigureAwait(false);

Multiple Parameters

// stream:
-----------------------------41952539122868
Content-Disposition: form-data; name="checkbox"

likes_cake
-----------------------------41952539122868
Content-Disposition: form-data; name="checkbox"

likes_cookies
-----------------------------41952539122868--
// ===== Simple Parsing ====
// You can parse synchronously:
var parser = MultipartFormDataParser.Parse(stream);

// Or you can parse asynchronously:
var parser = await MultipartFormDataParser.ParseAsync(stream).ConfigureAwait(false);

// From this point the data is parsed, we can retrieve the
// form data from the GetParameterValues method
var checkboxResponses = parser.GetParameterValues("checkbox");
foreach(var parameter in checkboxResponses)
{
    Console.WriteLine("Parameter {0} is {1}", parameter.Name, parameter.Data)
}

Multiple Files

// stream:
-----------------------------41111539122868
Content-Disposition: form-data; name="files[]"; filename="photo1.jpg"
Content-Type: image/jpeg

MoreBinaryData
-----------------------------41111539122868
Content-Disposition: form-data; name="files[]"; filename="photo2.jpg"
Content-Type: image/jpeg

ImagineLotsOfBinaryData
-----------------------------41111539122868--
// ===== Simple Parsing ====
// You can parse synchronously:
var parser = MultipartFormDataParser.Parse(stream);

// Or you can parse asynchronously:
var parser = await MultipartFormDataParser.ParseAsync(stream).ConfigureAwait(false);

// Loop through all the files
foreach(var file in parser.Files)
{
    Stream data = file.Data;

    // Do stuff with the data.
}

// ==== Advanced Parsing ====
var parser = new StreamingMultipartFormDataParser(stream);
parser.ParameterHandler += parameter => DoSomethingWithParameter(parameter);
parser.FileHandler += (name, fileName, type, disposition, buffer, bytes, partNumber, additionalProperties) =>
{
    // Write the part of the file we've received to a file stream. (Or do something else)
    // Assume that filesreamsByName is a Dictionary<string, FileStream> of all the files
    // we are writing.
    filestreamsByName[name].Write(buffer, 0, bytes);
};
parser.StreamClosedHandler += () 
{
    // Do things when my input stream is closed
};

// You can parse synchronously:
parser.Run();

// Or you can parse asynchronously:
await parser.RunAsync().ConfigureAwait(false);

Licensing

This project is licensed under MIT.

http-multipart-data-parser's People

Contributors

bkrbtc avatar catester avatar dnik2000 avatar dtewinkel avatar geoperez avatar jericho avatar kevogich avatar mrjoe avatar nakamura2000 avatar otf avatar richardpoes avatar vodurden avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

http-multipart-data-parser's Issues

Improve .Net Standard support

I propose the following improvements:

  • One single solution (instead of one for .NET 4.0 and another for netcore). This solution will produce both versions of the library.
  • Target net452 instead of net40 since net40, net45 and net451 are no longer supported by Microsoft
  • Target netstandard instead of netcore because netcore is intended for apps such as console apps for example while netstandard is intended for libraries. As it currently stands, a library targeting netstandard is prevented from using the HttpMultipartDataParser nuget package because it targets netcore.
  • use xUnit for unit testing because it's multi-target friendly (meaning you write your unit tests only once and it will automatically run then against net452 and netstandard versions automatically)

I recently implemented these improvements in my own library StrongGrid. I now have a single solution which targets net452 and netstandard1.3 so I am familiar with what needs to be done. My motivation is that I would like to reference your library but I can't since you don't support netstandard.

Couldn't install assembly to GAC

The latest 2.2.3 is not signed and doesn't have a strong name, so couldn't be installed to GAC. I have verified all versions below with ILSpy and found that the latest signed is 2.1.6 after which the project was migrated to NETCORE. I suspect that migration has a side effect of loosing signing. Please verify.

Benchmark

I think it would be great to have benchmark in order to measure the speed of the parser as well as the amount of memory allocated. This would serve as a baseline if we want to make performance improvements.

I volunteer to submit PR if you are interested. (I can write the code to produce the benchmark but I would need help with generating various samples such as small, medium, large multipart forms, forms with many small files, many large files, etc.)

MultipartFormDataParser.cs Line 561 Bug

HI
Because "content-type" and "content-disposition" keys in parameters dictionary save like "Content-Type" and "Content-Disposition" so this to line :

var contentType = parameters.ContainsKey("content-type") ? parameters["content-type"] : "text/plain";
var contentDisposition = parameters.ContainsKey("content-disposition") ? parameters["content-disposition"] : "form-data";

must change to :
var contentType = parameters.ContainsKey("Content-Type") ? parameters["Content-Type"] : "text/plain";
var contentDisposition = parameters.ContainsKey("Content-Disposition") ? parameters["Content-Disposition"] : "form-data";

XPROJ is no longer supported

Have you considered upgrading the project files to the supported format so that we can open your project in VS.NET 2019?

If you are interested, I can submit a PR.

The given key 'name' was not present in the dictionary

I am using a service which gives me responses in the following format:

--MIMEBoundary_db1ef2fb8e2d1f5a9c90f0eed9a482da52be09f800a0069e
Content-Type: text/xml; charset=utf-8
Content-Transfer-Encoding: binary
Content-ID: <[email protected]>

<?xml version="1.0" encoding="utf-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><response>1</response></soapenv:Body></soapenv:Envelope>
--MIMEBoundary_db1ef2fb8e2d1f5a9c90f0eed9a482da52be09f800a0069e
Content-Type: application/zip
Content-Transfer-Encoding: binary
Content-ID: <[email protected]>

PK���

The PK��� would be the content of a zip file that I'm trying to load into a byte array.

I tried to use your package to that:

var stream = await response.Content.ReadAsStreamAsync();
var parser = new MultipartFormDataParser(stream);
FilePart zipFile = parser.Files.First();

When I try to do that I receive the following error though:

System.Collections.Generic.KeyNotFoundException: The given key 'name' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at HttpMultipartParser.StreamingMultipartFormDataParser.ParseParameterPart(Dictionary`2 parameters, RebufferableBinaryReader reader)
   at HttpMultipartParser.StreamingMultipartFormDataParser.ParseSection(RebufferableBinaryReader reader)
   at HttpMultipartParser.StreamingMultipartFormDataParser.Parse(RebufferableBinaryReader reader)
   at HttpMultipartParser.MultipartFormDataParser..ctor(Stream stream, String boundary, Encoding encoding, Int32 binaryBufferSize)
   at HttpMultipartParser.MultipartFormDataParser..ctor(Stream stream)

Is your package able to give me the byte array for the content in the second part of the response? If yes: What am I doing wrong?

Optimizing parsing speed

Hi Jake,
Thanks for your efforts in creating this great parser, it's really well designed.
I have some suggestions though:

  • When I read this on the description: "This particular parser is well suited to parsing large data from streams as it doesn't attempt to read the entire stream at once and procudes a set of streams for file data.", I was glad that at last I found a parser which doesn't store all the data in memory. However I was surprised when I saw you were storing the resulting file data in a MemoryStream at ParseFilePart method. In my opinion, this kills the premise. No big deal though, I implemented an interface and integrated it with the parser to call methods like BeginFile, WriteData, EndFile when necessary. This way I can redirect the stream whereever I want without any memory pressure on the server. This is especially good for large files (> 2GB).
  • Do you think the parser can be optimized further? On my local tests, I can see 20 Mb/s with the parser which is not bad but when I upload via pure stream (chunked though) I can see 200 Mb/s to 300 Mb/s. Note that these speed tests exclude processing the file, i.e. writing to disk. I used a buffer size of 80kb. I came along the below post which discusses parsing file uploads at 500 mb/s:
    http://debuggable.com/posts/parsing-file-uploads-at-500-mb-s-with-node-js:4c03862e-351c-4faa-bb67-4365cbdd56cb
    If they can do this with node.js , we should be able to do it better with C# :) I know you are using KMP Algorithm and the poster seems to be using Boyer-Moore algorithm with an additional trick.

Let me know.

Doesn't seem to support international encoding.. BinaryStreamStack line 293 throwing error.

when trying to parse a file containing the following sniplet the accent é in ( Bonjour poignée ) is not parsed correctly.. and instead BinaryStreamStack line 293 is throwing the following error when attempting to read the é character.

Error:
"The output char buffer is too small to contain the decoded characters, encoding 'Unicode (UTF-8)' fallback 'System.Text.DecoderReplacementFallback'.\r\nParameter name: chars"

here is a sniplet of what my multipart/form-data input file looks like..

--boundary_.oOo._MjQ1NTU=OTk3Ng==MjcxODE=
Content-Disposition: form-data; name="psAdTitle"

Bonjour poignée
--boundary_.oOo._MjQ1NTU=OTk3Ng==MjcxODE=

Out of memory exception

Hi Jake,

I am trying out your multipart parser, but am running into some issues with out of memory exceptions when constructing an instance. The file.bin file is 2.09 GB in size.

Here is my code, in VB:

Dim fileStream As New System.IO.FileStream("file.bin", IO.FileMode.Open, IO.FileAccess.Read)
Dim parser As New HttpMultipartParser.MultipartFormDataParser(fileStream, System.Text.Encoding.UTF8)

Running this code returns an OutOfMemoryException. Any ideas why this is? Thanks! :D

Support for .Net 4.0

I want to use this package with .Net 4.0 but the latest NuGet package does not support it.
Is there an older version of the NuGet package that I can install?

File Name with diacritics causes the problem

I found a problem when I'm trying to upload file with diacrtics in filename

filename: Domašov návrh výměny svítidel.xlsx

data dump:
raw stream data:
------WebKitFormBoundaryL5QOWPMWAFioo0QJ
Content-Disposition: form-data; name="file"; filename="Domašov návrh výměny svítidel.xlsx"
Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet

PK��� � � ! t6Z¦z� „� � ��[Content_Types].xml ˘��(  �
........ file data ...

file data from parser:
t

PK��� � � ! t6Z¦z� „� � ��[Content_Types].xml ˘��(  �
........ file data......

there are 5bytes more than in original file, I guess because there are 5more chars in name
Domašov návrh vĂ˝mÄ›ny svĂ­tidel.xlsx than in original Domašov návrh výměny svítidel.xlsx

Index out of range exception if the boundary was detected at the end of a file, but the end of the boundary is outside of the current buffer

Hello,

thank you very much for great solution!
I used it to upload about a million files in SharePoint.
it worked for most of the files, but sometimes I stumbled upon an "Index out of range" exception in the CalculateNewlineLength method at line 372 in MultipartFormDataParser.cs.

The reason for this exception was that the length of the byte array variable "data", which is the currently buffered part of the file stream, was smaller than the value of the variable "offset".
This error can be reproduced, if a boundary between a file stream and the metadata or another file is detected at the end of the current buffer (the variable "fullBuffer" in the method "ParseFilePart"), but the end of the boundary is outside of the buffer.

In order to fix this I changed the line 506 in MultipartFormDataParser.cs to:
if (endPos != -1 && fullBuffer.Length > endPos + endPosLength + 2)
Here I make sure, that the end of the boundary is inside the current buffer.

Has someone else seen this error?
Can you please verify the issue and apply the fix?

Thanks.

Question anout upload 2+ photos

Hi, I am developing a ASP.NET Web API project and you HttpMultipartDataParser helps me a lot. Recently, I met a confussion. When I want to upload 1 photo, it will be OK whether I assigned a name for the photo or not. However, when I want to upload 2+ photos, there is just 1 photo detected and uploaded if I don't assigned a name for every photo. Just like that:

default

(Yes, I am a Chinese. 选择文件 in English means "select a file".)

If I assign a name for every photo, then all photos will be uploaded as I wish.

2

Hence, my question is that may I upload 2+ photos without assigning names?

Automate the release process

To ensure we are consistent, I will automate the release process.

  • Use gitflow
  • Use a Cake script to increase version number, build, run unit tests and pack
  • Use Appveyor to automatically build the project when a commit is detected on some pre-determined branches (such as master, release/*, etc.)
  • Create a 'milestone' for each release and assign issues to the appropriate milestone.
  • Create a 'release' in GitHub and document bug fixes, enhancements, etc included in a given release (This is typically a manual step but I have a Cake script that can automate the creation of the release and also summarize resolved issues associated with the milestone). Users can refer to this 'release' after a package is published to Nuget for release notes.
  • By default, the 'release' is in draft mode. As soon as the release is published, a build is triggered on Appveyor and package published to nuget
  • Organize files to make it easier to automate process. For instance, move source code in a new folder called ’source’, etc.
  • Add various configuration file in root folder such as appveyor.yml, .editorconfig, etc.

Posting Large Files gives OutofMemoryException

I posted a 200mb file and it is throwing an error: Exception of type 'System.OutOfMemoryException' was thrown.

StackTrace:
at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at HttpMultipartParser.MultipartFormDataParser.<.ctor>b__1(String name, String fileName, String type, String disposition, Byte[] buffer, Int32 bytes) in c:\Users\RKsherman\Downloads\Http-Multipart-Data-Parser-master\Http-Multipart-Data-Parser-master\HttpMultipartParser\MultipartFormDataParser.cs:line 216
at HttpMultipartParser.StreamingMultipartFormDataParser.ParseFilePart(Dictionary`2 parameters, RebufferableBinaryReader reader) in c:\Users\RKsherman\Downloads\Http-Multipart-Data-Parser-master\Http-Multipart-Data-Parser-master\HttpMultipartParser\StreamingMultipartFormDataParser.cs:line 560
at HttpMultipartParser.StreamingMultipartFormDataParser.ParseSection(RebufferableBinaryReader reader) in c:\Users\RKsherman\Downloads\Http-Multipart-Data-Parser-master\Http-Multipart-Data-Parser-master\HttpMultipartParser\StreamingMultipartFormDataParser.cs:line 705
at HttpMultipartParser.StreamingMultipartFormDataParser.Parse(RebufferableBinaryReader reader) in c:\Users\RKsherman\Downloads\Http-Multipart-Data-Parser-master\Http-Multipart-Data-Parser-master\HttpMultipartParser\StreamingMultipartFormDataParser.cs:line 423
at HttpMultipartParser.StreamingMultipartFormDataParser.Run() in c:\Users\RKsherman\Downloads\Http-Multipart-Data-Parser-master\Http-Multipart-Data-Parser-master\HttpMultipartParser\StreamingMultipartFormDataParser.cs:line 239
at HttpMultipartParser.MultipartFormDataParser..ctor(Stream stream, String boundary, Encoding encoding, Int32 binaryBufferSize) in c:\Users\RKsherman\Downloads\Http-Multipart-Data-Parser-master\Http-Multipart-Data-Parser-master\HttpMultipartParser\MultipartFormDataParser.cs:line 219
at HttpMultipartParser.MultipartFormDataParser..ctor(Stream stream, String boundary, Encoding encoding) in c:\Users\RKsherman\Downloads\Http-Multipart-Data-Parser-master\Http-Multipart-Data-Parser-master\HttpMultipartParser\MultipartFormDataParser.cs:line 155
at FileNetRestService.Service1.AddDocument(String appid, String isrecord, Stream stream) in c:\Users\RKsherman\Documents\Workspaces\FileNet\FRD\Code\FileNetRestService\FileNetRestService\FileNetRestService\Service1.svc.cs:line 149

IIS Worker stuck in a loop....

Hi Vodurden,
Thanks for the great Http-Multipart-Data-Parser..

Assuming we forget to close the boundary..
for example:

curl -X POST -H "Content-Type: multipart/form-data; --00content0boundary00" --data-binary @my2.txt http://localhost/Service1.svc/SomeMethod

The content of my2.txt is:
--00content0boundary00
Content-Disposition: form-data; name="A"

123
--00content0boundary00
Content-Disposition: form-data; name="B"

this is B
--00content0boundary00
Content-Disposition: form-data; name="C"

Bili
--00content0boundary00

for some reason it's stuck in a loop... and the IIS Worker memory usage as expected.. leak.

Issue with Parser

I was going off your post here:
http://stackoverflow.com/questions/7460088/reading-file-input-from-a-multipart-form-data-post/14514351#14514351

I see that you changed your constructor to require a boundary, instead of just passing in a stream.

I have a restful WCF service, that I'm trying to get to accept multiple files in a single request. Your parser is the closest to what I was going to write. I've found some out there that work, but they are for a single file request only.

Parser doesn't parse, kinda just spins out of control, and I have to kill the AppPool.
When I didn't replace "-" with "", I could get the parse to blow up with this:

The output char buffer is too small to contain the decoded characters, encoding 'Unicode (UTF-8)' fallback 'System.Text.DecoderReplacementFallback'.

Here is the code snippet:

[WebInvoke(UriTemplate = "upload", Method = "POST", ResponseFormat=WebMessageFormat.Json)]
public uploadResponse UploadFile(Stream stream)
{
    string txtFile = @"D:\uploads.txt";
    string filePath = null;
    string url = null;
    uploadFileInfo info = null;
    uploadResponse response = null; 

    StreamReader reader = new StreamReader(stream);
    string data = reader.ReadToEnd();
    string boundary = data.Substring(data.IndexOf("\r\n")).Replace("-","");
    try
    {    
    NetTools.dvFileIO.WriteToFile(txtFile, boundary, true);
    HttpMultipartParser.MultipartFormDataParser parser = new HttpMultipartParser.MultipartFormDataParser(boundary, stream);
    NetTools.dvFileIO.WriteToFile(txtFile, parser.Parameters["adID"].Data, false);
    }
    catch (Exception ex)
    {
        NetTools.dvFileIO.WriteToFile(txtFile, ex.Message, false);
        NetTools.dvFileIO.WriteToFile(txtFile, ex.StackTrace, false);
    }

OutOfMemoryException when form is empty

Hi Jake,

Tried posting an empty form. The constructor of parser takes on some time and then throws System.OutOfMemoryException. Can you replicate the issue and verify?

An item with the same key has already been added.

Is it possible to parse where the same key appears several times? I'm only getting An item with the same key has already been added. Its an email sent by mailgun to http where there are several Content-Disposition: form-data; name="Received"

Could not install package 'HttpMultipartParser 2.2.1'.

When using the NuGet Package Manager inside VS2015, I get the following error:

Could not install package 'HttpMultipartParser 2.2.1'. You are trying to install this package into a project that targets '.NETFramework,Version=v4.5.1', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact the package author.

Do I need to do anything special to install the NuGet package? Obviously, my project is targeting 4.5.1 and this cannot be changed.

call the parse twice

So am trying to call the parser twice,I have two WCF services on two different servers.
I am able to parse the Stream on first web service however when I send the stream to the second server and try to parse it,it throws an exception "Value cannot be null. Parameter name: source" on

reader.ReadLine().Skip(2)

Issues when parsing files being uploaded across network

First off awesome project!

I am using the parser in a vb.net application that has a self hosted webserver. I am using it to parse out uploaded text files. Each text file is very small 1-4kb. I am uploading multiple files at a time and sometimes upwards of 10-15 files at once. When selecting files from my local machine everything is parsed flawlessly however, if I select files from across a network, I receive the debug assert as shown below and nothing is parsed. As I stated, these files are very small, and even combined, the total file size is less than 50kb. In the snippet below, bodyData is a byte array and regardless of whether the files are being uploaded locally or coming from across the network, the byte array is the same size. So, it seems that the same amount of data is there to be parsed. Do you have any suggestions?

Dim stream As New MemoryStream(bodyData)
Dim parser As New MultipartFormDataParser(stream)

If parser.Files.Count > 0 Then
For Each f In parser.Files
Using fileStream = File.Create(mDir & f.FileName)
f.Data.Seek(0, SeekOrigin.Begin)
f.Data.CopyTo(fileStream)
fileStream.Close()
End Using
Next
End If

debug

I/O calls should be async

I propose enhancing the project to support async I/O when parsing the stream containing the multi-part data.

  • All methods in RebufferableBinaryReader that perform I/O (such as Read, ReadLine, etc.) should have an async counter-part (i.e.: ReadAsync, ReadLineAsync, etc.)
  • All methods in StreamingMultipartFormDataParser that perform I/O (such as Run, DetectBoundary, Parse, etc.) should have an async counter-part (i.e.: RunAsync, DetectBoundaryAsync, ParseAsync, etc.)
  • Developers should be able to chose between the old synchronous MultiFormDataParser and a new async version of the parser
  • Unit tests to validate the new async methods

I volunteer to submit a PR if you're interrested

unable to upload more than one images

Hi,

while uploading one image i am able to get it, but when i upload more than images the parser object displays file count 1 only.

looking for the solution.

"Object reference not set to an instance of an object" in GetParameterValue

The documentation for the GetParameterValue says:

Returns the value of a parameter or null if it doesn't exist.

However, if I try to get the value of a parameter that doesn't exist, I get a Object reference not set to an instance of an object exception.

This problem is caused by the following code in MultipartFormDataParser.cs:

return Parameters.FirstOrDefault(p => p.Name == name).Data;

The issue is that FirstOrDefault returns a null value which cause the exception when trying to get the value in .Data.

I have a suggestion to resolve this issue and also to allow a default value to be specified when the parameter doesn't exist:

public string GetParameterValue(string name, string defaultValue = null)
{
    if (parser.HasParameter(name)) return parser.GetParameterValue(name);
    else return defaultValue;
}

I would be happy to submit PR with this improvement

Question about saving multiple images from large stream

Hi Guys

Apologies to post this as in issue. It's more of question since I am novice. I have a large stream (real estate property photos), and using StreamingMultipartFormDataParser to parse. I am looking to find a c# sample code to how to save images to my sever after fileHandler step. Parser.filehandler.

I am looking to loop through the files. But following code doesn't seem to work when using StreamingMultipartFormDataParser

// Loop through all the files
foreach(var file in parser.Files)
{
    Stream data = file.Data;

    // Do stuff with the data.
}

BinaryStreamStack.cs Line:294

I got this error .I think if you use StreamReader instead BinaryReader the problem will be solved.

The output char buffer is too small to contain the decoded characters, encoding 'Unicode (UTF-8)' fallback 'System.Text.DecoderReplacementFallback'.

Best Regards

FormData object support

I am unable to parse a FormData objects containing form information.
The received data is in the following format:

------WebKitFormBoundary05dwgxFHZcnKtx42
Content-Disposition: form-data; name="name"

John
------WebKitFormBoundary05dwgxFHZcnKtx42

IndexOutOfRangeException

Hi,

Sometimes we get an IndexOutOfRange exception. Unfortunately I don't have much info at the moment on what the actual input is.

at HttpMultipartParser.StreamingMultipartFormDataParser.<>c.<ParseSection>b__38_2(String[] x)  
at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
at HttpMultipartParser.StreamingMultipartFormDataParser.ParseSection(RebufferableBinaryReader reader)
at HttpMultipartParser.StreamingMultipartFormDataParser.Parse(RebufferableBinaryReader reader)
at HttpMultipartParser.StreamingMultipartFormDataParser.Run()

Stream.Null cannot be constructed

private MultipartFormDataParser p_data = new MultipartFormDataParser(Stream.Null);

Causes an exception to be thrown, I only downloaded the library about 10 minutes ago but figured I can bug report this as its quite good practice to check for null values. Maybe it should work this way by design please let me know.

Web Service responds with Bad Request on using MultipartFormDataParser

I was using MultipartParser to handle a single file and then I saw this project for handling multiple files and form data in one request, but as soon as I add the line var parser = new MultipartFormDataParser(data); the web service call starts responding with Bad Request. Please help.

Cannot release the Memory

Dear vodurden,

i used this codes on my httpserver project and i got 2 problems:

1: cannot release memory when client uploads some big files
i already disposed all fileparts after processed request, but the memory cannot be released

2: parsing speed is slowly when parsing a big request
i have tried some big files it's greater than 1gb , it's very sowly :(

can you provide some advises ?

MultipartFormDataParser does not accept quoted boundary value

The current version of the MultipartFormDataParser does not work when the boundary value specified in the header is enclosed in quotes, such as when produced by the System.Net.Http.MultipartFormDataContent class.

A workaround can be found in the answer to this stackoverflow question regarding the aforementioned .NET class, but, according to the RFC linked in the comment (https://tools.ietf.org/html/rfc2046#section-5.1.1), a quoted boundary is used when the boundary value may include 'invalid' characters, and as such should be accepted by the MultipartFormDataParser.

Index out of bounds exception when \r\n after boundary spans buffer

On line 485 in StreamingMultipartFormDataParser.cs https://github.com/Vodurden/Http-Multipart-Data-Parser/blob/master/HttpMultipartParser/StreamingMultipartFormDataParser.cs#L485

there is a check to see if the boundary is right up at the end of the buffer. This makes sense since you are expecting that the "--" of the end boundary could be cutoff. I offer up that the condition of the boundary being 1 char away from the end is also an issue because farther down in the code the check for the \r\n runs off the end of the buffer.

My error is occurring because the \r\n between parts is running across the buffer.

Possibly can fix this by skipping the current loop if the endBoundary was not found and the boudary that was found is not more than two char from the end

System.ArgumentNullException: Value cannot be null.

Hi Jake,

I ran across this issue today trying to use the parser. I'm going to take a stab at fixing it, but wanted to post this issue anyway in case you are either faster at fixing it, or know generally why this may be happening.

Here's the stack trace...

Unhandled Exception: System.ArgumentNullException: Value cannot be null.
Parameter name: source

at System.Linq.Enumerable.Skip[TSource](IEnumerable1 source, Int32 count)

at HttpMultipartParser.StreamingMultipartFormDataParser.DetectBoundary(RebufferableBinaryReader reader) in d:\VS Projects\HttpMultipartDataParser\HttpMultipartParser\StreamingMultipartFormDataParser.cs:line 312

at HttpMultipartParser.StreamingMultipartFormDataParser.Run() in d:\VS Projects\HttpMultipartDataParser\HttpMultipartParser\StreamingMultipartFormDataParser.cs:line 222

at HttpMultipartParser.MultipartFormDataParser..ctor(Stream stream, String boundary, Encoding encoding, Int32 binaryBufferSize) in d:\VS Projects\HttpMultipartDataParser\HttpMultipartParser\MultipartFormDataParser.cs:line 219

at HttpMultipartParser.MultipartFormDataParser..ctor(Stream stream) in d:\VS Projects\HttpMultipartDataParser\HttpMultipartParser\MultipartFormDataParser.cs:line 104

at MultipartMimeParser.MultipartMimeParser.Run()

at MultipartMimeParser.MultipartMimeParser.Main()

Bytes EFBBBF (BOM) randomly disappears from middle of binary content

Hello!
Here is very foolish bug when BOM marker cut out from middle of content. Problem in reading input stream into buffer. Raw stream read by chunks and every chunk examined on start with BOM marker (EFBBBF). Suddenly this sequence of bytes encounters not only as BOM marker, especially if transmitted large binary files. At current if accidentally some chunk started with this three bytes even it is not the BOM, they cut out. And as result received content is not valid.

Exception with no parameter name is found

This form raises an exception because there is no parameter "name":
--BOUNDARY
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=202_1708_09066660000001000000010016_20170527005018.data
Content-Length: 1549

Obviously the form is malformed, but the parser should be resillient to that error.

Stacktrace:
at System.Collections.Generic.Dictionary2.get_Item(TKey key) at HttpMultipartParser.StreamingMultipartFormDataParser.ParseFilePart(Dictionary2 parameters, RebufferableBinaryReader reader)
at HttpMultipartParser.StreamingMultipartFormDataParser.ParseSection(RebufferableBinaryReader reader)
at HttpMultipartParser.StreamingMultipartFormDataParser.Parse(RebufferableBinaryReader reader)
at HttpMultipartParser.StreamingMultipartFormDataParser.Run()
at HttpMultipartParser.MultipartFormDataParser..ctor(Stream stream, String boundary, Encoding encoding, Int32 binaryBufferSize)
at HttpMultipartParser.MultipartFormDataParser..ctor(Stream stream)

Unicode Problem file name

Hi

When I test file like "تست.jpg" the name that parser return is "3.jpg".

If you want I can email Multipart-Data file.

Best Regards.

boundaryBinary issue

##hello

in StreamingMultipartFormDataParser.cs file the method Run() recognizes the boundary by trying to find the matched bytes of "--" at the beginning and end of the file part ... what if by some mistake there is a false positive matching in the middle of the file's binary data, for instance, the post data contains a bate-encoded string "--" or even an HTML document (such as a web page http-posted from one server node to another for cache purpose) that contains a comment like "!--"? in such cases boundary detection is invalid since only specific portion of the actual binary file data being serialized into "FilePart" type

Sample implementation code for StreamingMultipartFormDataParser

Hi, i am using "MultipartFormDataParser" class for uploading files from my wcf service (c#). I am getting out of memory exception for large files and i have decided to use "StreamingMultipartFormDataParser". But i am little confused with implementation. Can you please share a sample code implementing this class.

NET Core support

Hi, I found your project helpful but I can't use it directly from nuget because you don't support NET Core. My workaround was to copy your source code and it works fine with NET Core (only a little change with Serializable).

So I wonder if you will publish soon NET Core support.

Thank you

Signed version of the assembly

Hi,

Can you please provide future versions as a SNK signed assembly? THis will help in using it out of the box in signed projects.

With regards,

Daniël

Support for dotnet core 1.0 (i.e. AWS Lambda)

It would be great to have a lightweight form data parser like this for use in AWS Lambda, which only supports dotnet core 1.0 at the moment (though that's likely to change soon). It looks like you can do this by changing the .Net Standard build to use 1.6.0 instead of 1.6.1 (yep, it matters). I did this locally, and it seems to work fine. Not sure if this justifies a pull request or not.

<NetStandardImplicitPackageVersion Condition=" '$(TargetFramework)' == 'netstandard1.3' ">1.6.0</NetStandardImplicitPackageVersion>

TaskCanceledException throwed when StreamingMultipartFormDataParser.Run()

When upload some small file,everything is ok.
but when i test with a large file(vs2015.ent_chs.iso,3.88GB),TaskCanceledException will throw.the exception sometimes thowed when 2%,somtimes when 80%.it's random.

HttpMultipartParser dll version:2.0.1,from nuget
Test Browser:Chrome 44.0.2403.107 m

the exception detail is below:

System.AggregateException: 发生一个或多个错误。 ---> System.Threading.Tasks.TaskCanceledException: 已取消一个任务。
   --- 内部异常堆栈跟踪的结尾 ---
   在 System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   在 System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   在 HttpMultipartParser.RebufferableBinaryReader.StreamData()
   在 HttpMultipartParser.RebufferableBinaryReader.Read(Byte[] buffer, Int32 index, Int32 count)
   在 HttpMultipartParser.StreamingMultipartFormDataParser.ParseFilePart(Dictionary`2 parameters, RebufferableBinaryReader reader)
   在 HttpMultipartParser.StreamingMultipartFormDataParser.ParseSection(RebufferableBinaryReader reader)
   在 HttpMultipartParser.StreamingMultipartFormDataParser.Parse(RebufferableBinaryReader reader)
   在 Quick.OwinMVC.Utils.MultipartFormDataUtils.HandleMultipartData(IOwinRequest req, ParameterDelegate parameterHandler, FileStreamDelegate fileHandler) 位置 E:\GitHub\Quick.OwinMVC\Source\Quick.OwinMVC\Utils\MultipartFormDataUtils.cs:行号 29
---> (内部异常 #0) System.Threading.Tasks.TaskCanceledException: 已取消一个任务。<---

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.