Comments (27)
Hi @tuxan11, can you please make sure that you're up-to-date with the master
branch of this repository? In the past, there have been issues with files over 4GB, but they should be corrected at this point. If that doesn't help you, I'd need to see how you're using the library (code), and the resulting zip (you an email it to [email protected]). Thanks!
from archivestream-php.
Hello @johnmaguire,
Thank you for the quick response.
I tried with ArchiveStream-php-master.zip from couple of days back, still the same issue.
Here is the code I am using.
$opt = array();
$opt['comment'] = 'Hello World';
$zip = new \ArchiveStream_Zip('download.zip'); //I am forcing Zip download. We don't want tar
$stat = fstat($fh); //$fh is the file stream
$zip->init_file_stream_transfer($filePath, $stat['size'], $opt); //$filePath path to store the file at
while (!feof($fh)) {
$zip->stream_file_part(fread($fh, 1048576));
}
$zip->complete_file_stream();
$zip->finish();
BTW, I will upload the zip created with this code to a fileshare and send you the link.
from archivestream-php.
@tuxan11 Thanks for the ZIP, I'll be looking into it as soon as possible. Just a couple more questions: You're not seeing any PHP errors or warnings in your log? Do you have output buffering turned off? Try adding this before your Zipstream code:
while (ob_get_level() > 0) {
ob_end_clean();
}
This will turn off output buffering if it's turned on. If you're using a framework like Kohana, Laravel, or CodeIgniter, it's quite likely that it starts buffering prior to your code getting executed. This can cause out of memory issues.
Thanks!
from archivestream-php.
Hello John,
I don't see any errors or warnings while using the zip library.
I will add the code that you've had to turn off memory buffering and will get back to you with the results. Thanks.
from archivestream-php.
@tuxan11 There's one more thing I'll ask you to try if that doesn't work for you... if you look at this issue you can see that @ianvonholt had some issues using the streaming capabilities of the library after we switched our code from using fgets
to fread
. I haven't been able to reproduce this, but in your example, you use fread
(which theoretically should be correct). I would ask however that you try switching it to fgets
and we can see if that fixes the issue.
If so, I can target a single bug instead of two. :) Thanks!
from archivestream-php.
@johnmaguire I did try that solution yesterday(but the result was same) as I was browsing through the issue list.
I've started same zip operation with the code change you've suggested. Will let you know how it goes.
from archivestream-php.
@johnmaguire I tried with explicit memory buffer off and still the same issue. Please let me know if you want me try anything different.
from archivestream-php.
@tuxan11 Thanks for giving it a shot. I should have asked sooner -- can I get the PHP version, distro version, and web server version? Also, does this issue occur with files full of just null
characters (or just 0
s for that matter?) If so, I can try to reproduce this on my side.
from archivestream-php.
@johnmaguire Here is the info:
Distro: Windows 8,
XAMPP :1.8.3
Apache: 2.4.9
PHP Version 5.5.15
This problem occurs with any files. I just created bunch of files using linux command and zipped them up.
from archivestream-php.
@johnmaguire, were you able to find any issues with the zip file that I uploaded ?
from archivestream-php.
@tuxan11 It definitely doesn't look correct. :) I'm hoping to get a tool from a colleague of mine that analyzes ZIP files to see exactly what's off. I'm also creating a test to reproduce this:
-> % cat issue_12.php
<?php
// Github issue: https://github.com/barracudanetworks/ArchiveStream-php/issues/12
// ZIPs with a size exceeding 4 GB are corrupt
require_once './ArchiveStream-php/stream.php';
require_once './ArchiveStream-php/zipstream.php';
$files = array(
'A.bin', // 2.5GB
'B.bin', // 2.5GB
);
$opt = array(
'comment' => 'Hello world',
);
$zip = new \ArchiveStream_Zip('download.zip');
foreach ($files as $file)
{
$fh = fopen($file, 'r');
$stat = fstat($fh, $stat['size'], $opt);
while (!feof($fh))
{
$zip->init_file_stream_transfer(fread($fh, 1024 * 1024));
}
}
$zip->complete_file_stream();
$zip->finish();
If I can't reproduce with this script, I'll attempt one using a single 5GB file to see if that creates a corrupt ZIP.
Also, did you have a comment on your ZIP file that reads "Generated by FileCloud"? I don't see this in the sample code you provided. If so, I think the code that adds a comment to the ZIP file may be a little messed up. If not, you're outputting more to the page than simply the ZIP file, and that could be part of your issue.
from archivestream-php.
Looks like that reproduced it. Will look into this after lunch hopefully.
-> % 7z x issue_12.zip
7-Zip [64] 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18
p7zip Version 9.20 (locale=utf8,Utf16=on,HugeFiles=on,8 CPUs)
Processing archive: issue_12.zip
Error: Can not open file as archive
from archivestream-php.
Err, scratch that, my script was totally messed up, haha. Going to give it another go with this:
-> % cat issue_12.php
<?php
// Github issue: https://github.com/barracudanetworks/ArchiveStream-php/issues/12
// ZIPs with a size exceeding 4 GB are corrupt
ini_set('max_execution_time', 0);
require_once './ArchiveStream-php/stream.php';
require_once './ArchiveStream-php/zipstream.php';
$files = array(
'A.bin',
'B.bin',
);
$opt = array(
'comment' => 'Hello world',
);
$zip = new \ArchiveStream_Zip('issue_12.zip');
foreach ($files as $file)
{
// Note: Using stat instead of fstat
$fh = fopen($file, 'r');
$zip->init_file_stream_transfer($file, $stat['size'], $opt);
while (!feof($fh))
{
$zip->stream_file_part(fread($fh, 1024 * 1024));
}
$zip->complete_file_stream();
}
$zip->finish();
from archivestream-php.
That seemed to work just fine:
-> % 7z x issue_12.zip
7-Zip [64] 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18
p7zip Version 9.20 (locale=utf8,Utf16=on,HugeFiles=on,8 CPUs)
Processing archive: issue_12.zip
Extracting A.bin
Extracting B.bin
Everything is Ok
Files: 2
Size: 5242880000
Compressed: 5242880432
Some info about the server used to produce the zip file:
john@dib [12:42:16] [~/www/dib.leftforliving.com/files/archivestream]
-> % uname -a
Linux dib.leftforliving.com 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2+deb7u2 x86_64 GNU/Linux
john@dib [12:42:19] [~/www/dib.leftforliving.com/files/archivestream]
-> % sudo nginx -v
nginx version: nginx/1.6.2
john@dib [12:42:20] [~/www/dib.leftforliving.com/files/archivestream]
-> % php --version
PHP 5.6.7-1 (cli) (built: Mar 24 2015 12:30:15)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2015, by Zend Technologies
Could you try running the script I posted above? To generate A.bin and B.bin, I used the following commands on Linux:
dd if=/dev/urandom of=A.bin bs=1M count=2500
dd if=/dev/urandom of=B.bin bs=1M count=2500
Any data should do.
from archivestream-php.
One last note for now... I added $opt into the ArchiveStream_Zip constructor, in order to make sure ZIP file-level comments work. No problems:
jmaguire@ZimsBase [01:01:57] [~/Downloads]
-> % 7z x issue_12\(1\).zip
7-Zip [64] 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18
p7zip Version 9.20 (locale=utf8,Utf16=on,HugeFiles=on,8 CPUs)
Processing archive: issue_12(1).zip
file A.bin
already exists. Overwrite with
A.bin?
(Y)es / (N)o / (A)lways / (S)kip all / A(u)to rename all / (Q)uit? A
Extracting A.bin
Extracting B.bin
Everything is Ok
Files: 2
Size: 5242880000
Compressed: 5242880432
from archivestream-php.
@johnmaguire, thanks for the info. I will try to run the script and get back to you.
from archivestream-php.
@johnmaguire I tried your code on my windows box and it resulted in a corrupted zip same as before.
Then I tried the same code in Linux and there it works. Looks like the problem is only on PHP on Windows. Can anything be done on Windows platform.
Please let me know if you need any more information.
from archivestream-php.
Hi tuxan11. I don’t have any Windows machines currently, nor do we use any here at Barracuda Networks. If I get some time over the next week or so, I’ll try to setup a VM to test this in. No guarantees. :(
Just to verify, do you have the GMP PHP extension installed correctly?
John
On May 14, 2015, at 6:34 PM, tuxan11 [email protected] wrote:
@johnmaguire https://github.com/JohnMaguire I tried your code on my windows box and it resulted in a corrupted zip same as before.
Then I tried the same code in Linux and there it works. Looks like the problem is only on PHP on Windows. Can anything be done on Windows platform.
Please let me know if you need any more information.—
Reply to this email directly or view it on GitHub #12 (comment).
from archivestream-php.
Also, when testing on Linux, did you use Apache? If so, version used there
would be great. Thanks!
On May 14, 2015 6:40 PM, "John Maguire" [email protected] wrote:
Hi tuxan11. I don’t have any Windows machines currently, nor do we use any
here at Barracuda Networks. If I get some time over the next week or so,
I’ll try to setup a VM to test this in. No guarantees. :(Just to verify, do you have the GMP PHP extension installed correctly?
John
On May 14, 2015, at 6:34 PM, tuxan11 [email protected] wrote:
@johnmaguire https://github.com/JohnMaguire I tried your code on my
windows box and it resulted in a corrupted zip same as before.
Then I tried the same code in Linux and there it works. Looks like the
problem is only on PHP on Windows. Can anything be done on Windows platform.
Please let me know if you need any more information.—
Reply to this email directly or view it on GitHub
#12 (comment)
.
from archivestream-php.
@johnmaguire, on Linux I used Apache version: 2.2.15.
On windows machine, I did install gmp php extension. Without installing it, some php calls such as(gmp_init) was failing.
from archivestream-php.
Thanks! I'm able to reproduce this in a Windows VM with the latest XAMPP using PHP 5.5. I'll try to debug this soon. :)
from archivestream-php.
Any news about this? I have been trying all day to make this work with files larger than 4GB.
Running all 64 bit software, with PHP 5.6.10, but on Windows (8). The archive is always corrupt, regardless of being Zip or Tar. The expected archive size seems to be correct. WinRAR lists the archive contents, but cannot extract anything and reports the large file's size as around 2GB instead of 4.5GB. So I am pretty sure there is an integer limitation somewhere. I do get this warning from PHP: PHP Warning: ArchiveStream::int64_split(): Unable to convert variable to GMP - wrong type in stream.php on line 361
But I'm not good with numbers and bits to troubleshoot this...
Also, for other people trying to run this on Windows, do note that PHP, even the latest version, and even 64 bit, reports incorrect number when using "filesize()". I am using COM to get the real filesize and that works every time.
from archivestream-php.
Are you saying that if you switch from filesize()
to COM the archives are no longer corrupt, or the filesizes are correct?
Found this on php.net/filesize: Note: Because PHP's integer type is signed and many platforms use 32bit integers, some filesystem functions may return unexpected results for files which are larger than 2GB.
Definitely sounds like it could be related. Thank you for the information. I haven't tracked down the problem yet, but the PHP warning and that info is definitely helpful!
Also, are you using an up-to-date version of the ArchiveStream library? Line 361 of stream.php doesn't seem to be a gmp_*()
call.
from archivestream-php.
On Windows, the sizes of large files (>2GB on 32bit systems and >4GB on 64bit) will always be reported incorrect by "filesize()". Regardless of fixing this problem, the library still provides corrupt archives.
I know that line doesn't specifically use any GMP functions, but the error points always to the "$low = $value & $right;" line. I have found however that this error occurs only with PHP 5.6, so I will skip that version for now.
from archivestream-php.
Interesting. Thanks again for all the info.
On Jun 24, 2015, at 1:53 PM, Vlad [email protected] wrote:
On Windows, the sizes of large files (>2GB on 32bit systems and >4GB on 64bit) will always be reported incorrect by "filesize()". Regardless of fixing this problem, the library still provides corrupt archives.
I know that line doesn't specifically use any GMP functions, but the error points always to the "$low = $value & $right;" line. I have found however that this error occurs only with PHP 5.6, so I will skip that version for now.
—
Reply to this email directly or view it on GitHub #12 (comment).
from archivestream-php.
Note: on windows before PHP 7 there is no support for 64 bit integers, even if running php 64 bit. For this reason filesize() function reports negative value for files > 2GB.
// Check if PHP is running with 64 bit integer support
if (PHP_INT_SIZE < 8) {
// NOT 64 bit
}
from archivestream-php.
@FabryB I believe that is what @vvllaadd was saying above. He switched out the filesize()
call for a COM call, and was still unable to get the library working. We store most numbers using GMP within the library, in order to get around 32-bit limitations. However, the line he said the error occured on ($low = $value & $right;
) does not use GMP, so I wonder if it could suffer from a bug if you don't have 64-bit ints. I'd have to look closer at what we're storing in those vars, and what we're aiming to do.
Thank you for the information.
from archivestream-php.
Related Issues (20)
- PHP 5.6 above: Zip will be corrupted because it can't split a 64-bit integer to two 32-bit integers in int64_split function HOT 7
- Offline zip HOT 1
- Zip archives cannot be opened on OS X (Archive Utility, unzip) HOT 2
- int64_split(): Unable to convert variable to GMP - wrong type HOT 1
- Object of class GMP could not be converted to boolean HOT 2
- file corrupt HOT 3
- Can you download from multiple pages simultaneously? HOT 2
- Feature Request: add_file_from_path remote file HOT 3
- Zip file is invalid in php version 5.5 and 5.6 HOT 2
- Undefined offset: 8 in ZipArchive.php
- Corrupt Zip archive on macOS HOT 4
- Speed & Time-Limit / stopping after some minutes
- Undefined property: $need_headers
- Support request loop over files in folder
- Build an archive from a stream HOT 2
- How to zip file from a url ?
- 7-Zip reports Headers Error HOT 4
- Output stream isn't written to with default param
- Add a way to not disable output buffering HOT 1
- PHP 8.1 - Deprecated: Implicit conversion from float
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from archivestream-php.