Large manifest file causes deb-s3 not to work about deb-s3

[jjshoe]

Did some poking, found a couple of things, though nothing that was a smoking gun for me.

https://github.com/krobertson/deb-s3/blob/master/lib/deb/s3/manifest.rb#L82 - It looks like to me that this is getting the results of the package.erb and cramming them all into an array, when it's done, it then calls a join. This was causing my desktop to kill the process for running out of memory. The way to do this would be to write the contents out to the file as generated. I played with this some, and while it prevented it from getting killed on my desktop, it didn't help with overall speed.

All the time was sucked in that generate method though. It seems like the correct approach would be:

1. See if I need to change an existing package
2. Write line by line until that existing package
3. Template
4. Write line by line until the end

If the package didn't exist it could just be tacked onto the end.

Another performance piece would be instead of writing out manifest into the FH for compressing you could just copy or compress Packages.

I ran it through ruby-prof, however github won't let me attach anything other then images, so you can grab the output here: https://www.dropbox.com/s/pl5kvkounilzvfl/stat.tar?dl=0

[jjshoe]

I found the root cause, my Packages file had lots of entrys of just periods:

[jjshoe@joel-desktop Downloads]$ du -hs Packages 
121M    Packages
[jjshoe@joel-desktop Downloads]$ grep -ve '^\s*.\s*$' Packages > test
[jjshoe@joel-desktop Downloads]$ du -hs test 
88K test
[jjshoe@joel-desktop Downloads]$

[jjshoe]

Note: going to hold the ticket open for a bit, see if I can figure out what caused this.

[leedm777]

I've looked into this a bit. It appears that blank continuation lines in a description (which are rendered as . in the Packages file) are duplicated when it goes through the parsing process. So for every successive upload, the number of blank lines are doubled.

[jjshoe]

@leedm777 - perhaps: https://github.com/krobertson/deb-s3/blob/master/lib/deb/s3/manifest.rb#L46

[leedm777]

The massive number of dots we were seeing were caused by a parsing bug. PR posted, which even includes a test :-)

[leedm777]

FWIW, it looks like this bug was introduced between 0.6.2 and 0.7.0.

[JeanFred]

I am having the same problem − dot lines get duplicated, resulting in a huge Packages file eventually leading deb-s3 to crash when manipulating it.

[leedm777]

@JeanFred Can you try out my patch to see if it works for you?

I also wrote a fixup command in a branch on my work. I used it to get our Packages files back in order. It's a quick hack, so I highly recommend backups before trying it.

[JeanFred]

@leedm777 I have tried your patch and I’m happy to report that it does work for me − thanks! I tried uploading packages with deb-s3 − which duplicated dots − and then your version − which did not.

I also used (after backup ;-) your fixup command, which worked like a charm and basically saved our lives. Thanks a lot for this.

[krobertson]

Merged the PR and released 0.7.1 earlier this morning. Thanks!

[JeanFred]

Thanks @krobertson and @leedm777 !