GithubHelp home page GithubHelp logo

phpmussel / sigtool Goto Github PK

View Code? Open in Web Editor NEW
7.0 5.0 3.0 273 KB

Generates signatures for phpMussel using main.cvd and daily.cvd from ClamAV.

Home Page: https://github.com/phpMussel/SigTool

License: MIT License

PHP 100.00%
phpmussel signatures generator sigtool clamav php hacktoberfest

sigtool's Issues

RAM usage benchmark.

Date: 2021.07.21
From Commit: 68be5e2
main.cvd used: ClamAV-VDB:14 Jul 2021 22-39 -0400:61:6607162:90:10dad18...sigmgr:1626316750
daily.cvd used: ClamAV-VDB:14 Jul 2021 14-29 -0400:26232:1961167:90:a24b438...sigmgr:1626287399
PHP version used: 8.0.5

$ php SigTool.php xp

 Decompressing daily.cvd ... Done! <RAM 174.61 MB>
 Extracting contents from daily.cvd to Cvd object ... Done! <RAM 174.61 MB>
 Writing COPYING ... Done! <RAM 174.63 MB>
 Writing COPYING ... Done! <RAM 174.63 MB>
 Writing daily.info ... Done! <RAM 174.62 MB>
 Writing daily.cfg ... Done! <RAM 174.62 MB>
 Writing daily.ign ... Done! <RAM 174.62 MB>
 Writing daily.ign2 ... Done! <RAM 174.62 MB>
 Writing daily.ftm ... Done! <RAM 174.63 MB>
 Writing daily.hdb ... Done! <RAM 174.61 MB>
 Writing daily.hdu ... Done! <RAM 174.62 MB>
 Writing daily.hsb ... Done! <RAM 288.61 MB>
 Writing daily.hsu ... Done! <RAM 174.61 MB>
 Writing daily.mdb ... Done! <RAM 180.61 MB>
 Writing daily.mdu ... Done! <RAM 174.68 MB>
 Writing daily.msb ... Done! <RAM 174.61 MB>
 Writing daily.msu ... Done! <RAM 174.61 MB>
 Writing daily.ndb ... Done! <RAM 175.12 MB>
 Writing daily.ndu ... Done! <RAM 175.40 MB>
 Writing daily.ldb ... Done! <RAM 226.61 MB>
 Writing daily.ldu ... Done! <RAM 178.61 MB>
 Writing daily.idb ... Done! <RAM 174.65 MB>
 Writing daily.fp ... Done! <RAM 174.64 MB>
 Writing daily.sfp ... Done! <RAM 174.61 MB>
 Writing daily.pdb ... Done! <RAM 174.62 MB>
 Writing daily.wdb ... Done! <RAM 174.63 MB>
 Writing daily.crb ... Done! <RAM 174.63 MB>
 Writing daily.cdb ... Done! <RAM 174.62 MB>
 Decompressing main.cvd ... Done! <RAM 436.61 MB>
 Extracting contents from main.cvd to Cvd object ... Done! <RAM 436.61 MB>
 Writing COPYING ... Done! <RAM 436.63 MB>
 Writing COPYING ... Done! <RAM 436.63 MB>
 Writing main.info ... Done! <RAM 436.62 MB>
 Writing main.hdb ... Done! <RAM 442.61 MB>
 Writing main.hsb ... Done! <RAM 596.61 MB>
 Writing main.mdb ... Done! <RAM 680.61 MB>
 Writing main.msb ... Done! <RAM 436.61 MB>
 Writing main.ndb ... Done! <RAM 460.61 MB>
 Writing main.ldb ... Done! <RAM 442.61 MB>
 Writing main.fp ... Done! <RAM 436.64 MB>
 Writing main.sfp ... Done! <RAM 436.61 MB>
 Writing main.crb ... Done! <RAM 436.61 MB>
 Writing main.cdb ... Done! <RAM 436.61 MB>
 Accessing signatures.dat ... Done! <RAM 929.00 KB>
 Deleting COPYING ... Done! <RAM 928.94 KB>
 Deleting daily.cdb ... Done! <RAM 928.94 KB>
 Deleting daily.cfg ... Done! <RAM 928.94 KB>
 Deleting daily.crb ... Done! <RAM 928.94 KB>
 Deleting daily.fp ... Done! <RAM 928.94 KB>
 Deleting daily.ftm ... Done! <RAM 928.94 KB>
 Deleting daily.hdu ... Done! <RAM 928.94 KB>
 Deleting daily.hsb ... Done! <RAM 928.94 KB>
 Deleting daily.hsu ... Done! <RAM 928.94 KB>
 Deleting daily.idb ... Done! <RAM 928.94 KB>
 Deleting daily.ign ... Done! <RAM 928.94 KB>
 Deleting daily.ign2 ... Done! <RAM 928.94 KB>
 Deleting daily.info ... Done! <RAM 928.94 KB>
 Deleting daily.ldb ... Done! <RAM 928.94 KB>
 Deleting daily.ldu ... Done! <RAM 928.94 KB>
 Deleting daily.mdu ... Done! <RAM 928.94 KB>
 Deleting daily.msb ... Done! <RAM 928.94 KB>
 Deleting daily.msu ... Done! <RAM 928.94 KB>
 Deleting daily.ndu ... Done! <RAM 928.94 KB>
 Deleting daily.pdb ... Done! <RAM 928.94 KB>
 Deleting daily.sfp ... Done! <RAM 928.94 KB>
 Deleting daily.wdb ... Done! <RAM 928.94 KB>
 Deleting main.cdb ... Done! <RAM 928.94 KB>
 Deleting main.crb ... Done! <RAM 928.94 KB>
 Deleting main.fp ... Done! <RAM 928.94 KB>
 Deleting main.hsb ... Done! <RAM 928.94 KB>
 Deleting main.info ... Done! <RAM 928.94 KB>
 Deleting main.ldb ... Done! <RAM 928.94 KB>
 Deleting main.msb ... Done! <RAM 928.94 KB>
 Deleting main.sfp ... Done! <RAM 928.94 KB>
 Accessing daily.hdb ... Done! <RAM 929.06 KB>
 Accessing main.hdb ... Done! <RAM 6.91 MB>
 Writing clamav.hdb ... Done! <RAM 8.93 MB>
 Deleting daily.hdb ... Done! <RAM 955.66 KB>
 Deleting main.hdb ... Done! <RAM 955.66 KB>
 Accessing daily.mdb ... Done! <RAM 6.93 MB>
 Accessing main.mdb ... Done! <RAM 18.93 MB>
 Writing clamav.mdb ... Done! <RAM 18.93 MB>
 Deleting daily.mdb ... Done! <RAM 955.90 KB>
 Deleting main.mdb ... Done! <RAM 955.90 KB>
 Accessing daily.ndb ... Done! <RAM 1.44 MB>
 Accessing main.ndb ... Done! <RAM 24.93 MB>
 Writing clamav.ndb ... Done! <RAM 56.93 MB>
 Deleting daily.ndb ... Done! <RAM 955.93 KB>
 Deleting main.ndb ... Done! <RAM 955.93 KB>
 Accessing clamav.ndb ... Done! <RAM 24.93 MB>
 Processing ... Done! <RAM 46.25 MB>
 Writing clamav.db ... Done! <RAM 22.25 MB>
 Writing clamav_regex.db ... Done! <RAM 22.25 MB>
 Writing clamav.htdb ... Done! <RAM 22.25 MB>
 Writing clamav_regex.htdb ... Done! <RAM 22.25 MB>
 Writing clamav.ndb ... Done! <RAM 22.25 MB>
 Writing clamav_regex.ndb ... Done! <RAM 22.25 MB>
 Writing clamav_elf.db ... Done! <RAM 22.25 MB>
 Writing clamav_elf_regex.db ... Done! <RAM 22.25 MB>
 Writing clamav_email.db ... Done! <RAM 22.25 MB>
 Writing clamav_email_regex.db ... Done! <RAM 22.25 MB>
 Writing clamav_exe.db ... Done! <RAM 22.25 MB>
 Writing clamav_exe_regex.db ... Done! <RAM 22.25 MB>
 Writing clamav_graphics.db ... Done! <RAM 22.25 MB>
 Writing clamav_graphics_regex.db ... Done! <RAM 22.25 MB>
 Writing clamav_java.db ... Done! <RAM 22.25 MB>
 Writing clamav_java_regex.db ... Done! <RAM 22.25 MB>
 Writing clamav_macho.db ... Done! <RAM 22.25 MB>
 Writing clamav_macho_regex.db ... Done! <RAM 22.25 MB>
 Writing clamav_ole.db ... Done! <RAM 22.25 MB>
 Writing clamav_ole_regex.db ... Done! <RAM 22.25 MB>
 Writing clamav_pdf.db ... Done! <RAM 22.25 MB>
 Writing clamav_pdf_regex.db ... Done! <RAM 22.25 MB>
 Writing clamav_swf.db ... Done! <RAM 22.25 MB>
 Writing clamav_swf_regex.db ... Done! <RAM 22.25 MB>
 Writing signatures.dat ... Done! <RAM 22.30 MB>

Currently, it seems to peak at Writing main.mdb ... Done! <RAM 680.61 MB>.

The currently documented minimum RAM recommendation for using SigTool is ~1 GB. I wonder whether I should lower the minimum recommendation..? Not sure. Anyway, based on this benchmark, it seems that anything above at least ~680 MB should actually be sufficient (though asking for higher is always safer, since there may be occasions where it spikes, where something happens sometimes causing it to need more RAM or whatever). Different PHP versions may also beget different results (the currently documented minimum PHP version recommendation is 7.0.3; this benchmark uses 8.0.5).

Anyway, just posting for the sake of sharing info. :-)

Current master/dev code doesn't work.

Current master/dev code doesn't work properly for processing, due to incomplete changes.

The current latest available tagged release (v1.0.2) works fine though, for the most part (downloading doesn't work properly though).

For now, for those wanting to use phpMussel/SigTool, just stick with tagged releases, and download the necessary database files manually prior to processing (i.e., use phpMussel/SigTool just for processing, not for downloading).

I'll hold off tagging any new releases until it's working properly again. This issue will remain open until it's working properly again, and will be closed afterwards.

Invalid signatures

Hi,

(I move conversation over here as it is mostly just related to SigTool process now).

That last change in SigTool somehow made it worse I think. Now every file is flagged for something. var dump on $VN shows for example:

array(2) { [0]=> string(25) "� ��Java.Trojan.Boonana-6" [1]=> string(4) "2005" }

With index 1 being just "2005" sure enough it will flag almost any file. I'm sure that is not correct signature?

Not many faulty signatures now (have not checked how many is bypassed due to !isset($VN[1])):

Anti-virus: Signature errors during file scan: 2 of 35563
Last error message: preg_match(): Compilation failed: number too big in {} quantifier at offset 78
Last signature: 7265766973696f6e20383630206c747b717569747d69662f417b2f4434303c(?:..){122880,}39303930393039303930393039303930393039304538303030303030303035

Automate updates to URL scanner signatures.

Future feature on the to-do list: Automating updates to URL scanner signatures.

Currently, SigTool just processes signatures from ClamAV. Would be useful to be able to also process URL scanner signatures too though.

Currently low-priority, but I should be able to make a start on this within the next month or so.

Creating this issue for reference and marking as "attention required".

Note about false positives and docs update.

Another low-priority item on the to-do list: Updating the documentation to sync translations with the main project and add a note about potential false positives to the documentation (i.e., may need to disable some AVs while generating new signatures due to the risk of false positives during the process). Will sort this out later.

Hacktoberfest 2017

Hacktoberfest by Digital Ocean will be running again this year through the month of October 2017. If anyone sees any room for improvement to the codebase, has ideas, suggestions, bug-fixes, or whatever else, if you want to win a free Hacktoberfest t-shirt and some stickers, consider signing up for this year and sending some pull requests to this repository. :-)

For more details: https://hacktoberfest.digitalocean.com/#details

Issue will remain open until the end of October.

"error code: 1020"

Posting this issue because I've just discovered that, when attempting to download the daily.cvd and main.cvd files through SigTool, the server returns:

error code: 1020

..thus causing errors to appear in SigTool (seeing as it would be attempting to act upon files which aren't the expected signature files):

(SigToolPath)>php sigtool.php xpmd
 Downloading main.cvd ... Done!
 Writing main.cvd ... Done!
 Downloading daily.cvd ... Done!
 Writing daily.cvd ... Done!
 Stripping ClamAV package header from daily.cvd ... Done!
 Decompressing daily.cvd (GZ) ... Done!
 Extracting contents from daily.cvd (TAR) to (SigToolPath) ...PHP Warning:  scandir(phar://(SigToolPath)/daily.cvd): Failed to open directory: internal corruption of phar "(SigToolPath)\daily.cvd" (__HALT_COMPILER(); not found)
phar url "phar://(SigToolPath)/daily.cvd" is unknown in (SigToolPath)\SigTool.php on line 393

Warning: scandir(phar://(SigToolPath)/daily.cvd): Failed to open directory: internal corruption of phar "(SigToolPath)\daily.cvd" (__HALT_COMPILER(); not found)
phar url "phar://(SigToolPath)/daily.cvd" is unknown in (SigToolPath)\SigTool.php on line 393
PHP Warning:  scandir(): (errno 0): No error in (SigToolPath)\SigTool.php on line 393

Warning: scandir(): (errno 0): No error in (SigToolPath)\SigTool.php on line 393
 Done!
 Deleting daily.cvd ... Done!
 Stripping ClamAV package header from main.cvd ... Done!
 Decompressing main.cvd (GZ) ... Done!
 Extracting contents from main.cvd (TAR) to (SigToolPath) ...PHP Warning:  scandir(phar://(SigToolPath)/main.cvd): Failed to open directory: internal corruption of phar "(SigToolPath)\main.cvd" (__HALT_COMPILER(); not found)
phar url "phar://(SigToolPath)/main.cvd" is unknown in (SigToolPath)\SigTool.php on line 393

Warning: scandir(phar://(SigToolPath)/main.cvd): Failed to open directory: internal corruption of phar "(SigToolPath)\main.cvd" (__HALT_COMPILER(); not found)
phar url "phar://(SigToolPath)/main.cvd" is unknown in (SigToolPath)\SigTool.php on line 393
PHP Warning:  scandir(): (errno 0): No error in (SigToolPath)\SigTool.php on line 393

Warning: scandir(): (errno 0): No error in (SigToolPath)\SigTool.php on line 393
 Done!
 Deleting main.cvd ... Done!
 Accessing signatures.dat ... Done!
 Writing signatures.dat ... Done!

I've only discovered this just now, so I haven't looked into it too deeply yet, nor committed any fix yet.

In case anyone encounters this, as a temporary workaround, I would recommend, instead of running $ php SigTool.php xpmd, to just download the necessary files (daily.cvd and main.cvd) through your browser, dump those files into SigTool's working directory, and run $ php SigTool.php xp. Should still work fine without any problems just doing it that way.

Probably won't look any further into this tonight, since it's getting late. I'll look into this when I've got a moment and post any updates/progress/etc here at that time.

YAML reconstruction bug

A reconstructed signatures.dat, after processing new signature files, should be prefixed by ---; not by 0: false. The code doesn't currently accommodate this properly and needs to be fixed (and is also the cause of this problem here). I had intended to fix this a while ago, and already have a solution in mind, but have unfortunately been busy with other commitments and interests, and so haven't gotten around to sorting it out yet. Creating this issue as a personal reminder and to track progress, and I'll get it sorted out soon.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.