enygma / expose

An Intrusion Detection System library loosely based on PHP IDS

enygma @ Twitter || ccornutt @ LinkedIn

---

🔭 I’m currently working on...

Right now I'm working on building out a security education and training program here at GitHub. Our group, the Security Partners team, is chartered with introducing new (and maintaining current) educational content, training, and activities across the GH org, spreading the security knowledge as far as possible!

🌱 I’m currently learning...

There's lots of things I'm learning at the moment including:

• building out effective training programs
• practicing the definition of learning and performance objectives
• building team culture

oh, and there's some development thrown in there somewhere! 😁

💬 Ask me about...

If you're interested in tallking about application security, training approaches, educational programs or even just general development, hit me up at one of my profiles above!

expose's Issues

What is Manager Config for?

Im not quite sure, what's the purpose of the \Expose\Config class.
It is used in Manager, where you can set and get it, but it doesn't seem to be used ever.

Coulnd't it be removed?

Impact rating

Hello!

I would like to know which is the impact level (rating), especially the max impact number that it can return.

Thank you!

Uncaught exception: Class 'Mongo' not found

Running in Centos 7.4 PHP 7.2.
A package dependency missing from composer?

Cached entries always have an impact 0

Hi there,

first of all many thanks for your great work, very appreciated.

I just wonder if this is an intended behaviour or kind of a bug:
If caching is activated via $manager->setCache($cache); every request on a cached request will leave impact as 0 since the code setting impact is never reached (done in Manager.php function processFilters()). Thus any further processing based on impact assumes a clean, non-offending request.
Imho caching should just avoid the re-application of filter rules but it should not bypass actions taken on impact. In other words why should an offending request be considered harmless just by the fact that it has been cached?
Bests
Harald

Manager::setException() problem

Hi,
working with exceptions I've seen one problem. If I use manager's method setException() and I declare and exception with value = "(.)+" in this way (and and for sure there is more string combinations):

$manager->setException('(.)+');

I exclude all variables to the Expose's analysis. In other words: I could baypass Expose. One posible solution may be adding preg_quote() into the code of isException() function. Replacing:

if($path === $exception || preg_match('/^'.$exception.'$/', $path) !== 0) {

with:

if($path === $exception || preg_match('/^'.preg_quote($exception, '/').'$/', $path) !== 0) {

Regards.

Calling FilterCollection::getFilterData( $id ) results in an error

The function getFilterData works two different ways, though only one is used in Expose code - if called with no argument, it returns all filter data.

When called with an id, it's supposed to return ONLY that filter object, but throws an error instead.

In a new unit test:
Expose\FilterConnectionTest::testGetSetFilterImpact
Trying to get property of non-object

Will be submitting a pull request shortly, just documenting the separate issues.

allow setting of thresholds

Be able to configure the manager to only report/log things that match a certain impact threshold

Feature: Allow the ability to dynamically adjust the impact of individual rules

At php[world] 2015, I talked with Gregory Wilson about this, he suggested I submit a ticket. Here it is.

Essentially, it would be good to be able to adjust the impact level of individual rules in a filterset without having to copy and edit the rules into a new filterset. For example, if you find a lot of false positives for rule id 77, you could adjust the impact to 0. If you are particularly concerned about a particular attack, you could adjust it to 50, to make it trigger higher levels of action.

With a brief glance at how the FilterCollection works, an end user could actually hack this together now - run load(), run getFilterData, adjust the datastructure manually, then run setFilterData with your modified filter set. But there could be a much easier way created.

Bypassing ConvertSQL->convertFromSQLHex() 2/2

An other problem with this function is that it doesn't detect the other ways to write hexadecimal literals in mysql server. Reading documentation [1] we see other ways to do it:

SELECT X'4D7953514C'
SELECT x'0a'+0

Good luck
[1] https://dev.mysql.com/doc/refman/5.7/en/hexadecimal-literals.html

Alerting functionality

Currently the system doesn't do alerting. This needs to be added to:

1. First, notify an email address
2. Send notifications to another system...

It should be implemented with a service layer so these alerting types can be switched out.

make the CLI processing work with the custom queue

Right now, the CLI queue processing still assumes it's using the local Mongo instance.
It needs to take in the dsn option (already there, just not used) and use that to connect to the right queue database.

event IP submission to blocklist

Would be interesting to add the ability to administratively submit IP addresses that have made attempts to some of the public block lists.

Allow wildcard expressions for exceptions

As a technical user of expose, I would like to use wildcards when specifying wildcards.

h3. Example:

$manager->setException('foo.*.__utmz');

This would exclude every element that is one level below "foo". AMQP uses "#" for zero or one and "*" for one. Following that convention might be easier than glob expressions.

How can I save the datetime field in ISODate format in MongoDB

I want to change the datetime format to ISODate in MongoDB so that I can use MongoDB’s “time to live” or TTL collection feature.

I want to store the timestamp like this:

 "datetime": ISODate("2015-09-08T17:43:25.678Z") 

I have changed the class in Log/Mongo.php to use new \MongoDate

    public function log($level, $message, array $context = array())
    {
        $logger = new \Monolog\Logger('audit');
        try {
            $handler = new \Monolog\Handler\MongoDBHandler(
                new \MongoClient($this->getConnectString()),
                $this->getDbName(),
                $this->getDbCollection()
            );
        } catch (\MongoConnectionException $e) {
            throw new \Exception('Cannot connect to Mongo - please check your server');
        }
        $logger->pushHandler($handler);
        $logger->pushProcessor(function ($record) {
            //$record['datetime'] = $record['datetime']->format('U');

            $exampleDate = new \MongoDate();

            $record['datetime'] =  $exampleDate;

            return $record;
        });

        return $logger->$level($message, $context);
    }

But that doesn't convert the timestamp to ISODate but still store as an object:

 "datetime": "[object] (MongoDate: 0.32700000 1441719561)",

Would you please tell me how to store the datetime field in ISODate format?

allow CLI to execute off of data in a file

Pull in the contents of an external file (either POST-encoded or just plain text) and give results

Allow "offline" processing of data

Instead of trying to process all data at runtime, give the option to "queue" up the requests and analyze them by a cron job or some other monitoring process.

To implement:

1. Store incoming data (serialized maybe?) into the data storage (Mongo right now)
2. Create a console command to pull out the items not marked as processed and evaluate
3. Report these back to an admin user or some other alerting system

Bump Twig Version Requirement

Right now I am using the latest version of Twig which at this time is 1.18 with "minimum-stability": "RC", however Expose requires 1.13 and will cause the following error when updating via composer:

Your requirements could not be resolved to an installable set of packages.

Problem 1
- Installation request for enygma/expose 2.2 -> satisfiable by enygma/expose[2.2].
- enygma/expose 2.2 requires twig/twig 1.13.* -> no matching package found.

Potential causes:

• A typo in the package name
• The package is not available in a stable-enough version according to your minimum-stability setting
  see https://groups.google.com/d/topic/composer-dev/_g3ASeIFlrc/discussion for more details.

I can get by it by doing "twig/twig": "1.18 as 1.13" but that's not ideal in my opinion. I checked the Twig code for Expose and it does not appear to contain anything that could potentially break with this version bump. Can you verify on your end that is the case and please consider bumping it? Thank you.

Bypassing ConvertSQL->convertFromSQLHex() 1/2

the function convertFromSQLHex() converts SQLHEX to plain text. This meaning that if I use this attack vector:

SELECT 0x6D7973716C

The function must detect it and decode the string. But not. Try yourself and you'll see that converted string doesn't is:

SELECT mysql

This is due to the function detects ' 0x6D7973716C' instead of '0x6D7973716C', an extra white space that disturbs the later conversion process. To fix the problem replace line:

foreach (str_split(trim($match), 2) as $hex_index) {

With:

if (strpos($match, ' ') !== false) $converted = ' ';
foreach (str_split(trim($match), 2) as $hex_index) {

Regards

Fatal error: Class 'Expose\FilterCollection' not found

From the example...

Help?

Bug? Regex issue with input of 1,11,11,11,120<

There may be a bug with this input...

1,11,11,11,120<

I found this when I had a bunch of numbers separated by commas and then follwed by a
it gives a non well formed numerical value encountered on the convertFromJSCharcode function.

expose/src/Expose/Converter/ConvertJS.php

Line 36 in 64538aa

$converted .= chr($char);

Note: It must be < and not < in the input that goes through the IDS.

Issue also exists in the ancestor PHPIDS implementation.

I'm not exactly sure how to fix this elegantly. Maybe $char = intval($char). The problem lies in the way php does a $char <= 127 comparison when $char = "120<" it converts the string to a number by truncating the < off the end, so it passes the comparison, but then dies when chr("120<") tries to execute.

Rules 11 and 73: incomplete rules

Hi all,

analyzing the rules I see that filter rules 11 and 73 are an incomplete try to detect use of common folders. What happens with /var folder on linux environments? And with C:\Users, %windir% and the other windows environment variables? It's necessary a revision to adjust these rules to modern Windows environments.

Regards

Performance improvement

Please change setFilterData($data) to:
$filter = new \Expose\Filter();
foreach ($data as $index => $config) {
if (is_object($config)) {
$config = get_object_vars($config);
}
$filter->load($config);
$this->addFilter($filter);
}
Thank you.

Test Rules Against Known List Of Attack Strings & False Positives

I think we should have a way to test current rules against a list of known attack strings. I couldn't find anything in the source that does that. We have no idea when we modify rules if it breaks a rule and an attack that was once detected for that rule is no longer detected, or false positives that were ignored are now being tripped. Correct me if I am wrong if this is already being done somewhere. I just see a lot of room for breakage to occur as rules are updated for new attack patterns.

This is the original thread for PHPIDS, which contains the attack strings that the original rules were created for.

I'd like to propose that the attack strings on there be extracted so that the rules can be run against them as part of the tests.

This seem like the simplest way. Run each string against all rules, and then note down which rules it sets off. Then when we edit the rules we run through the strings again as part of our tests, and make sure they match with the prior recorded rule matches. I'll be happy to volunteer and extract all the attack strings that were posted on the tread. I understand this won't be a 100% breakage detecting solution, but it's better than not checking.

Test ordering of Converter methods for optimization

The order makes a difference. Running the conversion twice catches more evasions. Need to see if there is a more optimal order, or just run the converter a selectable number of times.

Replace direct monolog dependency with PSR-3 logger

Instead of depending directly on monolog, expose should suggest installing monolog but use any PSR-3 compatible logger.

SQL Injection

Are the filters configured to detect a SQL injection attempt?

I passed

anything' OR '1'='1

to one form and it didn't detect it.

implement caching of results

Cache the results of the processing by a hash of the dataset for replay attacks.

Improve potential XSS attack

It is possible to circumvent the IDS filter using the following vectors:

<p onmouseover=-a() ></p>
<p onmouseover=(a()) ></p>
<p onmouseover=;a() ></p>
<p onmouseover=!a() ></p>

Other types of events can also be use used for the attack (e.g. "onClick").

The vulnerability is allowed because the filter requires an alphanumeric character (and the symbols "_" "$") after an event followed by the "=" symbol in order to flag the input as an attack and increase the impact. By adding a symbol that does not break the JavaScript syntax (such as the "-" or the parenthesis "()" ) just after the "=" character permits an adversary to bypass this pattern check. Rules 71, 27, 2 and 65 are related to this insufficient pattern match. It is recommended to extend these regular expressions in order to also include the symbols in the accepted pattern.

The issue is clone of PHPIDS/PHPIDS#74

Is somebody using this package with Symfony 4 or newer?

I tried install package, but found that it is incompatible with my "symfony/console": "4.4.*"

So, is somebody using this package with Symfony 4 or newer?

Replace FilterCollection::load() with FilterCollectionFactory

To allow changing the data source, a filter collection is read from, FilterCollection::load() should be replaced by an abstraction that returns FilterCollections.

Fatal error: Uncaught Error: Class 'Mongo' not found

I'm using win 10 and install expose using:

composer require enygma/expose

And do the example code but i got this error:

Fatal error: Uncaught Error: Class 'Mongo' not found in C:\xampp\htdocs\test_expose\vendor\enygma\expose\src\Expose\Log\Mongo.php on line 206

Extra backslash in filter rules

The regexs pulled from PHPIDS have the forward slashes ("/") escaped with a backslash. This is not necessary in PHP strings, and is creating an extra backslash in the expose regex string.

For example: "/" in the PHPIDS regex has been escaped to "\/" instead of "/".

create a web-formatted output page

Default is to just show the most recent issues
Additional features could include graphs of trends over time

Things to add to the configuration

The following things need to be added to the configuration:

Mongo host (default localhost)
Mongo username (default none)
Mongo password (default none)

Contact email(s) for results

from CLI, pull out report summary of recent results

using the configured logging data source, pull out latest (default of 7 days) worth of results.
output to stdout by default, option to write to file

Decouple queueing mechanism from MongoDB

To allow using a different queueing system, like e.g. RabbitMQ, the Queue functionality should be decoupled from the actual queueing implementation. To allow as many queue technologies as possible, the requirements should be as simple as possible. A list of things from my knowledge more or less every queue supports:

• Append a message to the queue
• Read a single message
• Mark a single message as processed

Bypassing ConvertSQL->convertFromSQLKeywords()

The convertFromSQLKeywords() function try to convert several chars to others for a later analysis. But the line below has a problem:

$value = preg_replace('/,null/ims', ',0', $value);

If you send the string "select 1,null;" convertFromSQLKeywords() converts it into "select 1,0;" but adding an extra white space after semicolon ("select 1, null;") the function is bypassed. A possible solution is replace:

$value = preg_replace('/,null/ims', ',0', $value);

with:

$value = preg_replace('/,\s+null/ims', ',0', $value);

Regards

Support custom Converters

From @enygma : "having a method that can get the method names in the Converter class as an array and execute them that way. Then potentially we could allow the addition of custom converters and possibly closures as converters...could be helpful."

why convertFromNestedBase64() check only strings longer than 50 characters?

If you use an exploit vector that it's base64_encode version is less than 50 characters, you can bypass this function.Try with these two string: first of them is detected and the second one not.

dGhpcyBpcyBhIGRhbmdlcm91cyBzdHJpbmcsIGJlIGNhcmVmdWxs
c3RyaW5nIG5vdCBkZXRlY3RlZCBieSBleHBvc2V=

Is there any reason that I can't see for limit at 50 characters?

How to use expose?

from code, I think expose use symfony framework, if I want use expose to protect my website, should I setup symfony firstly? Can someone write down the steps to use expose detailedly

New release?

Any chance of getting a new release rolled? I'd rather not have to fork just to create a tag.

caaeb87 in particular prevents using this library to notify about possible intrusion attempts.

Integrate the concept of thresholds

As it stands (for now) all results are processed and reported back to the admin of the system. A "threshold" system needs to be put in place to help reduce the noise from false positives.

These thresholds should be related to the Impact scores so that only runs (inline or the queued) that meet or are equal to the threshold have notifications sent.

Future idea: have thresholds/watches set on individual filters so that if they're tripped, the admin is notified regardless of if the total Impact level falls below the threshold.

Expose doesn't detect encoded variables

If you do a test with the example code from [1] and change the $data variable adding 'baz':

$data = array( 'POST' => array( 'test1' => 'foo', 'bar' => array( 'baz' => '%3C%69%6D%67%20%73%72%63%3D%22%22%20%6F%6E%65%72%72%6F%72%3D%22%6A%61%76%61%73%63%72%69%70%74%3A%64%6F%63%75%6D%65%6E%74%2E%77%72%69%74%65%22%3E', 'testing' => '<script>test</script>', 'path' => '../transversal/path' ) ), );

Expose doesn't detect attack on 'baz' variable ( baz's value = HTTP_encoded ('<img src="" onerror="javascript:document.write">') ). Any encoded variable isn't detected. If you see how PHPIDS do this you'll see that there is a class, IDS_Converter [2], to check these encoded variables.

[1] https://github.com/enygma/expose
[2] https://github.com/PHPIDS/PHPIDS/blob/master/lib/IDS/Converter.php

symfony/console v3

Need to update reqs to support symfony/console 3. Current setup makes this difficult on Laravel (and probably others)

composer require enygma/expose:dev-master