GithubHelp home page GithubHelp logo

teler-sh / teler-waf Goto Github PK

View Code? Open in Web Editor NEW
323.0 6.0 30.0 893 KB

teler-waf is a Go HTTP middleware that protects local web services from OWASP Top 10 threats, known vulnerabilities, malicious actors, botnets, unwanted crawlers, and brute force attacks.

Home Page: https://test.teler.sh

License: Apache License 2.0

Go 98.71% Makefile 1.29%
ids teler waf teler-ids teler-waf go golang http middleware router

teler-waf's Introduction

teler-waf

GoDoc codecov tests OpenSSF Scorecard Mentioned in Awesome Go

teler-waf is a comprehensive security solution for Go-based web applications. It acts as an HTTP middleware, providing an easy-to-use interface for integrating IDS functionality with teler IDS into existing Go applications. By using teler-waf, you can help protect against a variety of web-based attacks, such as cross-site scripting (XSS) and SQL injection.

The package comes with a standard net/http.Handler, making it easy to integrate into your application's routing. When a client makes a request to a route protected by teler-waf, the request is first checked against the teler IDS to detect known malicious patterns. If no malicious patterns are detected, the request is then passed through for further processing.

In addition to providing protection against web-based attacks, teler-waf can also help improve the overall security and integrity of your application. It is highly configurable, allowing you to tailor it to fit the specific needs of your application.

See also:

Features

teler-waf offers a range of powerful features designed to enhance the security of your Go web applications:

  • HTTP middleware for Go web applications.
  • Integration of teler IDS functionality.
  • Detection of known malicious patterns using the teler IDS.
    • Common web attacks, such as cross-site scripting (XSS) and SQL injection, etc.
    • CVEs, covers known vulnerabilities and exploits.
    • Bad IP addresses, such as those associated with known malicious actors or botnets.
    • Bad HTTP referers, such as those that are not expected based on the application's URL structure or are known to be associated with malicious actors.
    • Bad crawlers, covers requests from known bad crawlers or scrapers, such as those that are known to cause performance issues or attempt to extract sensitive information from the application.
    • Directory bruteforce attacks, such as by trying common directory names or using dictionary attacks.
  • Providing increased flexibility for creating your own custom rules.
  • Configuration options to whitelist specific types of requests based on their URL or headers.
  • Easy integration with many frameworks.
  • High configurability to fit the specific needs of your application.

Overall, teler-waf provides a comprehensive security solution for Go-based web applications, helping to protect against web-based attacks and improve the overall security and integrity of your application.

Install

Dependencies:

  • gcc (GNU Compiler Collection) should be installed & configured to compile teler-waf.

To install teler-waf in your Go application, run the following command to download and install the teler-waf package:

go get github.com/teler-sh/teler-waf

Usage

Warning

Deprecation notice: Threat exclusions (Excludes) will be deprecated in the upcoming release (v2). See #73 & #64.

Here is an example of how to use teler-waf in a Go application:

  1. Import the teler-waf package in your Go code:
import "github.com/teler-sh/teler-waf"
  1. Use the New function to create a new instance of the Teler type. This function takes a variety of optional parameters that can be used to configure teler-waf to suit the specific needs of your application.
waf := teler.New()
  1. Use the Handler method of the Teler instance to create a net/http.Handler. This handler can then be used in your application's HTTP routing to apply teler-waf's security measures to specific routes.
handler := waf.Handler(http.HandlerFunc(yourHandlerFunc))
  1. Use the handler in your application's HTTP routing to apply teler-waf's security measures to specific routes.
http.Handle("/path", handler)

That's it! You have configured teler-waf in your Go application.

Options:

For a list of the options available to customize teler-waf, see the teler.Options struct.

Examples

Here is an example of how to customize the options and rules for teler-waf:

// main.go
package main

import (
	"net/http"

	"github.com/teler-sh/teler-waf"
	"github.com/teler-sh/teler-waf/request"
	"github.com/teler-sh/teler-waf/threat"
)

var myHandler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	// This is the handler function for the route that we want to protect
	// with teler-waf's security measures.
	w.Write([]byte("hello world"))
})

var rejectHandler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	// This is the handler function for the route that we want to be rejected
	// if the teler-waf's security measures are triggered.
	http.Error(w, "Sorry, your request has been denied for security reasons.", http.StatusForbidden)
})

func main() {
	// Create a new instance of the Teler type using the New function
	// and configure it using the Options struct.
	telerMiddleware := teler.New(teler.Options{
		// Exclude specific threats from being checked by the teler-waf.
		Excludes: []threat.Threat{
			threat.BadReferrer,
			threat.BadCrawler,
		},
		// Specify whitelisted URIs (path & query parameters), headers,
		// or IP addresses that will always be allowed by the teler-waf
		// with DSL expressions.
		Whitelists: []string{
			`request.Headers matches "(curl|Go-http-client|okhttp)/*" && threat == BadCrawler`,
			`request.URI startsWith "/wp-login.php"`,
			`request.IP in ["127.0.0.1", "::1", "0.0.0.0"]`,
			`request.Headers contains "authorization" && request.Method == "POST"`
		},
		// Specify file path or glob pattern of custom rule files.
		CustomsFromRule: "/path/to/custom/rules/**/*.yaml",
		// Specify custom rules for the teler-waf to follow.
		Customs: []teler.Rule{
			{
				// Give the rule a name for easy identification.
				Name:      "Log4j Attack",
				// Specify the logical operator to use when evaluating the rule's conditions.
				Condition: "or",
				// Specify the conditions that must be met for the rule to trigger.
				Rules: []teler.Condition{
					{
						// Specify the HTTP method that the rule applies to.
						Method: request.GET,
						// Specify the element of the request that the rule applies to
						// (e.g. URI, headers, body).
						Element: request.URI,
						// Specify the pattern to match against the element of the request.
						Pattern: `\$\{.*:\/\/.*\/?\w+?\}`,
					},
				},
			},
			{
				// Give the rule a name for easy identification.
				Name: `Headers Contains "curl" String`,
				// Specify the conditions that must be met for the rule to trigger.
				Rules: []teler.Condition{
					{
						// Specify the DSL expression that the rule applies to.
						DSL: `request.Headers contains "curl"`,
					},
				},
			},
		},
		// Specify the file path to use for logging.
		LogFile: "/tmp/teler.log",
	})

	// Set the rejectHandler as the handler for the telerMiddleware.
	telerMiddleware.SetHandler(rejectHandler)

	// Create a new handler using the handler method of the Teler instance
	// and pass in the myHandler function for the route we want to protect.
	app := telerMiddleware.Handler(myHandler)

	// Use the app handler as the handler for the route.
	http.ListenAndServe("127.0.0.1:3000", app)
}

For more examples of how to use teler-waf or integrate it with any framework, take a look at examples/ directory.

Custom Rules

Tip

If you want to explore configurations, delve into crafting custom rules and composing DSL expressions, you can practice and gain hands-on experience by using this teler WAF playground. Here, you can also simulate requests customized to fulfill the specific needs of your application.

To integrate custom rules into the teler-waf middleware, you have two choices: Customs and CustomsFromFile. These options offer flexibility to create your own security checks or override the default checks provided by teler-waf.

  • Customs option

You can define custom rules directly using the Customs option, as shown in the example above.

In the Customs option, you provide an array of teler.Rule structures. Each teler.Rule represents a custom rule with a unique name and a condition that specifies how the individual conditions within the rule are evaluated (or or and). The rule consists of one or more teler.Condition structures, each defining a specific condition to check. Conditions can be based on the HTTP method, element (headers, body, URI, or any), and a regex pattern or a DSL expression to match against.

  • CustomsFromFile option

Alternatively, the CustomsFromFile option allows you to load custom rules from external files, offering even greater flexibility and manageability. These rules can be defined in YAML format, with each file containing one or more rules. Here is an example YAML structure representing a custom rule:

- name: <name>
  condition: <condition> # Valid values are: "or" or "and", in lowercase or uppercase.
  rules:
    - method: <method> # Valid methods are: "ALL", "CONNECT", "DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT", and "TRACE". Please refer to https://pkg.go.dev/github.com/teler-sh/teler-waf/request for further details.
      element: <element> # Valid elements are: "headers", "body", "uri", and "any", in lowercase, uppercase, or title case (except for "uri").
      pattern: "<pattern>" # Regular expression pattern
    - dsl: "<expression>" # DSL expression

Important

Please note that the condition, method, and element are optional parameters. The default values assigned to them are as follows: condition is set to or, method is set to ALL, and element is set to ANY. Therefore, if desired, you can leave those parameters empty. The pattern parameter is mandatory, unless you specify a dsl expression. In such cases, when a dsl expression is provided, teler-waf will disregard any values assigned to method and element, even if they are defined. To see some examples, you can refer to the tests/rules/ directory.

You can specify the CustomsFromFile option with the actual file path or glob pattern pointing to the location of your custom rule files. For example:

// Create a new instance of the Teler middleware and
// specify custom rules with the CustomsFromFile option.
telerMiddleware := teler.New(teler.Options{
    CustomsFromFile: "/path/to/custom/rules/**/*.yaml",
})

With CustomsFromFile, you provide the file path or glob pattern where your custom rule files are located. The pattern can include wildcards to match multiple files or a directory and its subdirectories. Each file should contain one or more custom rules defined in the proper YAML format.

By utilizing either the Customs, CustomsFromFile, or both option, you can seamlessly integrate your custom rules into the teler-waf middleware, enhancing its security capabilities to meet your specific requirements.

DSL Expression

DSL (Domain-Specific Language) expressions provide a powerful means of defining conditions that are used to evaluate incoming requests within the context of custom rules or whitelists. With DSL expressions, you can create sophisticated and targeted conditions based on different attributes of the incoming requests. Here are some illustrative examples of DSL expression code:

Examples of DSL expression code:

Check if the incoming request headers contains "curl":

request.Headers contains "curl"

Check if the incoming request method is "GET":

request.Method == "GET"

Check if the incoming request method is "GET" or "POST" using regular expression, matches operator:

request.Method matches "^(POS|GE)T$"

Check if the incoming request IP address is from localhost:

request.IP in ["127.0.0.1", "::1", "0.0.0.0"]

Check if the any element in request contains the string "foo":

one(request.ALL, # contains "foo")

Check if the incoming request body contains "foo":

request.Body contains "foo"

Check whether the current threat category being analyzed is bad crawler or directory bruteforce:

threat in [BadCrawler, DirectoryBruteforce]

Those examples provide a glimpse into the expressive capabilities of DSL expressions, allowing you to define intricate conditions based on various request attributes. By leveraging these expressions, you can effectively define the criteria for evaluating incoming requests and tailor your custom rules or whitelists accordingly, enabling fine-grained control over your application's behavior.

Available variables

When working with DSL expressions, you have access to various variables that provide valuable information about the incoming requests and the threat category being analyzed. Here is a detailed description of the available variables:

  • Threat category

    All constant identifiers of the threat.Threat type can be used as valid variables. These identifiers represent different threat categories that are relevant to your analysis.

  • request

    The request variable represents the incoming request and provides access to its fields and corresponding values. The following sub-variables are available within the request variable:

    • request.URI: Represents the URI of the incoming request, including the path, queries, parameters, and fragments.
    • request.Headers: Represents the headers of the incoming request, presented in multiple lines.
    • request.Body: Represents the body of the incoming request.
    • request.Method: Represents the method of the incoming request.
    • request.IP: Represents the client IP address associated with the incoming request.
    • request.ALL: Represents all the string values from the request fields mentioned above in a slice.
  • threat

    The threat variable represents the threat category being analyzed. It is of the threat.Threat type and allows you to evaluate and make decisions based on the specific threat category associated with the request.

By utilizing those variables within your DSL expressions, you can effectively access and manipulate the attributes of the incoming requests and assess the relevant threat categories. This enables you to create custom rule conditions that tailored to your specific use case.

Available functions

Also, you have access to a variety of functions. These functions encompass both the built-in functions provided by the expr package and those specifically defined within the DSL package. The functions utilize the functionalities offered by the built-in strings Go package. Here is a detailed list of the functions available:

  • cidr: Get all IP addresses in range with given CIDR.
  • clone: Create a copy of a string.
  • containsAny: Check if a string contains any of the specified substrings.
  • equalFold: Compare two strings in a case-insensitive manner.
  • hasPrefix: Check if a string has a specified prefix.
  • hasSuffix: Check if a string has a specified suffix.
  • join: Concatenate multiple strings using a specified separator.
  • repeat: Repeat a string a specified number of times.
  • replace: Replace occurrences of a substring within a string.
  • replaceAll: Replace all occurrences of a substring within a string.
  • request: Access request-specific information within the DSL expression.
  • threat: Access information related to the threat category being analyzed.
  • title: Convert a string to title case.
  • toLower: Convert a string to lowercase.
  • toTitle: Convert a string to title case.
  • toUpper: Convert a string to uppercase.
  • toValidUTF8: Convert a string to a valid UTF-8 encoded string.
  • trim: Remove leading and trailing whitespace from a string.
  • trimLeft: Remove leading whitespace from a string.
  • trimPrefix: Remove a specified prefix from a string.
  • trimRight: Remove trailing whitespace from a string.
  • trimSpace: Remove leading and trailing whitespace and collapse consecutive whitespace within a string.
  • trimSuffix: Remove a specified suffix from a string.

For more comprehensive details on operators and built-in functions, please refer to the Expr documentation. It provides a comprehensive guide to utilizing operators and exploring the available built-in functions in your DSL expressions.

Streamlined Configuration Management

For effective configuration, it's essential to define a range of settings, including whitelists, custom rule definitions, logging preferences, and other parameters. The option package streamlines this configuration workflow by enabling you to efficiently unmarshal or load configuration data from JSON and YAML formats into a format that teler-waf can readily comprehend and implement.

// Load configuration from a YAML file.
opt, err := option.LoadFromYAMLFile("/path/to/teler-waf.conf.yaml")
if err != nil {
    panic(err)
}

// Create a new instance of the Teler type with
// the loaded options.
telerMiddleware := teler.New(opt)

Development

By default, teler-waf caches all incoming requests for 15 minutes & clear them every 20 minutes to improve the performance. However, if you're still customizing the settings to match the requirements of your application, you can disable caching during development by setting the development mode option to true. This will prevent incoming requests from being cached and can be helpful for debugging purposes.

// Create a new instance of the Teler type using
// the New function & enable development mode option.
telerMiddleware := teler.New(teler.Options{
	Development: true,
})

Logs

Here is an example of what the log lines would look like if teler-waf detects a threat on a request:

{"level":"warn","ts":1672261174.5995026,"msg":"bad crawler","id":"654b85325e1b2911258a","category":"BadCrawler","caller":"teler-waf","listen_addr":"127.0.0.1:36267","request":{"method":"GET","path":"/","ip_addr":"127.0.0.1:37702","headers":{"Accept":["*/*"],"User-Agent":["curl/7.81.0"]},"body":""}}
{"level":"warn","ts":1672261175.9567692,"msg":"directory bruteforce","id":"b29546945276ed6b1fba","category":"DirectoryBruteforce","caller":"teler-waf","listen_addr":"127.0.0.1:36267","request":{"method":"GET","path":"/.git","ip_addr":"127.0.0.1:37716","headers":{"Accept":["*/*"],"User-Agent":["X"]},"body":""}}
{"level":"warn","ts":1672261177.1487508,"msg":"Detects common comment types","id":"75412f2cc0ec1cf79efd","category":"CommonWebAttack","caller":"teler-waf","listen_addr":"127.0.0.1:36267","request":{"method":"GET","path":"/?id=1%27%20or%201%3D1%23","ip_addr":"127.0.0.1:37728","headers":{"Accept":["*/*"],"User-Agent":["X"]},"body":""}}

The id is a unique identifier that is generated when a request is rejected by teler-waf. It is included in the HTTP response headers of the request (X-Teler-Req-Id), and can be used to troubleshoot issues with requests that are being made to the website.

For example, if a request to a website returns an HTTP error status code, such as a 403 Forbidden, the teler request ID can be used to identify the specific request that caused the error and help troubleshoot the issue.

Teler request IDs are used by teler-waf to track requests made to its web application and can be useful for debugging and analyzing traffic patterns on a website.

Custom Response

By default, teler-waf employs the DefaultHTMLResponse as the standard response when a request is rejected or blocked. However, teler-waf offers a high degree of customization, empowering you to tailor the response to your specific requirements. The customization can be achieved using the Status, HTML, or HTMLFile options, all of which are part of the Response interface.

Here's how you can make use of these options in your code:

// Create a new instance of the Teler middleware
telerMiddleware := teler.New(teler.Options{
	// Customize the response for rejected requests
	Response: teler.Response{
		Status: 403,
		HTML: "Your request has been denied for security reasons. Ref ID: {{ID}}.",
		// Alternatively, you can use HTMLFile to point to a custom HTML file
		HTMLFile: "/path/to/custom-403.html",
	},
})

With this level of customization, you can construct personalized and informative responses to be shown when teler-waf blocks or rejects a request. The HTML option permits you to directly specify the desired HTML content as a string, whereas the HTMLFile option enables you to reference an external file containing the custom HTML response.

Moreover, to enhance the user experience, you can leverage placeholders in your HTML content to generate dynamic elements. During runtime, these placeholders will be substituted with actual values, resulting in more contextually relevant responses. The available and supported placeholders include:

  • {{ID}}: Request IDs, allowing for unique identification of each rejected request.
  • {{message}}: Rejected messages conveying the reason for request blocking.
  • {{threat}}: Threat categories, providing insights into the detected security threat.

By incorporating these placeholders, you can create detailed and comprehensive responses that effectively communicate the rationale behind request rejections or blocks.

Falco Sidekick

Falco Sidekick is a tool that receives events from Falco, an open-source cloud-native runtime security project, and sends them to different output channels. It allows you to forward security alerts to various third-party systems such as Slack, Elasticsearch, Loki, Grafana, Datadog and more. This enables security teams to efficiently monitor and respond to security threats and events in real-time.

Integrating Falco Sidekick with teler-waf is also possible. By using the FalcoSidekickURL option, you can configure teler-waf to send events to Falco Sidekick, which will receive and process them for you. To do this, simply create a new instance of the Teler type using the New function and provide the FalcoSidekickURL option with the URL of your Falco Sidekick instance. For example:

// Create a new instance of the Teler type using
// the New function & integrate Falco Sidekick.
telerMiddleware := teler.New(teler.Options{
	FalcoSidekickURL: "http://localhost:2801",
})

Once you have set up this integration, any threats detected by teler-waf will be sent to Falco Sidekick, which can then take appropriate actions based on the configuration you have set up. For instance, you can set up Falco Sidekick to automatically send alerts to your incident response team.

teler-waf's Falco Sidekick event

Event

The event forwarded to Falco Sidekick instance includes the following information:

  • output: Represents the alert message.
  • priority: The priority is consistently denoted as warning, a nod to the urgency associated with security-related events.
  • rule: Indicates the specific rule (message) that matched the associated request.
  • time: Event's generation timestamp.
  • Output fields:
    • teler.caller: Identifies the application source that invoked teler-waf.
    • teler.id: Represents a unique identifier for the rejected request.
    • teler.threat: Specifies the category of the threat.
    • teler.listen_addr: Denotes the network address on which teler-waf is listening for incoming requests.
      • request.body: Contains the body of the associated request.
      • request.headers: Lists the headers from the associated request.
      • request.ip_addr: Discloses the IP address of the associated request.
      • request.method: States the HTTP method employed in the associated request.
      • request.path: Refers to the path of the associated request.

Overall, Falco Sidekick is a versatile tool that can help you automate your security response process and improve your overall security posture. By leveraging its capabilities, you can ensure that your cloud-native applications are secure and protected against potential threats.

Wazuh

You can enhance your security monitoring by integrating teler WAF logs into Wazuh. To do this, use custom rules available in the extras/ directory.

Add the localfile element block below inside ossec_config element in the local configuration file:

<ossec_config>
  <!-- ... -->
  <localfile>
    <log_format>syslog</log_format>
    <location>/path/to/your/teler.log</location>
  </localfile>
</ossec_config>

Note

The value of location should be the teler WAF log file path you specified in Options.LogFile.

By doing this, Wazuh will be able to read and analyze the teler WAF logs, enhancing your network protection and providing better insights.

Datasets

The teler-waf package utilizes a dataset of threats to identify and analyze each incoming request for potential security threats. This dataset is updated daily, which means that you will always have the latest resource. The dataset is initially stored in the user-level cache directory (on Unix systems, it returns $XDG_CACHE_HOME/teler-waf as specified by XDG Base Directory Specification if non-empty, else $HOME/.cache/teler-waf. On Darwin, it returns $HOME/Library/Caches/teler-waf. On Windows, it returns %LocalAppData%/teler-waf. On Plan 9, it returns $home/lib/cache/teler-waf) on your first launch. Subsequent launch will utilize the cached dataset, rather than downloading it again.

Note

The threat datasets are obtained from the teler-sh/teler-resources repository.

However, there may be situations where you want to disable automatic updates to the threat dataset. For example, you may have a slow or limited internet connection, or you may be using a machine with restricted file access. In these cases, you can set an option called NoUpdateCheck to true, which will prevent the teler-waf from automatically updating the dataset.

Caution

Enabling the InMemory takes precedence and ensures that automatic updates remain enabled.

// Create a new instance of the Teler type using the New
// function & disable automatic updates to the threat dataset.
telerMiddleware := teler.New(teler.Options{
	NoUpdateCheck: true,
})

Finally, there may be cases where it's necessary to load the threat dataset into memory rather than saving it to a user-level cache directory. This can be particularly useful if you're running the application or service on a distroless or runtime image, where file access may be limited or slow. In this scenario, you can set an option called InMemory to true, which will load the threat dataset into memory for faster access.

// Create a new instance of the Teler type using the
// New function & enable in-memory threat datasets store.
telerMiddleware := teler.New(teler.Options{
	InMemory: true,
})

Caution

This may also consume more system resources, so it's worth considering the trade-offs before making this decision.

Resources

  • teler WAF tester! โ€” You are free to use the following site for testing, https://test.teler.sh.
  • teler WAF playground โ€” Simulate your requests tailored to meet the specific needs of your app, https://play.teler.sh.

Security

If you discover a security issue, please bring it to their attention right away, we take security seriously!

Reporting a Vulnerability

If you have information about a security issue, or vulnerability in this teler-waf package, and/or you are able to successfully execute such as cross-site scripting (XSS) and pop-up an alert in our demo site (see resources), please do NOT file a public issue โ€” instead, kindly send your report privately via the vulnerability report form.

Limitations

Here are some limitations of using teler-waf:

  • Performance overhead: teler-waf may introduce some performance overhead, as the teler-waf will need to process each incoming request. If you have a high volume of traffic, this can potentially slow down the overall performance of your application significantly. See benchmark below:
$ go test -bench "^BenchmarkAnalyze" -cpu=4 
goos: linux
goarch: amd64
pkg: github.com/teler-sh/teler-waf
cpu: 11th Gen Intel(R) Core(TM) i9-11900H @ 2.50GHz
BenchmarkAnalyzeDefault-4                      	  266018	      4018 ns/op	    2682 B/op	      74 allocs/op
BenchmarkAnalyzeCommonWebAttack-4              	  374410	      3126 ns/op	    2090 B/op	      68 allocs/op
BenchmarkAnalyzeCVE-4                          	  351668	      3296 ns/op	    2402 B/op	      68 allocs/op
BenchmarkAnalyzeBadIPAddress-4                 	  416152	      2967 ns/op	    1954 B/op	      63 allocs/op
BenchmarkAnalyzeBadReferrer-4                  	  410858	      3033 ns/op	    2098 B/op	      64 allocs/op
BenchmarkAnalyzeBadCrawler-4                   	  346707	      2964 ns/op	    1953 B/op	      63 allocs/op
BenchmarkAnalyzeDirectoryBruteforce-4          	  377634	      3062 ns/op	    1953 B/op	      63 allocs/op
BenchmarkAnalyzeCustomRule-4                   	  432568	      2594 ns/op	    1954 B/op	      63 allocs/op
BenchmarkAnalyzeWithoutCommonWebAttack-4       	  354930	      3460 ns/op	    2546 B/op	      69 allocs/op
BenchmarkAnalyzeWithoutCVE-4                   	  304500	      3491 ns/op	    2234 B/op	      69 allocs/op
BenchmarkAnalyzeWithoutBadIPAddress-4          	  288517	      3924 ns/op	    2682 B/op	      74 allocs/op
BenchmarkAnalyzeWithoutBadReferrer-4           	  298168	      3667 ns/op	    2538 B/op	      73 allocs/op
BenchmarkAnalyzeWithoutBadCrawler-4            	  276108	      4023 ns/op	    2682 B/op	      74 allocs/op
BenchmarkAnalyzeWithoutDirectoryBruteforce-4   	  276699	      3627 ns/op	    2682 B/op	      74 allocs/op
PASS
ok  	github.com/teler-sh/teler-waf	32.093s

Note

Benchmarking results may vary and may not be consistent. The teler-resources dataset may have increased since then, which may impact the results.

  • Configuration complexity: Configuring teler-waf to suit the specific needs of your application can be complex, and may require a certain level of expertise in web security. This can make it difficult for those who are not familiar with application firewalls and IDS systems to properly set up and use teler-waf.
  • Limited protection: teler-waf is not a perfect security solution, and it may not be able to protect against all possible types of attacks. As with any security system, it is important to regularly monitor and maintain teler-waf to ensure that it is providing the desired level of protection.

Known Issues

To view a list of known issues with teler-waf, please filter the issues by the "known-issue" label.

Community

We use the Google Groups as our dedicated mailing list. Subscribe to teler-announce via [email protected] for important announcements, such as the availability of new releases. This subscription will keep you informed about significant developments related to teler IDS, teler WAF, teler Proxy, teler Caddy, and teler Resources.

For any inquiries, discussions, or issues are being tracked here on GitHub. This is where we actively manage and address these aspects of our community engagement.

License

This package is made available under a dual license: the Apache License 2.0 and the Elastic License 2.0 (ELv2) (for the main package, teler).

You can use it freely inside your organization to protect your applications. However, you cannot use the main package to create a cloud, hosted, or managed service, or for any commercial purposes, as you would need a commercial license for that โ€“ though this option is not currently available. If you are interested in obtaining this commercial license for uses not authorized by ELv2, please reach out to @dwisiswant0.

teler-waf and any contributions are copyright ยฉ by Dwi Siswanto 2022-2024.

teler-waf's People

Contributors

dependabot[bot] avatar dwisiswant0 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

teler-waf's Issues

[proposal] Remove `dsl` package

Summary

The dsl package will be removed from the codebase and migrated into a new repository, https://github.com/teler-sh/dsl.

Motivation

The current codebase has become too complex, and maintaining the dsl package within it has become increasingly difficult. Migrating the dsl package into its own repository will help in better organization, management, and versioning of this specific functionality. It will also facilitate easier collaboration and contribution from external parties who may only be interested in the DSL aspect.

Design

The dsl package will be extracted from the existing codebase and placed into its own repository. This will involve creating a new repository dedicated to the dsl (done at https://github.com/teler-sh/dsl) package and transferring all relevant code, documentation, and tests into it. The package will then be published as a standalone module, allowing it to be independently versioned and distributed.

Drawbacks

  • Increased complexity in managing multiple repositories.
  • Potential fragmentation of documentation and contribution efforts.
  • Additional overhead in maintaining separate versions for the dsl package.

Alternative Approaches

An alternative approach would be to refactor the existing codebase to simplify the dsl package within it. However, given the current complexity of the codebase, this may not be a feasible solution and could lead to further complications.

Unresolved Questions

What parts of the design are still undecided? Are there any outstanding questions or concerns that need to be addressed? For instance:

[bug] race condition on checkFalcoEvents

$ go test -v -cover -race -count=1 -run="TestNewWithFalcoSidekickURL" ./
=== RUN   TestNewWithFalcoSidekickURL
==================
WARNING: DATA RACE
Read at 0x00c0005b4760 by goroutine 106:
  github.com/kitabisa/teler-waf.(*Teler).checkFalcoEvents()
      /home/dw1/Tools/teler-waf/falcosidekick.go:65 +0x1c5
  github.com/kitabisa/teler-waf.New.func2()
      /home/dw1/Tools/teler-waf/teler.go:348 +0x33

Previous write at 0x00c0005b4760 by goroutine 112:
  github.com/kitabisa/teler-waf.(*Teler).sendLogs()
      /home/dw1/Tools/teler-waf/teler.go:429 +0x1265
  github.com/kitabisa/teler-waf.(*Teler).postAnalyze()
      /home/dw1/Tools/teler-waf/teler.go:371 +0x144
  github.com/kitabisa/teler-waf.TestNewWithFalcoSidekickURL.(*Teler).Handler.func2()
      /home/dw1/Tools/teler-waf/handler.go:56 +0xd2
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2136 +0x47
  net/http.serverHandler.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2938 +0x2a1
  net/http.(*conn).serve()
      /usr/local/go/src/net/http/server.go:2009 +0xc24
  net/http.(*Server).Serve.func3()
      /usr/local/go/src/net/http/server.go:3086 +0x4f

Goroutine 106 (running) created at:
  github.com/kitabisa/teler-waf.New()
      /home/dw1/Tools/teler-waf/teler.go:348 +0x2676
  github.com/kitabisa/teler-waf.TestNewWithFalcoSidekickURL()
      /home/dw1/Tools/teler-waf/teler_test.go:273 +0xfd
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /usr/local/go/src/testing/testing.go:1648 +0x44

Goroutine 112 (running) created at:
  net/http.(*Server).Serve()
      /usr/local/go/src/net/http/server.go:3086 +0x80c
  net/http/httptest.(*Server).goServe.func1()
      /usr/local/go/src/net/http/httptest/server.go:310 +0xaa
==================
    testing.go:1465: race detected during execution of test
--- FAIL: TestNewWithFalcoSidekickURL (9.04s)
=== NAME  
    testing.go:1465: race detected during execution of test
FAIL
coverage: 46.6% of statements
FAIL	github.com/kitabisa/teler-waf	9.102s
FAIL

[docs] teler-waf v1.1.7 comparison

System Information
Operating System: Linux
Architecture: amd64
CPU: 11th Gen Intel(R) Core(TM) i9-11900H @ 2.50GHz
GOMAXPROCS: 4
teler-waf version: v1.1.7

Compilation

ms/op Mb/op allocs/op threat rules
teler WAF 21.2 44.7 114K 25828
OWASP Coraza WAF v3 66.3 68.8 694K 6

Note
teler WAF result of BenchmarkInitializeDefault, run yours with make bench-initalize command.
OWASP Coraza WAF v3 result of BenchmarkCRSCompilation.

Analyzed requests

ฮผs/op Mb/op allocs/op
teler WAF 4.0 2.7 74
OWASP Coraza WAF v3 1425.1 312.6 6938

Note
teler WAF result of BenchmarkAnalyzeDefault, run yours with make bench-analyze command.
OWASP Coraza WAF v3 result of BenchmarkCRSSimplePOST.

reqs/s transfer/s avg. req time fastest req slowest req
teler WAF 71167 7.53MB 351.285ยตs 26.25ยตs 6.80ms
OWASP Coraza WAF v2 624 - - - -
OWASP Coraza WAF v3 892 - - - -
ModSecurity 842 - - - -

Note
teler WAF result of go-wrk -c 25 -d 10 "http://localhost:3000/path?query=value#fragments" -H "Referrer: https://teler.sh/" -H "User-Agent: X" -body "some=body".
OWASP Coraza WAF v2, v3, and ModSecurity results of Simple URLENCODED Request according to https://coraza.io/docs/reference/benchmarks/.

Conclusion

In conclusion, the benchmark results clearly indicate that teler WAF outperforms OWASP Coraza WAF v3 in terms of both compilation and request analysis. teler WAF demonstrates significantly lower values in the compilation phase, highlighting its efficiency in preparing and initializing the threat rules.

Moreover, teler WAF excels with remarkable lower values in the analysis of requests compared to OWASP Coraza WAF v3. This indicates that teler WAF is highly efficient at processing incoming requests, offering faster response times and lower memory consumption. Additionally, teler WAF maintains a substantially higher throughput with an impressive request per second rate and it operates at approx. 80x faster than OWASP Coraza WAF v3. This impressive performance can be attributed to teler WAF default behavior of caching all analyzed requests, a crucial feature for applications that demand high-speed traffic handling.

In summary, based on these benchmark results, teler WAF stands out as a high-performance web application firewall solution, providing faster compilation times and superior request handling performance when compared to OWASP Coraza WAF v3. These findings make teler WAF a compelling choice for organizations seeking enhanced security without compromising on speed and resource efficiency in their web applications.

[feature] caching requests before further analysis

Is your feature request related to a problem? Please describe.

caching requests before further analysis

Describe the solution you'd like

Added a Cache field that has a value of time.Duration, and a Development mode that has a bool value to the Options.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[bug] nil pointer dereference in checkBadReferrer method

Describe the bug

A clear and concise description of what the bug is.

To Reproduce

Steps to reproduce the behavior:

Your teler usage & options...

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • OS: [e.g. mac, linux]
  • OS version: [uname -a]
  • teler Version [see go.mod]

Additional context

Add any other context about the problem here. Full output log is probably a helpful thing to add here.

[feature] Caching any DSL expressions

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[feature] Load options from file

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[proposal] Redesign Custom Response Configuration

Summary

This proposal outlines the design changes for configuration related to custom response handling. The primary modifications involve introducing a new structure for custom response headers and replacing the html and html_file fields with content and content_file, and expression support for dynamic content generation. Additionally, it suggests the inclusion of placeholders for dynamic content generation in the response.

Motivation

The motivation behind this proposal is to enhance the flexibility and functionality of custom response configuration for rejected requests. Users can set custom headers as needed by introducing response headers as a map. The shift from html to content and content_file allows for arbitrary content types by enabling users to specify the Content-Type header. Introducing placeholders in the response content empowers users to create dynamic responses tailored to specific conditions, improving the versatility of teler-waf.

Design

The proposed design changes are as follows:

1. Custom Response Headers

Introduce a new field in the configuration called headers, which is of type map[string]string. Users can define custom response headers using this map. This change allows for greater control over the HTTP response.

2. Content Handling

Substitute the existing html and html_file fields with content and content_file. The content field is designed to accommodate response content as a string, while content_file provides the flexibility for users to specify a file from which the response content can be sourced. These updates enhance the configuration's ability to effectively manage a wide range of content types. Furthermore, these changes introduce support for expressions, enabling the creation of dynamic response content that can be tailored to specific conditions.

Consequently, the use of the DefaultHTMLResponse will be deprecated.

3. New Placeholders

To support dynamic content generation, introduce placeholders within the response content. Placeholders like {{request.Method}}, {{request.Path}}, {{request.URI}}, {{request.Headers}}, and {{request.IP}} can be used to inject request-related information into the response. This enhances the ability to tailor responses based on specific conditions.

Example

Marshalled in YAML.

# ...
response:
    status: 403
    headers:
        X-Custom-Header: foo
        X-New-Header: bar
        Content-Type: text/html
    content: |
        {{ if request.Path == "/login" && request.Method == "POST" }}
            Your request has been denied for security reasons. Ref ID: {{ID}}.
        {{ else }}
            You have been blocked from accessing our website. Contact support to unblock. (IP: {{request.IP}})
        {{ end }}
    content_file: ""
# ...

Drawbacks

The primary drawback is that these changes may require existing users to update their configurations to align with the new structure. This may temporarily cause compatibility issues for those upgrading from older versions.

Alternative Approaches

An alternative approach would be to keep the existing configuration structure as it is. However, this would limit the flexibility and customization options available to users. The proposed changes aim to provide a more versatile.

Unresolved Questions

TBD

[feature] live reload datasets

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[feature] Caddy module/plugin integration

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[feature] built-in rate-limit

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[feature] more log output option

Is your feature request related to a problem? Please describe.
I want to view the logs in the browser, so I need to write the logs to the mysql database, but the logs in teler-waf can only be output to files. Can you provide more output methods ?

Describe the solution you'd like
provide more output methods

Describe alternatives you've considered
teler.Options Provider custom log output field, like LogOutput, It needs to implement the io.Writer interface

Additional context
no

[bug] improper client IP is captured in getClientIP util func

Describe the bug

A clear and concise description of what the bug is.

To Reproduce

Steps to reproduce the behavior:

Your teler usage & options...

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • OS: [e.g. mac, linux]
  • OS version: [uname -a]
  • teler Version [see go.mod]

Additional context

Add any other context about the problem here. Full output log is probably a helpful thing to add here.

[feature] embedded threat datasets in-memory

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[feature] Verify datasets with checksum

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[proposal] consolidating to a single regexp engine

Background

Currently, teler-waf utilizes two different regexp engines for pattern matching: one from the built-in regexp and another from go-pcre. However, to enhance maintainability and performance, we're considering the transition to a single regexp engine. This decision is driven by the need to streamline our codebase and ensure consistent behavior across our library.

Objective

The primary objective of this proposal is to select a single regexp engine that best aligns with our requirements. It's crucial that the chosen engine supports PCRE patterns, as these patterns are integral to our use cases, specifically to check the CommonWebAttack threat.

Options

We have evaluated several alternative regexp engines that are commonly used:

Criteria

To make an informed decision, we should consider the following criteria:

  1. PCRE Pattern Support: The selected engine must fully support PCRE patterns, as they are a fundamental part of our library's functionality.
  2. Performance: Evaluate the performance of each engine, particularly in scenarios where our library is used intensively.
  3. Community and Maintenance: Consider the activity and responsiveness of the engine's maintainers and community. A vibrant community often leads to quicker issue resolution and ongoing improvements.
  4. Documentation: Well-documented engines simplify integration, troubleshooting, and future maintenance.
  5. Compatibility: Ensure that the chosen engine aligns with our library's other dependencies and doesn't introduce conflicts.

Proposed Steps

  1. Evaluate Engines: Assess the proposed engines based on the criteria outlined above. This evaluation should include hands-on testing and performance benchmarking.
  2. Engine Selection: Based on the comparison and community judgment, decide on the most suitable regexp engine for our library's needs.
  3. Implementation Plan: Once an engine is chosen, outline a clear plan for migrating our existing codebase to the selected engine. This should include steps for modifying and testing the code, as well as addressing any compatibility issues that may arise.
  4. Testing and Validation: Rigorously test the migration to ensure that the chosen engine behaves as expected and doesn't introduce regressions or new issues.

Conclusion

By transitioning to a single regexp engine that supports PCRE patterns, we can simplify our codebase, improve maintainability, and ensure consistent behavior. This proposal outlines the steps to evaluate and select the most appropriate engine for our library's needs. Let's work collaboratively to make this transition successful and beneficial for our users and the longevity of our project.


Note
This issue proposal is meant to serve as a starting point for discussion. Feel free to contribute by suggesting modifications or adjustments as required. Your input is highly appreciated.

[proposal] deprecating `Excludes` with `Whitelists`

Description

Currently, the configuration for handling threat exclusions in our package involves the use of Excludes option along with a slice of threat.Threat that should be excluded from the security checks. While this approach has served its purpose, we have identified some limitations and complexities associated with it. To improve the configuration's clarity, flexibility, and consistency, we propose deprecating the in favor of a new approach using Whitelists option.

Proposed Solution

Deprecating Excludes

In the current configuration, threat exclusions are defined using the Excludes field, as shown in the example below:

teler.Options{
	Excludes: []threat.Threat{
		threat.BadIPAddress,
		threat.BadCrawler,
	},
}

To replace the Excludes option, we propose using an option called Whitelists. The Whitelists field will allow users to define a list of items that should be included or allowed for security check processes by using DSL expression. This approach provides better clarity and makes it easier to manage the threat exclusions.

Advantages of Whitelists

  1. Clarity: Since v1.0.0-alpha.1, the use of Whitelists explicitly states "[...] DSL expressions that match request elements that should be excluded from the security checks", improving the readability and understanding of the configuration.

  2. Flexibility: With Whitelists, users can easily manage and customize the threat exclusions with DSL (Domain Specific Language) expressions, allowing for more fine-grained control over the allowed items.

  3. Consistency: By standardizing the configuration approach, we can simplify the maintenance process and reduce user confusion.

Example Workaround

To aid users during the transition from Excludes to Whitelists, we can provide an example of how the new configuration would work. Given the original Excludes configuration:

teler.Options{
	Excludes: []threat.Threat{
		threat.BadIPAddress,
		threat.BadCrawler,
	},
}

The equivalent configuration using Whitelists would be:

teler.Options{
	Whitelists: []string{
		`threat in [BadCrawler, BadIPAddress]`, // or
		`threat == BadCrawler || threat == BadIPAddress`,
	},
}

Impact on Existing Implementations

To ensure a smooth transition for our users, we will provide clear documentation and guidelines on how to migrate from Excludes to Whitelists. Additionally, we can include documentation to assist users in updating their configurations.

Recommended Timeline

To give users sufficient time to adapt to the new configuration approach, we propose the following timeline:

  1. Deprecation Announcement: We will announce the deprecation of Excludes and the introduction of Whitelists in the next minor release, v1.1.*.

  2. Deprecation Warning: Starting from the subsequent minor release, we will provide deprecation warnings whenever the Excludes field is used, encouraging users to migrate to Whitelists.

  3. End of Support: After version v2.*.* release, we will officially remove support for the Excludes field and provide support solely for Whitelists.

Conclusion

By deprecating the use of Excludes in favor of Whitelists, we can enhance the clarity, flexibility, and consistency of our package's configuration. This change will benefit both new and existing users, making it easier to manage and understand the threat exclusions.

Please feel free to share your thoughts and feedback on this proposal. We are open to discussing any concerns or suggestions regarding this configuration update.

Thank you for your attention.

[docs] benchmarking

Summary

The current issue pertains to the process of benchmarking and initializing the *Teler instance within the codebase. The proposed solution suggests that benchmarking should be conducted by directly invoking the corresponding threat check functions (e.g., checkCommonWebAttack, checkCVE, checkDirectoryBruteforce, etc.), instead of employing the http.Client{}.Do method approach, which incorporates the use of httptest.NewServer function.

Motivation

Documenting this change in the benchmarking process is essential to clarify how benchmarking should be performed and to ensure that the process is consistent and well understood-by all contributors. This documentation will benefit both existing and future developers who work on the codebase. It will provide clear instructions on how to conduct benchmarking, leading to better code quality and more streamlined development processes.

[proposal] deprecating `Excludes` with `Whitelists`

Description

Currently, the configuration for handling threat exclusions in our package involves the use of Excludes option along with a slice of threat.Threat that should be excluded from the security checks. While this approach has served its purpose, we have identified some limitations and complexities associated with it. To improve the configuration's clarity, flexibility, and consistency, we propose deprecating the in favor of a new approach using Whitelists option.

Proposed Solution

Deprecating Excludes

In the current configuration, threat exclusions are defined using the Excludes field, as shown in the example below:

teler.Options{
	Excludes: []threat.Threat{
		threat.BadIPAddress,
		threat.BadCrawler,
	},
}

To replace the Excludes option, we propose using an option called Whitelists. The Whitelists field will allow users to define a list of items that should be included or allowed for security check processes by using DSL expression. This approach provides better clarity and makes it easier to manage the threat exclusions.

Advantages of Whitelists

  1. Clarity: Since v1.0.0-alpha.1, the use of Whitelists explicitly states "[...] DSL expressions that match request elements that should be excluded from the security checks", improving the readability and understanding of the configuration.

  2. Flexibility: With Whitelists, users can easily manage and customize the threat exclusions with DSL (Domain Specific Language) expressions, allowing for more fine-grained control over the allowed items.

  3. Consistency: By standardizing the configuration approach, we can simplify the maintenance process and reduce user confusion.

Example Workaround

To aid users during the transition from Excludes to Whitelists, we can provide an example of how the new configuration would work. Given the original Excludes configuration:

teler.Options{
	Excludes: []threat.Threat{
		threat.BadIPAddress,
		threat.BadCrawler,
	},
}

The equivalent configuration using Whitelists would be:

teler.Options{
	Whitelists: []string{
		`threat in [BadCrawler, BadIPAddress]`, // or
		`threat == BadCrawler && threat == BadIPAddress`,
	},
}

Impact on Existing Implementations

To ensure a smooth transition for our users, we will provide clear documentation and guidelines on how to migrate from Excludes to Whitelists. Additionally, we can include documentation to assist users in updating their configurations.

Recommended Timeline

To give users sufficient time to adapt to the new configuration approach, we propose the following timeline:

  1. Deprecation Announcement: We will announce the deprecation of Excludes and the introduction of Whitelists in the next minor release, v1.1.*.

  2. Deprecation Warning: Starting from the subsequent minor release, we will provide deprecation warnings whenever the Excludes field is used, encouraging users to migrate to Whitelists.

  3. End of Support: After version v2.*.* release, we will officially remove support for the Excludes field and provide support solely for Whitelists.

Conclusion

By deprecating the use of Excludes in favor of Whitelists, we can enhance the clarity, flexibility, and consistency of our package's configuration. This change will benefit both new and existing users, making it easier to manage and understand the threat exclusions.

Please feel free to share your thoughts and feedback on this proposal. We are open to discussing any concerns or suggestions regarding this configuration update.

Thank you for your attention.

[feature] implement FalcoSidekick for metric, alert, log, .etc

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[known-issue] unable to compile some CommonWebAttack & BadCrawler patterns

Here are some patterns that won't compile based on threat category.

  1. CommonWebAttack:
{Description:Detects hash-contained xss payload attacks, setter usage and property overloading ID:5 Impact:5 Rule:(?:\W\s*hash\s*[^\w\s-])|(?:\w+=\W*[^,]*,[^\s(]\s*\()|(?:\?"[^\s"]":)|(?:(?<!\/)__[a-z]+__)|(?:(?:^|[\s)\]\}])(?:s|g)etter\s*=) Tags:map[tag:[xss csrf]] pattern:<nil>}
{Description:Detects self-executing JavaScript functions ID:8 Impact:5 Rule:(?:\/\w*\s*\)\s*\()|(?:\([\w\s]+\([\w\s]+\)[\w\s]+\))|(?:(?<!(?:mozilla\/\d\.\d\s))\([^)[]+\[[^\]]+\][^)]*\))|(?:[^\s!][{([][^({[]+[{([][^}\])]+[}\])][\s+",\d]*[}\])])|(?:"\)?\]\W*\[)|(?:=\s*[^\s:;]+\s*[{([][^}\])]+[}\])];) Tags:map[tag:[xss csrf]] pattern:<nil>}
{Description:Detects JavaScript DOM/miscellaneous properties and methods ID:15 Impact:6 Rule:([^*:\s\w,.\/?+-]\s*)?(?<![a-z]\s)(?<![a-z\/_@\-\|])(\s*return\s*)?(?:create(?:element|attribute|textnode)|[a-z]+events?|setattribute|getelement\w+|appendchild|createrange|createcontextualfragment|removenode|parentnode|decodeuricomponent|\wettimeout|(?:ms)?setimmediate|option|useragent)(?(1)[^\w%"]|(?:\s*[^@\s\w%",.+\-])) Tags:map[tag:[xss csrf id rfe]] pattern:<nil>}
{Description:Detects possible includes and typical script methods ID:16 Impact:5 Rule:([^*\s\w,.\/?+-]\s*)?(?<![a-mo-z]\s)(?<![a-z\/_@])(\s*return\s*)?(?:alert|inputbox|showmod(?:al|eless)dialog|showhelp|infinity|isnan|isnull|iterator|msgbox|executeglobal|expression|prompt|write(?:ln)?|confirm|dialog|urn|(?:un)?eval|exec|execscript|tostring|status|execute|window|unescape|navigate|jquery|getscript|extend|prototype)(?(1)[^\w%"]|(?:\s*[^@\s\w%",.:\/+\-])) Tags:map[tag:[xss csrf id rfe]] pattern:<nil>}
{Description:Detects JavaScript object properties and methods ID:17 Impact:4 Rule:([^*:\s\w,.\/?+-]\s*)?(?<![a-z]\s)(?<![a-z\/_@])(\s*return\s*)?(?:hash|name|href|navigateandfind|source|pathname|close|constructor|port|protocol|assign|replace|back|forward|document|ownerdocument|window|top|this|self|parent|frames|_?content|date|cookie|innerhtml|innertext|csstext+?|outerhtml|print|moveby|resizeto|createstylesheet|stylesheets)(?(1)[^\w%"]|(?:\s*[^@\/\s\w%.+\-])) Tags:map[tag:[xss csrf id rfe]] pattern:<nil>}
{Description:Detects JavaScript array properties and methods ID:18 Impact:4 Rule:([^*:\s\w,.\/?+-]\s*)?(?<![a-z]\s)(?<![a-z\/_@\-\|])(\s*return\s*)?(?:join|pop|push|reverse|reduce|concat|map|shift|sp?lice|sort|unshift)(?(1)[^\w%"]|(?:\s*[^@\s\w%,.+\-])) Tags:map[tag:[xss csrf id rfe]] pattern:<nil>}
{Description:Detects JavaScript string properties and methods ID:19 Impact:4 Rule:([^*:\s\w,.\/?+-]\s*)?(?<![a-z]\s)(?<![a-z\/_@\-\|])(\s*return\s*)?(?:set|atob|btoa|charat|charcodeat|charset|concat|crypto|frames|fromcharcode|indexof|lastindexof|match|navigator|toolbar|menubar|replace|regexp|slice|split|substr|substring|escape|\w+codeuri\w*)(?(1)[^\w%"]|(?:\s*[^@\s\w%,.+\-])) Tags:map[tag:[xss csrf id rfe]] pattern:<nil>}
{Description:Detects JavaScript language constructs ID:20 Impact:4 Rule:(?:\)\s*\[)|([^*":\s\w,.\/?+-]\s*)?(?<![a-z]\s)(?<![a-z_@\|])(\s*return\s*)?(?:globalstorage|sessionstorage|postmessage|callee|constructor|content|domain|prototype|try|catch|top|call|apply|url|function|object|array|string|math|if|for\s*(?:each)?|elseif|case|switch|regex|boolean|location|(?:ms)?setimmediate|settimeout|setinterval|void|setexpression|namespace|while)(?(1)[^\w%"]|(?:\s*[^@\s\w%".+\-\/])) Tags:map[tag:[xss csrf id rfe]] pattern:<nil>}
{Description:Detects very basic XSS probings ID:21 Impact:3 Rule:(?:,\s*(?:alert|showmodaldialog|eval)\s*,)|(?::\s*eval\s*[^\s])|([^:\s\w,.\/?+-]\s*)?(?<![a-z\/_@])(\s*return\s*)?(?:(?:document\s*\.)?(?:.+\/)?(?:alert|eval|msgbox|showmod(?:al|eless)dialog|showhelp|prompt|write(?:ln)?|confirm|dialog|open))\s*(?:[^.a-z\s\-]|(?:\s*[^\s\w,.@\/+-]))|(?:java[\s\/]*\.[\s\/]*lang)|(?:\w\s*=\s*new\s+\w+)|(?:&\s*\w+\s*\)[^,])|(?:\+[\W\d]*new\s+\w+[\W\d]*\+)|(?:document\.\w) Tags:map[tag:[xss csrf id rfe]] pattern:<nil>}
{Description:Detects data: URL injections, VBS injections and common URI schemes ID:27 Impact:5 Rule:(?:(?:vbs|vbscript|data):.*[,+])|(?:\w+\s*=\W*(?!https?)\w+:)|(jar:\w+:)|(=\s*"?\s*vbs(?:ript)?:)|(language\s*=\s?"?\s*vbs(?:ript)?)|on\w+\s*=\*\w+\-"? Tags:map[tag:[xss rfe]] pattern:<nil>}
{Description:Detects possible event handlers ID:32 Impact:4 Rule:(?:[^\w\s=]on(?!g\&gt;)\w+[^=_+-]*=[^$]+(?:\W|\&gt;)?) Tags:map[tag:[xss csrf]] pattern:<nil>}
{Description:Detects obfuscated script tags and XML wrapped HTML ID:33 Impact:4 Rule:(?:\<\w*:?\s(?:[^\>]*)t(?!rong))|(?:\<scri)|(<\w+:\w+) Tags:map[tag:xss] pattern:<nil>}
{Description:Detects classic SQL injection probings 2/2 ID:43 Impact:6 Rule:(?:"\s*\*.+(?:or|id)\W*"\d)|(?:\^")|(?:^[\w\s"-]+(?<=and\s)(?<=or\s)(?<=xor\s)(?<=nand\s)(?<=not\s)(?<=\|\|)(?<=\&\&)\w+\()|(?:"[\s\d]*[^\w\s]+\W*\d\W*.*["\d])|(?:"\s*[^\w\s?]+\s*[^\w\s]+\s*")|(?:"\s*[^\w\s]+\s*[\W\d].*(?:#|--))|(?:".*\*\s*\d)|(?:"\s*or\s[^\d]+[\w-]+.*\d)|(?:[()*<>%+-][\w-]+[^\w\s]+"[^,]) Tags:map[tag:[sqli id lfi]] pattern:<nil>}
{Description:Detects MySQL comment-/space-obfuscated injections and backtick termination ID:57 Impact:5 Rule:(?:,.*[)\da-f"]"(?:".*"|\Z|[^"]+))|(?:\Wselect.+\W*from)|((?:select|create|rename|truncate|load|alter|delete|update|insert|desc)\s*\(\s*space\s*\() Tags:map[tag:[sqli id]] pattern:<nil>}
{Description:Detects basic XSS DoS attempts ID:65 Impact:5 Rule:(?:(^|\W)const\s+[\w\-]+\s*=)|(?:(?:do|for|while)\s*\([^;]+;+\))|(?:(?:^|\W)on\w+\s*=[\w\W]*(?:on\w+|alert|eval|print|confirm|prompt))|(?:groups=\d+\(\w+\))|(?:(.)\1{128,}) Tags:map[tag:[rfe dos]] pattern:<nil>}
  1. BadCrawler, just Yandex(?!Search)

[feature] Kong plugin integration

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[feature] Copy downloaded teler-resources into temp dir for fallback on threat.Get

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[bug] can't running echo server

Describe the bug

I can't running code because i got some error, it likes screenshoot

To Reproduce

Steps to reproduce the behavior:

  • first i install the library
  • second i install the echo framework
  • third i copy the code it likes code in folder example/integration/echo
  • fourth if i running the code i got the error it likes screenshoot
go run main.go

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.
Screenshot 2023-04-20 222500

Environment (please complete the following information):

  • OS: [windows]
  • OS version: [11]
  • teler Version [v0.5.0]

Additional context

Add any other context about the problem here. Full output log is probably a helpful thing to add here.

[feature] log rotation

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[feature] load custom rules from path

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context

[feature] refactor whitelists

Is your feature request related to a problem? Please describe.

The problem is that when using a whitelist, any request that matches it will be returned early, regardless of the type of threat it may contain. This means that some requests with payloads that could potentially be harmful may not be thoroughly analyzed for common web attack threats.

Describe the solution you'd like

One solution could be to implement the whitelist in each function of the respective threat type. This would allow for more targeted whitelisting and ensure that requests with payloads are thoroughly analyzed for common web attack threats. For example, the proposed solution could involve modifying the whitelist to include a threat type associated with each whitelist entry, as shown in the following code snippet:

		Whitelists: []teler.Whitelist{
			{`(curl|Go-http-client|okhttp)/*`, threat.BadCrawler},
			{`^/wp-login\.php`, threat.DirectoryBruteforce},
			{`(?i)Referer: https?:\/\/www\.facebook\.com`, threat.BadReferrer},
			{`192\.168\.0\.1`, threat.BadIPAddress},
		},

Describe alternatives you've considered

An alternative solution could be to implement the whitelist like a custom rules format. This would allow for more granular control over which requests are allowed through the teler-waf, and would enable the specification of complex rules based on various request elements (e.g. URI, headers, body). The proposed implementation includes a Customs field that would contain an array of custom rules that the teler-waf would follow.

Additional context

Warning: Implementing this solution would require changes to existing code and may potentially break existing functionality.

It is essential to balance the need for security with the need for efficient processing of requests. Adding too many checks can slow down the analysis process, while insufficient checks can increase the risk of security breaches. Therefore, any proposed solution should aim to strike a balance between these two competing needs.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.