GithubHelp home page GithubHelp logo

danghvu / mod_dumpost Goto Github PK

View Code? Open in Web Editor NEW
68.0 68.0 22.0 26 KB

Small and lightweight Apache module to log POST data of a HTTP request

License: GNU General Public License v3.0

C 97.24% Makefile 2.76%

mod_dumpost's People

Contributors

danghvu avatar zaccheob avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mod_dumpost's Issues

Make File Path Configurable

Thanks for the module - works great!

It would be really nice if there was a configuration setting to have the mod_dumpost log go somewhere other than the error_log; that log file is noisy, especially when you're using TLS/SSL.

Log file security

@danghvu Thank you for this great extension.

There is one security concern I have.
I am using Ubuntu Trusty.
The log mod_dumpost creates are owned by www-data:www-data and chmod 644.
All other logs Apache creates are owned by root:adm. They can be stored in a dir with chmod 700 + chown root, making it impossible for users to read the logs.
The logs mod_dumpost creates are a lot less secure.
Is it possible for you to improve this?

No logging from Apache/2.2.15 CentOS 6

Hi, Just been trying to use this module. Having an issue not getting any output from it.

apachectl -V | grep version
Server version: Apache/2.2.15 (Unix)

apachectl -D DUMP_MODULES | grep dump
Syntax OK
dumpost_module (shared)

I then have this in /etc/httpd/conf/httpd.conf

DumpPostLogFile /var/log/httpd/post.log

I then make a post request which is logged by apache

[root@localhost mod_dumpost]# tail /var/www/vhosts/website1.com/logs/access_log | grep POST
10.211.55.2 - - [17/Dec/2015:08:16:17 +0000] "POST /wp-login.php?action=lostpassword HTTP/1.0" 200 3415 "http://website1.com/wp-login.php?action=lostpassword" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"

But nothing is ever logged in var/log/httpd/post.log

Any help?

Cheers.

Store POST data as Base64

Would it be possible to set an option so that the POST data is stored as base64? the things I log need to be base64'd, as they contain characters that stop the log halfway through

DumpPostLogFile is ignored

Running on OSX Mavericks, I put the following in /etc/apache2/httpd.conf:

DumpPostLogFile "/private/var/log/apache2/post_log"

After saving httpd.conf and restarting Apache, the dumpost output is still logged in my error_log file.

The ErrorLog is in the same directory (/private/var/log/apache2/).

Issue in reading big post data

While using mod_dumpost module, we observed partial read of post body.We tried to read a post body of 246K, in apache error.log body was received in several parts whereas we were expecting complete body in one response.

Observation:
Default read buffer size used was "DEFAULT_MAX_SIZE 1024*1024".

Function "dumpost_input_filter ()" is getting called several times, whereas the given for loop executes once on every call of this function, due to this we are not getting post body in one buffer. . We have modified code to meet our requirements.

apr_size_t buf_len_lol = 0 //current size written by dumpit ()
int count = 0; //Local var to count loop execution
for (b = APR_BRIGADE_FIRST(bb); b != APR_BRIGADE_SENTINEL(bb); b = APR_BUCKET_NEXT(b))
{
if (state->log_size != LOG_IS_FULL && buf_len < cfg->max_size)
dumpit(f, b, buf + (state->log_size + buf_len_lol), &buf_len_lol);
count++;
}
buf_len = state->log_size + buf_len_lol;

Can you please suggest us some solution for the above problem.
Looking forward for your reply

Manpreet

Customizable log format

Hi,

it would be great to have a customizable log format. E.g. I have a use case, where I need response times and users sending the request.

Thanks!

Filtering & Dumping binary data

Hi, i was looking for a way to dump POST&PUT data on a production instance using jakarta to serve a java servlet, with the particularity that i also need binary dumping, and also filtering the hook to some urls due to high memory needed on some binary request i had to log (~300k) also ive choosed to dump binary data as hex string, so buffer has to be duplicated to allocate all buffer. I wanted to avoid
state->buffer = apr_palloc(state->mp, cfg->max_size + 1);
on non filtered urls, so added this at the begginning of the input filter

apr_status_t dumpost_input_filter (ap_filter_t *f, apr_bucket_brigade *bb, ap_input_mode_t mode, apr_read_type_e block, apr_off_t readbytes) {

		apr_status_t ret;
		if (strstr(f->r->the_request, "/file/to/log1/") != NULL || strstr(f->r->the_request, "/file/to/log2") != NULL) {

		}  else {
			if ((ret = ap_get_brigade(f->next, bb, mode, block, readbytes)) != APR_SUCCESS)
				return ret;
			return APR_SUCCESS;
		}

Ive never worked with APR or apache so i dont know if this is good or not, but for sure, i should had to create a new AP_INIT_ITERATE cmd on the config to add filtered urls on config file instead hardcode them. As i wont need it and due to lazyness i will keep this...

Then i foundout that the way dumpit is doing the copy to the buffer truncates the data on nulls [strncpy(buf, ibuf, nbytes);] so ive replaced with byte to byte copy

/*strncpy(buf, ibuf, nbytes);*/
				
				for (int kk = 0; kk < nbytes; kk++) {
					buf[kk]=ibuf[kk];
				}
                *current_size += nbytes;

Then ive added a simple routine to perform binary to hex string conversion

void hexArrayToStr(unsigned char* info, unsigned int infoLength, char **buffer, unsigned int start) {
    const char* pszNibbleToHex = {"0123456789ABCDEF"};
    int nNibble, i;
            for (i = 0; i < infoLength; i++) {
					nNibble = info[i] >> 4;
					buffer[0][2 * i + start] = pszNibbleToHex[nNibble];
					nNibble = info[i] & 0x0F;
					buffer[0][2 * i + 1 + start] = pszNibbleToHex[nNibble];
            }
}

and edited logit to test the buffer to find binary bytes and set a proper output, i know i should check more bytes than null, for example (<0x20 && >0x7e && !=0x0A && !=0x0D) but in my case i encounter lots of 0x00 on my binary buffer so it will be fine as is.

apr_status_t logit(ap_filter_t *f) {
    request_state *state = f->ctx;
    request_rec *r = f->r;

    if (state == NULL || state->log_size == 0) return -1;
    state->buffer[state->log_size] = 0;
///ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r, "mod_dumpost: wat");

    if (state->fd == NULL) {
      // no file to write to, write to error log
      // data is truncated to MAX_STRING_LEN ~ 8192 in apache
      ap_log_rerror(APLOG_MARK, APLOG_INFO, 0, r, "\"%s\" %s", r->the_request, state->buffer);
    } else {
      // need to manually get the time and ip address -- too lazy to make these cusomizable
      char *time = apr_palloc(r->pool, 50);
      apr_ctime(time, r->request_time);
      char *ip = r->connection->client_ip;
      apr_size_t nbytes_written;
      char *text = apr_psprintf(r->pool, "[%s] %s \"%s\" ",time, ip, r->the_request);
	  
	  //test bin
	  	  int not_bin=1;
		  for (int i = 0; i < state->log_size; i++)
		  {
			 if (state->buffer[i]==0) not_bin=0;
		  }
	  
	 int jj=strlen(text);
	 if (not_bin==0) {
		 char *text2 = apr_palloc(r->pool, (state->log_size*2) + 2 + jj);
		 sprintf(text2, "%s", text);
		 hexArrayToStr(state->buffer, state->log_size, &text2, jj);
		 jj=jj+(state->log_size*2);
		 text2[jj]='\n';
		 jj++;
		 text2[jj]='\0';
		 apr_status_t rc = apr_file_write_full(state->fd, text2, jj, &nbytes_written);

		  if (rc != APR_SUCCESS) {
			ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r, "mod_dumpost: error while writing to log");
			return rc;
		  }
		  apr_file_close(state->fd);
	 } else {
		 char *text2 = apr_palloc(r->pool, (state->log_size) + 1 + jj);
		 sprintf(text2, "%s%s\n", text, state->buffer);
		 jj=strlen(text2);
		 apr_status_t rc = apr_file_write_full(state->fd, text2, jj, &nbytes_written);

		  if (rc != APR_SUCCESS) {
			ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r, "mod_dumpost: error while writing to log");
			return rc;
		  }
		  apr_file_close(state->fd);
	 }
	 
      
    }
    return APR_SUCCESS;
}

I do not know if i missed any requeriment by apr to handle those buffers but seems to be working nice and not leaking any memory but i didnt tested that properly...

Difficulties installing mod_dumpost in Apache/2.4.41 on Ubuntu 20.04

Good Morning.

I am trying to install mod_dumpost in a virtual machine to do some tests, it shows below the following output:

$ make install
apxs2 -Wc,-Wall -c mod_dumpost.c
/usr/share/apr-1.0/build/libtool  --mode=compile --tag=disable-static x86_64-linux-gnu-gcc -prefer-pic -pipe -g -O2 -fstack-protector-strong -Wformat -Werror=format-security  -Wdate-time -D_FORTIFY_SOURCE=2   -DLINUX -D_REENTRANT -D_GNU_SOURCE  -pthread  -I/usr/include/apache2  -I/usr/include/apr-1.0   -I/usr/include/apr-1.0 -I/usr/include -Wall  -c -o mod_dumpost.lo mod_dumpost.c && touch mod_dumpost.slo
libtool: compile:  x86_64-linux-gnu-gcc -pipe -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -DLINUX -D_REENTRANT -D_GNU_SOURCE -pthread -I/usr/include/apache2 -I/usr/include/apr-1.0 -I/usr/include/apr-1.0 -I/usr/include -Wall -c mod_dumpost.c  -fPIC -DPIC -o .libs/mod_dumpost.o
In file included from /usr/include/string.h:495,
                 from /usr/include/apr-1.0/apr_want.h:63,
                 from /usr/include/apr-1.0/apr_pools.h:47,
                 from /usr/include/apr-1.0/apr_tables.h:26,
                 from /usr/include/apr-1.0/apr_hooks.h:22,
                 from /usr/include/apache2/ap_hooks.h:40,
                 from /usr/include/apache2/ap_config.h:25,
                 from /usr/include/apache2/httpd.h:44,
                 from mod_dumpost.c:19:
In function ‘strncpy’,
    inlined from ‘dumpost_input_filter’ at mod_dumpost.c:149:13:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10: warning: ‘__builtin_strncpy’ specified bound depends on the length of the source argument [-Wstringop-overflow=]
  106 |   return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest));
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mod_dumpost.c: In function ‘dumpost_input_filter’:
mod_dumpost.c:147:23: note: length computed here
  147 |             int len = strlen(s);
      |                       ^~~~~~~~~
/usr/share/apr-1.0/build/libtool  --mode=link --tag=disable-static x86_64-linux-gnu-gcc -Wl,--as-needed -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now    -o mod_dumpost.la  -rpath /usr/lib/apache2/modules -module -avoid-version    mod_dumpost.lo
libtool: link: rm -fr  .libs/mod_dumpost.la .libs/mod_dumpost.lai .libs/mod_dumpost.so
libtool: link: x86_64-linux-gnu-gcc -shared  -fPIC -DPIC  .libs/mod_dumpost.o    -Wl,--as-needed -Wl,-Bsymbolic-functions -Wl,-z -Wl,relro -Wl,-z -Wl,now   -Wl,-soname -Wl,mod_dumpost.so -o .libs/mod_dumpost.so
libtool: link: ( cd ".libs" && rm -f "mod_dumpost.la" && ln -s "../mod_dumpost.la" "mod_dumpost.la" )
sudo apxs2 -i -a -n dumpost mod_dumpost.la;\
echo Now restart your apache
/usr/share/apache2/build/instdso.sh SH_LIBTOOL='/usr/share/apr-1.0/build/libtool' mod_dumpost.la /usr/lib/apache2/modules
/usr/share/apr-1.0/build/libtool --mode=install install mod_dumpost.la /usr/lib/apache2/modules/
libtool: install: install .libs/mod_dumpost.so /usr/lib/apache2/modules/mod_dumpost.so
libtool: install: install .libs/mod_dumpost.lai /usr/lib/apache2/modules/mod_dumpost.la
libtool: finish: PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/sbin" ldconfig -n /usr/lib/apache2/modules
----------------------------------------------------------------------
Libraries have been installed in:
   /usr/lib/apache2/modules

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the 'LD_RUN_PATH' environment variable
     during linking
   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to '/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
chmod 644 /usr/lib/apache2/modules/mod_dumpost.so
[preparing module `dumpost' in /etc/apache2/mods-available/dumpost.load]
Module dumpost already enabled
Now restart your apache

I am using Apache/2.4.41 on Ubuntu 20.04. Am I doing something wrong? Do I need to do anything else to get it to actually work?
I don't know C, and this is the first C module I am installing.

Thanks a lot if you can guide me.

Sincerely,

Marcelo Vasconcellos

Compilation problems

Hi.

I had some problems compiling mod_dumpost on Centos 5.5:

  1. The trick you suggest to define a shell alias apxs2=apxs does not work because shell aliases are not expanded in non-interactive shells (like in make)
  2. In my installation module compilation is invoked with the -Wall option by default, this revealed some (dangerous?) warnings:

mod_dumpost.c:123:5: warning: implicit declaration of function 'ap_hook_insert_filter'

I solved this adding:
#include "http_request.h"

and then:
mod_dumpost.c:124:5: warning: passing argument 1 of 'ap_hook_insert_filter' from incompatible pointer type

I solved changing return type to void instead of int and removing return OK

Moreover there are some minor warnings related to ambiguous use of assignment:

https://github.com/danghvu/mod_dumpost/blob/master/mod_dumpost.c#L60

https://github.com/danghvu/mod_dumpost/blob/master/mod_dumpost.c#L73

And wrong casting:
https://github.com/danghvu/mod_dumpost/blob/master/mod_dumpost.c#L146

After corrections compiles without warnings and seems to work.

(REVERSE) installation process

I am working to create a pre-compiled binary for CentOS 5 + 6. The rpm spec file I'm creating isn't working as expected because it's doing a runtime compile of make install with apxs. Admittedly, this may be due to my lack of full understanding of all stages of the rpmbuild. So my new approach is to manually install the components from a pre-compiled installation. I end up with:
mod_dumpost.so
which gets copied to /usr/lib64/httpd/modules/ and the 1 line change in the httpd.conf. Is there anything else that apxs does that's special in this process? Besides the restart of httpd, is there anything I would want to consider in a fully manual process? Thanks!

Possible bug?

I think I see a bug in line 65:

65: state->buffer[state->log_size] = 0;

If we've filled the buffer completely, then state->log_size will equal LOG_IS_FULL, which is -1. So you are assigning a 0 to buffer[-1] which can cause some interesting behavior in rare cases. :)

[Feature Request] New options to log( GET params/ResponseTime/IP origin/Headers)

Hi @danghvu

I'm interested on this module and improve by adding some new options .

But , I've never built an apache module before, and will take long in a code review to understand if could be possible 3 new features in a more or less easy way ( I will be able to do it if you can ensure me that is possible)

  1. I would like add ability to log generic "request" parameters not only POST but also GET parameters.
  2. I would like add also ability to log response time ( current %D/%T on the access.log) once the request has been completed.
  3. I would like also add some other generic access.log data . Like HTTP Status, HTTP URL/ client IP / Some Header etc.

How difficult could be add them? There is any apache api limitations to do this work?

Thank you very much.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.