danghvu / mod_dumpost Goto Github PK
View Code? Open in Web Editor NEWSmall and lightweight Apache module to log POST data of a HTTP request
License: GNU General Public License v3.0
Small and lightweight Apache module to log POST data of a HTTP request
License: GNU General Public License v3.0
It'd be nice if I could include variables such as REMOTE_USER in the log, in addition to any HTTP headers.
Hi, i was looking for a way to dump POST&PUT data on a production instance using jakarta to serve a java servlet, with the particularity that i also need binary dumping, and also filtering the hook to some urls due to high memory needed on some binary request i had to log (~300k) also ive choosed to dump binary data as hex string, so buffer has to be duplicated to allocate all buffer. I wanted to avoid
state->buffer = apr_palloc(state->mp, cfg->max_size + 1);
on non filtered urls, so added this at the begginning of the input filter
apr_status_t dumpost_input_filter (ap_filter_t *f, apr_bucket_brigade *bb, ap_input_mode_t mode, apr_read_type_e block, apr_off_t readbytes) {
apr_status_t ret;
if (strstr(f->r->the_request, "/file/to/log1/") != NULL || strstr(f->r->the_request, "/file/to/log2") != NULL) {
} else {
if ((ret = ap_get_brigade(f->next, bb, mode, block, readbytes)) != APR_SUCCESS)
return ret;
return APR_SUCCESS;
}
Ive never worked with APR or apache so i dont know if this is good or not, but for sure, i should had to create a new AP_INIT_ITERATE cmd on the config to add filtered urls on config file instead hardcode them. As i wont need it and due to lazyness i will keep this...
Then i foundout that the way dumpit is doing the copy to the buffer truncates the data on nulls [strncpy(buf, ibuf, nbytes);] so ive replaced with byte to byte copy
/*strncpy(buf, ibuf, nbytes);*/
for (int kk = 0; kk < nbytes; kk++) {
buf[kk]=ibuf[kk];
}
*current_size += nbytes;
Then ive added a simple routine to perform binary to hex string conversion
void hexArrayToStr(unsigned char* info, unsigned int infoLength, char **buffer, unsigned int start) {
const char* pszNibbleToHex = {"0123456789ABCDEF"};
int nNibble, i;
for (i = 0; i < infoLength; i++) {
nNibble = info[i] >> 4;
buffer[0][2 * i + start] = pszNibbleToHex[nNibble];
nNibble = info[i] & 0x0F;
buffer[0][2 * i + 1 + start] = pszNibbleToHex[nNibble];
}
}
and edited logit to test the buffer to find binary bytes and set a proper output, i know i should check more bytes than null, for example (<0x20 && >0x7e && !=0x0A && !=0x0D) but in my case i encounter lots of 0x00 on my binary buffer so it will be fine as is.
apr_status_t logit(ap_filter_t *f) {
request_state *state = f->ctx;
request_rec *r = f->r;
if (state == NULL || state->log_size == 0) return -1;
state->buffer[state->log_size] = 0;
///ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r, "mod_dumpost: wat");
if (state->fd == NULL) {
// no file to write to, write to error log
// data is truncated to MAX_STRING_LEN ~ 8192 in apache
ap_log_rerror(APLOG_MARK, APLOG_INFO, 0, r, "\"%s\" %s", r->the_request, state->buffer);
} else {
// need to manually get the time and ip address -- too lazy to make these cusomizable
char *time = apr_palloc(r->pool, 50);
apr_ctime(time, r->request_time);
char *ip = r->connection->client_ip;
apr_size_t nbytes_written;
char *text = apr_psprintf(r->pool, "[%s] %s \"%s\" ",time, ip, r->the_request);
//test bin
int not_bin=1;
for (int i = 0; i < state->log_size; i++)
{
if (state->buffer[i]==0) not_bin=0;
}
int jj=strlen(text);
if (not_bin==0) {
char *text2 = apr_palloc(r->pool, (state->log_size*2) + 2 + jj);
sprintf(text2, "%s", text);
hexArrayToStr(state->buffer, state->log_size, &text2, jj);
jj=jj+(state->log_size*2);
text2[jj]='\n';
jj++;
text2[jj]='\0';
apr_status_t rc = apr_file_write_full(state->fd, text2, jj, &nbytes_written);
if (rc != APR_SUCCESS) {
ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r, "mod_dumpost: error while writing to log");
return rc;
}
apr_file_close(state->fd);
} else {
char *text2 = apr_palloc(r->pool, (state->log_size) + 1 + jj);
sprintf(text2, "%s%s\n", text, state->buffer);
jj=strlen(text2);
apr_status_t rc = apr_file_write_full(state->fd, text2, jj, &nbytes_written);
if (rc != APR_SUCCESS) {
ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r, "mod_dumpost: error while writing to log");
return rc;
}
apr_file_close(state->fd);
}
}
return APR_SUCCESS;
}
I do not know if i missed any requeriment by apr to handle those buffers but seems to be working nice and not leaking any memory but i didnt tested that properly...
@danghvu Thank you for this great extension.
There is one security concern I have.
I am using Ubuntu Trusty.
The log mod_dumpost creates are owned by www-data:www-data
and chmod 644
.
All other logs Apache creates are owned by root:adm
. They can be stored in a dir with chmod 700
+ chown root
, making it impossible for users to read the logs.
The logs mod_dumpost creates are a lot less secure.
Is it possible for you to improve this?
Hi.
I had some problems compiling mod_dumpost on Centos 5.5:
mod_dumpost.c:123:5: warning: implicit declaration of function 'ap_hook_insert_filter'
I solved this adding:
#include "http_request.h"
and then:
mod_dumpost.c:124:5: warning: passing argument 1 of 'ap_hook_insert_filter' from incompatible pointer type
I solved changing return type to void
instead of int
and removing return OK
Moreover there are some minor warnings related to ambiguous use of assignment:
https://github.com/danghvu/mod_dumpost/blob/master/mod_dumpost.c#L60
https://github.com/danghvu/mod_dumpost/blob/master/mod_dumpost.c#L73
And wrong casting:
https://github.com/danghvu/mod_dumpost/blob/master/mod_dumpost.c#L146
After corrections compiles without warnings and seems to work.
Hi, Just been trying to use this module. Having an issue not getting any output from it.
apachectl -V | grep version
Server version: Apache/2.2.15 (Unix)
apachectl -D DUMP_MODULES | grep dump
Syntax OK
dumpost_module (shared)
I then have this in /etc/httpd/conf/httpd.conf
DumpPostLogFile /var/log/httpd/post.log
I then make a post request which is logged by apache
[root@localhost mod_dumpost]# tail /var/www/vhosts/website1.com/logs/access_log | grep POST
10.211.55.2 - - [17/Dec/2015:08:16:17 +0000] "POST /wp-login.php?action=lostpassword HTTP/1.0" 200 3415 "http://website1.com/wp-login.php?action=lostpassword" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
But nothing is ever logged in var/log/httpd/post.log
Any help?
Cheers.
Hi,
it would be great to have a customizable log format. E.g. I have a use case, where I need response times and users sending the request.
Thanks!
I think I see a bug in line 65:
65: state->buffer[state->log_size] = 0;
If we've filled the buffer completely, then state->log_size will equal LOG_IS_FULL, which is -1. So you are assigning a 0 to buffer[-1] which can cause some interesting behavior in rare cases. :)
i just compiled the mod and only got these messages:
[warn] [client 10.65.116.142] mod_dumpost: body limit reach, referer: http....
Thanks for the module - works great!
It would be really nice if there was a configuration setting to have the mod_dumpost log go somewhere other than the error_log; that log file is noisy, especially when you're using TLS/SSL.
Hi @danghvu
I'm interested on this module and improve by adding some new options .
But , I've never built an apache module before, and will take long in a code review to understand if could be possible 3 new features in a more or less easy way ( I will be able to do it if you can ensure me that is possible)
How difficult could be add them? There is any apache api limitations to do this work?
Thank you very much.
Can you please document the full uninstallation process for this module?
Running on OSX Mavericks, I put the following in /etc/apache2/httpd.conf:
DumpPostLogFile "/private/var/log/apache2/post_log"
After saving httpd.conf and restarting Apache, the dumpost output is still logged in my error_log file.
The ErrorLog is in the same directory (/private/var/log/apache2/).
While using mod_dumpost module, we observed partial read of post body.We tried to read a post body of 246K, in apache error.log body was received in several parts whereas we were expecting complete body in one response.
Observation:
Default read buffer size used was "DEFAULT_MAX_SIZE 1024*1024".
Function "dumpost_input_filter ()" is getting called several times, whereas the given for loop executes once on every call of this function, due to this we are not getting post body in one buffer. . We have modified code to meet our requirements.
apr_size_t buf_len_lol = 0 //current size written by dumpit ()
int count = 0; //Local var to count loop execution
for (b = APR_BRIGADE_FIRST(bb); b != APR_BRIGADE_SENTINEL(bb); b = APR_BUCKET_NEXT(b))
{
if (state->log_size != LOG_IS_FULL && buf_len < cfg->max_size)
dumpit(f, b, buf + (state->log_size + buf_len_lol), &buf_len_lol);
count++;
}
buf_len = state->log_size + buf_len_lol;
Can you please suggest us some solution for the above problem.
Looking forward for your reply
Manpreet
Good Morning.
I am trying to install mod_dumpost in a virtual machine to do some tests, it shows below the following output:
$ make install
apxs2 -Wc,-Wall -c mod_dumpost.c
/usr/share/apr-1.0/build/libtool --mode=compile --tag=disable-static x86_64-linux-gnu-gcc -prefer-pic -pipe -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -DLINUX -D_REENTRANT -D_GNU_SOURCE -pthread -I/usr/include/apache2 -I/usr/include/apr-1.0 -I/usr/include/apr-1.0 -I/usr/include -Wall -c -o mod_dumpost.lo mod_dumpost.c && touch mod_dumpost.slo
libtool: compile: x86_64-linux-gnu-gcc -pipe -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -DLINUX -D_REENTRANT -D_GNU_SOURCE -pthread -I/usr/include/apache2 -I/usr/include/apr-1.0 -I/usr/include/apr-1.0 -I/usr/include -Wall -c mod_dumpost.c -fPIC -DPIC -o .libs/mod_dumpost.o
In file included from /usr/include/string.h:495,
from /usr/include/apr-1.0/apr_want.h:63,
from /usr/include/apr-1.0/apr_pools.h:47,
from /usr/include/apr-1.0/apr_tables.h:26,
from /usr/include/apr-1.0/apr_hooks.h:22,
from /usr/include/apache2/ap_hooks.h:40,
from /usr/include/apache2/ap_config.h:25,
from /usr/include/apache2/httpd.h:44,
from mod_dumpost.c:19:
In function ‘strncpy’,
inlined from ‘dumpost_input_filter’ at mod_dumpost.c:149:13:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10: warning: ‘__builtin_strncpy’ specified bound depends on the length of the source argument [-Wstringop-overflow=]
106 | return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest));
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mod_dumpost.c: In function ‘dumpost_input_filter’:
mod_dumpost.c:147:23: note: length computed here
147 | int len = strlen(s);
| ^~~~~~~~~
/usr/share/apr-1.0/build/libtool --mode=link --tag=disable-static x86_64-linux-gnu-gcc -Wl,--as-needed -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -o mod_dumpost.la -rpath /usr/lib/apache2/modules -module -avoid-version mod_dumpost.lo
libtool: link: rm -fr .libs/mod_dumpost.la .libs/mod_dumpost.lai .libs/mod_dumpost.so
libtool: link: x86_64-linux-gnu-gcc -shared -fPIC -DPIC .libs/mod_dumpost.o -Wl,--as-needed -Wl,-Bsymbolic-functions -Wl,-z -Wl,relro -Wl,-z -Wl,now -Wl,-soname -Wl,mod_dumpost.so -o .libs/mod_dumpost.so
libtool: link: ( cd ".libs" && rm -f "mod_dumpost.la" && ln -s "../mod_dumpost.la" "mod_dumpost.la" )
sudo apxs2 -i -a -n dumpost mod_dumpost.la;\
echo Now restart your apache
/usr/share/apache2/build/instdso.sh SH_LIBTOOL='/usr/share/apr-1.0/build/libtool' mod_dumpost.la /usr/lib/apache2/modules
/usr/share/apr-1.0/build/libtool --mode=install install mod_dumpost.la /usr/lib/apache2/modules/
libtool: install: install .libs/mod_dumpost.so /usr/lib/apache2/modules/mod_dumpost.so
libtool: install: install .libs/mod_dumpost.lai /usr/lib/apache2/modules/mod_dumpost.la
libtool: finish: PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/sbin" ldconfig -n /usr/lib/apache2/modules
----------------------------------------------------------------------
Libraries have been installed in:
/usr/lib/apache2/modules
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
- add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the 'LD_RUN_PATH' environment variable
during linking
- use the '-Wl,-rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to '/etc/ld.so.conf'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
chmod 644 /usr/lib/apache2/modules/mod_dumpost.so
[preparing module `dumpost' in /etc/apache2/mods-available/dumpost.load]
Module dumpost already enabled
Now restart your apache
I am using Apache/2.4.41 on Ubuntu 20.04. Am I doing something wrong? Do I need to do anything else to get it to actually work?
I don't know C, and this is the first C module I am installing.
Thanks a lot if you can guide me.
Sincerely,
Marcelo Vasconcellos
Would it be possible to set an option so that the POST data is stored as base64? the things I log need to be base64'd, as they contain characters that stop the log halfway through
I am working to create a pre-compiled binary for CentOS 5 + 6. The rpm spec file I'm creating isn't working as expected because it's doing a runtime compile of make install with apxs. Admittedly, this may be due to my lack of full understanding of all stages of the rpmbuild. So my new approach is to manually install the components from a pre-compiled installation. I end up with:
mod_dumpost.so
which gets copied to /usr/lib64/httpd/modules/ and the 1 line change in the httpd.conf. Is there anything else that apxs does that's special in this process? Besides the restart of httpd, is there anything I would want to consider in a fully manual process? Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.