Comments (15)
I am not not afraid to write C, I am afraid to push C to any sort of public repo... I looked at src/p2f.c and don't see where the actual output file is being opened. I am looking for that and I will just hard code (for my use) the setbuf(f, NULL) right after it's opened so I know I don't have buffering.
from joy.
from joy.
I'm not sure what it is that you're asking. Can you be more specific?
from joy.
I am noticing when I am running joy with the output set to a named pipe created with mkfifo (with no rotation etc set, and compressed output = 0) Records appear to be buffering somewhere. If I exit joy, a bunch write, or if a pile of records get written, there seems to be a flush, but it seems to be buffering. (this is monitoring the pipe with tail -f)
It's the difference between when I run a python script with output, and I don't want buffered output, I just python -u myscript.py and it just writes direct.
from joy.
Also, I during the tail -f I will get "incomplete" records as it purges the buffer, but part of a record is in one dump of the buffer and the rest is in the next buffer. By doing unbuffered writes, it ensure only full records get written
from joy.
Basically, I see output buffering is disabled in DEBUG mode, can we get a flag to disable output buffering but not have DEBUG mode turned on. (I can read C code, I am afraid to write it though...
/* in debug mode, turn off output buffering */
#if P2F_DEBUG
setvbuf(stderr, NULL, _IONBF, 0);
setbuf(stdout, NULL);
from joy.
Line 432 in https://github.com/cisco/joy/blob/master/src/joy.c
from joy.
This is where my C hacking sucks... so that's just for stdout, but I need to find the place that the outputfile for the JSON records is opened, and provide either an option to disabled buffering or just hack this into the file where the output is opened... that way it will write unbuffered... I didn't see that in joy.c, can you point me in the right direction to see where the output file is opened? Thanks!
setbuf(f, NULL);
from joy.
You shouldn't be afraid to write C code, in fact I recommend that you learn how to. It appears to me that you have an good idea of how to change the code to suit your specific needs.
Fixing bugs is one thing, but feature requests is entirely up to you to implement or at least partially contribute code. My team and I simply do not have the bandwidth to drop what we are doing and make changes such as this. Joy is offered as open-source to encourage people such as yourself to advance the project for the whole user base, not to be used as a service.
I look forward to seeing a pull-request from you in the future!
Philip
from joy.
You may want to take a look at src/p2f.c because that is where the flow_records are being written.
from joy.
Does it use zopen if I set compression = 0 in the input.h? This is where I am confused by the compile time compression...
from joy.
Yes it will still use zopen, but the macro definition changes. With compression = 0 it will define to fopen, and if compression = 1, it will be gzopen.
from joy.
Ah cool
Trying now
from joy.
So I tried putting the setbuf(output, NULL); in both places zopen was called in joy.c, and I am still getting buffered output. I would like to live tail this log, and the partial records is troublesome. I will continue to troubleshoot on my side, but if anyone else any ideas where I could disable buffering I would be very appreciative I am just guess and checking at this point.
from joy.
I think it's working now... doing testing. (the original suggestion worked, I just um, tested with the wrong binary... *sigh
from joy.
Related Issues (20)
- Using sleuth to calculate distribution of each individual offered cipher suite type HOT 2
- Incorrect TLS handshake types and TLS content types when generating TLS session metadata
- compilation error on macOS v 10.12.6
- The return value of 'sscanf()' should be checked
- Build failed with VS2019 on win10 HOT 4
- no VLAN support for data link type 12
- make error HOT 1
- Build Error while enable output compress HOT 1
- compile failed
- Number of packets is incorrect HOT 1
- A description of the parameters in joy's tool HOT 1
- Question with “one or more interfaces“
- 'type=1' option does not work
- How to use the FEATURE OPTIONS "payload=N (number)"?
- `sudo ./install_joy/install-sh -p /opt/app/joy` error
- Bytes in/out not correct
- is it prossible to use JOY to analysis malware network traffic online?
- Mac输出乱码
- some problem about sleuth
- some problem about sleuth
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from joy.