Comments (3)
- Wireshark is not able to fully parse the different streams (TCP/HTTP/etc). For example when you right click on an HTTP request then select follow-HTTP. It is unable to reassemble and follow the stream. Even if the dump has been running for a while with many requests wireshark identifies newer requests as an early tcp.stream (for example 1 or 2). This leads me to believe it's not able to properly discern between different TCP streams.
The wireshark HTTP follow code only support TCP filters, see http_follow, and the exported_pdu layer didn't register any tcp.* variables. So it's not possible to follow.
- The source and destination are of the proxy server and the udpdump receiver. Ideally these would be of the connecting client and remote server. Or at least between the proxy and the remote server.
It's the first exported layer, which is added by udpdump. You can see the client address and the proxy address at the second exported_pdu layer which is generated by g3proxy.
- I believe due to issue 1 we are seeing HTTP [Malformed Packet] errors often.
I haven't checked the details yet, but packet loss is expected as it's UDP based dump protocol.
from g3.
- Wireshark is not able to fully parse the different streams (TCP/HTTP/etc). For example when you right click on an HTTP request then select follow-HTTP. It is unable to reassemble and follow the stream. Even if the dump has been running for a while with many requests wireshark identifies newer requests as an early tcp.stream (for example 1 or 2). This leads me to believe it's not able to properly discern between different TCP streams.
The wireshark HTTP follow code only support TCP filters, see http_follow, and the exported_pdu layer didn't register any tcp.* variables. So it's not possible to follow.
Thank you for the link. I will investigate more how it is trying to restructure. I noticed you were including “synthetic” TCP data (such as sequence number etc) in the packet you were sending. I was seeing some tcp variables being registered (sequence etc) register in wireshark. I wonder if it can be aligned to allow it to be fully followed.
I also tried modifying the wireshark side udpdump source code so it does not add the second layer of exported_pdu. It had no issues identifying the TCP data you send.
The pcap capture feature you have written is an extremely useful feature. Having more in depth data and being able to follow the streams would really be great.
- The source and destination are of the proxy server and the udpdump receiver. Ideally these would be of the connecting client and remote server. Or at least between the proxy and the remote server.
It's the first exported layer, which is added by udpdump. You can see the client address and the proxy address at the second exported_pdu layer which is generated by g3proxy.
This is my mistake. Local machine testing. I was thinking the server would be the remote server.
For our use having the remote HTTP server would be useful (as well as the HTTP client). Any easy suggestions on passing that info down to the udpdump?
- I believe due to issue 1 we are seeing HTTP [Malformed Packet] errors often.
I haven't checked the details yet, but packet loss is expected as it's UDP based dump protocol.
That would make sense. It is possible to do pcap over IP using a TCP pipe. PolarProxy does this (sending decrypted TLS over TCP).
https://www.netresec.com/?page=Blog&month=2022-05&post=Real-time-PCAP-over-IP-in-Wireshark
https://www.netresec.com/?page=PolarProxy
————————-
It sounds like the few issues we have identified above are known. We will start planning on how to modify G3 to have more functionality in PCAP export. If you have any requirements or suggestions please let us know.
Some background - we were leveraging ICAP but many sites have started using websockets which the ICAP protocol was not built to handle. So we started going down the PCAP route.
Thank you again @zh-jq - your work on this project is much appreciated.
from g3.
Thank you for the link. I will investigate more how it is trying to restructure. I noticed you were including “synthetic” TCP data (such as sequence number etc) in the packet you were sending. I was seeing some tcp variables being registered (sequence etc) register in wireshark. I wonder if it can be aligned to allow it to be fully followed.
It may be possible if those tcp.* variables especially tcp.stream get registered in the exported_pdu code. May be we can try and then contribute back to wireshark.
I also tried modifying the wireshark side udpdump source code so it does not add the second layer of exported_pdu. It had no issues identifying the TCP data you send.
The udpdump added exported_pdu layer is also needed when you have multiple g3proxy process sending the traffic to the same wireshark instance. The src addr and dst addr may conflict within g3proxy processes on different hosts, the correct way to calculate the stream id is by using g3proxy's dump addr + src addr + dst addr
all together.
Don't forget to set the data
type to exported_pdu
when capturing.
This is my mistake. Local machine testing. I was thinking the server would be the remote server. For our use having the remote HTTP server would be useful (as well as the HTTP client). Any easy suggestions on passing that info down to the udpdump?
It's still the same problem, the src addr + remote addr
may conflict even on the same host, as the client may connect to different g3proxy ports but with the same remote target address.
The way to add more info is to add more fields in the exported_pdu layer, which need to be defined by wireshark first. I'm not sure whether they will accept it.
That would make sense. It is possible to do pcap over IP using a TCP pipe. PolarProxy does this (sending decrypted TLS over TCP).
I prefer to use UDP to do this, as packet loss is also allowed in internal processing. It's may be possible to replace the generated exported_pdu layer with IP+TCP, but this way we will lose the ability to set dissectors hints other than tcp.port.
It sounds like the few issues we have identified above are known. We will start planning on how to modify G3 to have more functionality in PCAP export. If you have any requirements or suggestions please let us know.
I think that the exported_pdu code in wireshark could be improved to match our needs. It will be good if you can help to do that.
Some background - we were leveraging ICAP but many sites have started using websockets which the ICAP protocol was not built to handle. So we started going down the PCAP route.
It's fine if you don't use the block&modify feature. You may also use ICAP for websocket upgrade request and then use udpdump to dump the traffic.
from g3.
Related Issues (20)
- 代理串联支持二级代理账号密码验证吗? HOT 4
- 如何只单独编译g3proxy的Linux版本 HOT 1
- rustls server performance doesn't scale well HOT 1
- Evaluate Profile-Guided Optimization (PGO) and LLVM BOLT HOT 1
- 后续可以支持PPV1 吗 HOT 1
- 请教完整的tlcp代理示例 HOT 3
- g3proxy: Ability to change ClientHello sent to upstream server HOT 18
- Track upstream features that we want to use
- Docker container support HOT 2
- 配置文件存放路径文档不对 HOT 2
- Documentation Improvements HOT 3
- Issues with SSL interception HOT 8
- ICAP issues HOT 3
- Bump g3fcgen version to v0.6.3 release HOT 1
- Feature Request: Full remote certificate inspection and duplication for TLS MITM HOT 2
- Allow hostnames in the config HOT 5
- g3-io-ext 编译出错 HOT 2
- Ability to disable HTTP2 connections? HOT 3
- 后续可以增加 支持分布式的user auth认证的功能吗 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from g3.