shadow / tgen

A powerful traffic generator that can model complex behaviors using Markov models and an action-dependency graph.

The Shadow Simulator

Quickstart

After installing the dependencies: build, test, and install Shadow into ~/.local:

$ ./setup build --clean --test
$ ./setup test
$ ./setup install

Read the usage guide or get started with some example simulations.

What is Shadow?

Shadow is a discrete-event network simulator that directly executes real application code, enabling you to simulate distributed systems with thousands of network-connected processes in realistic and scalable private network experiments using your laptop, desktop, or server running Linux.

Shadow experiments can be scientifically controlled and deterministically replicated, making it easier for you to reproduce bugs and eliminate confounding factors in your experiments.

How Does Shadow Work?

Shadow directly executes real applications:

• Shadow directly executes unmodified, real application code using native OS (Linux) processes.
• Shadow co-opts the native processes into a discrete-event simulation by interposing at the system call API.
• The necessary system calls are emulated such that the applications need not be aware that they are running in a Shadow simulation.

Shadow connects the applications in a simulated network:

• Shadow constructs a private, virtual network through which the managed processes can communicate.
• Shadow internally implements simulated versions of common network protocols (e.g., TCP and UDP).
• Shadow internally models network routing characteristics (e.g., path latency and packet loss) using a configurable network graph.

Why is Shadow Needed?

Network emulators (e.g., mininet) run real application code on top of real OS kernels in real time, but are non-determinsitic and have limited scalability: time distortion can occur if emulated processes exceed an unknown computational threshold, leading to undefined behavior.

Network simulators (e.g., ns-3) offer more experimental control and scalability, but have limited application-layer realism because they run application abstractions in place of real application code.

Shadow offers a novel, hybrid emulation/simulation architecture: it directly executes real applications as native OS processes in order to faithfully reproduce application-layer behavior while also co-opting the processes into a high-performance network simulation that can scale to large distributed systems with hundreds of thousands of processes.

Caveats

Shadow implements over 150 functions from the system call API, but does not yet fully support all API features. Although applications that make basic use of the supported system calls should work out of the box, those that use more complex features or functions may not yet function correctly when running in Shadow. Extending support for the API is a work-in-progress.

That being said, we are particularly motivated to run large-scale Tor Network simulations. This use-case is already fairly well-supported and we are eager to continue extending support for it.

More Information

Homepage:

• https://shadow.github.io

Documentation:

• User documentation
• Developer documentation

Community Support:

• https://github.com/shadow/shadow/discussions

Bug Reports:

• https://github.com/shadow/shadow/issues

tgen's Issues

Remove assertions and properly handle send/recv errors

The TGen plugin has several send/recv assertions when setting up a SOCKS connection. It is possible that the socket closed or has some other error, even during the setup phase, and these assertions would cause the program to abort in this case. We should properly handle the return vals from send/recv instead of asserting that the amount we tried to send/recv actually got sent/recved.

Link Error

Dear author. I encountered link errors when I run make. It happened on ubuntu20.04

How can I fix it? Thanks.

Problems installing tgen on arch

Hi, i'm trying to install tgen on arch and the process fail during the make command.

➜ build git:(main) make
[ 5%] Building C object src/CMakeFiles/tgen.dir/tgen-config.c.o
[ 10%] Building C object src/CMakeFiles/tgen.dir/tgen-driver.c.o
[ 15%] Building C object src/CMakeFiles/tgen.dir/tgen-generator.c.o
[ 20%] Building C object src/CMakeFiles/tgen.dir/tgen-graph.c.o
[ 25%] Building C object src/CMakeFiles/tgen.dir/tgen-io.c.o
[ 30%] Building C object src/CMakeFiles/tgen.dir/tgen-log.c.o
[ 35%] Building C object src/CMakeFiles/tgen.dir/tgen-main.c.o
[ 40%] Building C object src/CMakeFiles/tgen.dir/tgen-markovmodel.c.o
[ 45%] Building C object src/CMakeFiles/tgen.dir/tgen-optionparser.c.o
[ 50%] Building C object src/CMakeFiles/tgen.dir/tgen-peer.c.o
[ 55%] Building C object src/CMakeFiles/tgen.dir/tgen-pool.c.o
[ 60%] Building C object src/CMakeFiles/tgen.dir/tgen-server.c.o
[ 65%] Building C object src/CMakeFiles/tgen.dir/tgen-stream.c.o
[ 70%] Building C object src/CMakeFiles/tgen.dir/tgen-timer.c.o
[ 75%] Building C object src/CMakeFiles/tgen.dir/tgen-transport.c.o
[ 80%] Linking C executable tgen
/usr/bin/ld: warning: libcxsparse.so.3, needed by /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_qr' /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_ltsolve'
/usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_usolve' /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_lusol'
/usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_dropzeros' /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_fkeep'
/usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_multiply' /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_droptol'
/usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_lu' /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_dupl'
/usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_compress' /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_sfree'
/usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_utsolve' /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_spfree'
/usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_ipvec' /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_sprealloc'
/usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_add' /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_spalloc'
/usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_permute' /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_transpose'
/usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_entry' /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_gaxpy'
/usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_happly' /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_nfree'
/usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_sqr' /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_realloc'
/usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_cholsol' /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/libigraph.so: undefined reference to cs_di_lsolve'
collect2: error: ld returned 1 exit status
make[2]: *** [src/CMakeFiles/tgen.dir/build.make:323: src/tgen] Error 1
make[1]: *** [CMakeFiles/Makefile2:116: src/CMakeFiles/tgen.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

Add better documentation for the TGen state machine and timings

TGen reports transfer successes and failures in the log files using the stream-success and stream-failure keywords. These log messages report several timestamps that we use to compute download time and other metrics (e.g., circuit build times when downloading through Tor).

We should better document the steps of the state machine and clearly describe the times.

Here is a quick dump. From the TGen client's perspective, here is the transfer process when using Tor:

Step	Log file timestamp key	Description
1.	usecs-to-socket-create	Open's socket
2.	usecs-to-socket-connect	Connect to Tor’s socks server port
3.	usecs-to-proxy-init	Client initiates SOCKS handshake
4.	usecs-to-proxy-choice	SOCKS server responds to handshake
5.	usecs-to-proxy-request	Tell Tor to open connection to a tgen server
6.	usecs-to-proxy-response	Tor says OK connection opened
7.	usecs-to-command	TGen client sends (through Tor) command to tell the TGen server what to do
8.	usecs-to-response	Client received initial TGen server response to command
A9.	usecs-to-first-byte-send	Client sent first byte of transfer
A10.	usecs-to-last-byte-send	Client sent last byte of transfer
A11.	usecs-to-checksum-send	Client sent checksum computed over sent data
B9.	usecs-to-first-byte-recv	Client received first byte of transfer from server
B10.	usecs-to-last-byte-recv	Client received last byte of transfer from server
B11.	usecs-to-checksum-recv	Client received server's checksum computed over received data

The "A" and "B" phases happen concurrently as the client sends data to and receives data from the server, respectively.

The "transfer" time starts when the command is sent (step 7), and ends when checksums are both sent and received (after steps A11 and B11).

More human readable TGen config

Support for a more human-readable format such as YAML would be nice as it would be easier to read/write/modify config files and would help us eventually move away from XML configuration files.

We'll be making this change for Shadow, and it would be nice to keep TGen consistent too.

See shadow/shadow#773 for more details and discussion.

Doesn't build against igraph 0.10.3 (default version on fedora 38 and debian 12)

I get:

/tgen/src/tgen-graph.c: In function '_tgengraph_hasSelfLoop':
/tgen/src/tgen-graph.c:423:45: warning: passing argument 2 of 'igraph_neighbors' from incompatible pointer type [-Wincompatible-pointer-types]
  423 |         result = igraph_neighbors(g->graph, resultNeighborVertices, vertexIndex, IGRAPH_OUT);
      |                                             ^~~~~~~~~~~~~~~~~~~~~~
      |                                             |
      |                                             igraph_vector_t *
In file included from /usr/include/igraph/igraph.h:57,
                 from /tgen/src/tgen-graph.c:6:
/usr/include/igraph/igraph_interface.h:54:91: note: expected 'igraph_vector_int_t *' but argument is of type 'igraph_vector_t *'
   54 | IGRAPH_EXPORT igraph_error_t igraph_neighbors(const igraph_t *graph, igraph_vector_int_t *neis, igraph_integer_t vid,
      |                                                                      ~~~~~~~~~~~~~~~~~~~~~^~~~
/tgen/src/tgen-graph.c:427:17: warning: 'igraph_vector_e' is deprecated [-Wdeprecated-declarations]
  427 |                 igraph_integer_t dstVertexIndex = igraph_vector_e(resultNeighborVertices, i);
      |                 ^~~~~~~~~~~~~~~~
In file included from /usr/include/igraph/igraph_vector.h:70,
                 from /usr/include/igraph/igraph_random.h:29,
                 from /usr/include/igraph/igraph.h:34:
/usr/include/igraph/igraph_vector_pmt.h:72:47: note: declared here
   72 | IGRAPH_EXPORT IGRAPH_DEPRECATED BASE FUNCTION(igraph_vector, e)(const TYPE(igraph_vector)* v, igraph_integer_t pos);
      |                                               ^~~~~~~~~~~~~
/usr/include/igraph/igraph_pmt.h:24:23: note: in definition of macro 'CONCAT2x'
   24 | #define CONCAT2x(a,b) a ## _ ## b
      |                       ^
/usr/include/igraph/igraph_pmt.h:161:31: note: in expansion of macro 'CONCAT2'
  161 |         #define FUNCTION(a,c) CONCAT2(a,c)
      |                               ^~~~~~~
/usr/include/igraph/igraph_vector_pmt.h:72:38: note: in expansion of macro 'FUNCTION'
   72 | IGRAPH_EXPORT IGRAPH_DEPRECATED BASE FUNCTION(igraph_vector, e)(const TYPE(igraph_vector)* v, igraph_integer_t pos);
      |                                      ^~~~~~~~
/tgen/src/tgen-graph.c: In function '_tgengraph_countIncomingEdges':
/tgen/src/tgen-graph.c:456:41: warning: passing argument 2 of 'igraph_neighbors' from incompatible pointer type [-Wincompatible-pointer-types]
  456 |     result = igraph_neighbors(g->graph, resultNeighborVertices, vertexIndex, IGRAPH_IN);
      |                                         ^~~~~~~~~~~~~~~~~~~~~~
      |                                         |
      |                                         igraph_vector_t *
/usr/include/igraph/igraph_interface.h:54:91: note: expected 'igraph_vector_int_t *' but argument is of type 'igraph_vector_t *'
   54 | IGRAPH_EXPORT igraph_error_t igraph_neighbors(const igraph_t *graph, igraph_vector_int_t *neis, igraph_integer_t vid,
      |                                                                      ~~~~~~~~~~~~~~~~~~~~~^~~~
/tgen/src/tgen-graph.c: In function '_tgengraph_parseGraphProperties':
/tgen/src/tgen-graph.c:1094:5: warning: 'igraph_clusters' is deprecated [-Wdeprecated-declarations]
 1094 |     result = igraph_clusters(g->graph, NULL, NULL, &(g->clusterCount), IGRAPH_WEAK);
      |     ^~~~~~
In file included from /usr/include/igraph/igraph.h:63:
/usr/include/igraph/igraph_components.h:45:48: note: declared here
   45 | IGRAPH_EXPORT IGRAPH_DEPRECATED igraph_error_t igraph_clusters(const igraph_t *graph, igraph_vector_int_t *membership,
      |                                                ^~~~~~~~~~~~~~~
/tgen/src/tgen-graph.c:1120:56: warning: passing argument 3 of 'igraph_cattribute_list' from incompatible pointer type [-Wincompatible-pointer-types]
 1120 |     result = igraph_cattribute_list(g->graph, &gnames, &gtypes, &vnames, &vtypes, &enames, &etypes);
      |                                                        ^~~~~~~
      |                                                        |
      |                                                        igraph_vector_t *
In file included from /usr/include/igraph/igraph_conversion.h:34,
                 from /usr/include/igraph/igraph.h:73:
/usr/include/igraph/igraph_attributes.h:379:91: note: expected 'igraph_vector_int_t *' but argument is of type 'igraph_vector_t *'
  379 |                                          igraph_strvector_t *gnames, igraph_vector_int_t *gtypes,
      |                                                                      ~~~~~~~~~~~~~~~~~~~~~^~~~~~
/tgen/src/tgen-graph.c:1120:74: warning: passing argument 5 of 'igraph_cattribute_list' from incompatible pointer type [-Wincompatible-pointer-types]
 1120 |     result = igraph_cattribute_list(g->graph, &gnames, &gtypes, &vnames, &vtypes, &enames, &etypes);
      |                                                                          ^~~~~~~
      |                                                                          |
      |                                                                          igraph_vector_t *
/usr/include/igraph/igraph_attributes.h:380:91: note: expected 'igraph_vector_int_t *' but argument is of type 'igraph_vector_t *'
  380 |                                          igraph_strvector_t *vnames, igraph_vector_int_t *vtypes,
      |                                                                      ~~~~~~~~~~~~~~~~~~~~~^~~~~~
/tgen/src/tgen-graph.c:1120:92: warning: passing argument 7 of 'igraph_cattribute_list' from incompatible pointer type [-Wincompatible-pointer-types]
 1120 |     result = igraph_cattribute_list(g->graph, &gnames, &gtypes, &vnames, &vtypes, &enames, &etypes);
      |                                                                                            ^~~~~~~
      |                                                                                            |
      |                                                                                            igraph_vector_t *
/usr/include/igraph/igraph_attributes.h:381:91: note: expected 'igraph_vector_int_t *' but argument is of type 'igraph_vector_t *'
  381 |                                          igraph_strvector_t *enames, igraph_vector_int_t *etypes);
      |                                                                      ~~~~~~~~~~~~~~~~~~~~~^~~~~~
/tgen/src/tgen-graph.c:1130:9: error: too many arguments to function 'igraph_strvector_get'
 1130 |         igraph_strvector_get(&gnames, (glong) i, &name);
      |         ^~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/igraph/igraph.h:47:
/usr/include/igraph/igraph_strvector.h:70:27: note: declared here
   70 | IGRAPH_EXPORT const char* igraph_strvector_get(const igraph_strvector_t *sv, igraph_integer_t idx);
      |                           ^~~~~~~~~~~~~~~~~~~~
/tgen/src/tgen-graph.c:1137:9: error: too many arguments to function 'igraph_strvector_get'
 1137 |         igraph_strvector_get(&vnames, (glong) i, &name);
      |         ^~~~~~~~~~~~~~~~~~~~
/usr/include/igraph/igraph_strvector.h:70:27: note: declared here
   70 | IGRAPH_EXPORT const char* igraph_strvector_get(const igraph_strvector_t *sv, igraph_integer_t idx);
      |                           ^~~~~~~~~~~~~~~~~~~~
/tgen/src/tgen-graph.c:1154:9: error: too many arguments to function 'igraph_strvector_get'
 1154 |         igraph_strvector_get(&enames, (glong) i, &name);
      |         ^~~~~~~~~~~~~~~~~~~~
/usr/include/igraph/igraph_strvector.h:70:27: note: declared here
   70 | IGRAPH_EXPORT const char* igraph_strvector_get(const igraph_strvector_t *sv, igraph_integer_t idx);
      |                           ^~~~~~~~~~~~~~~~~~~~
/tgen/src/tgen-graph.c: In function 'tgengraph_getNextActionIDs':
/tgen/src/tgen-graph.c:1346:41: warning: passing argument 2 of 'igraph_neighbors' from incompatible pointer type [-Wincompatible-pointer-types]
 1346 |     result = igraph_neighbors(g->graph, resultNeighborVertices, srcVertexIndex, IGRAPH_OUT);
      |                                         ^~~~~~~~~~~~~~~~~~~~~~
      |                                         |
      |                                         igraph_vector_t *
/usr/include/igraph/igraph_interface.h:54:91: note: expected 'igraph_vector_int_t *' but argument is of type 'igraph_vector_t *'
   54 | IGRAPH_EXPORT igraph_error_t igraph_neighbors(const igraph_t *graph, igraph_vector_int_t *neis, igraph_integer_t vid,
      |                                                                      ~~~~~~~~~~~~~~~~~~~~~^~~~
/tgen/src/tgen-graph.c:1367:9: warning: 'igraph_vector_e' is deprecated [-Wdeprecated-declarations]
 1367 |         igraph_integer_t dstVertexIndex = igraph_vector_e(resultNeighborVertices, i);
      |         ^~~~~~~~~~~~~~~~
/usr/include/igraph/igraph_vector_pmt.h:72:47: note: declared here
   72 | IGRAPH_EXPORT IGRAPH_DEPRECATED BASE FUNCTION(igraph_vector, e)(const TYPE(igraph_vector)* v, igraph_integer_t pos);
      |                                               ^~~~~~~~~~~~~
/usr/include/igraph/igraph_pmt.h:24:23: note: in definition of macro 'CONCAT2x'
   24 | #define CONCAT2x(a,b) a ## _ ## b
      |                       ^
/usr/include/igraph/igraph_pmt.h:161:31: note: in expansion of macro 'CONCAT2'
  161 |         #define FUNCTION(a,c) CONCAT2(a,c)
      |                               ^~~~~~~
/usr/include/igraph/igraph_vector_pmt.h:72:38: note: in expansion of macro 'FUNCTION'
   72 | IGRAPH_EXPORT IGRAPH_DEPRECATED BASE FUNCTION(igraph_vector, e)(const TYPE(igraph_vector)* v, igraph_integer_t pos);
      |                                      ^~~~~~~~
make[2]: *** [src/CMakeFiles/tgen.dir/build.make:118: src/CMakeFiles/tgen.dir/tgen-graph.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:116: src/CMakeFiles/tgen.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

Trying to install tgen for shadow tor plugin

I'm trying to run the tor plugin for shadow, but i'm getting an error saying "Shadow XML parsing error 5: attribute 'path': '/home/me/.shadow/bin/tgen' is not a valid path to an existing regular file". I'm really having troubles figuring out how to get tgen inside that folder

Link Error of Tgen tools

Kindly look into the matter at the earliest

pip3 install -I path/to/tgen/tools

shows following error

ERROR: Cannot unpack file /tmp/pip-unpack-4_gr2uuj/tools (downloaded from /tmp/pip-req-build-6tyoaz6v, content-type: text/html; charset=utf-8); cannot detect archive format
ERROR: Cannot determine archive format of /tmp/pip-req-build-6tyoaz6v

Incorrect handling of SOCKS server reply to a CONNECT request

TGen is incorrectly handling a reply from a Socks5 server after sending the server a socks5 CONNECT request.

TGen always sends CONNECT commands, shown as the second byte (0x01) here and here:

The RFC states that the reply to a CONNECT should be handled like this:

In the reply to a CONNECT, BND.PORT contains the port number that the server assigned to connect to the target host, while BND.ADDR contains the associated IP address. The supplied BND.ADDR is often different from the IP address that the client uses to reach the SOCKS server, since such servers are often multi-homed. It is expected that the SOCKS server will use DST.ADDR and DST.PORT, and the client-side source address and port in evaluating the CONNECT.

However, TGen incorrectly interprets non-zero BND.ADDR and BND.PORT as a request from the server to reconnect to the server at that new BND ADDR:PORT. I believe this reconnection behavior is not part of the CONNECT command, but might be part of the BIND command (which TGen is not sending to the server). See the RFC linked above, under the BIND section.

The is a bug because TGen is incorrectly erroring out of connections to socks servers when the socks server reply contains non-zero BND ADDR and PORTs, here and here

I think the fix is that TGen should just ignore whatever the socks server sends back as the BND.ADDR and BND.PORT in the reply to the CONNECT request.

tgentools date filter not parsing any stream event

I have noticed that the date filter in tgentools is not parsing any stream event.

I think the problem is in this snippet of code:

https://github.com/shadow/tgen/blob/main/tools/tgentools/analysis.py#L362-L371

I believe the first elif should be an if instead.

       if self.date_filter is not None:
            parts = line.strip().split(' ', 3)
            if len(parts) < 4: # the 3rd is the timestamp, the 4th is the rest of the line
                return True
            unix_ts = float(parts[2])
            line_date = datetime.datetime.utcfromtimestamp(unix_ts).date()
            if not self.__is_date_valid(line_date):
                return True

       if do_complete and re.search("stream-status", line) is not None:

tgentools: set number of xz threads to -m arg

When tgentools saves its results, it currently uses a hard-coded thread count of 3 for xz. For large experiments on machines with lots of cores, I've found that this ends up being the processing bottleneck. Using as many threads as specified with the -m arg would be a nice feature to have.

tgen-stream.c:1449:_tgenstream_onWritable: assertion failed: (_tgenstream_getTime(stream) >= stream->send.deferBarrierMicros)

I tripped this assertion in two tgen instances in a large-ish shadow simulation. They were both onion-service servers. Here's the failed simulation job: https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/jobs/149713

End of stdout:

2000-01-01 00:15:51 946685751.677612 [message] [tgen-stream.c:1633] [_tgenstream_log] [stream-success] transport [fd=494,local=localhost:127.0.0.1:80,proxy=NULL:0.0.0.0:0,remote=localhost:127.0.0.1:38376,state=SUCCESS_OPEN,error=NONE] stream [id=11462,vertexid=passive-stream:traffic,name=server77onionservice,peername=markovclient807onionservice,sendsize=0,recvsize=0,sendstate=SEND_SUCCESS,recvstate=RECV_SUCCESS,error=NONE] bytes [total-bytes-recv=8875,total-bytes-send=107632,payload-bytes-recv=8604,payload-bytes-send=107550,payload-progress-recv=100.00%,payload-progress-send=100.00%] times [created-ts=946685750153565,usecs-to-socket-create=0,usecs-to-socket-connect=0,usecs-to-proxy-init=-1,usecs-to-proxy-choice=-1,usecs-to-proxy-request=-1,usecs-to-proxy-response=-1,usecs-to-command=-1,usecs-to-response=792461,usecs-to-first-byte-recv=792461,usecs-to-last-byte-recv=1524047,usecs-to-checksum-recv=-1,usecs-to-first-byte-send=792461,usecs-to-last-byte-send=1473773,usecs-to-checksum-send=-1,now-ts=946685751677612]
2000-01-01 00:15:51 946685751.688462 [message] [tgen-stream.c:1633] [_tgenstream_log] [stream-success] transport [fd=2468,local=localhost:127.0.0.1:80,proxy=NULL:0.0.0.0:0,remote=localhost:127.0.0.1:41346,state=SUCCESS_OPEN,error=NONE] stream [id=11426,vertexid=passive-stream:traffic,name=server77onionservice,peername=markovclient835onionservice,sendsize=0,recvsize=0,sendstate=SEND_SUCCESS,recvstate=RECV_SUCCESS,error=NONE] bytes [total-bytes-recv=14610,total-bytes-send=631042,payload-bytes-recv=14340,payload-bytes-send=630960,payload-progress-recv=100.00%,payload-progress-send=100.00%] times [created-ts=946685748865026,usecs-to-socket-create=0,usecs-to-socket-connect=0,usecs-to-proxy-init=-1,usecs-to-proxy-choice=-1,usecs-to-proxy-request=-1,usecs-to-proxy-response=-1,usecs-to-command=-1,usecs-to-response=654000,usecs-to-first-byte-recv=1061436,usecs-to-last-byte-recv=2823436,usecs-to-checksum-recv=-1,usecs-to-first-byte-send=654000,usecs-to-last-byte-send=2807700,usecs-to-checksum-send=-1,now-ts=946685751688462]
Bail out! ERROR:/builds/jnewsome/sponsor-61-sims/jobs/src/tgen/src/tgen-stream.c:1449:_tgenstream_onWritable: assertion failed: (_tgenstream_getTime(stream) >= stream->send.deferBarrierMicros)

stderr:

ERROR:/builds/jnewsome/sponsor-61-sims/jobs/src/tgen/src/tgen-stream.c:1449:_tgenstream_onWritable: assertion failed: (_tgenstream_getTime(stream) >= stream->send.deferBarrierMicros)

Full stdout:
server77onionservice.tgen.1000.stdout.gz

build warnings for igraph_i_set_attribute_table and g_memmove

When I installed tgen on Fedora34, I encountered the following warnings in the pictures. How should I fix the bugs?