wader / fq

jq for binary formats - tool, language and decoders for working with binary and text formats

fq

Tool, language and decoders for working with binary data.

Basic usage is fq . file or fq d file.

For details see usage.md.

Background

fq is inspired by the well known jq tool and language that allows you to work with binary formats the same way you would using jq. In addition it can present data like a hex viewer, transform, slice and concatenate binary data. It also supports nested formats and has an interactive REPL with auto-completion.

It was originally designed to query, inspect and debug media codecs and containers like mp4, flac, mp3, jpeg. But since then it has been extended to support a variety of formats like executables, packet captures (with TCP reassembly) and serialization formats like JSON, YAML, XML, ASN1 BER, Avro, CBOR, protobuf. In addition it also has functions to work with URLs, convert to/from hex, number bases, search for things etc.

In summary it aims to be jq, hexdump, dd and gdb for files combined into one.

NOTE: fq is still early in development so things might change, be broken or do not make sense. That also means that there is a great opportunity to help out!

Goals

• Make binaries accessible, queryable and sliceable.
• Nested formats and bit-oriented decoding.
• Quick and comfortable CLI tool.
• Bits and bytes transformations.

Hopes

• Make it useful enough that people want to help improve it.
• Inspire people to create similar tools.

Supported formats

aac_frame, adts, adts_frame, aiff, amf0, apev2, apple_bookmark, ar, asn1_ber, av1_ccr, av1_frame, av1_obu, avc_annexb, avc_au, avc_dcr, avc_nalu, avc_pps, avc_sei, avc_sps, avi, avro_ocf, bencode, bitcoin_blkdat, bitcoin_block, bitcoin_script, bitcoin_transaction, bits, bplist, bsd_loopback_frame, bson, bytes, bzip2, caff, cbor, csv, dns, dns_tcp, elf, ether8023_frame, exif, fairplay_spc, fit, flac, flac_frame, flac_metadatablock, flac_metadatablocks, flac_picture, flac_streaminfo, gif, gzip, hevc_annexb, hevc_au, hevc_dcr, hevc_nalu, hevc_pps, hevc_sps, hevc_vps, html, icc_profile, icmp, icmpv6, id3v1, id3v11, id3v2, ipv4_packet, ipv6_packet, jp2c, jpeg, json, jsonl, leveldb_descriptor, leveldb_log, leveldb_table, luajit, macho, macho_fat, markdown, matroska, moc3, mp3, mp3_frame, mp3_frame_vbri, mp3_frame_xing, mp4, mpeg_asc, mpeg_es, mpeg_pes, mpeg_pes_packet, mpeg_spu, mpeg_ts, msgpack, nes, ogg, ogg_page, opentimestamps, opus_packet, pcap, pcapng, pg_btree, pg_control, pg_heap, png, prores_frame, protobuf, protobuf_widevine, pssh_playready, rtmp, sll2_packet, sll_packet, tar, tcp_segment, tiff, tls, toml, tzif, udp_datagram, vorbis_comment, vorbis_packet, vp8_frame, vp9_cfm, vp9_frame, vpx_ccr, wasm, wav, webp, xml, yaml, zip

It can also work with some common text formats like URLs, hex, base64, PEM etc and for some serialization formats like XML, YAML, etc. it can transform both from and to jq values.

For details see formats.md and usage.md.

Presentations

• "fq - jq for binary formats" at FOSDEM 2023 - video & slides
• "fq - jq for binary formats" at No time to wait 6 - video - slides
• "fq - jq for binary formats" at Binary Tools Summit 2022 - video - slides

Install

Use one of the methods listed below or download a pre-built release for macOS, Linux or Windows. Unarchive it and move the executable to PATH etc.

On macOS if you don't install using one of the method below then you might have to manually allow the binary to run. This can be done by trying to run the binary, ignore the warning and then go into security preference and allow it. Same can be done with this command:

xattr -d com.apple.quarantine fq && spctl --add fq

Homebrew (macOS)

brew install wader/tap/fq

MacPorts

On macOS, fq can also be installed via MacPorts. More details here.

sudo port install fq

Windows

fq can be installed via scoop.

scoop install fq

Arch Linux

fq can be installed from the extra repository using pacman:

pacman -S fq

You can also build and install the development (VCS) package using an AUR helper:

paru -S fq-git

Nix

nix-shell -p fq

FreeBSD

Use the fq port.

Alpine

Currently in edge testing but should work fine in stable also.

apk add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing fq

Build from source

Make sure you have go 1.20 or later installed.

To install directly from git repository (no clone needed):

# build and install latest release
go install github.com/wader/fq@latest

# build and install latest master
go install github.com/wader/fq@master

# copy binary to $PATH if needed
cp "$(go env GOPATH)/bin/fq" /usr/local/bin

To build, run and test from source:

# build and run
go run .
# build and run with arguments
go run . -d mp3 . file.mp3
# just build
go build -o fq .
# run all tests and build binary
make test fq

TODO and ideas

See TODO.md

Development and adding a new decoder

See dev.md

Thanks and related projects

This project would not have been possible without itchyny's jq implementation gojq. I also want to thank HexFiend for inspiration and ideas and stedolan for inventing the jq language.

Similar or related works

Tools

• HexFiend - Hex editor for macOS with format template support.
• ImHex - A Hex Editor for Reverse Engineers.
• binspector - Binary format analysis tool with query language and REPL.
• kaitai - Declarative binary format parsing.
• Wireshark - Decodes network traffic (tip: tshark -T json).
• MediaInfo - Analyze media files (tip mediainfo --Output=JSON and mediainfo --Details=1).
• GNU poke - The extensible editor for structured binary data.
• ffmpeg/ffprobe - Powerful media libraries and tools.
• hexdump - Hex viewer tool.
• hex - Interactive hex viewer with format support via lua.
• hachoir - General python library for working binary data.
• scapy - Decode/Encode formats, focus on network protocols.

Projects and Standards

• Let's Solve the File Format Problem.
• PRONOM file format registry.
• Sustainability of Digital Formats at Library of Congress.
• Data Format Description Language (DFDL).

License

fq is distributed under the terms of the MIT License.

See the LICENSE file for license details.

Licenses of direct dependencies:

• Forked version of gojq - https://github.com/itchyny/gojq/blob/main/LICENSE (MIT)
• github.com/ergochat/readline - https://github.com/ergochat/readline/blob/master/LICENSE (MIT)
• github.com/BurntSushi/toml - https://github.com/BurntSushi/toml/blob/master/COPYING (MIT)
• github.com/creasty/defaults - https://github.com/creasty/defaults/blob/master/LICENSE (MIT)
• github.com/gomarkdown/markdown - https://github.com/gomarkdown/markdown/blob/master/LICENSE.txt (BSD)
• github.com/gopacket/gopacket - https://github.com/gopacket/gopacket/blob/master/LICENSE (BSD)
• github.com/mitchellh/copystructure - https://github.com/mitchellh/copystructure/blob/master/LICENSE (MIT)
• github.com/mitchellh/mapstructure - https://github.com/mitchellh/mapstructure/blob/master/LICENSE (MIT)
• github.com/pmezard/go-difflib - https://github.com/pmezard/go-difflib/blob/master/LICENSE (BSD)
• golang/snappy - https://github.com/golang/snappy/blob/master/LICENSE (BSD)
• golang/x/* - https://github.com/golang/text/blob/master/LICENSE (BSD)
• gopkg.in/yaml.v3 - https://github.com/go-yaml/yaml/blob/v3/LICENSE (MIT)
• Parts of go crypto/tls and github.com/zmap/zcrypto - https://github.com/zmap/zcrypto/blob/master/LICENSE (Apache)

fq's Issues

Rebrand as "jq for formats"?

One of reasons fq is not called bq is because i was aiming for decoding formats not only binary formats. Is "jq for formats" good or should be clearer? Suggestions welcome.

fq can now do things like this:

$ fq -n -d html '[inputs | {(input_filename): .html.head.title}?] | add' *.html
{
  "Media Internals.html": "Media Internals",
  "make_2.html": "GNU make - An Introduction to Makefiles",
  "search.html": "Search · decrypt"
}

$ cat file.csv | fq -d csv '.[1:] | map(.[1] | tonumber) | add'
486

$ fq -rd bencode 'torepr | del(.info.pieces) | {torrent: .} | toxml({indent: 2})' debian-live-10.7.0-amd64-gnome.iso.torrent
<torrent>
  <announce>http://bttracker.debian.org:6969/announce</announce>
  <comment>&#34;Debian CD from cdimage.debian.org&#34;</comment>
  <creation date>1607188931</creation date>
  <httpseeds>https://cdimage.debian.org/cdimage/release/10.7.0-live/amd64/iso-hybrid/debian-live-10.7.0-amd64-gnome.iso</httpseeds>
  <httpseeds>https://cdimage.debian.org/cdimage/archive/10.7.0-live/amd64/iso-hybrid/debian-live-10.7.0-amd64-gnome.iso</httpseeds>
  <info>
    <length>2654552064</length>
    <name>debian-live-10.7.0-amd64-gnome.iso</name>
    <piece length>1048576</piece length>
  </info>
</torrent>

$ echo 'https://host?a=b' | fq -Rr 'fromurl | .query.b = ("hello" | tobase64) | tourl'
https://host?a=b&b=aGVsbG8%3D

$ fq '.svg | [."-height", ."-width"]' file.svg
[
  "1684pt",
  "5664pt"
]

At the moment some formats are not real format (can't be used with -d) ex url and base64, they are just jq functions. Maybe they should be formats in addition to being functions? so you could do echo ... | fq -d url ....

feature request: support rdb

Support redis database dump.

Apple Binary Property List Enhancement: Timestamps

Per @wader 's comment in the original PR, it would be nice to add an option for bplist decoding to display time values as timestamps instead of just CocoaDate floating point values in torepr/tovalue representations (and maybe elsewhere?).

fq hangs when reading from STDIN

What version are you using (fq -v)?

Both:
0.0.10 (darwin amd64)

and:
0.2.0 (darwin amd64)

How was fq installed?

brew install wader/tap/fq

Can you reproduce the problem using the latest release or master branch?

Yes (see above).

What did you do?

#!/bin/bash

# works with jq, gojq but not fq
JQ=fq
< /dev/random tr -cd '0-9' | fold -w 1 | $JQ -Mcnr '

### Generic Utility Functions
# Output: a PRN in range(0;$n) where $n is .
def prn:
  if . == 1 then 0
  else . as $n
  | (($n-1)|tostring|length) as $w
  | [limit($w; inputs)] | join("") | tonumber
  | if . < $n then . else ($n | prn) end
  end;

2 | prn
'

What result did you expect?

Same as for jq and gojq

What did you see instead?

The script hangs.

[FORMAT] Analog Captions & DTVCC Specs

maybe open a new issue if you want to dump some specs and ideas.

The wikipedia article about it seems quite good https://en.wikipedia.org/wiki/EIA-608 but would be nice to get hands on the spec.

Line 21 Analog Captions (EIA-608)

Spec for Analog Line 21 Captions is available, free to everyone, from https://shop.cta.tech/products/line-21-data-services. EIA-608 was originally designed for line-21 analog captions, so the spec covers both the data format and control codes and how to transmit them as analog. The transmission in line 21 analog is irrelevant in a digital world, but the control codes and decoding part is still referenced by its successor, DTVCC (EIA-708).

However, there is also a great Web1.0 Page, which covers the mystery 608 control codes, data channels, fields and 7 bit and includes lookup tables for the two-byte words. The control codes are typically marked up in human readable as {ENM}, {EOC}, {EDC}, as these are the abbreviations used in the EIA-608 spec. The McPoodle tools are written in circa-2005 perl. You will have seen references to SCC captions, which is basically just the file-based representation of 608 data. When caption tools disassemble the 608/SCC data, they are usually represented in these human readable closed caption disassembly (CCD) codes, although each captioning and conversion tool has their own proprietary format of CCD file, but the codes contained within like {ENM}, {EOC}, {EDC} are usually represented in the same markup. This reverse-engineered documentation is what everyone used before the EIA-608 spec became freely available.

• http://www.theneitherworld.com/mcpoodle/SCC_TOOLS/DOCS/CC_CODES.HTML
• http://www.theneitherworld.com/mcpoodle/SCC_TOOLS/DOCS/CC_CHARS.HTML

Digital TV Closed Captions (DTVCC), aka EIA-708.

Spec for DTVCC (EIA-708) is available, free to everyone, from https://shop.cta.tech/products/digital-television-dtv-closed-captioning. The DTVCC spec then defines both the transmission format, ccdata() and a new encoding format called 708 (Service 1/2), but the same spec allows cross-references EIA-608 for CC1/2/3/4 compatibility and explains how EIA-608 is encoded in ccdata(), which is exactly what we have been playing with.

You'll have to register with CTA to "buy it for $0", but all the other standards shops charge for it. If you get it from the CTA (previously EIA/CEA), both are free. Historically, they were chargeable. Since it is copyrighted material, I have not attached these to your repo, not out of laziness, but out of respect.

Feel free to close this ticket.

Add CBOR support

Run expect script on windows CI

CI env is https://github.com/actions/virtual-environments/blob/main/images/win/Windows2022-Readme.md

The expect test is done from here https://github.com/wader/fq/blob/master/Makefile#L24 using https://github.com/wader/fq/blob/master/pkg/cli/test.sh and https://github.com/wader/fq/blob/master/pkg/cli/test.exp. The shell script is helper around the expect script to be able to pass a fq binary to test and also to be silent on success and print log if not.

The windows image has make and bash so hopefully it is mostly about installing expect and then checking if expect.exe is in path?

Feature idea: support ASN.1 (DER/BER)

Hey, it would be pretty cool with fq supported decoding binary¹ ASN.1 (Abstract Syntax Notation One) as it is used in various communication protocols.

¹ ASN.1 has many encoding options but pretty much everything that is binary encoded seems to be either BER or DER.

The main problem I see is that the field names are not included in the actual files, so either the user would have to point to an ASN.1 grammar/spec file or the output would be "nameless".

[question] Is there a way to output raw bytes for selected field?

For example I can view the text of a tag in MP3 file using

fq '.headers[0].frames[0].text' file.mp3

But is there a way to dump the raw bytes from this text to stdout for further processing? For example somethind like this:

fq --some-option '.headers[0].frames[0].text' file.mp3 | iconv

I looked through the documentation, but didn't see anything like it.

P.S. Thank you for this tool, it is very nice - simple, but extremely useful.

Incorrect version in some distribution packages

$ docker run --rm alpine:3.15 sh -c 'apk add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing fq && fq --version'
fetch http://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
(1/1) Installing fq (0.0.3-r0)
Executing busybox-1.34.1-r3.trigger
OK: 15 MiB in 15 packages
dev

Currently version is set by goreleaser/make when building using go build ldflag -X main.version=.... To make it work with distributions that builds manually using go build etc we probably have to include the version in the source somehow.

• Create some version file using goreleaser when packaging source archives. Wont fix issue if distribution clones the repo.
• Convince package maintainers to build using make and pass version as some argument? too much hassel i think
• Manually change/commit some version file before release
• Some other way?

Arch Package

hi,

great tool. just want you to know that i created an ARCH package, so users of arch linux can now simply install it by

$ yay -S fq-bin

Database files: sqlite, dbf etc?

out of scope?

JSON to maintain array on predefined objects

@wader , Opening discussion at right forum now.

Could you please provide a combined command to do both tasks at once

• I would like to only specify few objects to be an array like ".aitem, .oitem, .sitem" but not all the JSON.
• convert all numbers/floats/booleans without double quotes as not a string.

<root>
    <aitem>
        <name>abc</name>
        <value>123</value>
    </aitem>
    <bitem>
        <name>bbbb</name>
        <value>2222</value>
    </bitem>
    <bitem>
        <name>BB</name>
        <value>22</value>
    </bitem>
</root>

Expected Output , Only specified like .aitem, .oitem, .sitem must become array but not entire JSON & All numbers/floats/booleans are as non string without double quoted.

{
  "root": {
    "aitem": [
    {
      "name": "abc",
      "value": 123
    }
    ],
    "bitem": [
      {
        "name": "bbbb",
        "value": 2222
      },
      {
        "name": "BB",
        "value": true
      }
    ]
  }
 }

Seeking from end of stream

I see that it is possible to seek relative to the current position or the beginning of the file. Is it possible to seek from the end, or does that functionality need to be added? Apple's binary plist format has a 32-byte trailer at the end of the file that contains metadata required for decoding the rest of the objects.

Increase performance

What version are you using (fq -v)?

$ fq -v
0.0.9 (linux amd64)

How was fq installed?

fq is build from source.

My branch:
https://github.com/pnsafonov/fq/tree/postgres

Can you reproduce the problem using the latest release or master branch?

Yes

What did you do?

I am implementing PostgreSQL format parsers. I can offer perfomance fix.

time fq -d pgheap -o flavour=postgres14 ".Pages[0].PageHeaderData.pd_linp[0,-1] | tovalue" 16397_8
{
  "lp_flags": "LP_NORMAL",
  "lp_len": 121,
  "lp_off": 8064
}
{
  "lp_flags": "LP_NORMAL",
  "lp_len": 121,
  "lp_off": 384
}

real    1m20.079s
user    1m21.643s
sys     0m0.389s

Then i made fix, and run again:

time fq -d pgheap -o flavour=postgres14 ".Pages[0].PageHeaderData.pd_linp[0,-1] | tovalue" 16397_8
{
  "lp_flags": "LP_NORMAL",
  "lp_len": 121,
  "lp_off": 8064
}
{
  "lp_flags": "LP_NORMAL",
  "lp_len": 121,
  "lp_off": 384
}

real    0m2.076s
user    0m3.452s
sys     0m0.242s

This fix decrease execution time 1m20.079s -> 0m2.076s, file decode.go:

func (d *D) AddChild(v *Value) {
	v.Parent = d.Value

	switch fv := d.Value.V.(type) {
	case *Compound:
		//if !fv.IsArray {
		//	for _, ff := range fv.Children {
		//		if ff.Name == v.Name {
		//			d.Fatalf("%q already exist in struct %s", v.Name, d.Value.Name)
		//		}
		//	}
		//}
		fv.Children = append(fv.Children, v)
	}
}

Also i made some perf with go tools. This was how i found fix:

   283.22s 84.00% 84.00%    321.80s 95.44%  github.com/wader/fq/pkg/decode.(*D).AddChild
    36.65s 10.87% 94.87%     36.65s 10.87%  memeqbody
     2.42s  0.72% 95.58%      4.94s  1.47%  runtime.scanobject
     0.94s  0.28% 95.86%      3.75s  1.11%  runtime.mallocgc
     0.44s  0.13% 95.99%      2.98s  0.88%  github.com/wader/fq/pkg/decode.(*D).TryFieldScalarFn.func1
     0.25s 0.074% 96.07%         5s  1.48%  runtime.gcDrain
     0.13s 0.039% 96.11%      2.58s  0.77%  runtime.newobject
     0.10s  0.03% 96.14%    323.53s 95.95%  github.com/wader/fq/pkg/decode.(*D).FillGaps
     0.08s 0.024% 96.16%      3.68s  1.09%  github.com/wader/fq/pkg/decode.(*D).TryFieldScalarFn
     0.03s 0.0089% 96.17%      2.21s  0.66%  github.com/wader/fq/pkg/decode.(*D).TryFieldScalarU16
     0.03s 0.0089% 96.18%      3.60s  1.07%  github.com/wader/fq/pkg/decode.(*D).TryFieldValue
     0.02s 0.0059% 96.18%      4.47s  1.33%  github.com/wader/fq/pkg/decode.(*D).FieldStruct
     0.02s 0.0059% 96.19%      2.25s  0.67%  github.com/wader/fq/pkg/decode.(*D).FieldU16
     0.02s 0.0059% 96.19%      2.14s  0.63%  github.com/wader/gojq.(*compiler).compileModule
     0.02s 0.0059% 96.20%    330.10s 97.90%  github.com/wader/gojq.(*env).Next
     0.01s 0.003% 96.20%      5.86s  1.74%  runtime.systemstack
         0     0% 96.20%      4.47s  1.33%  github.com/wader/fq/format/postgres.decodePgheap
         0     0% 96.20%      4.18s  1.24%  github.com/wader/fq/format/postgres/flavours/pgproee14.DecodeHeap (inline)

Feature request: Torrent metainfo Support

I think it would be really great to be able to look into .torrent files. Is this in the scope of this project?

[Question] what do 'f' and 'q' mean?

The issue is just a simple question: What do f and q mean?

I only know jq is a JSON processor, so what does q mean?

And what does f in fq mean? File?

It's a cryptic name, so I want to record it in cryptic_computer dictionary

Support for MessagePack (msgpack)

Description

fq looks like a great tool! Checking out the supported formats, it doesn't appear that https://msgpack.org/ is supported at the moment.

MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON.

Is this format something that would be considered for this project? It would be great to be able to inspect and filter against msgpack data.

Elf parsed as mp3

Is that expected to see such result of parsing elf (/bin/ls) as mp3?

$ file /bin/ls
/bin/ls: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=2f15ad836be3339dec0e2e6a3c637e08e48aacbd, for GNU/Linux 3.2.0, stripped
$ fq d /bin/ls

exr file format

the EXR file format, the professional-grade image storage see https://github.com/AcademySoftwareFoundation/openexr
Now

fq . AllHalfValues.exr
error: AllHalfValues.exr: probe: failed to decode (try -d FORMAT)

fq -d exr . AllHalfValues.exr
error: AllHalfValues.exr: exr: format group not found

Mach-O Binary Support

ELF is already supported. I think it is reasonable to add Mach-O format as a supported binary format to this tool.

As a reference for implementation I suggest following this document.

mkvmerge's output with FLAC cannot be read

First of all, I'm not sure this is a problem with fq at all, but it seems most likely.

For the flac specification, I was studying encapsulation for FLAC in matroska (fq is a great help here, thanks!) and while fq does wonderfully for ffmpeg, I wanted to check with mkvmerge. However, fq is unable to read these files. From the fq output (when forced to matroska) it seems mkvmerge adds a byte in front of the sync code, but I'm not sure.

Here is a FLAC file muxed to mkv with ffmpeg and mkvmerge. The first works fine, the second does not.

File muxed with ffmpeg
File muxed with mkvmerge

I've skimmed over the mkvmerge sources but couldn't find anything obvious. Furthermore, the mkvmerge file plays fine on ffmpeg, and produces no warnings or notices related to this extra byte even on the highest verbosity level. Therefore I think this is either a problem with fq or something that is left unspecified in the matroska embedding spec. The latter would obviously be very interesting for me :)

I hope you can help me out here. Thanks in advance!

Differentiate color/format for item count and index

In the lovely color coded output, the same color/format ("key: [value]") is used to present both item count and item index.
This is no big deal and I guess it's because all integers get this color, but slightly confusing.

Stack overflow when decoding longer dictionaries in Apple Binary Propery Lists

What version are you using (fq -v)?

0.0.10

$ fq -v
0.0.10 (darwin amd64)

How was fq installed?

go install github.com/wader/fq@master

Can you reproduce the problem using the latest release or master branch?

Yes

What did you do?

In order to find the size of the dictionary that triggers the bug, I took a plist that was causing the error, converted it to xml using plutil, and performed a binary search for the dictionary size that would trigger the bug. I tested on two different samples, and there wasn't a set number at which the bug would occur. The bug arose when increasing the dictionary element count from 40 to 41 in the first sample, and from 127 to 128 in the second sample. The second number is more striking, since it is the point at which a signed 8-bit integer would overflow, but the results from the first sample makes things confusing.

samples.zip

What result did you expect?

Proper decoding of a well-formed dictionary.

What did you see instead?

Stack overflow.

Runtime/bespoke format support like kaitai struct

Not a bug report, but a suggestion that might make supporting any format useful (I tend to work with really weird bespoke binary formats).

Any thoughts on adding support for arbitrary format support. Something like "if the decoded format isn't found, then try to resolve it as a file and load that".

i.e. fq --decode zxtap.go ".headers" game.tap - where zxtap.go follows your format file convention. Then I could make my own formats without having to rely on native support inside of fq (also solves those requests asking for other formats that have been raised already).

HTTP 1.0/1.1 support

I've done some experiments with http 1.0/1.1 and there are some things to consider:

• How to deal with white space/line separator. Own fields or bake into next/prev field?
• Support header line folding? (multi-line headers)
• How to represent multiple request/responses, root is always an array of structs?
• Decode body. Introduce a "content-type" group to let other decoders probe mime?
• De-chunking.
• Range-requests.
• Content-Encoding.
• Transfer-Encoding.
• Separate multi-part decoder, pass long content-type for decoding.
• How to get good test data. Done som promising experiments generating input files using caddy and curl --trace and some adhoc fq scripts
• How to pass options in combination with passing content-type options etc?

Hopefully a future http2/3 decoders can reuse same decode structure.

Runtime format descriptions

Very interested in this project, but I often work on non-standard or proprietary formats. All of the format support thus far seems to be done at compile time with parsers written as go modules. It'd be nice to have something that can be used for real oddball formats that either already exist in other binary-templating tools (kaitai, 010, hexfiend) or are not appropriate for upstreaming. Additionally, the current status quo requires anyone wanting to contribute a format to know/learn Go.

As such, it'd be very nice to have some way of using a new format without extending and recompiling fq directly ("Runtime Loading").

There's a bunch of existing standards here:

• Kaitai Struct format (.ksy)
• 010 Binary Template format (.bt)
• HexFiend/WinHex TCL format (.tcl)
• Probably much more

In addition, Go's plugin system could be used for this. This probably is the easiest to implement (since all existing formats are written as Go modules), but can't reuse existing templates from other ecosystems.

Add TLS support

Slowing working on it, aiming for these steps:

• Decode of most unencrypted data, records, extensions, handshake messages
• Decrypt most common ciphers (CHACHA20/AES AEAD) using a NSS keylog option
  • Needs some reorg inside fq to allow passing CLI options down to nested decoders as TLS will be inside PCAP -o keylog=@/path/to/keylog
  • Need to somehow communicate between TCP client/server stream decoders to coordinate cipher used etc. Idea now is to allow TCP decoder to return a value and deferred/post process function that will be used with the other streams returned value.

Comment here if you want to help out

bplist: NSKeyedArchiver jq function

NSKeyedArchiver stores objects in a bplist format by flattening the object into a set of keys and values, which reference each other by index. A common example of these are the sfl2 files located in ~/Library/Application Support/com.apple.sharedfilelist. @wader proposed the following function for reconstructing these objects into a more meaningful JSON representation:

def from_ns_keyed_archiver:
  (  . as {"$objects": $objs, "$top": {root: $root_uid}}
  | def _f($id):
      ( . #debug({$id})
      | $objs[$id]
      # | debug
      | if type == "string" then .
        elif type == "number" then .
        else
          (. as {"$class": $class}
          | if $class == 13 then # NSDictionary?
              ( . as {"NS.keys": $ns_keys, "NS.objects": $ns_objects}
              | [$ns_keys, $ns_objects]
              | transpose
              | map(
                  ( . as [$k, $o]
                  | {key: _f($k), value: _f($o)}
                  )
                )
              # | debug
              | from_entries
              )
            elif $class == 58 then #?
              ( . as {"NS.objects": $ns_objects}
              | $ns_objects
              | map(_f(.))
              )
            else "class-\($class)"
            end
          )
        end
      );
    _f($root_uid)
  );

However, it was found that the class numbers are not consistent across multiple files, so relying on them for interpreting underlying types is not a general solution. The following seems to work:

def from_ns_keyed_archiver:
  (  . as {"$objects": $objs, "$top": {root: $root_uid}}
  | def _f($id):
      ( . #| debug({$id})
      | $objs[$id]
      #| debug
      | if type == "string" then .
        elif type == "number" then .
        else
          (. as {"$class": $class}
          | . #debug
          | if ."NS.keys" != null and ."NS.objects" != null then
              ( . as {"NS.keys": $ns_keys, "NS.objects": $ns_objects}
              | [$ns_keys, $ns_objects]
              | transpose
              | map
                (
                  ( . as [$k, $o]
                  | {key: _f($k), value: _f($o)}
                  )
                )
              | from_entries
              )
            elif ."NS.objects" != null then
              ( . as {"NS.objects": $ns_objects}
              | $ns_objects
              | map(_f(.))
              )
            else "class-\($class)"
            end
          )
        end
      );
    _f($root_uid)
  );

However, we are not yet sure that this is a best practice since it is was created from a heuristic approach that is not based on any known reference documentation. More work is needed to identify the best way of identifying arrays and objects within NSKeyedArchiver representations.

_

_

Link file formats to existing format registries

@wader wrote:

👍 I wonder if some formats in fq could have links to these projects? currently format refrences are kept in <format-name>.md ex https://github.com/wader/fq/blob/master/format/mp4/mp4.md#references

I think the best registries of file formats to link to are

• Just Solve the File Format Problem
• Wikidata (currently with ~14.000 formats)

Both can be edited freely and both link to other registries such as PRONOM. Wikidata further contains more structured information such as default file extension and MIME Type. I once wrote a Wikidata module for jq this may help to get information from Wikidata (e.g. MP4).

Msgpack: fixstr gets wrong length

What version are you using (fq -v)?

$ fq -v
0.4.0 (darwin amd64)

How was fq installed?

Homebrew

Can you reproduce the problem using the latest release or master branch?

No changes to the msgpack module since the 0.4 as far as I can tell.

What did you do?

When encoding the string "string of length 19" (which has length 19), it erroneously gets length 3 instead.

string of length 19 in hex = 737472696e67206f66206c656e6774682031390a
fixstr of length 19 type = 0xb3 (see https://github.com/msgpack/msgpack/blob/master/spec.md#str-format-family)

Command: echo -n 'b3737472696e67206f66206c656e6774682031390a' | xxd -r -p | fq -d msgpack d

What result did you expect?

A fixstr of length 19.

What did you see instead?

A fixstr of length 3.

$ echo -n 'b3737472696e67206f66206c656e6774682031390a' | xxd -r -p | fq -d msgpack d
    |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13|0123456789abcdef0123|.{}: <stdin> (msgpack)
0x00|b3                                                         |.                   |  type: "fixstr" (0xb3)
0x00|b3                                                         |.                   |  length: 3
0x00|   73 74 72                                                | str                |  value: "str"
0x00|            69 6e 67 20 6f 66 20 6c 65 6e 67 74 68 20 31 39|    ing of length 19|  gap0: raw bits
0x14|0a|                                                        |.|                  |

Thank you for your great work on fq!

Support for PE COFF (.exe, .dll, .scr, etc Windows binaries) format ?

What version are you using (fq -v)?

$ fq -v
0.0.9 (linux amd64)

How was fq installed?

Downloaded from https://github.com/wader/fq/releases as that old ubuntu20 doesn't have packages.

Can you reproduce the problem using the latest release or master branch?

I believe it was.

What did you do?

I did just run fq . file.exe and turns out fq doens't know about PE files ! :D

$ fq . cobalt-strike-sample.exe 
error: cobalt-strike-sample.exe: probe: failed to decode (try -d FORMAT)
$ fq . -d raw cobalt-strike-sample.exe 
       |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.{}: cobalt-strike-sample.exe (raw)
0x00000|4d 5a 90 00 03 00 00 00 04 00 00 00 ff ff 00 00|MZ..............|  unknown0: raw bits
*      |until 0x4e1ff.7 (end) (320000)                 |                |

What result did you expect?

I did expect some knowledge of the PE format, but I understand that parsing PE files isn't simple as there are edge cases and ahem implementation details.

• https://paper.bobylive.com/Meeting_Papers/BlackHat/USA-2011/BH_US_11_VuksanPericin_PECOFF_WP.pdf

But if you lads have already implemented ELF then it shouldn't be unreachable to get PE COFF implemented ?

What did you see instead?

No knowledge of the PE/COFF format.

$ fq . cobalt-strike-sample.exe 
error: cobalt-strike-sample.exe: probe: failed to decode (try -d FORMAT)
$ fq . -d raw cobalt-strike-sample.exe 
       |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.{}: cobalt-strike-sample.exe (raw)
0x00000|4d 5a 90 00 03 00 00 00 04 00 00 00 ff ff 00 00|MZ..............|  unknown0: raw bits
*      |until 0x4e1ff.7 (end) (320000)                 |                |

Further reading

• https://learn.microsoft.com/en-us/windows/win32/debug/pe-format
• https://paper.bobylive.com/Meeting_Papers/BlackHat/USA-2011/BH_US_11_VuksanPericin_PECOFF_WP.pdf
• https://en.wikipedia.org/wiki/Portable_Executable
• https://justine.lol/ape.html & https://github.com/jart/cosmopolitan

Thanks a lot for reading, cheers !

Build and publish Snap image for releases

"Snap" is a popular package format that enables Linux users to install and run applications cross-distro.
In order to help potential users discover and use fq, it would be neat to automatically build and publish a snap during the release process.

fq utilizes GoReleaser in the "release" workflow, which according to the document supports building and publishing snaps.

I don't have any experience packaging snaps, so help would be greatly appreciated.

Is there HMAC-SHA256 hashing with a secret or something similar available?

What version are you using (fq -v)?

$ fq -v
0.1.0 (linux amd64)

I'm looking for some one-way hashing with a secret, ideally HMAC-SHA256. Does FQ have any?

Investigate bplist decode gaps

$ fq -o line_bytes=16 .  format/bplist/testdata/recentapps.sfl2
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│.{}: format/bplist/testdata/recentapps.sfl2 (bplist)
0x0000│62 70 6c 69 73 74 30 30                        │bplist00        │  header{}:
0x0000│                        d4 00 01 00 02 00 03 00│        ........│  objects{}:
0x0010│04 00 05 00 06 00 07 00 0a 58 24 76 65 72 73 69│.........X$versi│
*     │until 0x209f.7 (8344)                          │                │
0x01a0│   80 0d                                       │ ..             │  gap0: raw bits
0x01e0│                           80 0d               │         ..     │  gap1: raw bits
0x0510│                     80 06 80 07 80 08 80 09 80│       .........│  gap2: raw bits
0x0520│0a                                             │.               │
0x0520│                                    80 0b      │            ..  │  gap3: raw bits
0x0530│                  80 0d                        │      ..        │  gap4: raw bits
0x0540│                     80 0d                     │       ..       │  gap5: raw bits
0x07e0│                        80 06 80 07 80 08 80 09│        ........│  gap6: raw bits
0x07f0│80 0a                                          │..              │
0x07f0│                                       80 0b   │             .. │  gap7: raw bits
0x0800│                     80 0d                     │       ..       │  gap8: raw bits
0x0810│                        80 0d                  │        ..      │  gap9: raw bits
0x0ae0│                                    80 06 80 07│            ....│  gap10: raw bits
0x0af0│80 08 80 09 80 0a                              │......          │
0x0b00│   80 0b                                       │ ..             │  gap11: raw bits
0x0b00│                                 80 0d         │           ..   │  gap12: raw bits
0x0b10│                                    80 0d      │            ..  │  gap13: raw bits
0x0de0│               80 06 80 07 80 08 80 09 80 0a   │     .......... │  gap14: raw bits
0x0df0│                              80 0b            │          ..    │  gap15: raw bits
0x0e00│            80 0d                              │    ..          │  gap16: raw bits
0x0e10│               80 0d                           │     ..         │  gap17: raw bits
0x10c0│80 06 80 07 80 08 80 09 80 0a                  │..........      │  gap18: raw bits
0x10d0│               80 0b                           │     ..         │  gap19: raw bits
0x10d0│                                             80│               .│  gap20: raw bits
0x10e0│0d                                             │.               │
0x10f0│80 0d                                          │..              │  gap21: raw bits
0x13c0│   80 06 80 07 80 08 80 09 80 0a               │ ..........     │  gap22: raw bits
0x13d0│                  80 0b                        │      ..        │  gap23: raw bits
0x13e0│80 0d                                          │..              │  gap24: raw bits
0x13f0│   80 0d                                       │ ..             │  gap25: raw bits
0x16d0│                                             80│               .│  gap26: raw bits
0x16e0│06 80 07 80 08 80 09 80 0a                     │.........       │
0x16f0│            80 0b                              │    ..          │  gap27: raw bits
0x16f0│                                          80 0d│              ..│  gap28: raw bits
0x1700│                                             80│               .│  gap29: raw bits
0x1710│0d                                             │.               │
0x1a50│80 06 80 07 80 08 80 09 80 0a                  │..........      │  gap30: raw bits
0x1a60│               80 0b                           │     ..         │  gap31: raw bits
0x1a60│                                             80│               .│  gap32: raw bits
0x1a70│0d                                             │.               │
0x1a80│80 0d                                          │..              │  gap33: raw bits
0x1d90│80 06 80 07 80 08 80 09 80 0a                  │..........      │  gap34: raw bits
0x1da0│               80 0b                           │     ..         │  gap35: raw bits
0x1da0│                                             80│               .│  gap36: raw bits
0x1db0│0d                                             │.               │
0x1dc0│80 0d                                          │..              │  gap37: raw bits
0x2070│               80 0d                           │     ..         │  gap38: raw bits
0x20a0│00 08 00 19 00 22 00 2c 00 31 00 3a 00 3f 00 51│.....".,.1.:.?.Q│  offset_table[0:271]:
0x20b0│00 56 00 5b 00 5d 00 e4 00 ea 00 f7 00 ff 01 0a│.V.[.]..........│
*     │until 0x22bd.7 (542)                           │                │
0x22b0│                                          00 00│              ..│  trailer{}:
0x22c0│00 00 00 00 02 02 00 00 00 00 00 00 01 0f 00 00│................│
0x22d0│00 00 00 00 00 00 00 00 00 00 00 00 20 a0│     │............ .│ │

They all seems to be unused integer values between 0x06 and 0x0d (excluding 0x0c for some reason) and they match the bytes when showing 2 bytes for each entry (excluding 0) in the offset table that don't show up in any *_index field:

$ fq -o line_bytes=16 'tobytes as $b | ([range(.offset_table | length)] - [0,fgrep("_index$")]) as $missing | .offset_table[$missing[]] | $b[.:.+2]'  format/bplist/testdata/recentapps.sfl2
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1a0│   80 0d                                       │ ..             │.: raw bits 0x1a1-0x1a2.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1e0│                           80 0d               │         ..     │.: raw bits 0x1e9-0x1ea.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x510│                     80 06                     │       ..       │.: raw bits 0x517-0x518.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x510│                           80 07               │         ..     │.: raw bits 0x519-0x51a.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x510│                                 80 08         │           ..   │.: raw bits 0x51b-0x51c.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x510│                                       80 09   │             .. │.: raw bits 0x51d-0x51e.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x510│                                             80│               .│.: raw bits 0x51f-0x520.7 (2)
0x520│0a                                             │.               │
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x520│                                    80 0b      │            ..  │.: raw bits 0x52c-0x52d.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x530│                  80 0d                        │      ..        │.: raw bits 0x536-0x537.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x540│                     80 0d                     │       ..       │.: raw bits 0x547-0x548.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x7e0│                        80 06                  │        ..      │.: raw bits 0x7e8-0x7e9.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x7e0│                              80 07            │          ..    │.: raw bits 0x7ea-0x7eb.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x7e0│                                    80 08      │            ..  │.: raw bits 0x7ec-0x7ed.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x7e0│                                          80 09│              ..│.: raw bits 0x7ee-0x7ef.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x7f0│80 0a                                          │..              │.: raw bits 0x7f0-0x7f1.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x7f0│                                       80 0b   │             .. │.: raw bits 0x7fd-0x7fe.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x800│                     80 0d                     │       ..       │.: raw bits 0x807-0x808.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x810│                        80 0d                  │        ..      │.: raw bits 0x818-0x819.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xae0│                                    80 06      │            ..  │.: raw bits 0xaec-0xaed.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xae0│                                          80 07│              ..│.: raw bits 0xaee-0xaef.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xaf0│80 08                                          │..              │.: raw bits 0xaf0-0xaf1.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xaf0│      80 09                                    │  ..            │.: raw bits 0xaf2-0xaf3.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xaf0│            80 0a                              │    ..          │.: raw bits 0xaf4-0xaf5.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xb00│   80 0b                                       │ ..             │.: raw bits 0xb01-0xb02.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xb00│                                 80 0d         │           ..   │.: raw bits 0xb0b-0xb0c.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xb10│                                    80 0d      │            ..  │.: raw bits 0xb1c-0xb1d.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xde0│               80 06                           │     ..         │.: raw bits 0xde5-0xde6.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xde0│                     80 07                     │       ..       │.: raw bits 0xde7-0xde8.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xde0│                           80 08               │         ..     │.: raw bits 0xde9-0xdea.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xde0│                                 80 09         │           ..   │.: raw bits 0xdeb-0xdec.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xde0│                                       80 0a   │             .. │.: raw bits 0xded-0xdee.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xdf0│                              80 0b            │          ..    │.: raw bits 0xdfa-0xdfb.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xe00│            80 0d                              │    ..          │.: raw bits 0xe04-0xe05.7 (2)
     │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0xe10│               80 0d                           │     ..         │.: raw bits 0xe15-0xe16.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x10c0│80 06                                          │..              │.: raw bits 0x10c0-0x10c1.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x10c0│      80 07                                    │  ..            │.: raw bits 0x10c2-0x10c3.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x10c0│            80 08                              │    ..          │.: raw bits 0x10c4-0x10c5.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x10c0│                  80 09                        │      ..        │.: raw bits 0x10c6-0x10c7.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x10c0│                        80 0a                  │        ..      │.: raw bits 0x10c8-0x10c9.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x10d0│               80 0b                           │     ..         │.: raw bits 0x10d5-0x10d6.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x10d0│                                             80│               .│.: raw bits 0x10df-0x10e0.7 (2)
0x10e0│0d                                             │.               │
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x10f0│80 0d                                          │..              │.: raw bits 0x10f0-0x10f1.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x13c0│   80 06                                       │ ..             │.: raw bits 0x13c1-0x13c2.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x13c0│         80 07                                 │   ..           │.: raw bits 0x13c3-0x13c4.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x13c0│               80 08                           │     ..         │.: raw bits 0x13c5-0x13c6.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x13c0│                     80 09                     │       ..       │.: raw bits 0x13c7-0x13c8.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x13c0│                           80 0a               │         ..     │.: raw bits 0x13c9-0x13ca.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x13d0│                  80 0b                        │      ..        │.: raw bits 0x13d6-0x13d7.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x13e0│80 0d                                          │..              │.: raw bits 0x13e0-0x13e1.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x13f0│   80 0d                                       │ ..             │.: raw bits 0x13f1-0x13f2.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x16d0│                                             80│               .│.: raw bits 0x16df-0x16e0.7 (2)
0x16e0│06                                             │.               │
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x16e0│   80 07                                       │ ..             │.: raw bits 0x16e1-0x16e2.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x16e0│         80 08                                 │   ..           │.: raw bits 0x16e3-0x16e4.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x16e0│               80 09                           │     ..         │.: raw bits 0x16e5-0x16e6.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x16e0│                     80 0a                     │       ..       │.: raw bits 0x16e7-0x16e8.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x16f0│            80 0b                              │    ..          │.: raw bits 0x16f4-0x16f5.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x16f0│                                          80 0d│              ..│.: raw bits 0x16fe-0x16ff.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1700│                                             80│               .│.: raw bits 0x170f-0x1710.7 (2)
0x1710│0d                                             │.               │
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1a50│80 06                                          │..              │.: raw bits 0x1a50-0x1a51.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1a50│      80 07                                    │  ..            │.: raw bits 0x1a52-0x1a53.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1a50│            80 08                              │    ..          │.: raw bits 0x1a54-0x1a55.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1a50│                  80 09                        │      ..        │.: raw bits 0x1a56-0x1a57.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1a50│                        80 0a                  │        ..      │.: raw bits 0x1a58-0x1a59.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1a60│               80 0b                           │     ..         │.: raw bits 0x1a65-0x1a66.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1a60│                                             80│               .│.: raw bits 0x1a6f-0x1a70.7 (2)
0x1a70│0d                                             │.               │
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1a80│80 0d                                          │..              │.: raw bits 0x1a80-0x1a81.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1d90│80 06                                          │..              │.: raw bits 0x1d90-0x1d91.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1d90│      80 07                                    │  ..            │.: raw bits 0x1d92-0x1d93.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1d90│            80 08                              │    ..          │.: raw bits 0x1d94-0x1d95.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1d90│                  80 09                        │      ..        │.: raw bits 0x1d96-0x1d97.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1d90│                        80 0a                  │        ..      │.: raw bits 0x1d98-0x1d99.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1da0│               80 0b                           │     ..         │.: raw bits 0x1da5-0x1da6.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1da0│                                             80│               .│.: raw bits 0x1daf-0x1db0.7 (2)
0x1db0│0d                                             │.               │
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x1dc0│80 0d                                          │..              │.: raw bits 0x1dc0-0x1dc1.7 (2)
      │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
0x2070│               80 0d                           │     ..         │.: raw bits 0x2075-0x2076.7 (2)

Don't see anything obviously missing when comparing torepr and plutil -p format/bplist/testdata/recentapps.sfl2. Some lazy encoder including redundant integers for som reason?

demo.svg looks wired in my environment

demo.svg(https://github.com/wader/fq/blob/master/doc/demo.svg) looks wired in my browser.

I think that the font specified in demo.svg (monospace) and the coordinate of SVG text elements don't match in my environment.

environment:

• macOS Ventura 13.3.1（22E261）
• Google Chrome 112.0.5615.137

Fail to run test

What version are you using (fq -v)?

$ fq -v
0.0.10

How was fq installed?

Can you reproduce the problem using the latest release or master branch?

Yes, same behavior as the 0.0.10 version.

What did you do?

go build -trimpath -buildmode=pie -mod=readonly -modcacherw \
     -ldflags "-linkmode external -extldflags \"-Wl,-O1,--sort-common,--asneeded,-z,relro,-z,now\"" \
     -o dist/fq \
     .
make test

Additional environment context

Tests are running in Arch Linux RISC-V qemu-user.

What result did you expect?

Test pass successfully

What did you see instead?

Test mess up the terminal, and exit with non-zero status but no error reported.

Full log

go test -timeout 20m    ./format ./internal/aheadreadseeker ./internal/ansi ./internal/asciiwriter ./internal/ctxreadseeker ./internal/hexdump ./internal/hexpairwriter ./internal/mathex ./internal/recoverfn ./internal/shquote ./pkg/bitio ./pkg/interp ./pkg/ranges
go: downloading github.com/pmezard/go-difflib v1.0.0
ok  	github.com/wader/fq/format	44.486s
ok  	github.com/wader/fq/internal/aheadreadseeker	0.068s
ok  	github.com/wader/fq/internal/ansi	0.085s
ok  	github.com/wader/fq/internal/asciiwriter	0.076s
ok  	github.com/wader/fq/internal/ctxreadseeker	0.097s
?   	github.com/wader/fq/internal/hexdump	[no test files]
ok  	github.com/wader/fq/internal/hexpairwriter	0.079s
ok  	github.com/wader/fq/internal/mathex	0.054s
ok  	github.com/wader/fq/internal/recoverfn	0.076s
ok  	github.com/wader/fq/internal/shquote	0.072s
ok  	github.com/wader/fq/pkg/bitio	0.212s
ok  	github.com/wader/fq/pkg/interp	20.591s
ok  	github.com/wader/fq/pkg/ranges	0.070s
CGO_ENABLED=0 go build -o fq -ldflags "-s -w" -trimpath .
testjq pkg/interp/args_test.jq
testjq pkg/interp/funcs_test.jq
testjq pkg/interp/query_test.jq
spawn sh -c NO_DECODE_PROGRESS=1 fq -o color=false -i . pkg/interp/testdata/test.mp3 2>&1

 ��[6nmp3> 
.
 ��[6nmp3> .
     |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13|0123456789abcdef0123|.{}: pkg/interp/testdata/test.mp3 (mp3)
0x000|49 44 33 04 00 00 00 00 00 23 54 53 53 45 00 00 00 0f 00 00|ID3......#TSSE......|  headers[0:1]:
*    |until 0x2c.7 (45)                                          |                    |
0x028|               ff fb 40 c0 00 00 00 00 00 00 00 00 00 00 00|     ..@............|  frames[0:3]:
0x03c|00 00 00 00 00 00 49 6e 66 6f 00 00 00 0f 00 00 00 02 00 00|......Info..........|
*    |until 0x283.7 (end) (599)                                  |                    |
     |                                                           |                    |  footers[0:0]:
 ��[6nmp3> 
 ��[6nmp3> range(100000) | d
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
^C ��[6nmp3> 
 ��[6nmp3> range(100000)
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
^C ��[6nmp3> 
[range(100000)] | d
 ��[6nmp3> [range(100000)] | d
[
  0,
  1,
  2,
  3,
  4,
  5,
  6,
  7,
  8,
  9,
  10,
  11,
  12,
  13,
  14,
  15,
  16,
  17,
  18,
  19,
  20,
  21,
  22,
  23,
  24,
  25,
  26,
  27,
  28,
  29,
  30,
  31,
  32,
  33,
  34,
  35,
  36,
  37,
  38,
  39,
  40,
  41,
  42,
  43,
  44,
  45,
  46,
  47,
  48,
  49,
  50,
  51,
  52,
  53,
  54,
  55,
  56,
  57,
  58,
  59,
  60,
  61,
  62,
  63,
  64,
  65,
  66,
  67,
  68,
  69,
  70,
  71,
  72,
  73,
  74,
  75,
  76,
  77,
  78,
  79,
  80,
  81,
  82,
  83,
  84,
  85,
  86,
  87,
  88,
  89,
  90,
  91,
  92,
  93,
  94,
  95,
  96,
  97,
  98,
  99,
  100,
  101,
  102,
  103,
  104,
  105,
  106,
  107,
  108,
  109,
  110,
  111,
  112,
  113,
  114,
  115,
  116,
  117,
  118,
  119,
  120,
  121,
  122,
  123,
  124,
  125,
  126,
  127,
  128,
  129,
  130,
  131,
  132,
  133,
  134,
  135,
  136,
  137,
  138,
  139,
  140,
  141,
  142,
  143,
  144,
  145,
  146,
  147,
  148,
  149,
  150,
  151,
  152,
  153,
  154,
  155,
  156,
  157,
  158,
  159,
  160,
  161,
  162,
  163,
  164,
  165,
  166,
  167,
  168,
  169,
  170,
  171,
  172,
  173,
  174,
  175,
  176,
  177,
  178,
  179,
  180,
  181,
  182,
  183,
  184,
  185,
  186,
  187,
  188,
  189,
  190,
  191,
  192,
  193,
  194,
  195,
  196,
  197,
  198,
  199,
  200,
  201,
  202,
  203,
  204,
  205,
  206,
  207,
  208,
  209,
  210,
  211,
  212,
  213,
  214,
  215,
  216,
  217,
  218,
  219,
  220,
  221,
  222,
  223,
  224,
  225,
  226,
  227,
  228,
  229,
  230,
  231,
  232,
  233,
  234,
  235,
  236,
  237,
  238,
  239,
  240,
  241,
  242,
  243,
  244,
  245,
  246,
  247,
  248,
  249,
  250,
  251,
  252,
  253,
  254,
  255,
  256,
  257,
  258,
  259,
  260,
  261,
  262,
  263,
  264,
  265,
  266,
  267,
  268,
  269,
  270,
  271,
  272,
  273,
  274,
  275,
  276,
  277,
  278,
  279,
  280,
  281,
  282,
  283,
  284,
  285,
  286,
  287,
  288,
  289,
  290,
  291,
  292,
  293,
  294,
  295,
  296,
  297,
  298,
  299,
  300,
  301,
  302,
  303,
  304,
  305,
  306,
  307,
  308,
  309,
  310,
  311,
  312,
  313,
  314,
  315,
  316,
  317,
  318,
  319,
  320,
  321,
  322,
  323,
  324,
  325,
  326,
  327,
  328,
  329,
  330,
  331,
  332,
  333,
  334,
  335,
  336,
  337,
  338,
  339,
  340,
  341,
  342,
  343,
  344,
  345,
  346,
  347,
  348,
  349,
  350,
  351,
  352,
  353,
  354,
  355,
  356,
  357,
  358,
  359,
  360,
  361,
  362,
  363,
  364,
  365,
  366,
  367,
  368,
  369,
  370,
  371,
  372,
  373,
  374,
  375,
  376,
  377,
  378,
  379,
  380,
  381,
  382,
  383,
  384,
  385,
  386,
  387,
  388,
  389,
  390,
  391,
  392,
  393,
  394,
  395,
  396,
  397,
  398,
  399,
  400,
  401,
  402,
  403,
  404,
  405,
  406,
  407,
  408,
  409,
  410,
  411,
  412,
  413,
  414,
  415,
  416,
  417,
  418,
  419,
  420,
  421,
  422,
  423,
  424,
  425,
  426,
  427,
  428,
  429,
  430,
  431,
  432,
  433,
  434,
  435,
  436,
  437,
  438,
  439,
  440,
  441,
  442,
  443,
  444,
  445,
  446,
  447,
  448,
  449,
  450,
  451,
  452,
  453,
  454,
  455,
  456,
  457,
  458,
  459,
  460,
  461,
  462,
  463,
  464,
  465,
  466,
  467,
  468,
  469,
  470,
  471,
  472,
  473,
  474,
  475,
  476,
  477,
  478,
  479,
  480,
  481,
  482,
  483,
  484,
  485,
  486,
  487,
  488,
  489,
  490,
  491,
  492,
  493,
  494,
  495,
  496,
  497,
  498,
  499,
  500,
  501,
  502,
  503,
  504,
  505,
  506,
  507,
  508,
  509,
  510,
  511,
  512,
  513,
  514,
  515,
  516,
  517,
  518,
  519,
  520,
  521,
  522,
  523,
  524,
  525,
  526,
  527,
  528,
  529,
  530,
  531,
  532,
  533,
  534,
  535,
  536,
  537,
  538,
  539,
  540,
  541,
  542,
  543,
  544,
  545,
  546,
  547,
  548,
  549,
  550,
  551,
  552,
  553,
  554,
  555,
  556,
  557,
  558,
  559,
  560,
  561,
  562,
  563,
  564,
  565,
  566,
  567,
  568,
  569,
  570,
  571,
  572,
  573,
  574,
  575,
  576,
  577,
  578,
  579,
  580,
  581,
  582,
  583,
  584,
  585,
  586,
  587,
  588,
  589,
  590,
  591,
  592,
  593,
  594,
  595,
  596,
  597,
  598,
  599,
  600,
  601,
  602,
  603,
  604,
  605,
  606,
  607,
  608,
  609,
  610,
  611,
  612,
  613,
  614,
  615,
  616,
  617,
  618,
  619,
  620,
  621,
  622,
  623,
  624,
  625,
  626,
  627,
  628,
  629,
  630,
  631,
  632,
  633,
  634,
  635,
  636,
  637,
  638,
  639,
  640,
  641,
  642,
  643,
  644,
  645,
  646,
  647,
  648,
  649,
  650,
  651,
  652,
  653,
  654,
  655,
  656,
  657,
  658,
  659,
  660,
  661,
  662,
  663,
  664,
  665,
  666,
  667,
  668,
  669,
  670,
  671,
  672,
  673,
  674,
  675,
  676,
  677,
  678,
  679,
  680,
  681,
  682,
  683,
  684,
  685,
  686,
  687,
  688,
  689,
  690,
  691,
  692,
  693,
  694,
  695,
  696,
  697,
  698,
  699,
  700,
  701,
  702,
  703,
  704,
  705,
  706,
  707,
  708,
  709,
  710,
  711,
  712,
  713,
  714,
  715,
  716,
  717,
  718,
  719,
  720,
  721,
  722,
  723,
  724,
  725,
  726,
  727,
  728,
  729,
  730,
  731,
  732,
  733,
  734,
  735,
  736,
  737,
  738,
  739,
  740,
  741,
  742,
  743,
  744,
  745,
  746,
  747,
  748,
  749,
  750,
  751,
  752,
  753,
  754,
  755,
  756,
  757,
  758,
  759,
  760,
  761,
  762,
  763,
  764,
  765,
  766,
  767,
  768,
  769,
  770,
  771,
  772,
  773,
  774,
  775,
  776,
  777,
  778,
  779,
  780,
  781,
  782,
  783,
  784,
  785,
  786,
  787,
  788,
^C ��[6nmp3> 
[range(100000)]
 ��[6nmp3> [range(100000)]
[
  0,
  1,
  2,
  3,
  4,
  5,
  6,
  7,
  8,
  9,
  10,
  11,
  12,
  13,
  14,
  15,
  16,
  17,
  18,
  19,
  20,
  21,
  22,
  23,
  24,
  25,
  26,
  27,
  28,
  29,
  30,
  31,
  32,
  33,
  34,
  35,
  36,
  37,
  38,
  39,
  40,
  41,
  42,
  43,
  44,
  45,
  46,
  47,
  48,
  49,
  50,
  51,
  52,
  53,
  54,
  55,
  56,
  57,
  58,
  59,
  60,
  61,
  62,
  63,
  64,
  65,
  66,
  67,
  68,
  69,
  70,
  71,
  72,
  73,
  74,
  75,
  76,
  77,
  78,
  79,
  80,
  81,
  82,
  83,
  84,
  85,
  86,
  87,
  88,
  89,
  90,
  91,
  92,
  93,
  94,
  95,
  96,
  97,
  98,
  99,
  100,
  101,
  102,
  103,
  104,
  105,
  106,
  107,
  108,
  109,
  110,
  111,
  112,
  113,
  114,
  115,
  116,
  117,
  118,
  119,
  120,
  121,
  122,
  123,
  124,
  125,
  126,
  127,
  128,
  129,
  130,
  131,
  132,
  133,
  134,
  135,
  136,
  137,
  138,
  139,
  140,
  141,
  142,
  143,
  144,
  145,
  146,
  147,
  148,
  149,
  150,
  151,
  152,
  153,
  154,
  155,
  156,
  157,
  158,
  159,
  160,
  161,
  162,
  163,
  164,
  165,
  166,
  167,
  168,
  169,
  170,
  171,
  172,
  173,
  174,
  175,
  176,
  177,
  178,
  179,
  180,
  181,
  182,
  183,
  184,
  185,
  186,
  187,
  188,
  189,
  190,
  191,
  192,
  193,
  194,
  195,
  196,
  197,
  198,
  199,
  200,
  201,
  202,
  203,
  204,
  205,
  206,
  207,
  208,
  209,
  210,
  211,
  212,
  213,
  214,
  215,
  216,
  217,
  218,
  219,
  220,
  221,
  222,
  223,
  224,
  225,
  226,
  227,
  228,
  229,
  230,
  231,
  232,
  233,
  234,
  235,
  236,
  237,
  238,
  239,
  240,
  241,
  242,
  243,
  244,
  245,
  246,
  247,
  248,
  249,
  250,
  251,
  252,
  253,
  254,
  255,
  256,
  257,
  258,
  259,
  260,
  261,
  262,
  263,
  264,
  265,
  266,
  267,
  268,
  269,
  270,
  271,
  272,
  273,
  274,
  275,
  276,
  277,
  278,
  279,
  280,
  281,
  282,
  283,
  284,
  285,
  286,
  287,
  288,
  289,
  290,
  291,
  292,
  293,
  294,
  295,
  296,
  297,
  298,
  299,
  300,
  301,
  302,
  303,
  304,
  305,
  306,
  307,
  308,
  309,
  310,
  311,
  312,
  313,
  314,
  315,
  316,
  317,
  318,
  319,
  320,
  321,
  322,
  323,
  324,
  325,
  326,
  327,
  328,
  329,
  330,
  331,
  332,
  333,
  334,
  335,
  336,
  337,
  338,
  339,
  340,
  341,
  342,
  343,
  344,
  345,
  346,
  347,
  348,
  349,
  350,
  351,
  352,
  353,
  354,
  355,
  356,
  357,
  358,
  359,
  360,
  361,
  362,
  363,
  364,
  365,
  366,
  367,
  368,
  369,
  370,
  371,
  372,
  373,
  374,
  375,
  376,
  377,
  378,
  379,
  380,
  381,
  382,
  383,
  384,
  385,
  386,
  387,
  388,
  389,
  390,
  391,
  392,
  393,
  394,
  395,
  396,
  397,
  398,
  399,
  400,
  401,
  402,
  403,
  404,
  405,
  406,
  407,
  408,
  409,
  410,
  411,
  412,
  413,
  414,
  415,
  416,
  417,
  418,
  419,
  420,
  421,
  422,
  423,
  424,
  425,
  426,
  427,
  428,
  429,
  430,
  431,
  432,
  433,
  434,
  435,
  436,
  437,
  438,
  439,
  440,
  441,
  442,
  443,
  444,
  445,
  446,
  447,
  448,
  449,
  450,
  451,
  452,
  453,
  454,
  455,
  456,
  457,
  458,
  459,
  460,
  461,
  462,
  463,
  464,
  465,
  466,
  467,
  468,
  469,
  470,
  471,
  472,
  473,
  474,
  475,
  476,
  477,
  478,
  479,
  480,
  481,
  482,
  483,
  484,
  485,
  486,
  487,
  488,
  489,
  490,
  491,
  492,
  493,
  494,
  495,
  496,
  497,
  498,
  499,
  500,
  501,
  502,
  503,
  504,
  505,
  506,
  507,
  508,
  509,
  510,
  511,
  512,
  513,
  514,
  515,
  516,
  517,
  518,
  519,
  520,
  521,
  522,
  523,
  524,
  525,
  526,
  527,
  528,
  529,
  530,
  531,
  532,
  533,
  534,
  535,
  536,
  537,
  538,
  539,
  540,
  541,
  542,
  543,
  544,
  545,
  546,
  547,
  548,
  549,
  550,
  551,
  552,
  553,
  554,
  555,
  556,
  557,
  558,
  559,
  560,
  561,
  562,
  563,
  564,
  565,
  566,
  567,
  568,
  569,
  570,
  571,
  572,
  573,
  574,
  575,
  576,
  577,
  578,
  579,
  580,
  581,
  582,
  583,
  584,
  585,
  586,
  587,
  588,
  589,
  590,
  591,
  592,
  593,
  594,
  595,
  596,
  597,
  598,
  599,
  600,
  601,
  602,
  603,
  604,
  605,
  606,
  607,
  608,
  609,
  610,
  611,
  612,
  613,
  614,
  615,
  616,
  617,
  618,
  619,
  620,
  621,
  622,
  623,
  624,
  625,
  626,
  627,
  628,
  629,
  630,
  631,
  632,
  633,
  634,
  635,
  636,
  637,
  638,
  639,
  640,
  641,
  642,
  643,
  644,
  645,
  646,
  647,
  648,
  649,
  650,
  651,
  652,
  653,
  654,
  655,
  656,
  657,
  658,
  659,
  660,
  661,
  662,
  663,
  664,
  665,
  666,
  667,
  668,
  669,
  670,
  671,
  672,
  673,
  674,
  675,
  676,
  677,
  678,
  679,
  680,
  681,
  682,
  683,
  684,
  685,
  686,
  687,
  688,
  689,
  690,
  691,
  692,
  693,
  694,
  695,
  696,
  697,
  698,
  699,
  700,
  701,
  702,
  703,
  704,
  705,
  706,
  707,
  708,
  709,
  710,
  711,
  712,
  713,
  714,
  715,
  716,
  717,
  718,
  719,
  720,
  721,
  722,
  723,
  724,
  725,
  726,
  727,
  728,
  729,
  730,
  731,
  732,
  733,
  734,
  735,
  736,
  737,
  738,
  739,
  740,
  741,
  742,
  743,
  744,
  745,
  746,
  747,
  748,
  749,
  750,
  751,
  752,
  753,
  754,
  755,
  756,
  757,
  758,
  759,
  760,
  761,
  762,
  763,
  764,
  765,
  766,
  767,
  768,
  769,
  770,
  771,
  772,
  773,
  774,
  775,
  776,
  777,
  778,
  779,
  780,
  781,
  782,
  783,
  784,
  785,
  786,
  787,
  788,
  789,
  790,
  791,
  792,
  793,
  794,
  795,
  796,
  797,
  798,
  799,
  800,
  801,
  802,
  803,
  804,
  805,
  806,
  807,
  808,
  809,
  810,
  811,
  812,
  813,
  814,
  815,
  816,
  817,
  818,
  819,
  820,
  821,
  822,
  823,
  824,
  825,
  826,
  827,
  828,
  829,
  830,
  831,
  832,
  833,
  834,
  835,
  836,
  837,
  838,
  839,
  840,
  841,
  842,
  843,
  844,
  845,
  846,
  847,
  848,
  849,
  850,
  851,
  852,
  853,
  854,
  855,
  856,
  857,
  858,
  859,
  860,
  861,
  862,
  863,
  864,
  865,
  866,
  867,
  868,
  869,
  870,
  871,
  872,
  873,
  874,
  875,
  876,
  877,
  878,
  879,
  880,
  881,
  882,
  883,
  884,
  885,
  886,
  887,
  888,
  889,
  890,
  891,
  892,
  893,
  894,
  895,
  896,
  897,
  898,
  899^C ��[6nmp3> 
 ��[6nmp3> �[1G�[Jmake: *** [Makefile:32: testcli] Error 1

Support FLV format

Is it possible to support FLV format files? I am using some video sites and they are still using this format. See also https://docs.fileformat.com/video/flv/

install on the ubuntu

I try to install fq on ubuntu

go install github.com/wader/fq@latest
The 'go' command was not found, but it can be installed using:
sudo snap install go         # version 1.18.8, or
sudo apt  install golang-go  # version 2:1.18~0ubuntu2
sudo apt  install gccgo-go   # version 2:1.18~0ubuntu2
Zobacz 'snap info go' dla dodatkowych wersji.
a@zalman:~$ sudo apt  install golang-go
[sudo] hasło użytkownika a: 
Czytanie list pakietów... Gotowe
Budowanie drzewa zależności... Gotowe
Odczyt informacji o stanie... Gotowe   
The following additional packages will be installed:
  golang-1.18-go golang-1.18-src golang-src
Sugerowane pakiety:
  bzr | brz mercurial subversion
Zostaną zainstalowane następujące NOWE pakiety:
  golang-1.18-go golang-1.18-src golang-go golang-src
0 aktualizowanych, 4 nowo instalowanych, 0 usuwanych i 49 nieaktualizowanych.
Konieczne pobranie 82,2 MB archiwów.
Po tej operacji zostanie dodatkowo użyte 436 MB miejsca na dysku.
Kontynuować? [T/n] t
Pobieranie:1 http://pl.archive.ubuntu.com/ubuntu jammy/main amd64 golang-1.18-src all 1.18.1-1ubuntu1 [16,2 MB]
Pobieranie:2 http://pl.archive.ubuntu.com/ubuntu jammy/main amd64 golang-1.18-go amd64 1.18.1-1ubuntu1 [66,0 MB]                                                                                                                                                                         
Ign.:2 http://pl.archive.ubuntu.com/ubuntu jammy/main amd64 golang-1.18-go amd64 1.18.1-1ubuntu1                                                                                                                                                                                         
Pobieranie:3 http://pl.archive.ubuntu.com/ubuntu jammy/main amd64 golang-src all 2:1.18~0ubuntu2 [4 438 B]                                                                                                                                                                               
Pobieranie:4 http://pl.archive.ubuntu.com/ubuntu jammy/main amd64 golang-go amd64 2:1.18~0ubuntu2 [41,8 kB]                                                                                                                                                                              
Pobieranie:2 http://pl.archive.ubuntu.com/ubuntu jammy/main amd64 golang-1.18-go amd64 1.18.1-1ubuntu1 [66,0 MB]                                                                                                                                                                         
Pobrano 57,8 MB w 1min 32s (627 kB/s)                                                                                                                                                                                                                                                    
Wybieranie wcześniej niewybranego pakietu golang-1.18-src.
(Odczytywanie bazy danych ... 415077 plików i katalogów obecnie zainstalowanych.)
Przygotowywanie do rozpakowania pakietu .../golang-1.18-src_1.18.1-1ubuntu1_all.deb ...
Rozpakowywanie pakietu golang-1.18-src (1.18.1-1ubuntu1) ...
Wybieranie wcześniej niewybranego pakietu golang-1.18-go.
Przygotowywanie do rozpakowania pakietu .../golang-1.18-go_1.18.1-1ubuntu1_amd64.deb ...
Rozpakowywanie pakietu golang-1.18-go (1.18.1-1ubuntu1) ...
Wybieranie wcześniej niewybranego pakietu golang-src.
Przygotowywanie do rozpakowania pakietu .../golang-src_2%3a1.18~0ubuntu2_all.deb ...
Rozpakowywanie pakietu golang-src (2:1.18~0ubuntu2) ...
Wybieranie wcześniej niewybranego pakietu golang-go:amd64.
Przygotowywanie do rozpakowania pakietu .../golang-go_2%3a1.18~0ubuntu2_amd64.deb ...
Rozpakowywanie pakietu golang-go:amd64 (2:1.18~0ubuntu2) ...
Konfigurowanie pakietu golang-1.18-src (1.18.1-1ubuntu1) ...
Konfigurowanie pakietu golang-src (2:1.18~0ubuntu2) ...
Konfigurowanie pakietu golang-1.18-go (1.18.1-1ubuntu1) ...
Konfigurowanie pakietu golang-go:amd64 (2:1.18~0ubuntu2) ...
Przetwarzanie wyzwalaczy pakietu man-db (2.10.2-1)...
a@zalman:~$ go install github.com/wader/fq@latest
go: downloading github.com/wader/fq v0.0.10
go: downloading github.com/wader/readline v0.0.0-20220928125628-732951d41240
go: downloading github.com/golang/snappy v0.0.4
go: downloading golang.org/x/crypto v0.0.0-20220926161630-eccd6366d1be
go: downloading golang.org/x/text v0.3.7
go: downloading github.com/gomarkdown/markdown v0.0.0-20220905174103-7b278df48cfb
go: downloading github.com/mitchellh/copystructure v1.2.0
go: downloading github.com/wader/gojq v0.12.1-0.20220929141349-8874f5c7907c
go: downloading golang.org/x/exp v0.0.0-20220927162542-c76eaa363f9d
go: downloading github.com/BurntSushi/toml v1.2.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading golang.org/x/net v0.0.0-20220927171203-f486391704dc
go: downloading github.com/creasty/defaults v1.6.0
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading github.com/gopacket/gopacket v0.0.0-20220819214934-ee81b8c880da
go: downloading github.com/mitchellh/reflectwalk v1.0.2
go: downloading github.com/itchyny/timefmt-go v0.1.4
a@zalman:~$ fq
The 'fq' command was not found, but it can be installed using:
sudo apt install fq  # version 0.0.5-1, or
sudo apt install nq  # version 0.3.1-4
a@zalman:~$ go version
go version go1.18.1 linux/amd64

is there possible conflict wit fq https://manpages.ubuntu.com/manpages/bionic/en/man1/fq.1.html
?

Extend support of CSV with CSV Dialect

First thanks for this great work! CSV format can be troublesome because of its many dialects. If support of CSV is going to be extended, I recommend using the CSV Dialect Specification or a compatible subset of it. By now, fq supports two CSV Dialect properties with differing name and default. An alternative is CSVW dialect description (thanks xkcd #972!). Here is a comparision:

fq	default	csvddf	default	csvw	default
comma	,	delimiter	,	delimiter	,
comment	#	commentChar	not set	commentPrefix	null or # (spec is ambiguous)

Names can be adjusted by aliases, I prefer short names anyway. Remaining properties found in csvddf and csvw are:

csvddf	default	csvw	default
quoteChar	"	quoteChar	"
skipInitialSpace	false	skipInitialSpace	false
header	true	header	true
being discussed		headerRowCount	1 if header set else 0
lineTerminator	\r\n	lineTerminators	["\r\n", "\n"]
doubleQuote	true
		doubleQuote	true
escapeChar	not set
nullSequence	not set
		skipBlankRows	false
		skipColumns	0
		skipRows	0
		encoding	utf-8
		trim	true

Property doubleQuote differs in meaning between the two (csvw uses it to also set escapeChar). csvddf further has property caseSensitiveHeader (default false) but there are discussions to remove it.

problems with zip support

I tried fq on zip archives, and I had two problems so far:

• fq seems to have zip64 support, but it's not complete - at least on one file I tried it on - the eocd64 marker is shown as 'gap0' (example file: https://github.com/nih-at/libzip/blob/main/regress/zip64.zip):

# fq . zip64.zip
    |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b|0123456789abcdef0123456789ab|.{}: zip64.zip (zip)
0x00|50 4b 03 04 2d 00 00 00 00 00 4f 72 5b 40 07 a1 ea dd ff ff ff ff ff ff ff ff 01 00|PK..-.....Or[@..............|  local_files[0:1]:
0x1c|14 00 2d 01 00 10 00 02 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 61 0a         |..-....................a.   |
0x1c|                                                                           50 4b 01|                         PK.|  central_directories[0:1]:
0x38|02 1e 03 2d 00 00 00 00 00 4f 72 5b 40 07 a1 ea dd 02 00 00 00 02 00 00 00 01 00 00|...-.....Or[@...............|
0x54|00 00 00 00 00 01 00 00 00 80 11 00 00 00 00 2d                                    |...............-            |
0x54|                                                50 4b 06 06 2c 00 00 00 00 00 00 00|                PK..,.......|  gap0: raw bits
0x70|1e 03 2d 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00|..-.........................|
*   |until 0xaf.7 (76)                                                                  |                            |
0xa8|                        50 4b 05 06 00 00 00 00 01 00 01 00 2f 00 00 00 35 00 00 00|        PK........../...5...|  end_of_central_directory_record{}:
0xc4|00 00|                                                                             |..|                         |

• I wanted to try the interactive mode to look inside a zip inside a zip, and fq just hung completely, eating ~33GB RAM and 100% CPU:

fq -i . bigzero-zip.zip

using https://github.com/nih-at/libzip/blob/main/regress/bigzero-zip.zip

Unbreak on 32-bit architectures

The MakeTable() function in pkg/checksum/crc.go accepts poly int parameter (which is later being cast to uint), but called with 0xedb88320 argument which overflows it on 32 bits. Consider this trivial patch.

Add UEFI support

Just in case wondering: does it make sense to add support of UEFI files (using https://github.com/linuxboot/fiano)?

[FORMAT] Parse ITU-T T.35 data & SCTE-128 DTVCC Captions in H.264

Firstly can I state that fq absolutely rocks for the analysis of media files, and is the best parser for H.264 headers.

There are a very limited number of open source tools that go to the effort of allowing H.264 SEI headers to be easily decoded. On behalf of your user-base, many thanks for all your work!

In fact, I like fq so much, I would like to ask for the AVC & SEI parser to be enhanced so that it can decode DTVCC / SCTE-128 closed caption headers in order to index the tracks contained within. My interest is specifically H.264 files, but a generic parser could also service other /formats/mpeg/* such as H.262 picture user data and H.265 SEI data.

I appreciate that fq started with a heavy focus on media files, and thus I hope this request would be interpreted as being on topic.

At present, fq successfully decodes as far as detecting "user_data_registered_itu_t_t35" in the SEI. This is cool, and beyond that of some basic parsers.

SCTE 128 Closed Captioning

Closed captioning is important for audiences with accessibility requirements to be able to enjoy and consume content with equal access as to those without accessibility needs. Video engineers need to be able to debug closed captioning in order to service those users.

There are many standards for Closed Captioning, some of which define how to transmit the data in an MPEG stream (MPEG2 SCTE-20, ATSC A/53, H.264 SCTE-128), some of which define the encoding and decoding of the characters in the payload (EIA-608/708). The standards for ATSC A/53 closed captioning in SEI side data for H.264 share commonality with H.262/mpeg2 closed captioning in picture user data. Higher-level data-structures such as EIA-608/708 are common to all of H.262/264/265 use cases.

My primary interest is DTVCC SCTE-128 Closed Captioning in H.264, since AVC is the most widely used video codec. While SCTE-128 is a North America-centric standard, it is now commonly used worldwide in HLS & DASH protocols for the carriage of EIA-608/708. SCTE-128 DTVCC (with EIA-608) is the most common method of delivering closed captions to end-user players and devices in HLS and DASH streams.

All of the following format request request refers to freely available documentation.

• https://www.scte.org/documents/373/ANSI_SCTE-128-1-2020-1586877225672.pdf
• https://www.itu.int/pub/T-SP-T.35-2012
• https://www.atsc.org/documents/code-point-registry/

Current support in fq

Here is the fq command that I use to inspect a sample file containing DTVCC closed captioning.

At present, fq is capable of identifying that the SEI side data contains a T.35 Country Code.

$ ffmpeg -loglevel warning -hide_banner -i './testsrc.with608captions.mp4' -map '0:v:0' -codec:v 'copy' -bsf:v 'h264_mp4toannexb' -f 'h264' 'pipe:1' | fq --decode avc_annexb '[ .[] | objects | select(.nal_unit_type=="sei") | select(.sei.payload_type=="user_data_registered_itu_t_t35") | d ]'

      |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d|0123456789abcd|.[121]{}: nalu (avc_nalu)
      |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d|0123456789abcd|  sei{}: (avc_sei)
  0x00|04                                       |.             |    payload_type: "user_data_registered_itu_t_t35" (4)
  0x00|   68                                    | h            |    payload_size: 104
  0x00|      b5 00 31 47 41 39 34 03 5f ff fc 94|  ..1GA94._...|    data: raw bits
  0x00|ae fc 94 20 fc 91 40 fc 54 68 fc e9 73 fc|... [email protected].|
  *   |until 0x69.7 (104)                       |              |
  0x06|                        04 5c b5 00 31 47|        .\..1G|    rbsp_trailing_bits: raw bits
  0x07|41 39 34 03 5b ff fc 94 20 fc e9 6e fc 20|A94.[... ..n. |
  *   |until 0xc8.7 (end) (95)                  |              |
0x3c52|         06                              |   .          |  forbidden_zero_bit: false
0x3c52|         06                              |   .          |  nal_ref_idc: 0
0x3c52|         06                              |   .          |  nal_unit_type: "sei" (6) (Supplemental enhancement information)
0x3c52|            04 68 b5 00 31 47 41 39 34 03|    .h..1GA94.|  data: raw bits
0x3c60|5f ff fc 94 ae fc 94 20 fc 91 40 fc 54 68|_...... [email protected]|
*     |until 0x3d1e.7 (201)                     |              |

I would like to use fq to be able to dig deeper into the ITU-T.35 header, specifically for the purpose of inspecting SCTE-128 DTVCC closed captions, which in media terms is a very common type of closed captioning. There are very few closed caption tools that provide raw debug analysis - most tools (such as ccextractor or caption inspector) go the extra step and perform a conversion, which is can often change the measured result. I like fq because it attempts to decode rather than interpret or convert data.

The closed caption rawdata starts as user_data_registered_itu_t_t35, which fq already knows about. Within the user_data_registered_itu_t_t35 there is rawdata....

b5 00 31 47 41 39 34 03 5f ff fc 94
ae fc 94 20 fc 91 40 fc 54 68 fc e9 73

SCTE-128 DTVCC Format (Summary)

From: https://www.scte.org/documents/373/ANSI_SCTE-128-1-2020-1586877225672.pdf

itu_t_35_country_code

Within the user_data_registered_itu_t_t35, the ITU publishes a full lookup table, with country codes. Here is a link with all the current T.35 country codes. Standard is freely and publicly available and is maintained by ITU.

If a kind developer were to add an ITU T.35 Country Code Parser, I have extracted the values from the ITU documentation and formatted in what would be a useful way, in the hope of making a developer's task easier.

t35_country_codes.txt

The T.35 country code for DVTCC closed captioning is always B5 (since it is a standard derived from ATSC in the United States). A more comprehensive, generic T.35 parser in fq could include all country codes from the above pre-formatted file.

b5 00 31 47 41 39 34 03 5f ff fc 94
^^ itu_t_35_country_code
ae fc 94 20 fc 91 40 fc 54 68 fc e9 73

Where itu_t_35_country_code of value {0xb5: "United States"}

itu_t_35_provider_type

Each countries' standards body gets to define and scope their provider_type. DTVCC closed captioning is a US-standard. DTVCC 608/708 closed captions are used around the world in protocols such as HLS).

The provider_type for SCTE-128 Closed Captions is always 0x0031.

b5 00 31 47 41 39 34 03 5f ff fc 94
   ^^ ^^ itu_t_35_provider_type
ae fc 94 20 fc 91 40 fc 54 68 fc e9 73

Where itu_t_35_provider_type is {0x0031: SCTE-128 DTVCC}

scte_128_user_identifier

In SCTE-128, the following user identifiers are defined:

{
  0x47413934:          ATSC data ("GA94"),
  0x44544731:          Active Format Description (AFD), (“DTG1”)
}

From a selfish perspective, I am interested in the ATSC data, however the same SCTE-128 document does also cover Active Format Description signalling which may prove valuable to others (in case it also piques your interest or was on your to-do-sometime-in-the-future list).

b5 00 31 47 41 39 34 03 5f ff fc 94
         ^^ ^^ ^^ ^^ scte_128_user_identifier
ae fc 94 20 fc 91 40 fc 54 68 fc e9 73

Where scte_128_user_identifier is {0x31474139: "ATSC data (GA94)"}

ATSC1_data ("GA94")

Within the ATSC1_data header, there are registered codes. The one which is interesting from a Closed Caption perspective is 0x03 cc_data.

This information is also published and maintained by ATSC in the last tab of this freely accessible, public Code Point Registry spreadsheet...

b5 00 31 47 41 39 34 03 5f ff fc 94
                     ^^ atsc1_data
ae fc 94 20 fc 91 40 fc 54 68 fc e9 73

Where atsc1_data is {0x03: "ccdata"}.

ccdata() = EIA-708

b5 00 31 47 41 39 34 03 5f ff fc 94
                        ^^ ^^ ^^ ^^ ccdata() et cetera...
ae fc 94 20 fc 91 40 fc 54 68 fc e9 73

The ccdata() is defined in EIA-708, and while I would love to extend this format request to include EIA-708 headers, enabling fq to list headers that relate to the captions tracks (608 CC1/2/3/4) and (708 SERVICE1-6), I appreciate that this post is already long enough. In simple terms the ccdata() payload contains headers which list the CC1/2/3/4 tracks header and payload (aka 608 compatibility mode), followed by the headers and payload for full 708 data. The EIA-708 spec has traditionally been pay-to-play and not available for the general public, but has now been made available free-of-charge and "Available to Everyone" from the CTA (registration required).

https://shop.cta.tech/products/digital-television-dtv-closed-captioning

The EIA-708 standard defines the methods to decode ccdata() headers.

Please do not interpret this as a request for the decoding of closed captioning to human readable text. I consider fq to be a header inspector and thus indexing and listing the available closed captioning tracks in the SEI would be of great benefit to me and other video technicians, but fq does not attempt to be a video decoder, so therefore it need not attempt to decode the actual encoded text in the payload.

While the above list may be long, even fq was to implement the lower-level headers without ccdata(), it would be very beneficial.

I am not a developer, and while I can just about understand what https://github.com/wader/fq/blob/master/format/mpeg/avc_annexb.go and https://github.com/wader/fq/blob/master/format/mpeg/avc_sei.go are doing, I am not capable of writing a format interpreter in go, that is beyond my skillset. But I hope that you will appreciate that I have tried to present data such as t35_country_codes.txt in a data structure which would be compatible with the current parsers.

I hope you find the above easy to read. i have put a lot of care and attention into making the github markup clear and I have referenced any standards in the format request.

I do have a repository of some (copyrighted) media streams for the above from US broadcasters, which I could share privately, in the interests of legitimate research, however I have not attempted to attach these to your github repo to as to avoid you being flagged.

I am including my version of fq out of politeness since that is requested on all new tickets, although it is unlikely to be relevant for a format request.

# Version
fq -v
0.1.0 (darwin amd64)

# Installed by
brew install fq

Even if this format request is not of interest, thanks for developing such a great tool and taking the time to read this far!

Add gentoo install instructions

What is the proper way install fq on gentoo? emerge dev-util/fq? package is available in "stable" etc now?

Better documenation

Feedback and ideas:

• Don't use name/arity notation for functions unless it needed. Use name, name(arga; argb) instead
• Add some basic functions/syntax help to --help
• Inline documentation in go and jq source, extract it and generate documentation. Maybe use https://github.com/fadado/JBOL/blob/master/doc/JQ-Distilled.md#type-signatures and then render more user friendly versions?
• More examples

Build and publish container images

I'm running fq in a container by building it based on the Dockerfile locally.
It would be neat to automatically publish container images for releases and something I'm tempted to work on.
This however raises a few questions I would like feedback on:

To where should the container images be published? Simplest way is probably Github packages, as secrets management and similar is automagically handled.

Where in the CI pipeline should images be built? GoReleaser has support to build and publish images, but it could be done using other Github actions as well.

Which architectures should the images be built for? Considering that the "runtime" image is based on scratch, it should only be a matter of support for cross-compiling the Go binaries.

OOM when working with 1 GB PostgreSQL heap (table) file

What version are you using (fq -v)?

$ fq -v
0.0.9 (linux amd64)

How was fq installed?

fq is build from source.

My branch:
https://github.com/pnsafonov/fq/tree/postgres

Can you reproduce the problem using the latest release or master branch?

I can reproduce problem on my branch, where PostgreSQL parsers are implemented.

1 GB file:
https://github.com/pnsafonov/fq_testdata_postgres14
https://github.com/pnsafonov/fq_testdata_postgres14/raw/master/16397

What did you do?

I am trying to parse a heap (relation, table) file of maximum size (1 GB for PostgeSQL).
fq consumes 90-100 times memory than file size. For 80 mb file fq requires 7,5 GB RAM.

time fq -d pgheap -o flavour=postgres14 ".Pages[0].PageHeaderData.pd_linp[0, 1, 2, -1] | tovalue" 16397
Killed

real    0m50.794s
user    1m11.962s
sys     0m8.994s

Kenel messages:

sudo dmesg | tail -2
[193541.830725] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-1.scope,task=fq,pid=454783,uid=1000
[193541.830748] Out of memory: Killed process 454783 (fq) total-vm:31508780kB, anon-rss:26629332kB, file-rss:272kB, shmem-rss:0kB, UID:1000 pgtables:58860kB oom_score_adj:0

1 GB File:

ls -alh 16397
-rw-r----- 1 pavel pavel 1.0G Aug 31 08:38 16397

Memory profiler results:

go tool pprof mem.prof
File: postgres.test
Type: alloc_space
Time: Aug 30, 2022 at 6:16pm (MSK)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top 40
Showing nodes accounting for 31041.34MB, 97.35% of 31886.71MB total
Dropped 270 nodes (cum <= 159.43MB)
Showing top 40 nodes out of 87
      flat  flat%   sum%        cum   cum%
16563.23MB 51.94% 51.94% 17983.80MB 56.40%  github.com/wader/fq/pkg/decode.(*D).TryFieldScalarFn.func1
 5007.15MB 15.70% 67.65%  5007.15MB 15.70%  github.com/wader/fq/pkg/decode.(*D).fieldDecoder
 2168.92MB  6.80% 74.45%  3948.79MB 12.38%  github.com/wader/fq/pkg/decode.(*D).FillGaps
 2109.38MB  6.62% 81.06%  2109.38MB  6.62%  github.com/wader/fq/pkg/decode.(*D).AddChild (inline)
 1327.83MB  4.16% 85.23%  1327.83MB  4.16%  github.com/wader/fq/pkg/ranges.Gaps
  872.54MB  2.74% 87.96% 26209.93MB 82.20%  github.com/wader/fq/pkg/decode.(*D).FieldStruct
  587.02MB  1.84% 89.81%   587.02MB  1.84%  internal/reflectlite.Swapper
  513.59MB  1.61% 91.42%   513.59MB  1.61%  fmt.Sprintf
  425.01MB  1.33% 92.75%  1012.03MB  3.17%  github.com/wader/fq/pkg/decode.(*Value).postProcess.func1
  324.51MB  1.02% 93.77%   324.51MB  1.02%  github.com/wader/fq/pkg/bitio.NewSectionReader (inline)
     288MB   0.9% 94.67%   372.51MB  1.17%  github.com/wader/fq/pkg/decode.(*D).TryFieldScalarU16.func1
  236.02MB  0.74% 95.41%   696.61MB  2.18%  github.com/wader/fq/pkg/scalar.RawSym
  174.50MB  0.55% 95.96%  9928.56MB 31.14%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeInfomask
  104.50MB  0.33% 96.29%  2701.76MB  8.47%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeTChoice.func1
      61MB  0.19% 96.48%  1440.14MB  4.52%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeInfomask2
   56.53MB  0.18% 96.65%  1510.17MB  4.74%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.DecodeItemIds.func1
      48MB  0.15% 96.80%   311.03MB  0.98%  github.com/wader/gojq.(*compiler).compileCallInternal
      48MB  0.15% 96.95%   460.59MB  1.44%  github.com/wader/fq/pkg/scalar.glob..func10.1
   46.51MB  0.15% 97.10%   410.05MB  1.29%  github.com/wader/gojq.(*compiler).compileFuncDef
      24MB 0.075% 97.18%   369.99MB  1.16%  github.com/wader/gojq.(*compiler).compileTerm
   15.05MB 0.047% 97.22% 31404.72MB 98.49%  github.com/wader/gojq.(*env).Next
       7MB 0.022% 97.25%   169.67MB  0.53%  github.com/wader/gojq.(*compiler).compileCall
       7MB 0.022% 97.27%   186.53MB  0.58%  github.com/wader/gojq.(*compiler).compileIf
       5MB 0.016% 97.28%   358.54MB  1.12%  github.com/wader/gojq.(*compiler).compile
    4.51MB 0.014% 97.30%   222.25MB   0.7%  github.com/wader/fq/pkg/interp.(*Interp).Eval.func2
    4.50MB 0.014% 97.31%   375.04MB  1.18%  github.com/wader/gojq.(*compiler).compileQuery
    3.50MB 0.011% 97.32%   205.43MB  0.64%  github.com/wader/gojq.(*compiler).compileFunc
       3MB 0.0094% 97.33% 26212.93MB 82.21%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeHeapPages
       3MB 0.0094% 97.34% 26212.93MB 82.21%  github.com/wader/fq/pkg/decode.(*D).FieldArray
    1.50MB 0.0047% 97.35% 26198.29MB 82.16%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeHeapPage
    0.50MB 0.0016% 97.35%   545.91MB  1.71%  github.com/wader/gojq.Compile
    0.50MB 0.0016% 97.35%   164.87MB  0.52%  github.com/wader/gojq.(*compiler).compileCallPc
         0     0% 97.35% 26212.93MB 82.21%  github.com/wader/fq/format/postgres.decodePgheap
         0     0% 97.35% 26080.73MB 81.79%  github.com/wader/fq/format/postgres/flavours/pgproee14.DecodeHeap (inline)
         0     0% 97.35%  2322.01MB  7.28%  github.com/wader/fq/format/postgres/flavours/pgproee14/ee14.DecodePageHeaderData
         0     0% 97.35% 26212.93MB 82.21%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.DecodeHeap
         0     0% 97.35%  2235.26MB  7.01%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.DecodeItemIds
         0     0% 97.35%  5454.55MB 17.11%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeTChoice
         0     0% 97.35%   887.09MB  2.78%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeTChoice.func1.1
         0     0% 97.35%  1419.14MB  4.45%  github.com/wader/fq/format/postgres/flavours/postgres14/common14.decodeTChoice.func2

I check with disassembler. TryFieldScalarFn.func1 is unnamed lambda in TryFieldScalarFn.

Editing a binary

Hi,

With jq I'm used to being able to edit a json document in-situ:

$ echo '{"foo":0}' | jq '.foo = 42'
{
  "foo": 42
}

I figure you should be able to do the same sort of thing with fq with binaries, but if so, it's not documented how to do it exactly.

For example:

fq '.header.entry=0' a.out

... at the moment this prints out a json representation of the elf. The closest thing I saw in the documentation was the "Add query parameter to URL" in the screenshot at the beginning of the readme, but of course there is no toelf. tobytes reports: fq: value can't be a binary.

So, is there a way to edit fields and then dump it out in the original format, or is this not possible currently? Thanks!