osquery / osquery-go

Go bindings for osquery

osquery

osquery is a SQL powered operating system instrumentation, monitoring, and analytics framework.
Available for Linux, macOS, and Windows.

Information and resources

• Homepage: osquery.io
• Downloads: osquery.io/downloads
• Documentation: ReadTheDocs
• Stack Overflow: Stack Overflow questions
• Table Schema: osquery.io/schema
• Query Packs: osquery.io/packs
• Slack: Browse the archives or Join the conversation
• Build Status:
• CII Best Practices:

What is osquery?

osquery exposes an operating system as a high-performance relational database. This allows you to write SQL-based queries to explore operating system data. With osquery, SQL tables represent abstract concepts such as running processes, loaded kernel modules, open network connections, browser plugins, hardware events or file hashes.

SQL tables are implemented via a simple plugin and extensions API. A variety of tables already exist and more are being written: https://osquery.io/schema. To best understand the expressiveness that is afforded to you by osquery, consider the following SQL queries:

List the users:

SELECT * FROM users;

Check the processes that have a deleted executable:

SELECT * FROM processes WHERE on_disk = 0;

Get the process name, port, and PID, for processes listening on all interfaces:

SELECT DISTINCT processes.name, listening_ports.port, processes.pid
  FROM listening_ports JOIN processes USING (pid)
  WHERE listening_ports.address = '0.0.0.0';

Find every macOS LaunchDaemon that launches an executable and keeps it running:

SELECT name, program || program_arguments AS executable
  FROM launchd
  WHERE (run_at_load = 1 AND keep_alive = 1)
  AND (program != '' OR program_arguments != '');

Check for ARP anomalies from the host's perspective:

SELECT address, mac, COUNT(mac) AS mac_count
  FROM arp_cache GROUP BY mac
  HAVING count(mac) > 1;

Alternatively, you could also use a SQL sub-query to accomplish the same result:

SELECT address, mac, mac_count
  FROM
    (SELECT address, mac, COUNT(mac) AS mac_count FROM arp_cache GROUP BY mac)
  WHERE mac_count > 1;

These queries can be:

• performed on an ad-hoc basis to explore operating system state using the osqueryi shell
• executed via a scheduler to monitor operating system state across a set of hosts
• launched from custom applications using osquery Thrift APIs

Download & Install

To download the latest stable builds and for repository information and installation instructions visit https://osquery.io/downloads.

We use a simple numbered versioning scheme X.Y.Z, where X is a major version, Y is a minor, and Z is a patch. We plan minor releases roughly every two months. These releases are tracked on our Milestones page. A patch release is used when there are unforeseen bugs with our minor release and we need to quickly patch. A rare 'revision' release might be used if we need to change build configurations.

Major, minor, and patch releases are tagged on GitHub and can be viewed on the Releases page. We open a new Release Checklist issue when we prepare a minor release. If you are interested in the status of a release, please find the corresponding checklist issue, and note that the issue will be marked closed when we are finished the checklist. We consider a release 'in testing' during the period of hosting new downloads on our website and adding them to our hosted repositories. We will mark the release as 'stable' on GitHub when enough testing has occurred, this usually takes two weeks.

Build from source

Building osquery from source is encouraged! Check out our build guide. Also check out our contributing guide and join the community on Slack.

Osquery fleet managers

There are many osquery fleet managers out there. The osquery project does not endorse, recommend, or test these. They are provided as a starting point

Project	License
Fleet	Open Core
Kolide	Commercial
OSCTRL	Open Source
Zentral	Open Source

License

By contributing to osquery you agree that your contributions will be licensed as defined on the LICENSE file.

Vulnerabilities

We keep track of security announcements in our tagged version release notes on GitHub. We aggregate these into SECURITY.md too.

Learn more

The osquery documentation is available online. Documentation for older releases can be found by version number, as well.

If you're interested in learning more about osquery read the launch blog post for background on the project, visit the users guide.

Development and usage discussion is happening in the osquery Slack, grab an invite here!

osquery-go's Issues

Build C++ extentions

Sorry to use this forum but does anyone have document detailing compiling C++ extension (all the documents provided by OSQuery not working) . Any help will be much appreciated.

Failure to run custom logger example since eb3599e

Hey,

Today I updated the dependencies of my local osquery plugin that makes use of osquery-go and can no longer start a custom logger. As soon as I try and Run() a NewExtensionManagerServer() instance, I get a panic as follows:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x12a016a]

goroutine 35 [running]:
github.com/xxxxt/vendor/github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).RegisterExtension(0xc4201bc060, 0x150c400, 0xc4200be010, 0xc4201c2080, 0xc4201a8090, 0x1471de0, 0x1, 0xc4201c2080)
	/Users/xxxx/Documents/git/gopath/src/github.com/xxxxt/vendor/github.com/kolide/osquery-go/gen/osquery/osquery.go:1853 +0x8a
github.com/xxxxt/vendor/github.com/kolide/osquery-go.(*ExtensionManagerClient).RegisterExtension(0xc4201bc080, 0xc4201c2080, 0xc4201a8090, 0x0, 0x0, 0x0)
	/Users/xxxx/Documents/git/gopath/src/github.com/xxxxt/vendor/github.com/kolide/osquery-go/client.go:73 +0x5f
github.com/xxxxt/vendor/github.com/kolide/osquery-go.(*ExtensionManagerServer).Start.func1(0xc42001e080, 0xc4201d0f78, 0x0, 0x0)
	/Users/xxxx/Documents/git/gopath/src/github.com/xxxxt/vendor/github.com/kolide/osquery-go/server.go:143 +0xfe
github.com/xxxxt/vendor/github.com/kolide/osquery-go.(*ExtensionManagerServer).Start(0xc42001e080, 0x0, 0x0)
	/Users/xxxx/Documents/git/gopath/src/github.com/xxxxt/vendor/github.com/kolide/osquery-go/server.go:172 +0x41
github.com/xxxxt/vendor/github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func1(0xc4201c0060, 0xc42001e080)
	/Users/xxxx/Documents/git/gopath/src/github.com/xxxxt/vendor/github.com/kolide/osquery-go/server.go:186 +0x2b
created by github.com/xxxxt/vendor/github.com/kolide/osquery-go.(*ExtensionManagerServer).Run
	/Users/xxxx/Documents/git/gopath/src/github.com/xxxxt/vendor/github.com/kolide/osquery-go/server.go:185 +0x66

The example here currently shows the same behaviour. Could it be that there is a change in how the logger should be registered?

wait until socket is available

From @theopolis in slack:

I’m seriousness, yes, there is lag between socket creation and socket availability. Your thrift wrapper should take care of this, C++ and python do it by waiting up to —timeout seconds, and retrying every 200ms

``osqueryd`` claims shutdown request failed when exiting from SIGINT

I've written a basic osquery table extension using osquery-go. I confirmed that osquery{d,i} can load the extension and query my table as intended. However, when I try to quit osquery with ctrl+C, I see from osquery:

I0729 08:43:05.621091 134004736 extensions.cpp:220] Extension UUID 7951 shutdown request failed

I run osqueryd with:

sudo ./osqueryd --pidfile=osquery.pid --database_path=osquery.db --verbose=true --config_plugin=filesystem --config_path=osquery_test.conf --logger_plugin=filesystem --allow_unsafe --extensions_autoload=extensions.load

Is this expected with the default behavior of Shutdown()? Should I be implementing this signal handling logic myself? I should clarify that the extension process does quit, but I am wondering if the error is avoidable.

quitting osqueryi with an extension running freezes both osqueryi and the extension

Repro:

1. start osqueryi with

sudo osqueryi --nodisable_extensions

2. run a table extension
3. use CTRL+C to quit osqueryi

Both osqueryi and the extension become unresponsive. Possible bug with my handling of signal in server.Run() ?
Do we need to trap a specific signal?

quiting the extension while osqueryi is still running works fine.

`TestShutdownDeadlock` sporadically fails

git log -n 1 --oneline
d6f325f (HEAD -> master, origin/master, origin/HEAD) Include messages in distributed results (#115)

go test -v -run TestShutdownDeadlock 2>&1 | tee out.txt

=== RUN   TestShutdownDeadlock
=== RUN   TestShutdownDeadlock/#00
=== PAUSE TestShutdownDeadlock/#00
=== RUN   TestShutdownDeadlock/#01
=== PAUSE TestShutdownDeadlock/#01
=== RUN   TestShutdownDeadlock/#02
=== PAUSE TestShutdownDeadlock/#02
=== RUN   TestShutdownDeadlock/#03
=== PAUSE TestShutdownDeadlock/#03
=== RUN   TestShutdownDeadlock/#04
=== PAUSE TestShutdownDeadlock/#04
=== RUN   TestShutdownDeadlock/#05
=== PAUSE TestShutdownDeadlock/#05
=== RUN   TestShutdownDeadlock/#06
=== PAUSE TestShutdownDeadlock/#06
=== RUN   TestShutdownDeadlock/#07
=== PAUSE TestShutdownDeadlock/#07
=== RUN   TestShutdownDeadlock/#08
=== PAUSE TestShutdownDeadlock/#08
=== RUN   TestShutdownDeadlock/#09
=== PAUSE TestShutdownDeadlock/#09
=== RUN   TestShutdownDeadlock/#10
=== PAUSE TestShutdownDeadlock/#10
=== RUN   TestShutdownDeadlock/#11
=== PAUSE TestShutdownDeadlock/#11
=== RUN   TestShutdownDeadlock/#12
=== PAUSE TestShutdownDeadlock/#12
=== RUN   TestShutdownDeadlock/#13
=== PAUSE TestShutdownDeadlock/#13
=== RUN   TestShutdownDeadlock/#14
=== PAUSE TestShutdownDeadlock/#14
=== RUN   TestShutdownDeadlock/#15
=== PAUSE TestShutdownDeadlock/#15
=== RUN   TestShutdownDeadlock/#16
=== PAUSE TestShutdownDeadlock/#16
=== RUN   TestShutdownDeadlock/#17
=== PAUSE TestShutdownDeadlock/#17
=== RUN   TestShutdownDeadlock/#18
=== PAUSE TestShutdownDeadlock/#18
=== RUN   TestShutdownDeadlock/#19
=== PAUSE TestShutdownDeadlock/#19
=== CONT  TestShutdownDeadlock/#00
=== CONT  TestShutdownDeadlock/#10
=== CONT  TestShutdownDeadlock/#15
=== CONT  TestShutdownDeadlock/#13
=== CONT  TestShutdownDeadlock/#05
=== CONT  TestShutdownDeadlock/#08
=== CONT  TestShutdownDeadlock/#14
=== CONT  TestShutdownDeadlock/#03
=== CONT  TestShutdownDeadlock/#02
=== CONT  TestShutdownDeadlock/#01
=== CONT  TestShutdownDeadlock/#04
=== CONT  TestShutdownDeadlock/#18
=== CONT  TestShutdownDeadlock/#19
=== CONT  TestShutdownDeadlock/#09
=== CONT  TestShutdownDeadlock/#17
=== CONT  TestShutdownDeadlock/#07
=== CONT  TestShutdownDeadlock/#12
=== CONT  TestShutdownDeadlock/#16
=== CONT  TestShutdownDeadlock/#11
=== CONT  TestShutdownDeadlock/#06
=== NAME  TestShutdownDeadlock/#03
    server_test.go:175: hung on shutdown
=== NAME  TestShutdownDeadlock/#09
    server_test.go:175: hung on shutdown
--- FAIL: TestShutdownDeadlock (0.00s)
    --- PASS: TestShutdownDeadlock/#00 (0.02s)
    --- PASS: TestShutdownDeadlock/#15 (0.02s)
    --- PASS: TestShutdownDeadlock/#05 (0.02s)
    --- PASS: TestShutdownDeadlock/#14 (0.02s)
    --- PASS: TestShutdownDeadlock/#08 (0.02s)
    --- PASS: TestShutdownDeadlock/#10 (0.02s)
    --- PASS: TestShutdownDeadlock/#02 (0.01s)
    --- PASS: TestShutdownDeadlock/#04 (0.01s)
    --- PASS: TestShutdownDeadlock/#01 (0.01s)
    --- PASS: TestShutdownDeadlock/#18 (0.01s)
    --- PASS: TestShutdownDeadlock/#12 (0.01s)
    --- PASS: TestShutdownDeadlock/#07 (0.01s)
    --- PASS: TestShutdownDeadlock/#17 (0.01s)
    --- PASS: TestShutdownDeadlock/#06 (0.01s)
    --- PASS: TestShutdownDeadlock/#11 (0.01s)
    --- PASS: TestShutdownDeadlock/#13 (0.52s)
    --- PASS: TestShutdownDeadlock/#19 (0.51s)
    --- PASS: TestShutdownDeadlock/#16 (0.51s)
    --- FAIL: TestShutdownDeadlock/#03 (5.02s)
    --- FAIL: TestShutdownDeadlock/#09 (5.01s)
FAIL
exit status 1
FAIL	github.com/osquery/osquery-go	5.282s

Update docs for new command line args

hi friends! I noticed that osqueryi now passes the socket in the format --socket [...] instead of making it the first arg.

my quick hack was

	var socketPath string
	for index, flag := range os.Args {
		if flag == "--socket" {
			socketPath = os.Args[index+1]
		}
	}

though there should be bounds checking, etc.

just wanted to let you guys know to update the README!

Implement TPipe transport in Go

Right now, the osqueryd binary uses the TPipe thrift transport class to communicate between the daemon and extensions (on Windows only). In order for extensions to work in Go, we need to the Go thrift instance to use a compatible TPipe implementation. A TPipe implementation doesn't currently exist in Go, so we will have to create it. Ideally, once it works, we will upstream it as well.

Relevant Discussions

https://osquery.slack.com/archives/C0FHNQ2N6/p1496681569060902
https://kolide.slack.com/archives/C1WS15WAD/p1497308886933812

not enough arguments in call to iprot. [...]

go version go1.15.6 darwin/amd64

Running the default example in

https://github.com/osquery/osquery-go/blob/master/examples/query/main.go

results in compile errors :

# github.com/kolide/osquery-go/gen/osquery
../src/github.com/kolide/osquery-go/gen/osquery/osquery.go:134:37: not enough arguments in call to iprot.ReadStructBegin
	have ()
	want (context.Context)
../src/github.com/kolide/osquery-go/gen/osquery/osquery.go:140:57: not enough arguments in call to iprot.ReadFieldBegin
	have ()
	want (context.Context)
../src/github.com/kolide/osquery-go/gen/osquery/osquery.go:152:29: not enough arguments in call to iprot.Skip
	have (thrift.TType)
	want (context.Context, thrift.TType)
../src/github.com/kolide/osquery-go/gen/osquery/osquery.go:181:33: not enough arguments in call to iprot.ReadFieldEnd
	have ()
	want (context.Context)
../src/github.com/kolide/osquery-go/gen/osquery/osquery.go:185:32: not enough arguments in call to iprot.ReadStructEnd
	have ()
	want (context.Context)
../src/github.com/kolide/osquery-go/gen/osquery/osquery.go:192:32: not enough arguments in call to iprot.ReadString
	have ()
	want (context.Context)
../src/github.com/kolide/osquery-go/gen/osquery/osquery.go:201:32: not enough arguments in call to iprot.ReadString
	have ()
	want (context.Context)
../src/github.com/kolide/osquery-go/gen/osquery/osquery.go:210:32: not enough arguments in call to iprot.ReadString
	have ()
	want (context.Context)
../src/github.com/kolide/osquery-go/gen/osquery/osquery.go:219:35: not enough arguments in call to oprot.WriteStructBegin
	have (string)
	want (context.Context, string)
../src/github.com/kolide/osquery-go/gen/osquery/osquery.go:226:33: not enough arguments in call to oprot.WriteFieldStop
	have ()
	want (context.Context)
../src/github.com/kolide/osquery-go/gen/osquery/osquery.go:226:33: too many errors

Compilation finished with exit code 2

Is this expected to work out of the box, or is there any manual Thrift work that needs to be done?

New client not gracefully closed on error if it has grabbed the pipe/socket

I've seen this issue occur on Windows when the osquery process is terminated. The osquery-go end of the named pipe is not released and any attempts to re-establish communication begins to fail with error due to the pipe not being available. This behaviour continues unless we Shutdown() the extension manager. This will not be necessary if we call serverClient.Close() when error is encountered due to osquery process being killed.

Race condition when running example

While trying to track down a memory leak with my extension, I found there is a race condition.

When running the example and using the -race flag with go run, there is a race condition

C:\Users..\go\src\test\osquery>go run -race main.go \.\pipe\osquery.em

WARNING: DATA RACE
Read at 0x00c0000c6960 by goroutine 10:
git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:82 +0x59
github.com/kolide/osquery-go/gen/osquery.(*ExtensionClient).Ping()
C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1022 +0xfd
github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).Ping()
:1 +0x7c
github.com/kolide/osquery-go.(*ExtensionManagerClient).Ping()
C:/Users/../osquery-go.(*ExtensionManagerServer).Run.func2()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:194 +0x84

Previous write at 0x00c0000c6960 by goroutine 9:
git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:82 +0x70
github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).RegisterExtension()
C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1850 +0x1d5
github.com/kolide/osquery-go.(*ExtensionManagerClient).RegisterExtension()
C:/Users/../go/src/github.com/kolide/osquery-go/client.go:73 +0xa4
github.com/kolide/osquery-go.(*ExtensionManagerServer).Start.func1()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:143 +0x1d1
github.com/kolide/osquery-go.(*ExtensionManagerServer).Start()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:172 +0x55
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func1()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:186 +0x3f

Goroutine 10 (running) created at:
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:190 +0xa6
main.main()
C:/Users/../go/src/test/osquery/main.go:23 +0x30c

Goroutine 9 (running) created at:
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:185 +0x7a
main.main()
C:/Users/../go/src/test/osquery/main.go:23 +0x30c

==================
WARNING: DATA RACE
Read at 0x00c0000d0095 by goroutine 10:
git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteI32()
c:/go/src/encoding/binary/binary.go:116 +0x4f
git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteMessageBegin()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/binary_protocol.go:78 +0x6f
git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Send()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:37 +0x258
git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:85 +0x11d
github.com/kolide/osquery-go/gen/osquery.(*ExtensionClient).Ping()
C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1022 +0xfd
github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).Ping()
:1 +0x7c
github.com/kolide/osquery-go.(*ExtensionManagerClient).Ping()
C:/Users/../go/src/github.com/kolide/osquery-go/client.go:58 +0x8d
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func2()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:194 +0x84

Previous write at 0x00c0000d0095 by goroutine 9:
git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteI32()
c:/go/src/encoding/binary/binary.go:120 +0xbb
git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteMapBegin()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/binary_protocol.go:142 +0xc9
github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerRegisterExtensionArgs).writeField2()
C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2746 +0xa84
github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerRegisterExtensionArgs).Write()
C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2717 +0x241
git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Send()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:40 +0x29f
git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:85 +0x11d
github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).RegisterExtension()
C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1850 +0x1d5
github.com/kolide/osquery-go.(*ExtensionManagerClient).RegisterExtension()
C:/Users/../go/src/github.com/kolide/osquery-go/client.go:73 +0xa4
github.com/kolide/osquery-go.(*ExtensionManagerServer).Start.func1()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:143 +0x1d1
github.com/kolide/osquery-go.(*ExtensionManagerServer).Start()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:172 +0x55
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func1()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:186 +0x3f

Goroutine 10 (running) created at:
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:190 +0xa6
main.main()
C:/Users/../go/src/test/osquery/main.go:23 +0x30c

Goroutine 9 (running) created at:
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:185 +0x7a
main.main()
C:/Users/../go/src/test/osquery/main.go:23 +0x30c

==================
WARNING: DATA RACE
Write at 0x00c0000d0092 by goroutine 10:
git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteI32()
c:/go/src/encoding/binary/binary.go:117 +0x66
git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteMessageBegin()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/binary_protocol.go:78 +0x6f
git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Send()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:37 +0x258
git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:85 +0x11d
github.com/kolide/osquery-go/gen/osquery.(*ExtensionClient).Ping()
C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1022 +0xfd
github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).Ping()
:1 +0x7c
github.com/kolide/osquery-go.(*ExtensionManagerClient).Ping()
C:/Users/../go/src/github.com/kolide/osquery-go/client.go:58 +0x8d
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func2()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:194 +0x84

Previous write at 0x00c0000d0092 by goroutine 9:
git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteI32()
c:/go/src/encoding/binary/binary.go:117 +0x66
git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteMapBegin()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/binary_protocol.go:142 +0xc9
github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerRegisterExtensionArgs).writeField2()
C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2746 +0xa84
github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerRegisterExtensionArgs).Write()
C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2717 +0x241
git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Send()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:40 +0x29f
git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:85 +0x11d
github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).RegisterExtension()
C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1850 +0x1d5
github.com/kolide/osquery-go.(*ExtensionManagerClient).RegisterExtension()
C:/Users/../go/src/github.com/kolide/osquery-go/client.go:73 +0xa4
github.com/kolide/osquery-go.(*ExtensionManagerServer).Start.func1()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:143 +0x1d1
github.com/kolide/osquery-go.(*ExtensionManagerServer).Start()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:172 +0x55
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func1()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:186 +0x3f

Goroutine 10 (running) created at:
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:190 +0xa6
main.main()
C:/Users/../go/src/test/osquery/main.go:23 +0x30c

Goroutine 9 (running) created at:
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:185 +0x7a
main.main()
C:/Users/../go/src/test/osquery/main.go:23 +0x30c

==================
WARNING: DATA RACE
Write at 0x00c0000d0094 by goroutine 10:
git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteI32()
c:/go/src/encoding/binary/binary.go:119 +0x9e
git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteMessageBegin()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/binary_protocol.go:78 +0x6f
git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Send()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:37 +0x258
git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:85 +0x11d
github.com/kolide/osquery-go/gen/osquery.(*ExtensionClient).Ping()
C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1022 +0xfd
github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).Ping()
:1 +0x7c
github.com/kolide/osquery-go.(*ExtensionManagerClient).Ping()
C:/Users/../go/src/github.com/kolide/osquery-go/client.go:58 +0x8d
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func2()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:194 +0x84

Previous write at 0x00c0000d0094 by goroutine 9:
git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteI32()
c:/go/src/encoding/binary/binary.go:119 +0x9e
git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteMapBegin()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/binary_protocol.go:142 +0xc9
github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerRegisterExtensionArgs).writeField2()
C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2746 +0xa84
github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerRegisterExtensionArgs).Write()
C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2717 +0x241
git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Send()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:40 +0x29f
git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call()
C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:85 +0x11d
github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).RegisterExtension()
C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1850 +0x1d5
github.com/kolide/osquery-go.(*ExtensionManagerClient).RegisterExtension()
C:/Users/../go/src/github.com/kolide/osquery-go/client.go:73 +0xa4
github.com/kolide/osquery-go.(*ExtensionManagerServer).Start.func1()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:143 +0x1d1
github.com/kolide/osquery-go.(*ExtensionManagerServer).Start()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:172 +0x55
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func1()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:186 +0x3f

Goroutine 10 (running) created at:
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:190 +0xa6
main.main()
C:/Users/../go/src/test/osquery/main.go:23 +0x30c

Goroutine 9 (running) created at:
github.com/kolide/osquery-go.(*ExtensionManagerServer).Run()
C:/Users/../go/src/github.com/kolide/osquery-go/server.go:185 +0x7a
main.main()
C:/Users/../go/src/test/osquery/main.go:23 +0x30c

exit status 2

Go get for osquery-go throws error

I did go get on this repo and it threw the following error:

# github.com/kolide/osquery-go/gen/osquery
../../code/go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1272: cannot use extensionProcessorPing literal (type *extensionProcessorPing) as type thrift.TProcessorFunction in assignment:
	*extensionProcessorPing does not implement thrift.TProcessorFunction (wrong type for Process method)
		have Process(int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
		want Process(context.Context, int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
../../code/go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1273: cannot use extensionProcessorCall literal (type *extensionProcessorCall) as type thrift.TProcessorFunction in assignment:
	*extensionProcessorCall does not implement thrift.TProcessorFunction (wrong type for Process method)
		have Process(int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
		want Process(context.Context, int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
../../code/go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1274: cannot use extensionProcessorShutdown literal (type *extensionProcessorShutdown) as type thrift.TProcessorFunction in assignment:
	*extensionProcessorShutdown does not implement thrift.TProcessorFunction (wrong type for Process method)
		have Process(int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
		want Process(context.Context, int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
../../code/go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1284: not enough arguments in call to processor.Process
	have (int32, thrift.TProtocol, thrift.TProtocol)
	want (context.Context, int32, thrift.TProtocol, thrift.TProtocol)
../../code/go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2498: cannot use extensionManagerProcessorExtensions literal (type *extensionManagerProcessorExtensions) as type thrift.TProcessorFunction in argument to self33.ExtensionProcessor.AddToProcessorMap:
	*extensionManagerProcessorExtensions does not implement thrift.TProcessorFunction (wrong type for Process method)
		have Process(int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
		want Process(context.Context, int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
../../code/go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2499: cannot use extensionManagerProcessorOptions literal (type *extensionManagerProcessorOptions) as type thrift.TProcessorFunction in argument to self33.ExtensionProcessor.AddToProcessorMap:
	*extensionManagerProcessorOptions does not implement thrift.TProcessorFunction (wrong type for Process method)
		have Process(int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
		want Process(context.Context, int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
../../code/go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2500: cannot use extensionManagerProcessorRegisterExtension literal (type *extensionManagerProcessorRegisterExtension) as type thrift.TProcessorFunction in argument to self33.ExtensionProcessor.AddToProcessorMap:
	*extensionManagerProcessorRegisterExtension does not implement thrift.TProcessorFunction (wrong type for Process method)
		have Process(int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
		want Process(context.Context, int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
../../code/go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2501: cannot use extensionManagerProcessorDeregisterExtension literal (type *extensionManagerProcessorDeregisterExtension) as type thrift.TProcessorFunction in argument to self33.ExtensionProcessor.AddToProcessorMap:
	*extensionManagerProcessorDeregisterExtension does not implement thrift.TProcessorFunction (wrong type for Process method)
		have Process(int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
		want Process(context.Context, int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
../../code/go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2502: cannot use extensionManagerProcessorQuery literal (type *extensionManagerProcessorQuery) as type thrift.TProcessorFunction in argument to self33.ExtensionProcessor.AddToProcessorMap:
	*extensionManagerProcessorQuery does not implement thrift.TProcessorFunction (wrong type for Process method)
		have Process(int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
		want Process(context.Context, int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
../../code/go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2503: cannot use extensionManagerProcessorGetQueryColumns literal (type *extensionManagerProcessorGetQueryColumns) as type thrift.TProcessorFunction in argument to self33.ExtensionProcessor.AddToProcessorMap:
	*extensionManagerProcessorGetQueryColumns does not implement thrift.TProcessorFunction (wrong type for Process method)
		have Process(int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
		want Process(context.Context, int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
../../code/go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2503: too many errors

Breaking change in Apache Thrift NewTSocketFromAddrTimeout

This commit to the Apache Thrift go library adds a socket timeout param to NewTSocketFromAddrTimeout(), which causes a fatal error when compiling osquery-go in transport.go.

# github.com/kolide/osquery-go/transport
/go/src/github.com/kolide/osquery-go/transport/transport.go:31:43: not enough arguments in call to thrift.NewTSocketFromAddrTimeout
	have (*net.UnixAddr, time.Duration)
	want (net.Addr, time.Duration, time.Duration)

I tested on Linux and OSX with go 1.11 and 1.14. Not sure if the Windows version is affected.

The simplest fix looks like using the same timeout value for both the connectTimeout and socketTimeout params, changing transport.go#L31 from:

trans := thrift.NewTSocketFromAddrTimeout(addr, timeout)

to:

trans := thrift.NewTSocketFromAddrTimeout(addr, timeout, timeout)

I'm happy to open a PR for this if it looks reasonable.

transport.go does not compile on non Windows systems

I receive this error when trying to compile an extension written with osquery-go on Linux.

github.com/kolide/osquery-go/transport

/go/src/github.com/kolide/osquery-go/transport/transport.go:31:43: not enough arguments in call to thrift.NewTSocketFromAddrTimeout
have (*net.UnixAddr, time.Duration)
want (net.Addr, time.Duration, time.Duration)

Looking at github.com/apache/thrift/lib/go/thrift/socket.go the func NewTSocketFromAddrTimeout does indeed need 3 arguments

// Creates a TSocket from a net.Addr
func NewTSocketFromAddrTimeout(addr net.Addr, connTimeout time.Duration, soTimeout time.Duration) *TSocket {
return &TSocket{addr: addr, connectTimeout: connTimeout, socketTimeout: soTimeout}
}

I was able to get it to compile by locally modifying transport.go#31 to
trans := thrift.NewTSocketFromAddrTimeout(addr, timeout, timeout)

Constraints that are not `==` don't appear to be passed along

As pointed out on slack, it's not clear how to get a constraint that isn't ==

I added a spew to table.Call and I don't see the constraints coming in.

If I look at a trace in panic, I don't see much in between that, and the thrift layer.

Go extension processes do not die

We’ve been experiencing go extensions in Windows not dying when the service restarts. We believe this is due to the shutdown method not being implemented. I will update with more details later, but wanted the issue for posterity and record that this is an issue.

Glide and dependencies

I'm trying to compile logger.go from your samples.
I'm using Glide to manage dependencies.

"glide.exe init" generates a glide.yaml like this:

package: .
import:
- package: github.com/kolide/osquery-go
  subpackages:
  - gen/osquery

When I run "go build" the results is this:

C:\Users\Axel\go\src\github.com\kolide\osquery-go\gen\osquery\osquery.go:1154:15: assignment mismatch: 2 variables but 1 values
C:\Users\Axel\go\src\github.com\kolide\osquery-go\gen\osquery\osquery.go:1203:20: not enough arguments in call to oprot.Flush
have ()
want (context.Context)
C:\Users\Axel\go\src\github.com\kolide\osquery-go\gen\osquery\osquery.go:1227:15: assignment mismatch: 2 variables but 1 values
C:\Users\Axel\go\src\github.com\kolide\osquery-go\gen\osquery\osquery.go:1272:29: cannot use extensionProcessorPing literal (type *extensionProcessorPing) as type thrift.TProcessorFunction in assignment:
*extensionProcessorPing does not implement thrift.TProcessorFunction (wrong type for Process method)
have Process(int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
want Process(context.Context, int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
C:\Users\Axel\go\src\github.com\kolide\osquery-go\gen\osquery\osquery.go:1273:29: cannot use extensionProcessorCall literal (type *extensionProcessorCall) as type thrift.TProcessorFunction in assignment:
*extensionProcessorCall does not implement thrift.TProcessorFunction (wrong type for Process method)
have Process(int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
want Process(context.Context, int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
C:\Users\Axel\go\src\github.com\kolide\osquery-go\gen\osquery\osquery.go:1274:33: cannot use extensionProcessorShutdown literal (type *extensionProcessorShutdown) as type thrift.TProcessorFunction in assignment:
*extensionProcessorShutdown does not implement thrift.TProcessorFunction (wrong type for Process method)
have Process(int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
want Process(context.Context, int32, thrift.TProtocol, thrift.TProtocol) (bool, thrift.TException)
C:\Users\Axel\go\src\github.com\kolide\osquery-go\gen\osquery\osquery.go:1284:27: not enough arguments in call to processor.Process
have (int32, thrift.TProtocol, thrift.TProtocol)
want (context.Context, int32, thrift.TProtocol, thrift.TProtocol)
C:\Users\Axel\go\src\github.com\kolide\osquery-go\gen\osquery\osquery.go:1284:27: too many errors

Any idea?

Extension cannot connect on first try when loaded automatically

Context

osquery 4.2
Windows 10 Pro 64bit

Test extension

package main

import (
	"context"
	"flag"
	"log"

	"github.com/kolide/osquery-go"
	"github.com/kolide/osquery-go/plugin/table"
)

func main() {
	flSocket := flag.String("socket", "", "")
	flag.Int("timeout", 0, "")
	flag.Int("interval", 0, "")
	flag.Bool("verbose", false, "")
	flag.Parse()

	if *flSocket == "" {
		log.Fatalln("--socket flag cannot be empty")
	}

	server, err := osquery.NewExtensionManagerServer("dev_extension", *flSocket)
	if err != nil {
		log.Fatalf("Error creating osquery extension server: %s\n", err)
	}

	server.RegisterPlugin(
		table.NewPlugin(
			"test_table",
			[]table.ColumnDefinition{
				table.TextColumn("foo"),
			},
			func(ctx context.Context, queryContext table.QueryContext) ([]map[string]string, error) {
				return []map[string]string{
					map[string]string{
						"foo": "bar",
					},
				}, nil
			},
		),
	)

	if err := server.Run(); err != nil {
		log.Fatal(err)
	}
}

Running the above test extension with .\osqueryi.exe --allow_unsafe --extension=.\extension.exe will throw an error registering extension: i/o timeout. Then about a minute later the extension will connect successfully.
Adding a time.Sleep(1 * time.Second) to the beginning of the extension will make it connect on the first try instead.
Similar behavior happens when the extension is run via the extensions.load file.

osquery-go without watchdog.

There is no watchdog be launched unless the enable_extension_watchdog=true

windows extension memleak related to go-winio

Hi,

I notice there is memleak related to go-winio on windows only. Is anyone else noticing similar issue? Am I missing anything when using the lib?

this is found for commit: v0.0.0-20220706183148-4e1f83012b42

and mem leak is not found in latest commit at this comment: v0.0.0-20231006172600-d6f325f636a9

Is it possible to create event tables using osquery-go?

I'm hoping to create a custom "event" table (i.e. using https://osquery.readthedocs.io/en/stable/development/pubsub-framework rather than a traditional, on-demand generated table).

It seems like osquery-go might only support the on-demand generate table style though?

Entirely possible that this just isn't something you can do with a plugin, I couldn't find many details on the Thrift API to confirm this 🙂

logger plugin does not return useful values by default

right now, the logger Call method assumes that the format of a status log is

{status : "some log message"}
but a more accurate representations is
{status: true, "log": "some log message"}

Using the example logger:

I0619 22:28:20.300494 137629696 scheduler.cpp:74] Executing scheduled query users: SELECT * FROM users;
2017/06/19 22:28:21 status: true

After adding a fmt.Println(request) line:

map[log:{"":{"s":"0","f":"scheduler.cpp","i":"74","m":"Executing scheduled query users: SELECT * FROM users;"}} status:true]

this line returns the log string, which is json.

diff --git a/plugin/logger/logger.go b/plugin/logger/logger.go
index 68c2fe7..a4a8f8b 100644
--- a/plugin/logger/logger.go
+++ b/plugin/logger/logger.go
@@ -57,8 +57,8 @@ func (t *Plugin) Call(ctx context.Context, request osquery.ExtensionPluginReques
                err = t.logFn(ctx, LogTypeHealth, log)
        } else if log, ok := request["init"]; ok {
                err = t.logFn(ctx, LogTypeInit, log)
-       } else if log, ok := request["status"]; ok {
-               err = t.logFn(ctx, LogTypeStatus, log)
+       } else if _, ok := request["status"]; ok {
+               err = t.logFn(ctx, LogTypeStatus, request["log"])
        } else {
                return osquery.ExtensionResponse{
                        Status: &osquery.ExtensionStatus{

Query constraint parsing broken in osquery 3.2.0

With the addition of properly typed JSON in osquery 3, constraint parsing is broken.

Former JSON:

{  
   "constraints":[  
      {  
         "name":"domain",
         "list":[  
            {  
               "op":"2",
               "expr":"kolide.co"
            }
         ],
         "affinity":"TEXT"
      },
      {  
         "name":"email",
         "list":"",
         "affinity":"TEXT"
      }
   ]
}

New JSON:

{  
   "constraints":[  
      {  
         "name":"domain",
         "list":[  
            {  
               "op":2,
               "expr":"kolide.co"
            }
         ],
         "affinity":"TEXT"
      },
      {  
         "name":"email",
         "list":[  

         ],
         "affinity":"TEXT"
      }
   ]
}

Note that now integers have integer type rather than string, and empty lists are actually empty lists instead of empty strings. We need to update the parsing code (https://github.com/kolide/osquery-go/blob/master/plugin/table/table.go#L235) to handle both the old string types from osquery < 3 and the new integer types in osquery >= 3.

Should we add mutexs to osquery-go?

As I understand it, osquery's thrift interface is single threaded. But, this isn't mirrored in this SDK. We leave it to the caller to insure that Query is not called concurrently. If the caller does not respect that single-threaded nature, thrift will emit a slew of socket errors.

I see that the server side already has mutexes, but the client side does not. And while it's easy for the client to wrap the occasional Query in a mutex, it gets thornier with all the entry points. On the client side, we have at least Query, QueryRow, and QueryRows.

I think it would be pretty simple, and correct, to add a mutex to the Client.

I think it would be less simple, but still correct, to add a mutex that could be shared between the server and the client. I'm not sure how that would work with the current API. Best I've got is a package level global with a mutex per path. Or possible finding a way to expose serverClient from the server for client functionality on a shared mutex.

I'm curious what @zwass thinks

If the caller does not mutex, and there are conflicting writes, you'll see errors like i/o timeout or out of order

thrift git repo moved

https://issues.apache.org/jira/projects/THRIFT/issues/THRIFT-4542?filter=allopenissues

It looks like dep ensure -vendor-only still works, but attempting to update it fails

dep ensure -update -v
Root project is "github.com/kolide/osquery-go"
 14 transitively valid internal packages
 5 external packages imported from 4 projects
(0)   ✓ select (root)
(1)	  ← no more versions of git.apache.org/thrift.git to try; begin backtrack
  ✗ solving failed

Solver wall times by segment:
  b-source-exists: 447.612396ms
  b-list-versions: 158.997027ms
      select-root:    387.594µs
            other:     82.197µs
         new-atom:     39.252µs

  TOTAL: 607.118466ms

Solving failure: failed to list versions for https://git.apache.org/thrift.git: fatal: repository 'https://git.apache.org/thrift.git/' not found
: exit status 128

Support to pass the version in customized extension Info

Because current go SDK is not support add the customized extension Info
So I want to add this feature in go SDK.

When schedule Osquery packs in runtime config not getting callback

Issue - I created an osquery extension which includes a logger and a config plugin using osquery-go. Within this extension, I established a runtime config. When a scheduled query is triggered, it works fine. However, when Packs are utilized, it does not provide the expected callback.

Operating System - Windows 10
go version - go1.21.3

Code snippet

Flages :-

--disable_extensions=false 
 --disable_events=false
 --events_expiry=1 
 --events_optimize=true 
 --events_max=500000 
 --logger_plugin=testlogger 
--config_plugin=testconfig
 --extensions_timeout=600 
 --extensions_interval=5 
 --extensions_require=testextmgr
 --database_path="C:\Program Files\Test\testosqueryd.db" 
 --extensions_socket="\\.\pipe\testosquery.em"

Below is my runtime config :-

const configSchedule_processes string = `
{
   "schedule": {
   "processes": {
			  "query": "SELECT * FROM processes;",
			  "interval": 10
		}
   },
    "packs": {
        "windows-attacks": "C:\\Program Files\\osquery\\packs\\windows-attacks.conf"
    }  
}
`

Below are my logger and config function.

func LogString(ctx context.Context, typ logger.LogType, logText string) error {
        fmt.Println(logText)
        gCnt = gCnt + 1
        return nil
    }
    
    func ConfigCallback(ctx context.Context) (map[string]string, error) {
        return map[string]string{
            "config": configSchedule_processes,
        }, nil
    }

Output -
Only getting callback for processes. Not getting any callback for packs. For now trying with only one config. But in real scenario we can use multiple files

What does windows-attacks.conf contain?
Answer - This is general pack found on https://github.com/osquery/osquery/tree/master/packs

Few other observation -
I am no expert in this. Learning Osquery.
But,
https://osquery.readthedocs.io/en/stable/development/config-plugins/

in this doc it say's osquery packs needs to be achieve by implementing virtual method in cpp. (Follow Additional overloads from above link)

In osquery-go it seems that only genConfig is handled genPack is not handled.

Can some one PLEASE help here?
Thanks in advance.

Can't create socket file correctly, If defined an socket, but connect refused

• create an new socket, the result is context deadline exceeded
  err info: Error creating Thrift client: waiting for unix socket to be available: goosquery.em: context deadline exceeded
  exit status 1

• connect an exist socket file , the result is connection refused
  err info: Error creating Thrift client: opening socket transport: dial unix /var/osquery/osquery.em: connect: connection refused

I tried set timout to 200s, It's not available
the socket file can't be created successfully

Extensions dying after query that changes privileges

Whenever I run a query that attempts to drop privileges while I have extensions registered, the extensions die.

This output below is from a machine running Amazon Linux release 2 (Karoo), but I've run into the same problem on an Ubuntu machine. I haven't been able to reproduce on OSX and haven't yet tried on Windows.

Building and running the example extension from https://github.com/kolide/osquery-go/blob/master/examples/table/main.go , I get the following output:

osquery> select * from example_table;
+-------------+---------+-------------+---------+
| text        | integer | big_int     | double  |
+-------------+---------+-------------+---------+
| hello world | 123     | -1234567890 | 3.14159 |
+-------------+---------+-------------+---------+
osquery> select * from authorized_keys where uid in (select uid from users);
Thrift: Fri Dec 14 01:58:37 2018 TSocket::read() THRIFT_POLL() Interrupted system call
Thrift: Fri Dec 14 01:58:37 2018 TConnectedClient died: Unknown: Interrupted system call
<.....................QUERY RESULTS...................>
osquery> Thrift: Fri Dec 14 01:58:41 2018 TSocket::open() connect() <Host:  Port: 0>Connection refused
I1214 01:58:41.973999 32340 extensions.cpp:305] Extension UUID 18021 has gone away

osquery> select * from example_table;
Error: no such table: example_table
osquery> 

OSQuery version: Reproduced on both 3.3.0 (downloaded from the website) and built from source at hash 5188ce5288abe0e323b8e8bd364f452134a62d00
osquery-go version: current master (7486f08)

If you configure the extensions to re-register, this is fine as it doesn't actually prevent the query from running, but it means that it's also impossible to successfully execute a query from inside an extension.

I think this is related to osquery/osquery#1638.

I spoke to @groob briefly in slack. I'm not sure if this is an osquery-go issue or an issue with osquery, as I haven't yet tried building a C++ extension. Thanks so much for your help!

Results as map[string]interface{} rather then map[string]string

Would it be possible to have response return the actual data type like a map[string]interface{} rather then have the value always be a string.