We are currently using RuntimeDyld to relocate our JIT code in the target process. However, it is being deprecated and lack some features, like support for weak symbols. It has multiple other shortcomings described here.

BOLT went through the same change recently, which could be a good starting point to understand what moving to a more modern API would entice.
https://reviews.llvm.org/rG05634f7346a5

Typed TreeBuilder

This piece of work aims to solve four problems, in this priority order:

1. Level up the output from Object Introspection as a Library (OIL) to the same level as OI Debugger, enabling new use cases.
2. Unify the code path that OIL and OID take in our generated code, increasing maintainability.
3. Prevent desynchronisation between the logic that writes to the data segment (generated code) and the logic that reads from it (TreeBuilder/OIL consumers), similarly increasing maintainability.

While these may be a lofty set of goals, I believe by taking this approach we can turn them into one cohesive piece of work.

Glossary

Motivation

OI's traversal of arbitrary type hierarchies presents many opportunities, including: opportunities for compression, computing object differentials at a primitive level, and analysis based on captured memory.

OIL in its current form uses an accumulator to track each element which contributes to the total size. OID instead utilises a data segment to send information about the captured object from the target to the attached debugger. It is very easy to have OIL return instead a std::vector<uint8_t> which contains the contents of this data segment, providing the ability to access this already rich data in the end-user application. This is feasible because OIL runs on request in the target process, and unlike OID can utilise the memory allocator for a dynamic array.

While an array of bytes is a great way to store the data efficiently and share between processes, it provides an incredibly poor interface for the consumer. Our current consumer is TreeBuilder, a behemoth which infers the contents of the data segment in an entirely different way to the producer. That is, the generated code writes to the data segment based on dynamic logic for containers in .toml files, while TreeBuilder reads from the data segment using a statically compiled set of logic.

An ideal system maintains the efficiency of the data segment while presenting a friendly interface to the end user. I present Typed TreeBuilder, which should achieve both of these interface goals.

Approach

Provide statically typed objects which occupy the same amount of space as a uint8_t* (8-bytes) while having a rich type in the JIT code. Rely on the optimising compiler to inline the rich types into pointer operations, exactly as they were before in most cases. Provide a second dynamically typed representation to be used outside of the JIT code which describes the same data, and does not need to be so efficiency constrained.

The plan is to implement this in two phases. Phase One will implement types in the data segment and provide a representation of this type to TreeBuilder at runtime, ensuring that we put in and take out the same elements from the data segment. Phase Two will provide the additional information that TreeBuilder requires (type knowledge) through this runtime representation, vastly simplifying TreeBuilder and providing this data through the OIL interface as well. This allows an opportunity to test and evaluate the new system in production before making it required to use OIL or TreeBuilder, as Phase Two won't be easily gateable.

Type System

The static type system is simple here, based on the Simply Typed Lambda Calculus. Specifically, we require some very primitive types: Unit (no data, 0-bytes), Int, and Bool. Then two composite types exist: Product (pair), two data types both present; and Sum, a tagged union with only one of the two types present.

$$ T \Coloneqq \texttt{Unit} | \texttt{Int} | \texttt{Bool} | T_1 * T_2 | T_1 + T_2 $$

With these simple types we can represent everything in the datasegment. This is a sufficient set of types to store everything we store, by combining the types to create richer ones. However, we also need a couple more basic types to store values efficiently. Pointer makes sense as a number that isn't implemented as a Variable Length Integer (VarInt), as they will effectively always use most of the bits. Secondly, we'll need a List primitive for lists of values of known length, as because of implementation constraints fully recursive data types will be impossible (e.g. $\[T] \Coloneqq Unit + (T * [T])$) and lists of that form are spatially inefficient. Finally, we'll need to optimise Sum to accept N values rather than 2, limiting the number of tag bits required, again for space efficiency.

Specifically this will extend our array:

$$ T \Coloneqq ... | \texttt{Pointer} | [T] $$

Which should be sufficient. I'm no mathematician, so we'll leave it at that.

Generated Code Typing Examples

In our generated code we can perform static typing via template metaprogramming. By describing our Data Buffer as a rich type we can describe the possible operations at each stage.

namespace StaticTypes {
using String_DS = DataSegment::Containers::Tuple<
  StaticTypes::Bool,
  StaticTypes::VarInt,
  StaticTypes::VarInt>;
} // namespace StaticTypes

StaticTypes::Unit
getSizeType(
  const std::string &t,
  const StaticTypes::String out,
) {
    bool sso = ((uintptr_t)t.data() < (uintptr_t)(&t + sizeof(std::string)))
        &&
      ((uintptr_t)t.data() >= (uintptr_t)&t);

    return out.write(sso, t.capacity(), t.size());
}

A String is represented as a Tuple as a fixed number of fields of different types are used. In this case we use a Bool to describe whether Short-String Optimisation (SSO) is occurring, a VarInt for the capacity of the string, and a VarInt for the size. A possible implementation for a string getSizeType function is also shown, which confirms SSO with a pointer check before writing the three elements into the Tuple and returning the new offset as an unwritable Void type.

In the current implementation on main, primitives are completely ignored as only information that can't be recreated from the type information is stored in the data segment. We'll keep that for these examples, representing the underlying C++ primitive types as our static Unit type.

The class Foo would be represented in static types as follows:

class Foo {
  std::string name;
  uint64_t age;
  bool isEnabled;
  std::string nickname;
};

using Foo_DS = StaticTypes::Tuple<
  StaticTypes::String,
  StaticTypes::Unit,
  StaticTypes::Unit,
  StaticTypes::String>;

Following the philosophy of the data segment, we store the (relevant) fields of a struct as a tuple, as all that matters is that there are a fixed number of fields of potentially different types. A specific Struct type is unnecessary, as it is simply a collection of fields - the metadata can be re-added later. This Tuple will need to be automatically generated by codegen for arbitrary structs, but hopefully it is clear that this can be achieved with some quite simple recursion.

OIL Interface Typing Examples

For OIL we'll provide a new interface, alongside the existing ObjectIntrospection::getObjectSize, which produces the full OI interface to the type. This will look something like:

template <class T>
ObjectIntrospection::DataReader
ObjectIntrospection::introspect(
  const T& object,
  std::vector<uint8_t>& buffer = {},
);

The DataReader provides a compromise between the underlying byte array and a high level representation. While it's still a seeking data structure, it provides rich information about the underlying type and can compute values where necessary. Further, we can cache values as in the current Type Graph work.

Important here is the correlation between the type described in the previous section (Code Generation Typing Examples), and the C++ Object DataReader. I propose a function on each of the statically typed Code Generation types that produces a dynamic type for the equivalent type.

For example:

using String = Tuple<
  StaticTypes::Bool,
  StaticTypes::VarInt,
  StaticTypes::VarInt,
>;

using RuntimeTypes::Dynamic = std::variant<
  RuntimeTypes::Tuple,
  RuntimeTypes::Bool,
  RuntimeTypes::VarInt,
  ...>;

RuntimeTypes::Dynamic
String::describe() {
  return Tuple(
    Bool::describe(), VarInt::describe(), VarInt::describe()
  );
}

Where String::describe() won't be explicitly implemented, and will instead be automatically implemented by the Tuple::describe logic. By implementing this we can input and extract the same information to the data segment as we currently can, making this the division of phase 1. At the end of phase 1 we have a type safe serialization/deserialization between the JIT code and TreeBuilder.

Phase 2 moves on to providing all of the output TreeBuilder currently can from a RuntimeTypes::Dynamic object. This is what will bring OIL up to parity.

class RuntimeTypes::Struct: public Tuple {
  public:
    Struct(name, std::span<const char *> fieldNames, ...): Tuple(...), fieldNames_(fieldNames),  {
      assert(Tuple.size() == fieldNames.size());
    };

  private:
    const char *name_;
    std::span<const char *> fieldNames_;
};

class Foo_DS : public StaticTypes::Tuple<
  StaticTypes::String,
  StaticTypes::Unit,
  StaticTypes::Unit,
  StaticTypes::String> {

  static RuntimeTypes::Dynamic describe() {
    return Struct("Foo", {"name", "age", "isEnabled", "nickname"}, Tuple::describe());
  }
};

It's plausible, but skipped here for the sake of brevity (whoops), to make these describe functions and constructors constexpr. This means they could further be computed at compile time and stored in the text segment. For particularly compile time OIL which may have these values available but seldom used, keeping the information in the binary instead of dynamically generating them could reduce the resident memory burden. This can be explored in the future. Similar features like creating a static string pool in the JIT code can keep things memory efficient.

TreeBuilder Typing Examples

With some rearchitecting TreeBuilder becomes incredibly simple. If we use the OIL APIs (through a bit of fiddling) in OID, we can access all of the information we would need for TreeBuilder in a ready to go form.

Pseudocode for an implementation:

std::function<ObjectIntrospection::DataReader, uint8_t*, size_t> getDataReader =
  lookupInTextSegment(mangle(ObjectIntrospection::getDataSegmentReader<A,B,C>));

uint8_t* dataBlob = getDataSegmentStart();
size_t dataSize = getDataSegmentSize();

ObjectIntrospection::RuntimeTypes::Dynamic dataReader =
  getDataReader(dataBlob, dataSize);

recurse: {
  node.write("typePath", dataReader.typePath); // etc...
  for (ObjectIntrospection::DataReader& child : dataReader.children()) {
    recurse(child);
  }
}

The complexity here is mapping the text segment in both the target and the debugger. Once that is complete, TreeBuilder in OID becomes a small shim around the functionality already provided to OIL. That is, we can share a TreeBuilder implementation between OIL (the programmer's context) and OID (the debugger), allowing any programmer to exercise the full power of OI. This is particularly useful for uploading the OIL output to a database in a similar way to OID, but with significantly more control over where it is called.

Extensions

So far has been on gaining parity between OIL and what OID currently does in production. There are several experimental features that haven't been described, but I think fit well into this design. Although this won't be completed in the same piece of work, it is important to have enough of a design now that we know it won't become impossible after this work.

Part of this will likely involve a further explosion of template parameters. For example, we might start seeing bool MemoryCapture = false in the templates of getSizeType functions. This could be followed by if constexpr (MemoryCapture) { ... } within the function. An alternative would be to do something like:

namespace FeatureFlags {
enum FeatureFlags: uint8_t {
    None = 0,
    CaptureMemory = 1 << 1,
    ChaseRawPointers = 1 << 2,
};
}

template <uint8_t Flags>
void foo() {
    if constexpr (Flags & FeatureFlags::CaptureMemory) {
        std::cout << "CaptureMemory" << std::endl;
    }
    if constexpr (Flags & FeatureFlags::ChaseRawPointers) {
        std::cout << "ChaseRawPointers" << std::endl;
    }
    std::cout << "foo" << std::endl;
}

int main() {
    foo<FeatureFlags::CaptureMemory>();
    foo<FeatureFlags::ChaseRawPointers>();
    foo<FeatureFlags::None>();
    foo<FeatureFlags::CaptureMemory | FeatureFlags::ChaseRawPointers>();
}

This will be quick to resolve at compile (JIT) time as there is no template recursion, while allowing a good amount of flexibility and fast runtime execution with features turned off. It will also prevent adding a new template parameter every time an on or off feature is added, and allow us to reuse the same JIT code for multiple types with different feature sets - important for compile time OIL. More so, it'll mean we can use the same lump of code whether we have features turned on or off at a given callsite, keeping codegen relatively simple. This compares to using ifdefs or the like which would increase the complexity of codegen, particularly when multiple probes are codegenned as one blob.

To fit this piece in entirely with our type system, we need the ability to know at compile time what our output will look like in the face of feature flags (memory capture being the specific example). This is achievable in C++ using std::conditional, enabling us to decide with the same template arguments what the output will look like.

template <uint8_t Flags>
typename std::conditional<
  Flags & FeatureFlags::CaptureMemory,
  uintptr_t,
  void
>::type foo() { ... }

As we can change the type by modifying our call this type framework shouldn't impede any new features. You will need to establish how your new feature affects the output types, but really you should know this anyway...

Memory Capture

Memory capture in OIL can exist in two forms:

1. Providing pointers into the type that can be dereferenced in ordinary C++ code based on information captured.
2. Dumping the entire type hierarchy's primitives for post-processing and analysis.

Both of these fit cleanly with the above. Pointers involve adding a std::conditional gated Pointer everywhere it's relevant. Dumping primitives involves changing the Void type of the primitive getSizeType to Int or Bool, again conditionally gated.

### OIL Pointer Chasing

My recommended way to implement this would be a provided function which validates pointers. So in the current two cases, no pointer chasing would be implemented as bool check(uintptr_t p) { return false; } and --chase-raw-pointers would be implemented as bool check(uintptr_t p) { return true; }.

A pointer validation function would further allow us to have a set of pointers which it is valid to follow and can be determined dynamically, or anything else the user deems appropriate. We would pass this through as an additional argument to getSizeType, allowing it to be implemented either statically or dynamically. In all likelihood it would always be static, but the option would be nice. The type system helps significantly here as we'd have to represent a pointer that may or may not be chased as some sort of Optional<T> = Void + T, which can easily be implemented in the type system and removes that risk of desync.

Risk

Recursive templates like the $T_1 * T_2$ logic can be very slow to compile with a large number of parameters. For large C++ types we may have dozens of fields, perhaps hundreds. To alleviate this risk I will make large types one of the first test cases, and if it is too slow I can stop early and take a different approach.

We also risk breakage in changing existing behaviour that perhaps hasn't been touched in a long time. However, our (relatively) comprehensive test suite reduces this risk. Further, the risk after this is landed of such breakages in the future is much lower, so this should be worth it longer term.

The pointer address values aren't reported for pointers contained within a vector.

It currently doesn't work due to the dependence on unqualified type names in resolving drgn types.

e.g. this setup:

namespace nsA {
  class MyClass {
  public:
    virtual void myfunc() {}
    int int_a;
  };
} // namespace nsA

namespace nsB {
  class MyClass : public nsA::MyClass {
  public:
    virtual void myfunc() override {}
    std::vector<int> vec_b;
  };
} // namespace nsB

namespace nsC {
  class MyClass : public nsB::MyClass {
  public:
    virtual void myfunc() override {}
    int int_c;
  };
} // namespace nsC

Function pointers are handled slightly strangely in the current implementation.

Issues:

• C-style function pointers don't report the function's address in the pointer field of the output.
• Some attempt to follow a C-style function pointer is made, returning a nonsensical result - we shouldn't try to follow function pointers.
• C++ style function pointers (std::function) don't report the function's address in the pointer field of the output.

### Affected tests

• OidIntegration.pointers_function_raw
• OidIntegration.pointers_function_raw_chase
• OidIntegration.pointers_function_raw_null
• OidIntegration.pointers_function_std_function
• OidIntegration.pointers_function_std_function_chase
• OidIntegration.pointers_function_std_function_null

Racy isn't the word as you could drive a bus through the hole's in he we currently attach to the threads of a process. The mechanism currently used is to PTRACE_SEIZE all threads and then PTRACE_INTERRUPT individual threads when we want to manipulate them. Seizing a thread is simply making an association between oid (the debugger) and the target thread (the debuggee). However, it currently isn't possible to atomically seize all the threads in a process and that is a huge issue.

Currently in OIDebugger::targetAttach() we use a directory iterator to iterate over all the existing threads in a processes /proc/pid/task namespace and this is hopelessly racy. In fact there is a comment which I put in there a long, long time ago when I wrote the original code:

  /*
   * ptrace sucks. It doesn't have a mechanism for grabbing all the threads in
   * a process (even if it's a SEIZE operation). Grabbing all threads seems to
   * be a huge race condition so unless we find a better way to do it, we need
   * to be as defensive as we can.
   */

In the absence of an atomic PTRACE_SEIZE_ALL type operation a better mechanism is required or at least one that captures the thread lifecycle operations going off during the seize (once the seize occurs we should be good according to the contract offered by ptrace() owing to the options we seize with).

The issue we can end up with in processes with many threads that are creating and destroying threads quickly is that we miss to SEIZE a thread so it's not under our control but yet is is open to hitting breakpoint traps we have inserted. This leads to extreme badness. There are other issues but this is the most severe.

Related to #87, if a target thread generates a SEGV whilst under control of oid, oid doesn't handle the situation well:

Attached to pid 492977
E0228 19:27:12.555498 493446 OIDebugger.cpp:1096] SIGSEGV: faulting addr 0x28cc5000785b1 rip 7fc7ce64eacf (pid 492977)
E0228 19:27:12.748322 493446 OIDebugger.cpp:2579] restoreState Couldn't detach target pid 492977 (Reason: No such process)
E0228 19:27:12.748560 493446 OIDebugger.cpp:2017] process_vm_readv() error: No such process (3)
E0228 19:27:12.748571 493446 OIDebugger.cpp:2835] Failed to read data segment from target process
E0228 19:27:12.748581 493446 OID.cpp:434] Problems processing target data

oid should report something informative and much less shouty.

test/integration/pointers_incomplete.toml contains tests for which OIL successfully codegens but OID does not. This is unusual as most of the code path is the same. The affected tests are:

• OidIntegration.pointers_incomplete_raw_no_follow (has no oil equivalent but if enabled it passes)
• OidIntegration.pointers_incomplete_raw_no_follow
• OidIntegration.pointers_incomplete_raw_null

Some internal Meta cruft is hanging around in OITraceCode.cpp which needs removing.

Testing with the specific object should be done.

Containers with no template parameters are successfully found by OICodeGen::isContainer but not found by TreeBuilder::isContainer. This is because TreeBuilder relies on containerTypeMap (containerTypeMapDrgn in OICodeGen) which is only inserted to by OICodeGen::enumerateTemplateParamIdxs.

The naïve fix of inserting an empty vector into the map in OICodeGen::getContainerInfo does not work as it then gets skipped in OICodeGen::enumerateTemplateParamIdxs. Making it a std::optional also didn't work for reasons unknown. The final step of Typed TreeBuilder (#95) is going to fix this for free, so I don't intend to spend any more time on it for now.

Currently we have JLOG* macros inserted into the generated code. These are life savers for debugging JIT code execution issues but outside of that they are not needed. In fact they just bloat the text size and possibly slow us down with the addition of branches (although the macros are written with branch prediction guidance).

The JLOG* macros should only be generated when jit logging is explicitly enabled (-l on oid command line).

A Thrift struct such as:

  struct MyThriftStruct {
    1: optional i32 a;
    2: optional i32 b;
  }

Will get translated into a C++ struct such as:

  struct MyThriftStruct {
    int32_t __fbthrift_field_a;
    int32_t __fbthrift_field_b;
    apache::thrift::detail::isset_bitset<2, apache::thrift::detail::IssetBitsetOption::Unpacked> __isset;
  }

The "_fbthrift_field" prefix appears in OI's results and can make understanding them and searching for field names in code harder.

If a target process has a data segment setup from a precious run then a subsequent oid invocation specifying a different data segment size (-x option) will segv the target. Obviously that's extremely bad and needs sorting ASAP.

I personally struggle to find the -i option every time I look for it in the help list. I think it's because, like -c, the argument is stored separately from the flag. We should look at making the help more consistent and useful for newcomers.

Current help message:

usage: oid ...
  -h,--help                      Print this message and exit
  -p,--pid <pid>                 Target process to attach to
  -c,--config-file               </path/to/oid.toml>
  -x,--data-buf-size <bytes>     Size of data segment (default:1MB)
                                 Accepts multiplicative suffix: K, M, G, T, P, E
  -d,--debug-level <level>       Verbose level for logging
  -l,--jit-logging               Enable JIT's logs
  -r,--remove-mappings           Remove oid mappings from target process
  -s,--script                    </path/to/script.oid>
  -S,--script-source             type:symbol:arg
  -t,--timeout <seconds>         How long to probe the target process for
  -k,--custom-code-file          </path/to/code.cpp>
                                 Use your own CPP file instead of CodeGen
  -e,--compile-and-exit          Compile only then exit
  -o,--cache-path <path>         Enable caching using the provided directory
  -u,--cache-remote              Enable upload/download of cache files
                                 Pick from {both,upload,download}
  -i,--debug-path                </path/to/binary>
                                 Run oid on a executable with debug infos instead of a running process
  -J,--dump-json [oid_out.json]  File to dump the results to, as JSON
                                 (in addition to the default RocksDB output)
  -B,--dump-data-segment         Dump the data segment's content, before TreeBuilder processes it
                                 Each argument gets its own dump file: 'dataseg.<oid-pid>.<arg>.dump'
  -j,--generate-jit-debug        Output debug info for the generated JIT code
  -n,--chase-raw-pointers        Generate probe for raw pointers
  -a,--log-all-structs           Log all structures
  -z,--disable-packed-structs    Disable appending packed attributes to the definition of structs
  -w,--disable-padding-hunter    Disable Padding Hunter
                                 Padded structs will be written to file called PADDING
  -T,--capture-thrift-isset      Capture the isset value for Thrift fields
  -m,--mode [prod]               Allows to specify a mode of operation/group of settings

        For problem reporting, questions and general comments please pop along
        to the Object Introspection Workplace group at https://fburl.com/oid.

By default, folly::sorted_vector_map stores its elements in pairs in a vector, although this is configurable:

class Container = std::vector<std::pair<Key, Value>, Allocator>>

We do not model the underlying container, which means that we do not correctly record the size that it consumes.

e.g.

folly::sorted_vector_map<int8_t, int64_t> map { {1,2}, {3,4} };

To calculate the size of the map, we will only record the size of the individual elements, i.e. sizeof(int8_t) * 2 + sizeof(int64_t) * 2 = 18.
However, as map entries are stored as std::pair<int8_t, int64_t>, there will be padding which we are not counting: sizeof(std::pair<int8_t, int64_t>) * 2 = 32

Our pointer cycle detection currently relies on both JIT code and TreeBuilder to fully stop processing cycles.

This works in OID, but means that OIL ends up processing some nodes in a cycle (just the first?) more than once as it lacks the TreeBuilder component.

This seems like it should be able to work fully for both OID and OIL without the TreeBuilder component.

See test/integration/cycles.toml for tests which fail under OIL.

Code References

Core JIT cycle detection code:

TreeBuilder has its own independent cycle detection logic:

A test needs to be added to test conditional_t handling. Make sure it includes some conditional logic.

          I really like this as I have previously said. As you have previously said Jake, we need to remove the other options in conjunction with Strobelight changes.

Originally posted by @tyroguru in #125 (review)

Clean up prior use cases of non-feature command line flags to simplify the interface.

This environment variable is required by oilgen but isn't automatically set. This provides a surprising error to the end user. It should be set as such to avoid this:

if ((setenv("DRGN_ENABLE_TYPE_ITERATOR", "1", 1)) < 0) {
    std::cerr << "Failed to set environment variable\
       DRGN_ENABLE_TYPE_ITERATOR\n";
    exit(EXIT_FAILURE);
  }

Recently we've been experiencing an increased number of SIGPIPE errors when running the CI tests. I've seen this once on my devserver too. Fixing the symptom (failing CI tests) by allowing one retry of each test. No debugging done yet but creating this issue to come back and fix the underlying issue in the future.

Whilst debugging another issue I ran across an issue where oid permanently loops and has to be killed. The situation in this case is that the target process has generated a SIGILL and oid ends up looping on the SIGILL as we don't move the target processes ip on past the faulting instruction. We then kill the target process and oid is stuck in a loop trying to continue a thread that doesn't exist.

I'll try and put together an example to attach to this issue.

References in the original code become pointers once through codegen. We can't differentiate references from pointers, and therefore can't assume that references are always valid. This means that we require the --chase-raw-pointers flag for references when it shouldn't be necessary.

Code in base code (test case OidIntegration.references_int_ref):

struct IntRef {
  int &n;
};

Code in generated code:

struct IntRef{
/* 8         |     8 */int* n;
};

Further testing:

$ cat test.cc
struct Foo {
  int& a;
  int *b;
};

int main() {
  int x;
  Foo foo{ .a = x, .b = &x };
}
$ g++ -g test.cc && llvm-dwarfdump a.out
...
0x00000051:   DW_TAG_reference_type
                DW_AT_byte_size (0x08)
                DW_AT_type      (0x00000057 "int")
...
0x0000005e:   DW_TAG_pointer_type
                DW_AT_byte_size (0x08)
                DW_AT_type      (0x00000057 "int")
...

So there is enough information in the DWARF to generate the original example correctly. However we see that the drgn support for references doesn't differentiate them (added here). Therefore we can't support this without first supporting it in drgn.

Example:

  union MyUnion {
    std::vector<int> vec;
    std::unordered_map<std::string, std::string> map;
  };

  union TaggedUnion {
    MyUnion storage;
    uint8_t tag;
  };

We can't capture MyUnion by itself, as we have no way of telling which of vec or map to try to read.
However, the original developers would have had the same issue, so often create a wrapper struct to "tag" their union.

If we have some way of telling OI which field represents a tag for which union, then we'll be able to read this data safely.

This will likely take the form of some extra config settings:

• Option 1: Specify individual structs which represent tagged unions and which field is the tag for which union. e.g.:
  e.g.

  tagged_unions = [("TaggedUnion", "storage", "tag"), ...]
  

  This will always work, but will require a lot of configuration for codebases with a large number of tagged unioins.

• Option 2: Large codebases will often have a standard way of creating a tagged union, whether this be through a framework such as Thrift or just coding conventions. If we can codify these conventions then we could capture all tagged unions with a lot of configuration than Option 1.
  e.g. a Thrift union will have the form:

  struct MyType {
    storage_type value; // The union
    underlying_type_t type; // The tag
  };
  

  Which could be codified as:

  tagged_unions = [("storage_type", "value", "underlying_type_t", "type"), ...]
  

Raw arrays are currently handled by treating them as std::arrays.

This causes a couple of issues:

• std::array becomes a special container type, required just for supporting the base language. This may be unexpected for users and breaks the model where container types should be pluggable and able to be removed if not used
• Zero-length raw arrays do not behave the same as zero-length std::arrays. Because of this, we currently do not treat zero-length arrays as containers at all - see:

  object-introspection/test/integration/arrays.toml

  Lines 27 to 29 in d9b13ff

  	# WARNING: zero-length arrays are handled differently to non-empty arrays.
  	# They end up not being treated as containers. This should probably change
  	# in the future.

We have a few binaries in the 'tools/' directory that are sparsely documented.
It would be nice to improve their docs and also add a README to the directory.

The config section causes a new config to be created in a different temporary location (

Possibly the easiest solution to this is to canonicalise the paths in the base config first, allowing it to be moved anywhere after.

Oi needs to support the C++20 no_unique_address attribute which is detailed at: https://en.cppreference.com/w/cpp/language/attributes/no_unique_address .

Folly uses this under it's FOLLY_ATTR_NO_UNIQUE_ADDRESS define . It's a space optimization mechanism that causes problems with our offset calculations as we currently don't expect a member variable to occupy no space.

See tests at https://github.com/facebookexperimental/object-introspection/blob/main/test/integration/alignment.toml

We normally deal with alignment by adding explicit padding members to structs. However, this does not work for containers, where we do not control padding ourselves.

We need a way of determining what the alignment of a given type is, and then we must apply an alignas specifier to our generated types.

The help URL in oid --help goes to https://fburl.com/oid. This is useless to anyone outside of the company. We should create a landing page on the website with information about how to contact us/raise a GitHub issue/FAQs.

We currently get warnings from glog during our build. This is because FetchContent targets are built as opposed to treated as external.

When we update cmake_minimum_required(VERSION 3.25) we can add SYSTEM to each FetchContent_Declare which should correct this behaviour. Adding this issue to track it, as now is currently too early to update for most distributions (https://repology.org/project/cmake/versions).

Container configuration files must be explicitly listed in the configuration file. Extend this to support globbing them within a folder.

drgn_type_name fails an assertion if drgn_type_has_name is false. This is unlikely to happen but the logging print is dodgy.

A stubbed pointer (char[8]) is generated for an anonymous struct pointer even with --chase-raw-pointers enabled.

Actual code:

struct Node {
    int a, b, c;
  };
struct AnonStructPtrContainer {
    struct { struct Node *node; } *anon;
  };

Generated code:

// stubbed classes -----
struct alignas(8) __anon_struct_0_ { char dummy[8];};
// struct definitions -----
struct AnonStructPtrContainer{
/* 8         |     8 */__anon_struct_0_ anon;
};

### Affected tests

• OidIntegration.anonymous_anon_struct_ptr
• OilIntegration.anonymous_anon_struct_ptr

The top level pointer attribute is omitted when traced with OID. That is, the argument is treated as an argument of the underlying data type. For example:

Actual output:

[
  {
    "name": "a0",
    "typePath": "a0",
    "typeName": "int",
    "isTypedef": false,
    "staticSize": 4,
    "dynamicSize": 0
  }
]

Expected output:

[
  {
    "typeName": "int *",
    "staticSize": 8,
    "dynamicSize": 4,
    "pointer": XXX,
    "members": [
      {
        "typeName": "int",
        "staticSize": 4,
        "dynamicSize": 0
      }
    ]
  }
]

### Affected tests:

• OidIntegration.pointers_int
• OidIntegration.pointers_no_follow
• OidIntegration.pointers_int_null
• OidIntegration.pointers_void
• OidIntegration.pointers_void_no_follow
• OidIntegration.pointers_void_null
• OidIntegration.pointers_vector
• OidIntegration.pointers_vector_no_follow
• OidIntegration.pointers_vector_null

C++ shared pointers use a control block which contains the pointers to at least the deleter and the two atomic reference counts. This is heap allocated and shared between multiple shared pointers to the same object.

Given we can avoid double counting pointers, it might be possible to record the size taken by this blob. This is still tricky, as shared pointers with different get() results may share a control block - it is possible for two shared pointers to share ownership of an object while pointing at e.g. a subfield of it.

The only way that I see this data exposed is with std::shared_ptr<T>::owner_before, which compares the managed pointer of the shared pointers rather than the stored pointer (get() value) like operator==, !=, <, <=, >, >=, <=> (std::shared_ptr). As the underlying pointer isn't exposed, storing this data efficiently would be tricky.

oidDebugger->contTargetThread() returns a bool but it is never checked and is only ever true anyway! It's a historical thing. Make this a void.

Our CMakeLists.txt file currently always builds drgn with -O3 -march=broadwell. This prevents building OI on old machines or VMs with a limited CPU feature set.

This can be reliably reproduced on the CI by upping the parallelism from 4 to 16.

The target process receives a SIGILL, with the instruction pointer pointing to the dataBase address inserted by OID. The "illegal instruction" is the pointer which is stored at this address.

Info extracted from a core dump of the target process:

(gdb) x/i $rip
=> 0x7fc01aefe040:      (bad)  
(gdb) p $rip
$1 = (void (*)(void)) 0x7fc01aefe040
(gdb) x/xg $rip
0x7fc01aefe040: 0x00007fc01adfe027

In order to get a core dump from the target process, oid needs to be modified to not catch SIGILLs - this case needs commenting out:

Extracts from OID log:

I0202 15:46:23.986310 10953 OICompiler.cpp:595] Relocated code 0x7fba5281b000 to 7fc01aefe080                                                                         
I0202 15:46:23.986321 10953 OICompiler.cpp:604] Relocated data 0x7fba5281b400 to 7fc01aefe480                                                                         
I0202 15:46:23.986332 10953 OICompiler.cpp:604] Relocated data 0x7fba5291b420 to 7fc01affe4a0                                                                         
I0202 15:46:23.986343 10953 OICompiler.cpp:604] Relocated data 0x7fba5291b431 to 7fc01affe4b1                                                                                                                                                                                                                                                
I0202 15:46:23.986353 10953 OICompiler.cpp:604] Relocated data 0x7fba5291b460 to 7fc01affe4e0                                                                         
I0202 15:46:23.986377 10953 OICompiler.cpp:331] findSymbol(cookieValue) = Synth 7fc01aefe050                                                                          
I0202 15:46:23.986392 10953 OICompiler.cpp:331] findSymbol(dataBase) = Synth 7fc01aefe040                                                                             
I0202 15:46:23.986403 10953 OICompiler.cpp:331] findSymbol(dataSize) = Synth 7fc01aefe048                                                                             
I0202 15:46:23.986420 10953 OICompiler.cpp:331] findSymbol(logFile) = Synth 7fc01aefe058
...
I0202 15:46:24.101312 10953 OIDebugger.cpp:1954] Writing buffer 0x7fba5281b000, bufsz 1052672 into target 0x7fc01aefe080                                              
I0202 15:46:24.101773 10953 OIDebugger.cpp:1954] Writing buffer 0x55ecbdc56d58, bufsz 8 into target 0x7fc01aefe040                                                    
I0202 15:46:24.101800 10953 OIDebugger.cpp:1954] Writing buffer 0x55ecbdc56738, bufsz 8 into target 0x7fc01aefe048                                                    
I0202 15:46:24.101814 10953 OIDebugger.cpp:1954] Writing buffer 0x55ecbdc56d68, bufsz 8 into target 0x7fc01aefe050                                                    
I0202 15:46:24.101828 10953 OIDebugger.cpp:1954] Writing buffer 0x7ffe24a25eec, bufsz 4 into target 0x7fc01aefe058                                                    
I0202 15:46:24.101855 10953 SymbolService.cpp:667] Found funcDesc for oid_test_case_std_pair_uint64_uint32                                                            
I0202 15:46:24.101883 10953 OIDebugger.cpp:2136] Generating prologue for argument 'arg0', using probe at 0x7fc01aefe080                                               
I0202 15:46:24.101898 10953 OIDebugger.cpp:2168] INT3 at offset 16                                                                                                    
I0202 15:46:24.101912 10953 OIDebugger.cpp:1954] Writing buffer 0x7ffe24a25db0, bufsz 64 into target 0x7fc01aefe000                                                   
I0202 15:46:24.101996 10953 OID.cpp:364] Compilation Finished (8574745740 nsecs)                                                                                      
I0202 15:46:24.102063 10953 OIDebugger.cpp:2614] Seizing thread: 10950                                                                                                
I0202 15:46:24.102115 10953 OIDebugger.cpp:2672] stopTarget: Interrupting pid 10950 state: 0                                                                          
I0202 15:46:24.114239 10953 OIDebugger.cpp:148] (stopregs: ) dumpRegs: pid: 10950 rip 7fc01e5507fa rbp: 7ffdf922cff0 rsp 7ffdf922ae00 rdi 0 rsi 0 rdx 7ffdf922ae90 rbx 7ffdf922ae01 rcx 7fc01e5507fa r8 1 r9 7ffdf922ad90 r10 7ffdf922ae90 r11 246 r12 7ffdf922ae90 r13 0 r14 7ffdf922ae90 r15 7fc0247e7040 rax fffffffffffffdfc orig_rax e6 
eflags 246                                                                                                                                                            
I0202 15:46:24.114282 10953 OIDebugger.cpp:96] Type entry Func oid_test_case_std_pair_uint64_uint32 Args: arg0                                                        
I0202 15:46:24.114298 10953 SymbolService.cpp:667] Found funcDesc for oid_test_case_std_pair_uint64_uint32                                                            
I0202 15:46:24.114326 10953 OIDebugger.cpp:1357] Replay Instruction Base 7fc01b3fde00                                                                                 
I0202 15:46:24.114336 10953 OIDebugger.cpp:1365] Orig instructions for trap 0x4750df will get saved at 0x7fc01b3fde00                                                 
I0202 15:46:24.114352 10953 OIDebugger.cpp:1553] Patching function oid_test_case_std_pair_uint64_uint32 @0x4750df                                                     
I0202 15:46:24.114370 10953 OIDebugger.cpp:442] contTargetThread: About to PTRACE_CONT pid 10950 detach = 0                                                           
I0202 15:46:24.114389 10953 OIDebugger.cpp:934] processTrap: About to PTRACE_CONT pid 10950                                                                                                                                                                                                                                                  
I0202 15:46:24.114430 10953 OIDebugger.cpp:938] processTrap1: Error in PTRACE_CONT for pid 10950 No such process(3) status SLEEP (0)                                  
I0202 15:46:24.114444 10953 OIDebugger.cpp:951] About to wait for any child process                                                                                   
I0202 15:46:24.122257 10953 OIDebugger.cpp:979] Stop sig: 5
I0202 15:46:24.127225 10953 OIDebugger.cpp:984] PTRACE_GETSIGINFO pid 10950 signo 5 code 128 status 0 stopped: 0
I0202 15:46:24.127247 10953 OIDebugger.cpp:1891] removeTrap removing int3 at 4750df 
I0202 15:46:24.127264 10953 OIDebugger.cpp:567] 
Process Function Trap for pid=10950: Trap Vector Entry @0x4750df

I0202 15:46:24.127290 10953 OIDebugger.cpp:548] Entry: arg addr: 7ffdf922aed0
I0202 15:46:24.127301 10953 OIDebugger.cpp:1954] Writing buffer 0x7ffe24a25ff0, bufsz 8 into target 0x7fc01aefe002
I0202 15:46:24.127317 10953 OIDebugger.cpp:666] processTrap: redirect pid 10950 to address 0x7fc01aefe000 tInfo: Trap Vector Entry @0x4750df 7fc01aefe000 0
I0202 15:46:24.127331 10953 OIDebugger.cpp:689] Inserting Trapinfo for pid 10950
I0202 15:46:24.127342 10953 OIDebugger.cpp:693] processFuncTrap about to PTRACE_CONT pid 10950
I0202 15:46:24.127357 10953 OIDebugger.cpp:699] Finished Function Trap processing
I0202 15:46:24.127370 10953 OIDebugger.cpp:934] processTrap: About to PTRACE_CONT pid 10950
I0202 15:46:24.127384 10953 OIDebugger.cpp:951] About to wait for any child process  
I0202 15:46:24.127405 10953 OIDebugger.cpp:979] Stop sig: 4
I0202 15:46:24.127418 10953 OIDebugger.cpp:984] PTRACE_GETSIGINFO pid 10950 signo 4 code 2 status 0 stopped: 0
E0202 15:46:24.127427 10953 OIDebugger.cpp:1263] contAndExecute: Explictly unhandled signal. Forwarding Illegal instruction
I0202 15:46:24.127444 10953 OIDebugger.cpp:934] processTrap: About to PTRACE_CONT pid 10950
I0202 15:46:24.127475 10953 OIDebugger.cpp:938] processTrap1: Error in PTRACE_CONT for pid 10950 No such process(3) status RUNNING (2)
I0202 15:46:24.127486 10953 OIDebugger.cpp:951] About to wait for any child process  
I0202 15:46:25.713783 10953 OIDebugger.cpp:979] Stop sig: 5
I0202 15:46:25.713814 10953 OIDebugger.cpp:984] PTRACE_GETSIGINFO pid 10950 signo 5 code 1541 status 0 stopped: 0
I0202 15:46:25.713821 10953 OIDebugger.cpp:1043] Thread exiting!! pid 10950
I0202 15:46:25.713824 10953 OIDebugger.cpp:1053] extended wait About to cont pid 10950
E0202 15:46:25.713838 10953 OIDebugger.cpp:929] Thread list is empty: the target process must have died. Abort...

I was writing a simple test case for a comparator template parameter with a multimap and hit an apparent problem where oid instruments a specified function as it looks like the function is not inlined (well, we find a symbol for it) but it looks like the function has been inlined also. We therefore never execute the actual function instructions and never fire a probe.

Take the following code:

#include <map>
#include <iostream>
#include <chrono>
#include <thread>

struct CustomComparator {
        bool operator()(const int& lhs, const int& rhs) const {
                return lhs < rhs;
        }
};

struct Foo {
    std::multimap<int, int, CustomComparator> m1;
};

void doit(Foo& f) {
        for (auto p : f.m1) {
                        std::cout << p.first << " " << p.second << std::endl;
        }
}

int main(int argc, char *argv[])
{
        Foo foo;

        for (int i = 0; i < 10; ++i) {
                  foo.m1.insert(std::pair<int, int>(i, i*2));
        }

        while (1) {
                std::this_thread::sleep_for(std::chrono::seconds(10));
                doit(foo);
        }
}

Instrument the doit function entry point:

# /tmp/oid-integration-83I6pk]$ cat /tmp/mm.oid
entry:_Z4doitR3Foo:arg0

oid attaches to the running process but we never hit the probe even though the test program is dutifully printing outpput ever 10 secs. Debug shows the oid has instrumented the entry point that it found:

I0119 02:54:47.985539 3135718 OIDebugger.cpp:1365] Orig instructions for trap 0x400df0 will get saved at 0x7fde32de6e00
I0119 02:54:47.985548 3135718 OIDebugger.cpp:1553] Patching function _Z4doitR3Foo @0x400df0

And we see that this is the "correct" address for the doit function:

# nm mm | grep doit
0000000000400df0 T _Z4doitR3Foo

(gdb) pipe disas doit | head -5
Dump of assembler code for function doit(Foo&):
   0x0000000000400df0 <+0>:     push   %rbp
   0x0000000000400df1 <+1>:     push   %r15
   0x0000000000400df3 <+3>:     push   %r14
   0x0000000000400df5 <+5>:     push   %r12

However there are no calls to doit() in main:

(gdb) pipe disas main | grep doit
(gdb)

We can see that doit() has actually been inlined in main():

(gdb) info symbol doit
main + 1162 in section .text of /home/jonhaslam/mm

If we needed more data we can see that it is inlined using tools/inline_stats.py:

sqlite> select * from inlined_subprograms where function_name LIKE '%doit%';
_Z4doitR3Foo|1

We need to be smarter when selecting if we can instrument a function by using inlining information.

If the target process exits cleanly (I'm not talking about abnormal terminations here) before oid has reaped any data then oid spews a bunch of errors:

Attached to pid 365577
E0228 19:09:29.950443 365790 OIDebugger.cpp:929] Thread list is empty: the target process must have died. Abort...
E0228 19:09:29.950973 365790 OIDebugger.cpp:1846] Execute: Couldn't poke text: No such process
E0228 19:09:29.950996 365790 OIDebugger.cpp:1862] Couldn't continue target pid 365577 (Reason: No such process)
E0228 19:09:29.951006 365790 OID.cpp:421] Failed to remove instrumentation...
E0228 19:09:29.951015 365790 OIDebugger.cpp:929] Thread list is empty: the target process must have died. Abort...
E0228 19:09:29.951263 365790 OIDebugger.cpp:2017] process_vm_readv() error: No such process (3)
E0228 19:09:29.951273 365790 OIDebugger.cpp:2835] Failed to read data segment from target process
E0228 19:09:29.951283 365790 OID.cpp:434] Problems processing target data

oid needs to deal with clean terminations. It's almost bizarre that we've got those far along with this being the case!

Similar to how "cli_options" applies to OID integration tests, we should have a way of specifying custom options for OIL tests.

See integration test enums.anonymous and typedefs.anonymous

The container .toml should carry all but the treebuilder logic for a type. However, due to implementation details, there is currently only one code provided per enum that treebuilder expects. This means that although the container .toml can have different code bodies but the same treebuilder, only one of the codes is used. Fix this by carrying the ContainerInfo around, perhaps in a shared pointer, rather than using the string map.

On huge types we sometimes see the problem of uintptr_t being stubbed as struct alignas(8) uintptr_t { char dummy[8];};. This is a huge problem as we use uintptr_t for our pointer tracking/data saving, and this makes it a different type to std::uintptr_t.

Unfortunately I have no public repro for this yet, but I'm going to open an issue for tracking anyway,

std::vector<bool> is specialised to take just over 1 bit per element. OI currently has no way to represent the sub-byte space taken by each element, resulting in incorrect data.

Test case OidIntegration.std_vector_bool_empty:

[
  {
    "name": "a0",
    "typePath": "a0",
    "typeName": "vector<bool, std::allocator<bool> >",
    "isTypedef": false,
    "staticSize": 40,
    "dynamicSize": 0,
    "pointer": 140727152166496,
    "length": 0,
    "capacity": 0,
    "elementStaticSize": 1
  }
]

Where the expected elementStaticSize is 0.125 or 1/8 of a byte, which cannot be represented in the current output of OID (elementStaticSize is a size_t).

### Test case OidIntegration.std_vector_bool_some:

[
  {
    "name": "a0",
    "typePath": "a0",
    "typeName": "vector<bool, std::allocator<bool> >",
    "isTypedef": false,
    "staticSize": 40,
    "dynamicSize": 64,
    "pointer": 140724981840768,
    "length": 3,
    "capacity": 64,
    "elementStaticSize": 1,
    "members": [
      {
        "name": "",
        "typePath": "bool[]",
        "typeName": "bool",
        "isTypedef": false,
        "staticSize": 1,
        "dynamicSize": 0
      },
      {
        "name": "",
        "typePath": "bool[]",
        "typeName": "bool",
        "isTypedef": false,
        "staticSize": 1,
        "dynamicSize": 0
      },
      {
        "name": "",
        "typePath": "bool[]",
        "typeName": "bool",
        "isTypedef": false,
        "staticSize": 1,
        "dynamicSize": 0
      }
    ]
  }
]

Here we see the same issue with elementStaticSize, and this issue propagates to each element's staticSize. The inverse issue also occurs with dynamicSize, which is reported as 64 bytes but is actually 8 bytes (64 bits), due to being unable to multiply by a sub-bit size.

Proposed solutions

• Work in bits for all measures of size, then divide and round the output at the end.
• Work in bits in the output, though this is kind of nonsensical as anything which can be addressed must be byte aligned.
• Provide a float output... This wouldn't be too bad as things can only be 1/8s and should therefore be represented accurately, but we'd have to be significantly more careful about precision than we are so far. As I understand it we'd be fine on current AMD64 with 48 bit addressing + 3 bits for eights as doubles have a 53 bit mantissa, but it is more fuss than natural numbers.

OIDebugger is currently being reworked to be more tolerant of some of the more bizarre and arcane behaviour that can be observed in large, very aggressive multithreaded applications. One thing that is not being implemented currently is ptrace's group stops as AFAIK we have not had to deal with it yet. Yes, it needs doing and this is a reminder.

As per the description, we need a test writing for folly::sorted_vector_set.

Currently OIGenerator.cpp forms a relationship between ObjectIntrospection::getObjectSize<T>(...) and ObjectIntrospection::getObjectSizeImpl<T>(...) by symbol name because the Impl isn't in the debug due to being a weak symbol with no body.

By passing the -fstandalone-debug to the compiler we cause the function to appear in the DWARF, as well as ensuring all the necessary type information is available even through pointers. This removes the need for the quite fragile symbol matching.

This test is currently skipped due to lack of confidence in the expected values. These values would need to be validated and the code for std::unordered_map potentially corrected.