Compiling on macOS fails about sandsifter

[floatingatoll]

-       ((ucontext_t*)p)->uc_mcontext.gregs[IP]+=UD2_SIZE;
+       ((ucontext_t*)p)->uc_mcontext->__ss.__rip+=UD2_SIZE;

Variations of this seem to, at the very least, remove more warnings. Still doesn't compile and I'd be surprised if a straight swap out here worked.

[jaketesler]

I got this to compile, but I'm not sure if the executable still works as intended.
This is ONLY for 64-bit – I didn't take the time to convert these changes for 32-bit, but I imagine it wouldn't be too difficult.

---

Change 1: Dummy Struct

Modify the following snippet:

sandsifter/injector.c

Lines 174 to 177 in 8375e61

	struct {
	uint64_t dummy_stack_hi[256];
	uint64_t dummy_stack_lo[256];
	} dummy_stack __attribute__ ((aligned(PAGE_SIZE)));

Delete the entire dummy_stack definition. Then, copy the following (to anywhere before the inject_state typedef struct definition on line 90!):

typedef struct {
	uint64_t dummy_stack_hi[256];
	uint64_t dummy_stack_lo[256];
} dumb_stack;
static dumb_stack dummy_stack __attribute__ ((aligned(PAGE_SIZE)));

Then, in the __x86_64__ inject_state definition, modify the struct:

...
	uint64_t rsp;
	dumb_stack dumb;
} state_t;
...

and the inject_state definition:

...
	.rsp=0,
	.dumb.dummy_stack_lo=0,
	.dumb.dummy_stack_hi=0
};
...

We also need to modify the stack reset on line 766:

inject_state.dumb.dummy_stack_lo[0]=0;

And lastly, we need to modify the inline asm here:

sandsifter/injector.c

Lines 777 to 815 in 8375e61

	#if __x86_64__
	__asm__ __volatile__ ("\
	mov %[rax], %%rax \n\
	mov %[rbx], %%rbx \n\
	mov %[rcx], %%rcx \n\
	mov %[rdx], %%rdx \n\
	mov %[rsi], %%rsi \n\
	mov %[rdi], %%rdi \n\
	mov %[r8], %%r8 \n\
	mov %[r9], %%r9 \n\
	mov %[r10], %%r10 \n\
	mov %[r11], %%r11 \n\
	mov %[r12], %%r12 \n\
	mov %[r13], %%r13 \n\
	mov %[r14], %%r14 \n\
	mov %[r15], %%r15 \n\
	mov %[rbp], %%rbp \n\
	mov %[rsp], %%rsp \n\
	jmp *%[packet] \n\
	"
	: /* no output */
	: [rax]"m"(inject_state.rax),
	[rbx]"m"(inject_state.rbx),
	[rcx]"m"(inject_state.rcx),
	[rdx]"m"(inject_state.rdx),
	[rsi]"m"(inject_state.rsi),
	[rdi]"m"(inject_state.rdi),
	[r8]"m"(inject_state.r8),
	[r9]"m"(inject_state.r9),
	[r10]"m"(inject_state.r10),
	[r11]"m"(inject_state.r11),
	[r12]"m"(inject_state.r12),
	[r13]"m"(inject_state.r13),
	[r14]"m"(inject_state.r14),
	[r15]"m"(inject_state.r15),
	[rbp]"m"(inject_state.rbp),
	[rsp]"i"(&dummy_stack.dummy_stack_lo),
	[packet]"m"(packet)
	);

The asm on line 813 should read:

[rsp]"m"(inject_state.dumb.dummy_stack_lo),

---

Change 2: uc_mcontext

We need to modify uc_mcontext references for macOS. Make all of the following changes:

• Line 853:

((ucontext_t*)p)->uc_mcontext->__ss.__rip+=UD2_SIZE;

• Line 866:

(uintptr_t)uc->uc_mcontext->__ss.__rip-(uintptr_t)packet-preamble_length;

• Lines 883-885:

uc->uc_mcontext->__ss = fault_context->__ss;
uc->uc_mcontext->__ss.__rip =(uintptr_t)&resume;
uc->uc_mcontext->__ss.__rflags &=~TF;

---

Change 3: header file

Create a new file injector.h with the following contents:

extern char debug, resume, preamble_start, preamble_end;

Then modify the extern definition on line 366.

sandsifter/injector.c

Line 366 in 8375e61

extern char debug, resume, preamble_start, preamble_end;

Change line 366 to:

char debug, resume, preamble_start, preamble_end;

---

Change 4: pthread modifications

This code was borrowed from https://yyshen.github.io/2015/01/18/binding_threads_to_cores_osx.html.
Copy the following block underneath the top #include section.

#define SYSCTL_CORE_COUNT   "machdep.cpu.core_count"

typedef struct cpu_set {
  uint32_t    count;
} cpu_set_t;

static inline void
CPU_ZERO(cpu_set_t *cs) { cs->count = 0; }

static inline void
CPU_SET(int num, cpu_set_t *cs) { cs->count |= (1 << num); }

static inline int
CPU_ISSET(int num, cpu_set_t *cs) { return (cs->count & (1 << num)); }

int sched_getaffinity(pid_t pid, size_t cpu_size, cpu_set_t *cpu_set)
{
  int32_t core_count = 0;
  size_t  len = sizeof(core_count);
  int ret = sysctlbyname(SYSCTL_CORE_COUNT, &core_count, &len, 0, 0);
  if (ret) {
    printf("error while get core count %d\n", ret);
    return -1;
  }
  cpu_set->count = 0;
  for (int i = 0; i < core_count; i++) {
    cpu_set->count |= (1 << i);
  }

  return 0;
}

int pthread_setaffinity_np(pthread_t thread, size_t cpu_size,
                           cpu_set_t *cpu_set)
{
  thread_port_t mach_thread;
  int core = 0;

  for (core = 0; core < 8 * cpu_size; core++) {
    if (CPU_ISSET(core, cpu_set)) break;
  }
  printf("binding to core %d\n", core);
  thread_affinity_policy_data_t policy = { core };
  mach_thread = pthread_mach_thread_np(thread);
  thread_policy_set(mach_thread, THREAD_AFFINITY_POLICY,
                    (thread_policy_t)&policy, 1);
  return 0;
}

Then, modify line 1344 to read:

if (pthread_setaffinity_np(0, sizeof(mask), &mask)) {

Lastly, append this to the #include section

#include <mach/thread_policy.h>

---

Change 5: Python

The sifter.py file is designed for Linux and attempts to gather CPU info via /proc/cpuinfo, which doesn't exist on macOS. So, I made a small change to allow running on a Mac.

sandsifter/sifter.py

Lines 681 to 684 in 8375e61

	def get_cpu_info():
	with open("/proc/cpuinfo", "r") as f:
	cpu = [l.strip() for l in f.readlines()[:7]]
	return cpu

Modify the get_cpu_info() function to:

def get_cpu_info():
    p =  subprocess.Popen(['/usr/sbin/sysctl','-a'], stdout=subprocess.PIPE)
    p.wait()
    cpu = [l.strip('machdep.cpu.') for l in str(p.stdout.read()).split('\n') if 'machdep.cpu' in l]
    return cpu

---

Lastly, run make. If you encounter this error:

/usr/include/ucontext.h:43:2: error: The deprecated ucontext routines require _XOPEN_SOURCE to be defined

Run make with the CFLAGS variable:

CFLAGS="-D_XOPEN_SOURCE" make

---

Hope this helps! Also hopefully the injector binary still functions as expected. Please comment below with results or findings.

[orf]

Same, I've tried with the bundled MacOS llvm as well as the latest.

[floatingatoll]

Pull #6 fixes one of these errors.

Changing <ucontext.h> to <sys/ucontext.h> fixes another.

Changing uc->mcontext.gregs to uc->mcontext->gregs fixed another.

Unfortunately, there's no ->gregs member in uc_mcontext64 on Darwin:

  _STRUCT_MCONTEXT64
  {
          _STRUCT_X86_EXCEPTION_STATE64   __es;
          _STRUCT_X86_THREAD_STATE64      __ss;
          _STRUCT_X86_FLOAT_STATE64       __fs;
  };

Because, apparently, this is dependent on glibc's register reading stuff:

https://fossies.org/dox/glibc-2.25/structmcontext__t.html

This ancient article talks about trying to persuade an old OS X kernel to divulge register state:

http://web.archive.org/web/20090627062246/http://www.matasano.com/log/1100/what-ive-been-doing-on-my-summer-vacation-or-it-has-to-work-otherwise-gdb-wouldnt/

Ideally there's a better way these days, but no guarantee there.

[floatingatoll]

Happily, ->mcontext64->__ss is nothing but registers:

  #if __DARWIN_UNIX03
  #define _STRUCT_X86_THREAD_STATE64      struct __darwin_x86_thread_state64
  _STRUCT_X86_THREAD_STATE64
  {
          __uint64_t      __rax;
          __uint64_t      __rbx;
          __uint64_t      __rcx;

[floatingatoll]

cpu_set_t is GNU-only:

https://www.gnu.org/software/libc/manual/html_node/CPU-Affinity.html

You can fake the goal here, which is I believe to test one core only, using:

http://jesperrasmussen.com/2013/03/07/limiting-cpu-cores-on-the-fly-in-os-x/

Storing and restoring __ss might be as simple as using plain struct assignment:

-       memcpy(uc->uc_mcontext->gregs, fault_context->gregs, sizeof(fault_context->gregs));
+        uc->uc_mcontext->__ss = fault_context->__ss;

And then there's this:

injector.c:778:24: error: invalid operand for inline asm constraint 'i'
        __asm__ __volatile__ ("\

[floatingatoll]

Which I tried to resolve with 'static dummy', and then managed to get:

cc -c injector.c -o injector.o -Wall
fatal error: error in backend: 32-bit absolute addressing is not supported in 64-bit mode

So we're making progress! But this is where I run out of gas, because I can't debug assembler 32/64 differences, sorry.

https://stackoverflow.com/questions/35834937/error-in-backend-32-bit-absolute-addressing-is-not-supported-in-64-bit-mode

[IComplainInComments]

So it seems the issues we are facing for macOS deal with the XNU kernel architecture as a whole.

1. Instead of using gLibC, we need to use the built-in XNU kernel PTrace calls.

2. We need to modify CPU_SET to use Apples alternative. Which can be described here: http://yyshen.github.io/2015/01/18/binding_threads_to_cores_osx.html.

So in general this code will work, but will need a specific branch for any MACH based micro kernel.

[IComplainInComments]

So I am working on CPU_SET, and it seems all the functions needed can be called from the Affinity API (According to http://yyshen.github.io/2015/01/18/binding_threads_to_cores_osx.html)

So far the first code block works fine, and compiles with no errors. Also with these code blocks in, the current CPU_SET errors go away, but... this code block needs rewritten it seems.

`int pthread_setaffinity_np(pthread_t thread, size_t cpu_size,
cpu_set_t *cpu_set)
{
pthread_cond_t mach_thread;
int core = 0;

for (core = 0; core < 8 * cpu_size; core++) {
    if (CPU_ISSET(core, cpu_set)) break;
}
printf("binding to core %d\n", core);
thread_affinity_policy_data_t policy = { core };
mach_thread = pthread_mach_thread_np(thread);
thread_policy_set(mach_thread, THREAD_AFFINITY_POLICY,
                  (thread_policy_set)&policy, 1);
return 0;

}`

@floatingatoll Would you be able to debug this code? I unfortunately do not know enough C to try.

These are the errors I get:

injector.c:1383:5: error: use of undeclared identifier 'thread_affinity_policy_data_t' thread_affinity_policy_data_t policy = { core }; ^ injector.c:1384:17: error: assigning to 'pthread_cond_t' (aka 'struct _opaque_pthread_cond_t') from incompatible type 'mach_port_t' (aka 'unsigned int') mach_thread = pthread_mach_thread_np(thread); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ injector.c:1385:5: warning: implicit declaration of function 'thread_policy_set' is invalid in C99 [-Wimplicit-function-declaration] thread_policy_set(mach_thread, THREAD_AFFINITY_POLICY, ^ injector.c:1386:43: error: use of undeclared identifier 'policy' (thread_policy_set)&policy, 1); ^ injector.c:1385:36: error: use of undeclared identifier 'THREAD_AFFINITY_POLICY' thread_policy_set(mach_thread, THREAD_AFFINITY_POLICY, ^ injector.c:1396:7: warning: implicit declaration of function 'sched_setaffinity' is invalid in C99 [-Wimplicit-function-declaration] if (sched_setaffinity(0, sizeof(mask), &mask)) {
EDIT: The Affinity API documentation is here: https://developer.apple.com/library/content/releasenotes/Performance/RN-AffinityAPI/index.html

It says Leopard, but it is still supported and used in the current macOS (Including 10.13)

[floatingatoll]

Sorry, I haven't the foggiest how to *write* C code. It looks like you need to start with solving this, though:

error: use of undeclared identifier 'thread_affinity_policy_data_t'

Since providing the right header here should fix many of these errors. - R.

…

On Wed, Aug 2, 2017 at 6:08 PM, IComplainInComments < ***@***.***> wrote: So I am working on CPU_SET, and it seems all the functions needed can be called from the Affinity API (According to http://yyshen.github.io/2015/ 01/18/binding_threads_to_cores_osx.html) So far the first code block works fine, and compiles with no errors. Also with these code blocks in, the current CPU_SET errors go away, but... this code block needs rewritten it seems. `int pthread_setaffinity_np(pthread_t thread, size_t cpu_size, cpu_set_t *cpu_set) { pthread_cond_t mach_thread; int core = 0; for (core = 0; core < 8 * cpu_size; core++) { if (CPU_ISSET(core, cpu_set)) break; } printf("binding to core %d\n", core); thread_affinity_policy_data_t policy = { core }; mach_thread = pthread_mach_thread_np(thread); thread_policy_set(mach_thread, THREAD_AFFINITY_POLICY, (thread_policy_set)&policy, 1); return 0; } ` @floatingatoll <https://github.com/floatingatoll> would you possibly be able to rewrite this code to compile? Im getting these errors: injector.c:1383:5: error: use of undeclared identifier 'thread_affinity_policy_data_t' thread_affinity_policy_data_t policy = { core }; ^ injector.c:1384:17: error: assigning to 'pthread_cond_t' (aka 'struct _opaque_pthread_cond_t') from incompatible type 'mach_port_t' (aka 'unsigned int') mach_thread = pthread_mach_thread_np(thread); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ injector.c:1385:5: warning: implicit declaration of function 'thread_policy_set' is invalid in C99 [-Wimplicit-function-declaration] thread_policy_set(mach_thread, THREAD_AFFINITY_POLICY, ^ injector.c:1386:43: error: use of undeclared identifier 'policy' (thread_policy_set)&policy, 1); ^ injector.c:1385:36: error: use of undeclared identifier 'THREAD_AFFINITY_POLICY' thread_policy_set(mach_thread, THREAD_AFFINITY_POLICY, ^ injector.c:1396:7: warning: implicit declaration of function 'sched_setaffinity' is invalid in C99 [-Wimplicit-function-declaration] if (sched_setaffinity(0, sizeof(mask), &mask)) { — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFqDHX31KAXo6yiHCadvBTPJq7HdkYFks5sUR2mgaJpZM4OmW_m> .

[IComplainInComments]

Alright it seems like adding #include <mach/thread_policy.h> does indeed do as you said @floatingatoll

There are now just 8 total errors that need hammered out.

The new Errors go as follows:
injector.c:1385:17: error: assigning to 'pthread_cond_t' (aka 'struct _opaque_pthread_cond_t') from incompatible type 'mach_port_t' (aka 'unsigned int') mach_thread = pthread_mach_thread_np(thread);

and

injector.c:1387:42: error: invalid operands to binary expression ('int (*)()' and 'thread_affinity_policy_data_t' (aka 'struct thread_affinity_policy')) (thread_policy_set)&policy, 1); ~~~~~~~~~~~~~~~~~~~^~~~~~~

[IComplainInComments]

Found a fix, delete this code block:

`
int pthread_setaffinity_np(pthread_t thread, size_t cpu_size,
cpu_set_t *cpu_set)
{
pthread_cond_t mach_thread;
int core = 0;

for (core = 0; core < 8 * cpu_size; core++) {
    if (CPU_ISSET(core, cpu_set)) break;
}
printf("binding to core %d\n", core);
thread_affinity_policy_data_t policy = { core };
mach_thread = pthread_mach_thread_np(thread);

thread_policy_set(mach_thread, THREAD_AFFINITY_POLICY,
(thread_policy_set)&policy, 1);
return 0;
}`

6 Errors left! So all we need to do now is figuring out how to get the Program to use Pthread from the kernel!

[IComplainInComments]

Also stuck at injector.c:818:24: error: invalid operand for inline asm constraint 'i' __asm__ __volatile__ ("\ ^

:/

[floatingatoll]

There might be a hint here: rust-lang/rust#41337

The operand for the "i" constraint must be constant, so in general this

won't work. I have an asm fix earlier in the thread that might help you here.

…

On Wed, Aug 2, 2017 at 10:06 PM, IComplainInComments < ***@***.***> wrote: Also stuck at injector.c:818:24: error: invalid operand for inline asm constraint 'i' __asm__ __volatile__ ("\ ^ :/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFqDJxPNrDwGrf3OexdnB1yc1OFB5qDks5sUVVCgaJpZM4OmW_m> .

[IComplainInComments]

@jaketesler Thanks for writing the missing C code we needed! I really should brush up on my coding hehe