ioi / isolate Goto Github PK

Couldn't we avoid having a pool of UIDs, leading to potential races and adding constraints to the execution, by passing the CLONE_NEWUSER flag to clone(2) and always using the same uid and gid when dropping privileges?
If this change sounds good to you, I can work on it, it shouldn't take long!

Or is there any known kernel issues that could potentially arise?

CONFIG_CGROUP_MEM_RES_CTRL has been renamed to CONFIG_MEMCG in 3.6 (referenced at the end of isolate.1

Correct signal handling

int main(void) {
    return 1 / 0;
}

$ /var/lib/isolate/4/box/a.out
[1]    18386 floating point exception (core dumped)  /var/lib/isolate/4/box/a.out

$ isolate --box-id 4 --cg --run -- /box/a.out
Caught fatal signal 8

Result: SIGFPE is correctly caught by isolate.

Incorrect signal handling

#include <signal.h>
int main(void) {
    return raise(SIGFPE);
}

$ /var/lib/isolate/4/box/a.out
[1]    18433 floating point exception (core dumped)  /var/lib/isolate/4/box/a.out

$ isolate --box-id 4 --cg --run -- /box/a.out
OK (0.002 sec real, 0.064 sec wall)

Result: SIGFPE is raised but nothing is caught by isolate, as seen by the OK message.
I have the same result with other signals (eg. SIGINT). Always OK.

Any idea why this would happen? Is there a bug in isolate or is this expected behavior?

Hi,

when running isolate --cg --init as root user I get following error:

Failed to create control group /sys/fs/cgroup/memory/box-0/: No such file or directory

Running above command without --cg option returns me box path (as expected):

/var/local/lib/isolate/0

Why am I getting this error?

Is it possible to start software with interactive CLI and type commands from terminal?
I tried to run isolate --run -- /usr/bin/php -a (interactive PHP shell), but my input makes no effect.

There is a race condition when a lot of users are constantly using isolate in parallel, even if they try to play nice with each other.

Let's say you want to get a new box for your program. The "proper" way to do that is to list the directories in /var/lib/isolate/ and to --init with an available ID. But between the time you check that an ID is available and the time you call --init, another program might have done a --init for the same ID and you get the same cgroup and sandbox for both.

I see two solutions for that:

• The first one is really simple to implement in isolate, it would be an option like --require-empty (name suggestions welcome) that makes isolate --init fail if the box folder already exists (mkdir returns EEXIST).
• The second one is to move the responsibility of assigning box IDs from the calling program to isolate. When calling --init, you'd be able to specify, say, --box-id new. Then, isolate would try to find an available box id by looking at the directories in /var/lib/isolate, would try to mkdir it and if it gets EEXIST, it tries to find a new one until the directory has been properly created. Then, when --init is over, we print the box id along with the path of the box, so that --run and --cleanup can be called with this ID.

I'm okay to implement either solution as long as you give me approval for one of them and comments on the API & short/long option names (and any other implementation detail you might think of).

After the sandbox quits, there is a tmp directory with content of the /tmp folder in the sandboxed process. This directory should be chowned back to orig_uid, orig_gid when cleanup_ownership is set (basically the same way as it currently works for a box directory). Nowadays, it is practically impossible to remove tmp directory by non-root user, because the owner is the sandboxed-uid (60000 etc.).

isolate ... -q100,100 ... produces

Invalid numeric parameter: 100,100

This is probably due to the the fact that variable optarg points to 100,100 when calling opt_uint at line 1036.

Hello.

I have the next simple code:

 #include <iostream>
 #include <fstream>
 using namespace std;

   int main()
   {
   int n;
   cin >> n;

   cout << " Number is: " << n;

   return 0;
  }

I want to evaluate this program and to redirect the standard input and the standard output. I ran the next command: ./isolate --cg --meta=/tmp/muc2.txt --stdin=test1.in --stdout=fis1.out --cg-mem=3000 --time=2 --run -- prog
After I ran this command , in my sandbox folder the file "fis1.out" was generated and contains the results. My question is, I can redirect the output file to be created in another folder (not in the sandbox folder) ?

Thanks

I'm trying to read the cache miss counter of a program running in isolate with perf, but the result is always zero. I suspect this might have something to do with the program being in a cgroup, but that's just a thought. Is there a way to make perf work?

isolate -b 1 --cg --init
perf stat -e L1-dcache-load-misses isolate -b 1 --cg --cg-timing --run /usr/bin/ls

Hello!
I downloaded and successfully set up isolate on my 2nd computer (I've used it before on my server). But now I'm encountering problems with --dir option.
The easiest sample. I have file ab in /something directory i use this command

sudo isolate --dir=/something/:rw --run /something/ab

I get this error:
execve("/something/ab"): No such file or directory

When I do -vv option I get:

Started proxy_pid=21774 box_pid=21775 box_pid_inside_ns=2
Binding ./box on box (flags 1006)
Binding /bin on bin (flags 1007)
Binding /dev on dev (flags 1003)
Binding /lib on lib (flags 1007)
Binding /lib64 on lib64 (flags 1007)
Mounting proc on proc (flags 5)
Binding ./tmp/tmp on tmp (flags 1006)
Binding /usr on usr (flags 1007)
Binding //something/ on something/ (flags 1006)
Passing environment:
	LIBC_FATAL_STDERR_=1

What I'm doing wrong?

Thanks in advance

Best Regards

This is useful for maintaining distribution-specific packages (e.g. it's enough to change v1.1 to v1.2 in the packaging script instead of having to look for the specific commit id)

When trying to compile a java program with command

isolate --full-env --cg -p --time=5 --meta=log.txt --stderr=error.txt --run -- javac code.java

I receive the following message
execve("/usr/bin/javac") No such file or directory

Note:
$ which javac
/usr/bin/javac

Hi,
I have question about using isolate for compiling.

Aside from running a, for example C, code with isolate, should I also compile it with isolate? If yes, how is it done?

Thanks in advance,
Herman

The workaround for piping stderr to stdout suggested here: f7b60f2

doesn't seem to work:

$ isolate --box-id 0 --cg -p --stderr=/dev/fd/1 --stdout=stdout --run -- /bin/bash -c "echo a >&2; echo b; echo a >&2; echo b"     
OK (0.000 sec real, 0.019 sec wall)
$ cat /var/lib/isolate/0/box/stdout 
b
b

Any idea why that is the case?

https://github.com/ioi/isolate/blob/master/isolate.c#L121

This prevents me from doing things in the box if there was some problem in the sandbox, and I have to add checks everytime in my code to see if the permissions are the right ones. What's the rationale?

Debian/stable has cgroup-tools supersedding cgroup-bin and cgroup-lite, this should be reflected in the documentation

I plan to use isolate on a web service to run arbitrary code. So i want to run programs without having a fixed number of sandboxes.

Currently --init only refers to a certain box. Is it possible to init to a new sandbox if an empty one is not available ?

This way any arbitrary number of threads (from my web server) can run their programs in their own sandboxes.

Thanks

On Ubuntu 18.10, isolate quits with error message Caught fatal signal 11 when run with infinite stack limit (i.e. with no --stack parameter given). Strangely, however there seems to be some interplay with other parameters. More precisely, the above always happens in the following cases:

• when used inside CMS for evaluating a submission
• when run as isolate --env=HOME=. --run -- ./test (where test just prints hi); the same happens with other HOME paths

In both cases everything is fine once we supply --stack=<large integer>. However, isolate works just fine even with infinite stack limit in the following cases:

• when used inside CMS for compiling a submission
• when run as isolate --run -- ./test

While it is possible to work around this issue for usage in CMS (cf. e.g. this branch) it would be preferrable if this could be fixed in isolate directly.

(To be entirely precise, this happens in the German fork of CMS; however, this uses just the same version of isolate as the current cms-dev main branch. The above error occurs on two different computers, one of them with a completely fresh Ubuntu install.)

Hi. I have the next program:

#include
using namespace std;

int main()
{
int x[1000000];
for(int i = 0; i < 500000000; ++i);

cout << "Good life";

return 0;

}

I want to measure the memory and the time for my program. I used the following command:

./isolate --cg --meta=/tmp/log4.txt --time=1 --run -- prog

In my log4.txt i have the next values:
time:0.844
time-wall:0.898
max-rss:3052
csw-voluntary:1
csw-forced:96
cg-mem:536

What represent the value 536 (cg-mem) ? If I make a short memory calculation of my array (x[1000000]) i will have ~3500kb of memory.

How I can measure the memory ?
I'm using version 1.4.1.

Thanks.

when i run this command,
isolate --run -- '/usr/bin/g++ -lm -W -o code code.cpp'

the following error comes.
execve("/usr/bin/g++ -lm -W -o code code.cpp" ): No such file or directory

But if I run without quote it runs properly but number of process count have to increased from 1

The wall-time option is wall-time and the output is time-wall. Could this be changed to be more consistent?

Can isolate limit the size of output file?

Repro:

1. Initialize a box with quota:

   obj/isolate/isolate -b1 -q50000,50 --init
   

2. Create an executable e.g. test; copy to /tmp/box/1/box.

3. Run:

   obj/isolate/isolate -b1 --full-env --dir=/etc:rw --dir=/tmp/box/1/box:rw --run -- ./test
   

4. Try to clean up:

   obj/isolate/isolate -b1 --cleanup
   

5. Expected: no output and box gets cleaned up, actual:

   Cannot rm /tmp/box/1: directory not empty
   

6. Upon another investigation, it turns out that after each run (step 3), the mounted chroot root directory inside /tmp/box/1 is not empty, so the cleanup command cannot remove it.

Tested on

Linux XXX 4.4.0-36-generic #55-Ubuntu SMP XXX x86_64 x86_64 x86_64 GNU/Linux

The same steps worked perfectly on Ubuntu 14.04.

cc: @dragoon20

Hi,

I just upgraded to Linux kernel 4.15 and it looks like isolate is broken because there is no more /sys/fs/cgroup/cpuacct here. I checked the man page and it looks like it has been replaced by cpu. I don't know too much about control groups and how isolate uses them, take this with a grain of salt.

Or can I or run java through bash indirectly?

Hi. I tried to run "--init" command but I got the next error:
Error in config file, line 0: Configuration is not complete.

I ran the sandbox in an ubuntu-16.04.4-desktop-amd64 environment. The command that I try to execute in terminal is : sudo ./isolate --init

Thanks.

Hello, I'm trying to use isolate in an alpine-linux docker image. Alpine linux uses musl for the libc implementation, and there's a difference in ftw.h implementation between musl and glibc: FTW_CONTINUE enum flag only exists in glibc/ftw.h.

FTW_CONTINUE is used in util.c so I cannot directly compile this unless you define FTW_CONTINUE explicitly, but I'm a newbie in linux and I don't know the proper way to fix the problem, so I'm creating the issue.

Hi, I have a question about running Java programs with isolate.

I am trying to run a simple Java program, these are steps I made:

1. Init isolate:

   $ isolate --cg --init
   

2. Save Main.class file to /var/local/lib/isolate/0/box/ which is compiled version of following program (I compiled it on the same machine, not using isolate):

   import java.io.BufferedReader;
   import java.io.InputStreamReader;
   import java.io.IOException;
   
   public class Main {
     public static void main(String[] args) throws IOException {
       BufferedReader reader = new BufferedReader(new InputStreamReader(System.in));
       String name = reader.readLine();
       System.out.println("hello, " + name);
     }
   }

3. Save input.txt to /var/local/lib/isolate/0/box/ with following content:

   world
   

4. Run program with isolate:

   $ isolate -p --cg -i input.txt --run /usr/local/jdk1.8.0_91/bin/java Main
   hello, world
   OK (0.066 sec real, 0.248 sec wall)
   

That's fine, but I as soon as I add memory limit, for example 256MB, I get following error:

$ isolate -p -m 256000 --cg -i input.txt --run /usr/local/jdk1.8.0_91/bin/java Main
Error occurred during initialization of VM
Could not reserve enough space for code cache
Exited with error status 1

Of course I can add much higher memory limit and everything will work fine:

$ isolate -p -m 4096000 --cg -i input.txt --run /usr/local/jdk1.8.0_91/bin/java Main 
hello, world
OK (0.063 sec real, 0.105 sec wall)

but how can I then limit my Java code to 256MB?

So, my question is: How do I need to run Java code using isolate if I want to set some memory limit?

What are the best practices for running Java code with isolate?

Thanks in advance,
Herman

Not sure if this is intended or a bug, so let's hear from you. I think from 7dbc4e1, isolate might fail or succeed with the same arguments just by switching the order of two --dirs. For example: isolate [...] --dir=a/b=/a/b --dir=a=/a succeeds, but switching the two, fails.

When using --fsize (-f) flag programs output to stdout is also being limited to specified value (eq. 1MB). Is this expected behaviour and if it is why is it expected behaviour?

Thanks!

Running statically compiled C programs failed.

# isolate --run -- "./a" --time=1
execve("./a"): No such file or directory

The source code of the program is a simple while(1){}; wrapped in int main(), compiled with gcc -static -no-pie.

The existence of the a executable in the directory isolate was invoked can be confirmed.

If a BTRFS filesystem is mounted, and included in the sandbox using the --dir= command, quotas will fail silently on this filesystem. This allows arbitrary large files to be written to disk. I believe the quotas are still applied to the other filesystems, if any.

The preferable behavior would be that isolate sets up subvolumes and applies quotas on them.

Hello. I'm a newbie at container & sandboxing environment and linux kernel. I'm trying to run isolate inside docker container (alpine-linux base image) and it fails to do a clone() call at run() function. The test used simple hello world program written in C.

#include <stdio.h>
int main(int argc, char* argv[]) {
  printf("hello world\n");
  return 0;
}

Here's what console says:

~ # isolate --run hello
Cannot run proxy, clone failed: Operation not permitted

I wonder what does this situation mean and how can I fix it. I haven't tested whether this occurs on other linux images (i.e. whether this is a docker problem), and I'll update the issue as soon as I conduct further tests.

Environment: Docker 17.12.0 (community edition) running on macOS high sierra 10.13.3
Base image: gliderlabs/alpine:3.7
Installed package:

• gcc, alpine-sdk, git using apk add command
• isolate using cloned repository.

Note: Since alpine uses different implementation for libc (not glibc but musl) I had to define FTW_CONTINUE in util.c, which is the enum value that only exists in glibc version of ftw.h.

Introduction of the proxy process in Isolate 1.5 broke non-group time limits. I do not have a quick fix for that, so I am posting this issue as a warning. Please use either --cg-timing or an older version of Isolate.

When isolate --run is executed in quick succession, the later commands take sigificantly longer than the first.

To reproduce, save this as a shell script, run it, and check the real time (as output by time).

isolate --init
time isolate --run -- /usr/bin/echo 1
isolate --cleanup
isolate --init
time isolate --run -- /usr/bin/echo 2
isolate --cleanup

It is expected that all executions take roughly the same. However, the first execution takes around 0.05s and the second takes around 0.1s on my machine. On another machine the first execution also takes around 0.05s but the second takes at least 0.8s!

Both machine are x86_64 and running latest Arch Linux and isolate built from commit 7f55e36. The first machine (the less severe one) is using usual Arch Linux kernel 4.10.5. The second machine is inside lxc with linux-userns kernel 4.10.1.

I'm talking about this line:

Rationale: no need for full paths when specifying command to run.

Not sure if it's a good idea, although.

I want to calculate the memory usage of my program and reproduce the a memory limit exceeded. No luck so far.

To run isolate I'm using this:

./isolate --cg -i $SANDBOX_INPUT --cg-mem=$SANDBOX_MEMORY --cg-timing -t $SANDBOX_TIME -M run-stats --run ./main

I've always get time limit but not memory limit.

To reproduce the memory limit I'm using this:

int main() {
    while(true) new int;
}

I installed isolate using Makefile and i'm trying to init it. Every time I'm receiving

Cannot open /usr/local/etc/isolate: No such file or directory

How to fix it and create sandbox ?

Hi,

I have a question about using isolate. How can I run Java and Python programs with it? I have read all documentation you guys provided, but I really don't understand how can I run Java and Python programs.

Thanks in advance.

Best,
Herman

Can't configure Isolate
After running
root@computer:/usr/local/etc/isolate# make isolate
and
root@computer:/usr/local/etc/isolate# ./isolate-check-environment --execute

and everything has a PASS on it, I can't run :

root@computer:/usr/local/etc/isolate# ./isolate --init
The next message appears, but can't figure out what is the missing configuration

"Error in config file, line 0: Configuration is not complete"

see the image under:

The --init and --cleanup options need to be called everytime we want an isolated program to be run. The only thing they do is to create the directories and initialize the cgroups and the quotas. These operations are not really costly in time. Why couldn't we run them directly before and after running the program with --run?

The benefits would be:

• A simpler interface (only call isolate <options> -- <cmd>)
• No need for the caller to manage a pool of box ids themself
• No need to remember to init and cleanup the boxes, this is done automatically
• Simplify the code a lot (no more "modes", warnings for boxes that were not created, etc.)

I can work on that if you think this change is worth it, I already have prepared some changes.

Is there a rationale behind having these two separate options?

I am just reopening issue #29 since I am really concerned with this issue. In my server, the delay is about 0.4-0.5s, which is significant in contests which has usual time limit 1 sec.

Before I saw that issue, I just decided to use another sandbox only for C/C++ (but using isolate for other languages, because they are slow itself), but it would be good for me to use isolate for all languages.

I'm not sure whether this is a kernel issue. We checked the time of successive isolate executions in two machines on Linode, using this bash script.

sumtime=0
for ((i=1; i<=10; i++)); do
   isolate --init --box-id=$(echo $i % 100 | bc) >/dev/null
   /usr/bin/time -f %e -o /tmp/memotime isolate --run -b $(echo $i % 100 | bc) -- /bin/echo 1>/dev/null 2>/dev/null
   isolate --cleanup -b $(echo $i % 100 | bc) >/dev/null
   echo iteration $i : $(cat /tmp/memotime)
   sumtime=$(echo $sumtime + $(cat /tmp/memotime) | bc)
done
echo total time = $sumtime

Machine 1: working bad

root@ubuntu:~# uname -a
Linux ubuntu 4.9.36-x86_64-linode85 #1 SMP Thu Jul 6 15:31:23 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu:~# bash test.sh
iteration 1 : 0.06
iteration 2 : 0.49
iteration 3 : 0.50
iteration 4 : 0.52
iteration 5 : 0.47
iteration 6 : 0.45
iteration 7 : 0.48
iteration 8 : 0.48
iteration 9 : 0.43
iteration 10 : 0.46
total time = 4.34

Machine 2: working nicely

root@ubuntu:~# uname -a
Linux ubuntu 4.9.36-x86_64-linode85 #1 SMP Thu Jul 6 15:31:23 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu:~# bash test.sh
iteration 1 : 0.01
iteration 2 : 0.04
iteration 3 : 0.06
iteration 4 : 0.04
iteration 5 : 0.06
iteration 6 : 0.06
iteration 7 : 0.04
iteration 8 : 0.07
iteration 9 : 0.06
iteration 10 : 0.06
total time = .50

Also I executed the script on a Docker container (on my Mac): working bad

root@8b518e8841a9:~# uname -a
Linux 8b518e8841a9 4.9.36-moby #1 SMP Wed Jul 12 15:29:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
root@8b518e8841a9:~# bash test/m.sh
iteration 1 : 0.08
iteration 2 : 0.97
iteration 3 : 0.90
iteration 4 : 0.91
iteration 5 : 0.89
iteration 6 : 0.91
iteration 7 : 0.87
iteration 8 : 0.94
iteration 9 : 0.96
iteration 10 : 0.99
total time = 8.42

Maybe the difference is just Linode's fault, but I think this difference is notable.

I'm suffering insufficient memory error when running Java code by isolate within docker container
[gist link]

Running code Main.java is a simple hello world program, and it works without error if executed outside of isolate.

I have no clue where the error had begun and how to deal with it. I wonder anybody's using isolate within docker container and runs non-native binary/codes (e.g. Java, Python, etc.)

Environment detail

• Macbook Pro 2016
• Docker 18.03.1-ce-mac65 (latest at the moment I'm writing)
  • Reserved CPU: 4
  • Reserved Memory: 4.0GiB
  • Swap: 1.0GiB
• latest isolate (commit 60caa49)
• Docker image openjdk:8u151-jre-alpine3.7

I also tried Java VM options such as

• stack size option -Xss16M
• heap size option -Xms16M, -Xmx16m
• code cache size -XX:ReservedCodeCacheSize=16M