subutai-attic / subos Goto Github PK

Peer installed from scratch.

After Wi-Fi was disabled - IP address has not been changed

Also SS Console not accessible because of network disabled.

While downloading and importing after the download it would be nice to show some progress in this process. Now that we have content length available from Kurjun we can actually show download progress based on the percentage of the file already pulled down. This can also be used by the agent to send back information to SS about progress which can give better feedback to the user.

I created x-peer env from Hub. Later I grew it. Then I destroyed it (all from Hub, though might be not related).
After this i don't see then env and its containers neither on Hub nor on SS.
However, i still can see p2p swarm and associated network interface on both peers.

mng-net interface contains two IP-addresses in heartbeat.

{
    "response": {
        "type": "HEARTBEAT",
        "hostname": "rh1464490740.gw.intra.lan",
        "id": "FA0D5C60CFD76571FF43B3CD03D7F4698B534F9A",
        "arch": "AMD64",
        "instance": "LOCAL",
        "interfaces": [{
            "interfaceName": "eth1",
            "ip": "10.0.3.15"
        }, {
            "interfaceName": "eth2",
            "ip": "192.168.56.135"
        }, {
            "interfaceName": "nat",
            "ip": "10.10.0.254"
        }, {
            "interfaceName": "mng-net",
            "ip": "10.10.10.254"
        }, {
            "interfaceName": "mng-net",
            "ip": "10.10.10.160"
        }, {
            "interfaceName": "wan",
            "ip": "192.168.0.58"
        }],
        "containers": [{
            "hostname": "management",
            "id": "01EC3A5068DB6C458EB34E3C44DEC64DA4007272",
            "arch": "AMD64",
            "interfaces": [{
                "interfaceName": "eth0",
                "ip": "10.10.10.1"
            }],
            "status": "RUNNING",
            "templateName": "openjre8",
            "name": "management"
        }]
    }
}

This is something that we need to work on all together and will impact the installer people as well: @Lezh1k @btemir @p1gmale0n @soffokl @tzununbekov @crioto.

Here's the general situation:

VirtualBox has various issues with the MAC address it exposes over WiFi to enable bridged mode to get a DHCP lease. This makes it so MOST WiFi networks do not support our current configuration: hotels, conferences, and other public networks. The VM networking should be setup to support both NAT and bridged making the right decision at install time AND there after to account for users roaming from network to network. This capability needs to be built into our product.

Some of this functionality around intelligent VM networking needs to be outside of the VM guest and can be handled by the tray. The VM guest needs to know what environment it's sitting in too. We need a robust situation that handles all changes and all scenarios.

There might also be an impact on p2p and p2p might also do somethings for us with UPnP. *Please make sure to have a general online discussion about this topic and post the conclusions here. *

Previously we has switched from old service (on port 6881) to a new one (port 6882). Now we need to switch back and shutdown all services, that was working on port 6882 completely. But we need to do this in a proper way, to avoid conflicts between new and old services and clients, so this need to be done in a highly coordinated manner.

We need to add ability connect to container using SSH from outside.
Port forwarding should be based on p2p connections to delimit environments.

We want to be able to have users import dockerfile generated templates directly into their peer. It would be nice if this facility is integrated straight into subos. This way it can be used by the SS UI to add docker templates.

Mac laptop on home network created test env m3-single with 3 containers after a fresh install about 1hr ago. The environment worked well with EZ-SSH.

Then switched laptop to 3G on phone. SS console came up nicely with new IP. However the old environment stopped working without ability to EZ-SSH. Then to check I created a new environment ap2-single and EZ-SSH worked just fine with it, while the old still remained disabled.

Then I restarted the Subutai VM. Both environments worked perfectly with EZ-SSH. Something must be left stale when switching.

If we have several environments on one resource host, containers from these environments can ping each other. This is unacceptable and should be fixed.

I think this was because I was using the default small container size btw. I got the following error:

I guess we need to go larger?

To facilitate work with Subutai directory layout, we should move application and data path to single directory - /opt/subutai-mng

We need to catch problem with absent iptables rule after creating container.
In most cases everything works fine, but sometimes rules missing.

this command was returning IPv6 for a prettly long period until it finally returned correct ip

We are using v4.1.1-dirty, while current version 4.4.1.
This is very important component and need to be updated.

Moved from p2p issues: subutai-io/p2p#153

I created env using 3 peers:

• Askat Peer (behind NAT)
• Sydyk Peer (behind NAT)
• AWS Germany (accessible directly)

I can successfully ping containers to each other (see ping-ssh-check.txt). But SSH fails between them (askat-peer-p2p-logs.txt). Seems this issue is mostly between peers behind NAT.

Note: for each command's response I waited ~20 sec. If there was none I typed Ctrl-C.

ping-ssh-check.txt
askat-peer-p2p-logs.txt

All CDN artifacts should be digitally signed by their owners.

We need to start integrating embedded Gorjun to replace Kurjun.
Also we need to add a swtich to subutai import/close command to be able to import/clone template by its id.

Right now embedded Kurjun downloads CDN artifacts from from nearest CDN nodes. After implementing #486 we should enable peer-to-peer downloads of large public artifacts from all CDN nodes and other embedded Gorjun instances. Implementing digital signatures is a requirement as well for this feature: #492.

The download progress delay is probably due to going out to external standalone Kurjun for the template, but then we see the progress occur, however it is very fast obviously because we're now pull down from local kurjun from subutai cli command.

This does not seem to be happening with the other templates or maybe I am not noticing it. There should be a way for us to consistently track download progress even if being downloaded from standalone by our internal nginx proxy.

1. Take any ubuntu dockerfile and use our ubuntu master docker image in the docker index as the base
2. Run docker on a resource host just for the purpose of generating bring up a normal docker container
3. Stop and squash the file system snapshots of the docker container converting it into our template format

This might be a better way to generate templates from dockerfiles is what I am thinking. WDYT?

I tried to executesubutai import management -t <token> on RH where the management is located.
After executing command we see following output:

2016-03-14 10:50:04,021 | INFO  | h for user admin | CommandProcessor                 | 190 - io.subutai.core.executor.impl - 4.0.0.RC7-SNAPSHOT | Sending:
{
  "request": {
    "type": "EXECUTE_REQUEST",
    "id": "50E24A0DF7A1A6EC25410DCD5EBD1EE0E1B88073",
    "commandId": "6c40a158-c1fe-4222-8629-6cfc937d854c",
    "workingDirectory": "/",
    "command": "subutai import management -t eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI5MjI2NmRhNi0zMjEwLTQwZDYtYTUzNy1jMzMzMWIzNjQ1M2EiLCJpc3MiOiJpby5zdWJ1dGFpIn0.d3icdeUytDAqQ27a6a17NFgUT3YgskhUTOd7S30gwuc",
    "stdOut": "RETURN",
    "stdErr": "RETURN",
    "runAs": "root",
    "timeout": 30,
    "isDaemon": 0,
    "pid": 0
  }
}
2016-03-14 10:50:04,154 | INFO  | tp1812568228-208 | RestServiceImpl                  | 193 - io.subutai.core.executor.rest - 4.0.0.RC7-SNAPSHOT | Requested commands for RH 50E24A0DF7A1A6EC25410DCD5EBD1EE0E1B88073. Requests: [{
  "request": "-----BEGIN PGP MESSAGE-----\nVersion: BCPG v1.52\n\nhQEMA169HuDhuIBzAQgAoBtWIvhxcPVGQCRJy+GBbv2vsnMwnjdXizOQRiC2+JpC\na6ak2HECz91E+b5vskgU6fd/KF8QbvnxJfFf9B6qZ+Mqd4PVXhkRX9EOz1wmoKrf\nwgbrX4Ce1LfGdNmplE+DNw44R4m0Uh4pFb53NyW/axgFI5UoJu0QLsyTIiARHXdR\nzVV8wgpJktS3ziNh0cs3CHEi3yj+wvcTpZpiu07P+daVzgmJ4xau1pd/P/K4kjUp\n5nSQmtcALbfaRkhxY5UD/bs5zO5fR/bnBXWYnm9pzhZhQ1s71YwL5D9vqV3yOItS\nzWvu97EEE9LoDCECRXc6ug+0G/2IDYrrsK2cnTbmgdLCWgE+zbdkaV91Lkx9u7gg\nXo4JK4+nXnVYX1/ReGA1EpD4zOJSxDxZHzHkjon9Z2hYIzMmGUS5OG0UQqcNB5Oy\nf58Mf5d4DElf/ArEH2siWJr5diqwswG6avf3i/bWeMl2UJGTLg52c6yPFksGqdzo\nWWB37M25ahtQvyDYCrcOQ55WGlJmaP81hDd0z5BU+dEgU1XQQnH2j30QGevAhcE2\ntBjg0K1wu0R1l7S4X3HyrcPmETnunblBJtbltNxK9tsvzdDJLDWxLgV+2xm93+wX\nNXxHjxi0kiiN6W55gHyaYls/4S981E09jl5H3uPsZ9pa27V4bwIJVj8RBRgfqQkR\ncSx+UCo4rwSunUSVwD7qJ9TQ614pjaL62Pf3ICCLaGqhXVWCw5Y9slHycn5lLW+F\nvD/gcy8uIj6FxvH5gpEL3qzlTcr+v65SnbZmQ077xXMeuCYgMmYfabmVZo7UCH3A\nn0mwJIDwdt5X8ZZBnqquyE6s/y9uCMF/tSvtnLaYZvS0vKaLnjL45v9QK7tIlwjM\niN8VrWzh2fr3SCmCxsR6eaQR7+4HOxq/L2w8FO45hUZmhlYVhap1XOh4oMlbYQbx\n7u+SEFlNwuVep4cXhnxY4pyIt9jZWY3pPQL0/0/QMK1GA2uPWcSbYi0ZMtyt5EBo\nyC2UA/fFYwem6D78UG6EQ9o8YAECbOTZpw2wRS1/hStgzr9MZaK4rKECaw0iw1y7\n0964a/al3ZhnVLulvZKEbEVxgh058BBY3fRi9MQKO3m7J5EnJcOK4HZR4Po7/hcf\n1/DL5pNlxqJfN8z8gwbxXzOliwjFAVTWTmXz4oKI+jsdMeM1FhjBhoUNVgk4Ztuu\nKqlRZB2ZSX03LvO4InfPChLuBpJLuum918TF0jvdOO2OTWNlLeaUwb9Xt8k/voXD\nuwlZYT5lrW7U60KdWEe2ajC144DNVOA9jOLNDCvvK8LZklsAQNUCg++JdS3a2E+M\nwcLP7Cmfx0MKO6q2C6KWJenQaUXboJ8NWc4ca1p6Op/Rl4jSMxNbKdbjc2RZOogL\nQ6Zsa+wzQqVPmj0j\n\u003dUHC4\n-----END PGP MESSAGE-----\n"×,
  "hostId": "50E24A0DF7A1A6EC25410DCD5EBD1EE0E1B88073"
}]
2016-03-14 10:50:04,487 | INFO  | tp1812568228-216 | CommandProcessor                 | 190 - io.subutai.core.executor.impl - 4.0.0.RC7-SNAPSHOT | Received:
{
  "type": "EXECUTE_RESPONSE",
  "id": "50E24A0DF7A1A6EC25410DCD5EBD1EE0E1B88073",
  "commandId": "6c40a158-c1fe-4222-8629-6cfc937d854c",
  "pid": 0,
  "responseNumber": 1,
  "stdErr": "time\u003d\"2016-03-14 10:50:04\" level\u003dinfo msg\u003d\"[Importing management]\" \ntime\u003d\"2016-03-14 10:50:04\" level\u003dfatal msg\u003d\"Encrypting stdin.txt, exit status 2\" \n",
  "exitCode": 1
}

How to reproduce:

1. create fresh RH
2. subutai import management
3. execute command in karaf console:localpeer:management-key-exchange

I would like the SS console to expose settings to users to control the update/upgrade policy settings for the SS console (itself) and the RH snap. These settings however MUST be set within the subos to control the update/upgrade process because the SS management console cannot handle updating itself or the snap, it can ONLY expose a UI for users to modify such settings.

Right now we're going to have a very simple policy which either enables or disables automatic updates along with a simple update check interval. We will elaborate on this so it's more robust as time progresses.

This will mean different things for release branches verses non-release branches:

• For release versions on tags ONLY new releases pushed to Gorjun will be used to upgrade and the default check interval should be every week on Thursday 7am GMT.
• For non-release versions on branches any updated snap or management template to Gorjun could trigger an upgrade, the default check interval should be daily at 7am GMT.

Based on these settings the subos should register a cron job that executes a Gorjun check for the snap or management template and if the installed MD5 differs from what is available in Gorjun it initiates the upgrade (or downgrade if we have a recall).

Faced following problem with cleaning up environment, should be fixed:

2016-03-17 05:10:33,055 | INFO  | pool-83-thread-1 | CommandProcessor                 | 190 - io.subutai.core.executor.impl - 4.0.0.RC7-SNAPSHOT | Sending:
{
  "request": {
    "type": "EXECUTE_REQUEST",
    "id": "4E242D2426E436750C95BB5762433E4360BA8527",
    "commandId": "379ea399-4afb-46b6-a2ea-77f5b270c3a6",
    "workingDirectory": "/",
    "command": "subutai cleanup 100",
    "stdOut": "RETURN",
    "stdErr": "RETURN",
    "runAs": "root",
    "timeout": 30,
    "isDaemon": 0,
    "pid": 0
  }
}
2016-03-17 05:10:34,357 | INFO  | tp1383013813-146 | CommandProcessor                 | 190 - io.subutai.core.executor.impl - 4.0.0.RC7-SNAPSHOT | Received:
{
  "type": "EXECUTE_RESPONSE",
  "id": "4E242D2426E436750C95BB5762433E4360BA8527",
  "commandId": "379ea399-4afb-46b6-a2ea-77f5b270c3a6",
  "pid": 0,
  "responseNumber": 1,
  "stdErr": "time\u003d\"2016-03-17 05:10:34\" level\u003dinfo msg\u003d\"[95785492-7da1-4181-bb91-1c41dbd8a0f6 destroyed]\" \ntime\u003d\"2016-03-17 05:10:34\" level\u003dfatal msg\u003d\"config.Agent.DataPrefix + /var/subutai-network/vni_reserve delete vni, open /var/lib/apps/subutai/current//var/subutai-network/vni_reserve: no such file or directory\" \n",
  "exitCode": 1
}

There is a problem with containers in cross-peer environment: they are not accessible to each other. Sometimes a container doesn't see its gateway too: 192.168.1.1.

Testing environment:

• One physical machine with two virtual RH.
• RHs get IP from a wifi router.
• Each RH has management.
• RHs are created with: (1) autobuild -v -t 201 (2) autobuild -v -t 202.

Steps to reproduce:

1. Create Appscale environment with 1 container
2. Go to RH
3. grep vlan /mnt/lib/lxc/container_name/config
4. sudo subutai proxy del 'vlan_id'

(amd64)ubuntu@rh1464247688:~$ grep vlan /mnt/lib/lxc/483a9a87-6961-42f1-b3c9-0484effd253b/config 
#vlan_id = 104
(amd64)ubuntu@rh1464247688:~$ sudo subutai proxy del 104

the same error appear when trying execute sudo subutai proxy add 104 -h container_name

To facilitate installation and update process for SS management we should try to move all extras to dependency list in management package. It will significantly improve SS template preparation workflow and also add ability to update additional components, such as java, influxdb, etc.

Right now after the network changes or a machine wakes up to re-initialize it's network a manual heartbeat is required to show the peer as online. Otherwise we have to wait a long time. It would be nice if when subos detects these changes it makes the management server induce a peer heartbeat.

Route 53 supports health checks for any endpoint in configured environment such as our CDN. We should add simple check of nodes availability with failover option.

Try to upload deb packages to standalone Kurjun. If it works, we need to change apt source list and remove forwarding from reverse proxy.

Steps to reproduce:

1. deploy RH, import Management
2. create environment
3. try to run any command via agent inside the container in env

Seems like this is bug in agent, sometimes it is works, sometime it's not. Agent's restart helps. Cannot define exact conditions for this case. Also, this bug leads to environment creation failure due to ssh commands inaccessibility. Perhaps some dependencies are required before agent starts.

We need to implement simple mechanism that will enable user to upgrade containers in existing environments. It might be "Update is available" notification message in UI with button bind to apt upgrade command in environment. In this case using apt cache on peer makes sense.

We need to add ability of setting thresholds of particular resources to fire alerts.

For example, {id:'management', resourceType:'ram',  threshold:'80'} should fire RAM alert when currently used RAM exceeded 80% of RAM quota.

The id filed used for identifying container and holds ID of container.
The possible values of resourceType field are "ram" , "cpu" , "opt" , "home" , "var" and "rootfs".
The possible values of threshold field may vary from "0" till "100".
"0" value of threshold field should disable alert firing.

Steps:

1. Register two peers to each other. Peers are not within LAN, and not registered to Hub.
2. Create environment using these peers: 4 containers based on the master template; 2 containers on each peer.
3. Check ping from each container to other ones.

Result:

1. Ping is successful to a container on same peer only.
2. Containers on remote peer are not accessible.

SS version:

• Commit ID: aeb32b52d00d78e965c87d63040c405e5df49e77
• SS version: 4.0.0-RC13-SNAPSHOT
• RH version: 4.0.0-RC13
• P2P version: 4.0.0-RC12. Packet version: 4

Notes:

• I could reproduce using AWS hosted peers: Ireland, Germany, California, Virginia. Tested between all of them and reproduces.
• The bug may be reproducible with home peers too, if they are not in LAN. But I am not sure b/c I could not test that. (Don't have peers accessible directly to each other).
• I guess, this bug was introduced somewhere in last month. I tested same scenario in April when was working with AppScale cluster. Then connection was OK to remote containers.

Impossible to destroy container if consistency check is not valid, log file is attached

1. tried to create container, but it somehow created no consistent containers
2. couldn't destroy them
   output.txt

We should prevent execution double parallel update processes.
Need to modify script for that

Since we can use kurjun as apt repository, we should move all deb packages from s3 to standalone kurjun. There is a known problem with some of subutai's big-data packages, which has invalid deb format - it should be fixed.

We have an updated standalone Kurjun with some changes in REST. Need to upgrade our existing Kurjun repositories and make corresponding changes in agent and CLI

In the monitoring UI we see a number of environment interfaces:

As you can see there are about 7 different environment p2p interfaces present with some gateways. But when you look at the environments there's just one environment visible:

Here's the "About" page on the peer. Note that this was @NikolaiB's peer which I just snuck into just now 🎱 :

Can not ssh to container nor from hub nor from tray. To reproduce:

1. Download and run win dev installer
2. Launch to SS. About page:
   
3. Register peer to hub
4. Create environment (ceph environment was created)
   5.Go to Environments-containers, click SSH button. Result:
   
5. Go to Hub-Peer-Environments-Container and click SSH button.
   
6. From tray: "Failed to run SSH because container is not ready".

LOGS:

1. karaf logs

karaflogs2016-06-04 09-51-12.txt

1. Agent logs

agent_log.txt

1. p2p logs from rh
   p2p_log.txt
2. On host (Ubuntu-VMware -win)
   

Since we have different geographical regions included in kurjun.cdn.subutai.io CDN, it might be a good idea to change geoip mode to latency-based Route53 routing. We should test and switch to it, if it is more efficient approach.

After rebooting resource host, management server could not connect to agent.
In logs we see:

:SubutaiLogEvent{timeStamp=1458817528105, loggerName='io.subutai.core.hostregistry.impl.HostRegistryImpl', renderedMessage='Error checking host SubutaiHost{peerId=7D5BC3CE5822D5A34D5CCBD54F4E479B0EB4F5C6, lastHeartbeat=0}', stackTrace='javax.ws.rs.ProcessingException: java.net.NoRouteToHostException: NoRouteToHostException invoking http://10.10.10.121:7070/ping: No route to host at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:557) at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:539) at org.apache.cxf.jaxrs.client.WebClient.doResponse(WebClient.java:1151) at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1101) at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:894) at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:865) at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:331) at org.apache.cxf.jaxrs.client.WebClient.get(WebClient.java:357) at io.subutai.core.hostregistry.impl.HostRegistryImpl.updateHost(HostRegistryImpl.java:307) at io.subutai.core.hostregistry.impl.HostRegistryImpl$2.run(HostRegistryImpl.java:287) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

After stop and start management container we cannot even ping container IP.
We need to solve this problem.