arkoscloud / core Goto Github PK

(Note that this bug should not result in a crash, rather the user should be prompted with a simple error message that the operation could not be completed.)

arkOS 0.8.3 Crash Report

Running in prod
System: Linux arkos 4.4.47-1-ARCH #1 SMP Mon Feb 6 18:48:50 MST 2017 armv7l GNU/Linux

Platform: armv7l Unknown
Python version 3.6.0
Config path: settings.json

Loaded applicatons:
basicsite
paperwork
haste
syncthing
wallabag
fs-samba
db-mariadb
db-sqlite3
wordpress
radicale
grav
xmpp
nextcloud
mailpile
lychee

Request: POST /api/system/domains

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/lib/python3.6/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/flask/views.py", line 149, in dispatch_request
return meth(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/kraken/auth.py", line 135, in decorator
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/kraken/frameworks/roles.py", line 139, in post
d.add()
File "/usr/lib/python3.6/site-packages/arkos/system/domains.py", line 54, in add
signals.emit("domains", "post_add", self)
File "/usr/lib/python3.6/site-packages/arkos/signals.py", line 87, in emit
x.trigger(data, crit)
File "/usr/lib/python3.6/site-packages/arkos/signals.py", line 47, in trigger
self.func(data)
File "/var/lib/arkos/applications/xmpp/backend.py", line 54, in add_domain
services.get("prosody").restart()
File "/usr/lib/python3.6/site-packages/arkos/system/services.py", line 176, in restart
"sudo arkosctl svc status {0}".format(self.name))
arkos.system.services.ActionError: ('svc', 'The service failed to restart. Please check sudo arkosctl svc status prosody')

(Note that this bug should not result in a crash, rather the user should be prompted with a simple error message that the device could not be mounted.)

arkOS 0.8.3 Crash Report

Running in prod
System: Linux arkos 4.4.48-2-ARCH #1 SMP Mon Feb 13 19:47:53 MST 2017 armv7l GNU/Linux

Platform: armv7l Unknown
Python version 3.6.0
Config path: settings.json

Loaded applicatons:
basicsite
fs-afp
syncthing
fs-samba
db-mariadb
db-sqlite3
wordpress
radicale
fs-nfs
xmpp
nextcloud
mailpile
lychee

Request: POST /api/mounts

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/lib/python3.6/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/flask/views.py", line 149, in dispatch_request
return meth(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/kraken/auth.py", line 135, in decorator
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/kraken/frameworks/sharers.py", line 91, in post
data.get("password"), data["read_only"]
File "/var/lib/arkos/applications/fs-samba/fileshare.py", line 43, in add_mount
s.add()
File "/usr/lib/python3.6/site-packages/arkos/sharers.py", line 133, in add
self.mount()
File "/var/lib/arkos/applications/fs-samba/fileshare.py", line 184, in mount
os.strerror(ctypes.get_errno())))
Exception: Failed to mount test: Invalid argument

e.g. http://mywebsite.com/blog

In short:

• Instead of creating one nginx server block for every site, a base nginx server block is created for each domain. This will also facilitate automatic Let's Encrypt validation
• When a new site is created, the base block for the domain is added. Everything required in the server block goes into a Location block for the subfolder
• The desired subfolder path is passed to the website on pre_install, as the subdomain path is required to customize addtoblock items that the website already requires
• nginx server blocks then go from being named like /etc/nginx/sites-available/mysite to /etc/nginx/sites-available/mysite.com:80

https://github.com/nicolargo/pymdstat

arkOS 0.8.3 Crash Report

Running in prod
System: Linux arkos 4.4.45-1-ARCH #1 SMP Fri Jan 27 18:52:49 MST 2017 armv7l GNU/Linux

Platform: armv7l Unknown
Python version 3.6.0
Config path: settings.json

Loaded applicatons:
basicsite
fs-afp
syncthing
fs-samba
db-mariadb
db-sqlite3
wordpress
cmsjs
grav
xmpp
nextcloud
haste

Request: PUT /api/websites/xxx

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/lib/python3.6/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/flask/views.py", line 149, in dispatch_request
return meth(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/kraken/auth.py", line 135, in decorator
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/kraken/frameworks/websites.py", line 79, in put
site.edit(data.get("new_name"))
File "/usr/lib/python3.6/site-packages/arkos/websites.py", line 517, in edit
self._edit(newname)
File "/usr/lib/python3.6/site-packages/arkos/websites.py", line 522, in _edit
self.id))
File "/usr/lib/python3.6/site-packages/nginx.py", line 488, in loadf
with open(path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/etc/nginx/sites-available/xxx'

https://git.coderouge.co/arkOS/mockldap/repository/archive.tar.gz?ref=0.3.0#egg=mockldap-0.3.0
https://git.coderouge.co/arkOS/python-cryptsetup/repository/archive.tar.gz?ref=1.7.2#egg=pycryptsetup-1.7.2

The nexcloud app seems to install just fine, but adding a nextcloud website afterwards, the installer fails

I got ithe following message when trying to create a personal Nextcloud;
'FileExistsError' object is not subscriptable
I deleted and installed nextcloud in the appstore

root@arkos ~]# arkosctl svc status mysqld
Traceback (most recent call last):
File "/usr/bin/arkosctl", line 11, in
sys.exit(cli())
File "/usr/lib/python3.6/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/click/core.py", line 676, in main
_verify_python3_env()
File "/usr/lib/python3.6/site-packages/click/_unicodefun.py", line 118, in _verify_python3_env
'for mitigation steps.' + extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult http://click.pocoo.org/python3/for mitigation steps.

This system lists a couple of UTF-8 supporting locales that
you can pick from. The following suitable locales where
discovered: en_US.utf8
[root@arkos ~]#

I added en_US.utf8 but still Nextcloud won't install

At the outset, support an arkOS server for peakwinter/DittyDNS. Add more popular solutions subsequently.

Currently there is no way for an app to perform specific actions right after it has been installed, save for 1) a hacky use of Signals, or 2) the creation of a PKGBUILD. And with both of those options, there is no way to provide custom information to the app as it is being installed and/or loaded for the first time, like there is upon website creation.

Ideally there would be an on_install.py in the base of an application that is executed after the app is successfully installed and loaded for the first time. This script could be provided with extra_vars, much like websites are now. This would eliminate the need for hacky things like what is being done presently when the Radicale app is initialized.

It would be great to see where the project stands with regards to features and stability (alpha, beta, release...). A roadmap could provide an overview of the project status. i.e. an estimate to when "Warning – arkOS and its constituent software are still in development." will disappear from the homepage would help.

Currently to my knowledge we're compatible with IPv6 in:

• Base OS
• Websites

What remains to be updated:

• Network management

Following this support discussion

arkosctl is slow due to a problem in requests that is imported line 15 in arkos/utilities/utils.py.
We are talking about ~5sec to launch any arkosctl command on a scaleway server.

See :

• requests: https://github.com/kennethreitz/requests/issues/3213
• that forward the issue to setuptools: pypa/setuptools#510

A possible workaround lies in https://github.com/ninjaaron/fast-entry_points

If sshd is configured to listen on another port than 22, after the installation of arkOS it won't be possible to ssh to the machine anymore

The port may have been changed in /etc/sshd/config
The code may be changed around arkos/tracked_services.py:406

id to name, type to site_type or app_type, etc

(Note that this bug should not result in a crash, rather the user should be prompted with a simple error message that the operation could not be completed.)

arkOS 0.8.3 Crash Report

Running in prod
System: Linux arkos 4.9.4-1-ARCH #1 SMP Sun Jan 15 23:08:44 MST 2017 armv7l GNU/Linux

Platform: armv7l Unknown
Python version 3.6.0
Config path: settings.json

Loaded applicatons:
db-mariadb
fs-samba

Request: POST /api/shares

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/lib/python3.6/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/flask/views.py", line 149, in dispatch_request
return meth(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/kraken/auth.py", line 135, in decorator
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/kraken/frameworks/sharers.py", line 47, in post
data["valid_users"], data["read_only"]
File "/var/lib/arkos/applications/fs-samba/fileshare.py", line 33, in add_share
s.add()
File "/usr/lib/python3.6/site-packages/arkos/sharers.py", line 74, in add
self.add_share()
File "/var/lib/arkos/applications/fs-samba/fileshare.py", line 137, in add_share
svc.restart()
File "/usr/lib/python3.6/site-packages/arkos/system/services.py", line 176, in restart
"sudo arkosctl svc status {0}".format(self.name))
arkos.system.services.ActionError: ('svc', 'The service failed to restart. Please check sudo arkosctl svc status smbd')

arkOS 0.8.3 on RaspberryPI Original

[root@arkos ~]# systemctl status krakend
* krakend.service - arkOS Kraken/Genesis
   Loaded: loaded (/usr/lib/systemd/system/krakend.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2017-01-11 20:53:26 CET; 1 weeks 5 days ago
  Process: 222 ExecStart=/usr/bin/krakend -d (code=exited, status=1/FAILURE)

Jan 11 20:53:26 arkos krakend[222]:     from .utils import *
Jan 11 20:53:26 arkos krakend[222]:   File "/usr/lib/python3.6/site-packages/arkos/utilities/utils.py", line 14, in <module>
Jan 11 20:53:26 arkos krakend[222]:     import random
Jan 11 20:53:26 arkos krakend[222]:   File "/usr/lib/python3.6/random.py", line 742, in <module>
Jan 11 20:53:26 arkos krakend[222]:     _inst = Random()
Jan 11 20:53:26 arkos krakend[222]: SystemError: <class 'random.Random'> returned a result with an error set
Jan 11 20:53:26 arkos systemd[1]: krakend.service: Control process exited, code=exited status=1
Jan 11 20:53:26 arkos systemd[1]: Failed to start arkOS Kraken/Genesis.
Jan 11 20:53:26 arkos systemd[1]: krakend.service: Unit entered failed state.
Jan 11 20:53:26 arkos systemd[1]: krakend.service: Failed with result 'exit-code'.

• Replace assorted references to Arch-specific directories and utilities.
• Users and groups: http -> www-data
• Replace netctl with networkd or similar, see #5
• Replace python-pacman with python-apt driver
• Verify Python version compatibility (features should work on at least 3.4 and up!)

See the following resources for tips.

• python-apt integration in applications.py: https://github.com/GetSkylark/core/blob/master/skylark/applications.py
• Image building and deployment: https://github.com/GetSkylark/scripts/blob/master/build.sh

Convert the network management system from netctl to networkd

In this tutorial the script for ArchLinux install references a mirror list at line 94 that no longer exists. It results in the following error:

# curl -s https://pkg.arkos.io/resources/install_arch.py | python
=== arkOS ===
Arch-to-arkOS Conversion Script

Using this script will convert your Arch Linux installation to a fully-functional arkOS server. It will irreversibly alter the software configurations on your machine. You should make no assumptions about your ability to manually
configure your server after this script has successfully completed, except by using the tools and methods included in arkOS.

If you do NOT want this to occur, press Ctrl+C now.
Installing in 8....
Installing in 1...
Installing new packages...
gpg: /etc/pacman.d/gnupg/trustdb.gpg: trustdb created
gpg: no ultimately trusted keys found
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/etc/pacman.d/gnupg/secring.gpg' to gpg-agent
gpg: migration succeeded
gpg: Generating pacman keyring master key...
gpg: key B0B8CA6131F90C40 marked as ultimately trusted
gpg: directory '/etc/pacman.d/gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/etc/pacman.d/gnupg/openpgp-revocs.d/4F88CC230F0148D38DB23B4EB0B8CA6131F90C40.rev'
gpg: Done
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: key 0155ED1E0585F850: public key "Jacob Cook (arkOS Package Signing) <[email protected]>" imported
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: Total number processed: 1
gpg:               imported: 1
==> Updating trust database...
gpg: no need for a trustdb check
  -> Locally signing key 0585F850...
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   1  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1  valid:   1  signed:   0  trust: 1-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2019-03-21
:: Starting full system upgrade...
 there is nothing to do
error: package 'arkos-mirrorlist' was not found
Traceback (most recent call last):
  File "<stdin>", line 174, in <module>
  File "<stdin>", line 95, in install
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

I have created a new user and set it to allow for ssh access. Trying login via ssh, I always get a "Access denied" message. Logging in as root (which I know is not based upon the ldap service) works fine.
The information given in https://bbs.arkos.io/t/ssh-login-issue/2403?u=erniepb details more on the corresponding logs.

It spits out an error message but does not seem to have any impact:

.....
>> Initializing configurations...
[INFO   ] ctl:init:redis: Copying files
[INFO   ] ctl:init:redis: Setting admin password
[SUCCESS] ctl:init:redis: Complete
(89/90) installing arkos-genesis                                                                           [################################################################] 100%
(90/90) installing patch                                                                                   [################################################################] 100%
Optional dependencies for patch
    ed: for patch -e functionality
:: Running post-transaction hooks...
(1/2) Updating system user accounts...
(2/2) Creating temporary files...
resolving dependencies...
looking for conflicting packages...

Packages (1) arkos-keyring-20150321-1

Total Download Size:   0.00 MiB
Total Installed Size:  0.02 MiB

:: Proceed with installation? [Y/n] ==> ERROR: The keyring file /usr/share/pacman/keyrings/arkos.gpg does not exist.
[INFO   ] ctl:init:ldap: Cleaning up old LDAP database
[INFO   ] ctl:init:ldap: Installing initial configuration
[INFO   ] ctl:init:ldap: Setting admin password
[INFO   ] ctl:init:ldap: Generating new LDAP database
[SUCCESS] ctl:init:ldap: Complete
[INFO   ] ctl:init:nslcd: Patching system files
[SUCCESS] ctl:init:nslcd: Complete
[INFO   ] ctl:init:nginx: Copying files
[SUCCESS] ctl:init:nginx: Completed
.....

Including file share mounts.

(from arkOScloud/pkgbuilds#56)

Currently long-running tasks are done in concurrent threads. This should be converted to a task/worker/queue system for efficiency and stability's sake.

When i create a virtualenv and install arkos-core it does not install some dependencies correctly (pycryptsetup & pyparted).

pip install -e git+https://github.com/arkoscloud/core.git#egg=arkosctl
Obtaining arkosctl from git+https://github.com/arkoscloud/core.git#egg=arkosctl
  git clone in ./.venv/src/arkosctl exists with URL https://github.com/pomarec/core.git
  Cloning https://github.com/arkoscloud/core.git to ./.venv/src/arkosctl
  Running setup.py (path:/home/deploy/test/.venv/src/arkosctl/setup.py) egg_info for package arkosctl produced metadata for project name arkos-core. Fix your #egg=arkosctl fragments.
Collecting click (from arkos-core)
  Using cached click-6.7-py2.py3-none-any.whl
Collecting cryptography (from arkos-core)
  Using cached cryptography-1.7.2.tar.gz
Collecting dbus-python (from arkos-core)
  Using cached dbus-python-1.2.4.tar.gz
Collecting free_tls_certificates (from arkos-core)
  Using cached free_tls_certificates-0.1.6.tar.gz
Collecting GitPython (from arkos-core)
  Using cached GitPython-2.1.1-py2.py3-none-any.whl
Collecting gnupg (from arkos-core)
  Using cached gnupg-2.2.0-py3-none-any.whl
Collecting miniupnpc (from arkos-core)
  Using cached miniupnpc-1.9.tar.gz
Collecting netifaces (from arkos-core)
  Using cached netifaces-0.10.5.tar.gz
Collecting ntplib (from arkos-core)
  Using cached ntplib-0.3.3.tar.gz
Collecting psutil (from arkos-core)
  Using cached psutil-5.0.1.tar.gz
Collecting pyldap (from arkos-core)
  Using cached pyldap-2.4.28.tar.gz
Requirement already satisfied: pyparted==3.10.7 in ./.venv/lib/python3.6/site-packages (from arkos-core)
Collecting pycryptsetup==1.7.2 (from arkos-core)
  Could not find a version that satisfies the requirement pycryptsetup==1.7.2 (from arkos-core) (from versions: )
No matching distribution found for pycryptsetup==1.7.2 (from arkos-core)

Testing should have decent coverage in every module.

arkOS 0.8.3 Crash Report

Running in prod
System: Linux arkos 4.4.47-1-ARCH #1 SMP Mon Feb 6 18:48:50 MST 2017 armv7l GNU/Linux

Platform: armv7l Unknown
Python version 3.6.0
Config path: settings.json

Loaded applicatons:
basicsite
paperwork
haste
wallabag
db-mariadb
db-sqlite3
radicale
xmpp
nextcloud
mailpile
lychee
syncthing

Request: PUT /api/certificates/xxx

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/lib/python3.6/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/flask/views.py", line 149, in dispatch_request
return meth(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/kraken/auth.py", line 135, in decorator
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/kraken/frameworks/certificates.py", line 121, in put
cert.assign(x)
File "/usr/lib/python3.6/site-packages/arkos/certificates.py", line 101, in assign
w.ssl_enable()
File "/usr/lib/python3.6/site-packages/arkos/websites.py", line 341, in ssl_enable
self._ssl_enable()
File "/usr/lib/python3.6/site-packages/arkos/websites.py", line 429, in _ssl_enable
raise errors.InvalidConfigError("Could not find metadata file")
arkos.utilities.errors.InvalidConfigError: Could not find metadata file