zgulde / cods

COdeup Deployment Scripts

Setup Scripts

For setup and management of a single remote server and virtual hosting with nginx, which serves mostly as a reverse proxy to application servers that (optionally) talk to a mysql database.

Who this is for

If you want to quickly deploy your application on the same server that hosts your database and not worry about the infastructure details that entails, this project can help you out.

Primarily, these scripts are intended to be used by Codeup students going through the Java program. You might also find this project useful if you have a Java, Python, PHP, or Node web application you want to deploy quickly, or if you want to host several hobby projects on the same server.

This project is probably not for you if:

• You want to use a database that is not MySQL
• You don't want to put nginx in front of your application
• You want to use an operating system that is not Debian on your server

Documentation

• Quick Reference
• Installation
• Initial Server Setup
• Domain DNS Configuration
• Usage Guide
• API Docs
• Frequently Asked Questions
• Step by Step Deployment Guide For a Spring Boot Application
• Step by Step Deployment Guide For a Node + Webpack Frontend Application
• Step by Step Deployment Guide For a Python Flask Application
• Getting More Help

cods's Issues

Credentials are created when user creation command fails

For example

1. myserver db create --user some_user --name some_db
2. Enter the wrong db admin password
3. new user credentials for some_user are still appended to the credentials.txt file

I suspect this would be the case for new user creation as well.

We should check to see if the creation command succeeds before appending to the credentials file

Consider creating default site pages when a site is created

Cods could create a default index.html page that says something like "java site is ready to go!" just to ensure all the internet plumbing is setup the right way.

The downside is we'd probably want to remove this page when deploying the application as we would want to route requests to / and not see the default index.html page.

Add users by github username

Instead of manually specifying a username and public key file, the script could use a github username and use the username and the public keys from github, located at https://github.com/USERNAME.keys

If we go this route, we should make sure to check for the public key(s) existence first, so we don't just create a user without any authorized keys

Githook should listen for main or master

With the latest changes on github, our default branch is most likely to be main, but the cods githook on the server-side only triggers the deployment commands if it comes from master.

Right now the solution is just to create a master that mirrors main.

New users don't have permissions for any existing sites

When a new user account is created on an existing server, the new user will not have permissions (i.e. they won't be a part of any groups) for any existing sites.

Once a user exists, they will be added to any future sites that are created, this issue only exists for new users getting permissions for existing sites.

We should fix this programmatically when we create the new user account.

Document option for solving slow spring security startup

haveged is one such possible fix. In general this seems like a problem with any app that needs a lot of randomness on startup (like spring security) with any VPS provider.

I don't think it falls on this tool to fix this, rather, we could add some docs around when you might want to do this, and advise users to consider the tradeoffs for themselves.

Database install is a manual process

Currently, no database is installed by default.

To install mysql 5.7, do the following after the initial server provisioning:

1. Login to the server

   myserver login
   

2. configure the mysql installation process

   # from the server
   sudo -s
   wget http://repo.mysql.com/mysql-apt-config_0.8.13-1_all.deb
   dpkg -i mysql-apt-config_0.8.13-1_all.deb
   

   1. From here, choose option 1, "MySQL Server & Cluster".
   2. Next choose version 5.7

3. Install mysql

   # still from the server
   apt update && apt install -y mysql-server
   

   Leave the root password blank for now

4. Setup your admin user

   1. Login to the mysql server

      # still logged in to the production server...
      mysql -uroot
      

   2. Create your admin mysql user

      CREATE USER you@localhost IDENTIFIED BY 'password';
      GRANT ALL on *.* TO you@localhost WITH GRANT OPTION;
      

      Replacing you with your server username, and password with your
      admin db password (found with myserver credentials).

Add a log of commands

This would be super helpful for debugging, maybe something like ~/.config/cods/myserver/history.txt that has a list of all the commands run with myserver?

I don't think appending to this file everytime a command is run would slow things down

cods list command

to show all the servers that are setup with cods, and list the domains setup on each one

When provisioning fails, the command and data directory are still created

Reproduction:

0. Have a "bad" (out of date) url for the tomcat download
1. Create a server cods init myserver
2. The provisioning will fail, but ~/.config/cods/myserver and $BIN_PREFIX/myserver will still be created

How do we setup a spring boot application to handle uploaded files?

We're assuming you have a spring boot application is setup to handle file
uploads, and the location that files are uploaded to are specified your
application.properties.

We'll also assume that you have set the site up already with the domain name
example.com.

1. Create a folder for the uploaded files in your application

   myserver run mkdir -p /srv/example.com/public/uploads
   

   The directory can be named whatever you want, so long as it is inside the
   public for your site (i.e. /srv/example.com/public).

2. Modify the application.properties file on the server:

   myserver run nano /srv/example.com/application.properties
   

   Add the file upload path (i.e. the path to the directory you created in the
   last step) to the application.properties file

   ...
   file-upload-path = /srv/example.com/public/uploads
   ...
   

3. Re-build your application so that it uses the updated application.properties

   myserver site build --domain example.com
   

How to remove a user from the server?

This command

myserver run sudo userdel -f -r USERNAME

Will remove a user account from the server, and in the process, remove that person's ssh access to the server.

---

perhaps we should create a seperate command for this

/srv/letencrypt.log should be group-writable

So that anyone can enable https for a site

probably need to add a check for this when enabling https.

In the meantime if you hit this issue:

chmod g+w /srv/letsencrypt.log

Upgrade to ubuntu 18.04

The java installation / setup process becomes a little more tricky, but I think everything else will work as is.

This isn't terribly urgent, as 16.04 (the version we are currently on) is still supported until april 2021

Upload static files

Use Case:

I have a bunch of static files on my laptop that I want to deploy to my public url. I don't want to build on the server because the build process takes too long, the environment setup is too tricky, or I'm just too lazy.

This could be a simple as more documentation around the upload subcommand, or could be as complicated as a new command.

E.g. this would work without any changes to cods:

myserver upload -f build/* -d /srv/example.com/public/

But maybe it would be nice to be able to do something like:

myserver site upload -d example.com -f build

Error out sooner

We should probably bail out of the site create command after we fail to create the user and the group

 rpal@Rons-MBP blog % codeup_blog site create --domain ronnelpalencia.com --java --spring-boot --port 8080
- Making Sure ronnelpalencia.com Isn't Already Setup...
  ok
- Checking To Make Sure Port 8080 Is Free...
  ok
- Checking DNS Records For ronnelpalencia.com...
  ok
- Logging In To Create ronnelpalencia.com...
- Creating User and Group For ronnelpalencia.com
useradd: group ronnelpalencia-com exists - if you want to add this user to that group, use -g.
Connection to 68.183.114.117 closed.
- Performing Extra Spring Boot Configuration
Performing spring boot app configuration...
sed: can't read /srv/ronnelpalencia.com/config: No such file or directory
- Finished Setting Up ronnelpalencia.com
rpal@Rons-MBP blog %

Also it seems like there's a bug here somewhere in creating/removing the user+group for a site

Allow static sites to be built "automatically"

Like we do for java sites, we just have a .build_config that contains the command to build and the output.

We should do this for static sites too.

Example .build_config for a static site:

BUILD_COMMAND='npm install && npm build'
OUTPUT='dist' # or whatever

Deploying Spring + a front end framework

If one wants to use a front end framework in conjunction with spring, your frontend files need to be in the same project as your spring application, i.e. in ~/frontend. You can use a maven plugin like https://github.com/eirslett/frontend-maven-plugin to automatically build your front end and include it in the /target folder so maven can package it correctly. So your pom.xml should include

<build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>com.github.eirslett</groupId>
                <artifactId>frontend-maven-plugin</artifactId>
                <version>1.6</version>
                <configuration>
                    <workingDirectory>frontend</workingDirectory>
                    <installDirectory>target</installDirectory>
                </configuration>
                <executions>
                    <execution>
                        <id>install node and npm</id>
                        <goals>
                            <goal>install-node-and-npm</goal>
                        </goals>
                        <configuration>
                            <nodeVersion>v12.6.0</nodeVersion>
                            <npmVersion>6.9.0</npmVersion>
                        </configuration>
                    </execution>
                    <execution>
                        <id>npm install</id>
                        <goals>
                            <goal>npm</goal>
                        </goals>
                        <configuration>
                            <arguments>install</arguments>
                        </configuration>
                    </execution>
                    <execution>
                        <id>npm run build</id>
                        <goals>
                            <goal>npm</goal>
                        </goals>
                        <configuration>
                            <arguments>run build</arguments>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <artifactId>maven-antrun-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>generate-resources</phase>
                        <configuration>
                            <target>
                                <copy todir="${project.build.directory}/classes/public">
                                    <fileset dir="${project.basedir}/frontend/build"/>
                                </copy>
                            </target>
                        </configuration>
                        <goals>
                            <goal>run</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

In addition, the command to build the jar is mvn clean install rather than ./mvnw package so your .cods file should look like this

BUILD_COMMAND='mvn clean install'
JAR_FILE=target/FILENAME.jar

`db create` command silently fails if user or database already exist

allow user other than root for initial server setup

When setting up, e.g. an ec2 instance, the user account you have ssh access to is ubuntu, not root.

How to add a server command for an existing user to a different client?

Edit 2019-08-22: The below information is out of date. The way to do this is with cods add, see https://github.com/zgulde/cods/blob/master/docs/api.md#add

---

Scenario: You've created a server and added another one of your computer's ssh keys to it, and you want to add a server interaction command (e.g. myserver) to your other box. You don't want to create a new user account, you just want to use the one you already have.

Solution: The functionality is already there for this, all you would need to do is use cods share.

On the computer that setup the server:

1. Get a copy of the other machine's ssh key
2. use myserver addkey to add your other computer's ssh key

On the other machine:

1. Install cods
2. Run cods share myserver and enter your (existing) username
3. (optional) add generated credentials to the credentials file (we probably need some more docs for this step as well)

---

We should document this somewhere more formal than a github issue, also we should probably rename the cods share subcommand, as it is a little misleading. Right now I'm thinking it should be renamed to cods add as that is a little more meaningful.

More deployment guide docs

• Add skip tests to the pom.xml

For multiple java versions installed + spring boot 1.x, we'll need to

echo "JAVA_HOME=$(/usr/libexec/java_home -v 1.8)" > ~/.mavenrc
# maybe instruct to put this in the .bashrc?
export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)

when running from the command line

FAQ still shows old commands

A good example is how to share the server with others:

myserver adduser to myserver user add args...

Specify what OS we're using

This information should be in the deployment guide