typesafehub / conductr-doc

ConductR Documentation

Note that this project is EOL as of 2018-01-31.

Documentation for conductr.lightbend.com

Feel free to contribute and correct!

Dev Notes

This project has a few branches for the different versions of ConductR. master tracks the unreleased 2.2 version. Most changes should be made on both the master and 2.1 branches. You'll want to look into git cherry-pick to help facilitate this.

How to run ConductR doc locally

You will require sass or compass on the PATH. Refer to compass installation or sass installation for the actual installation step for your platform.

Clone the project-doc:

$ git clone https://github.com/typesafehub/project-doc

cd into the project-doc directory:

$ cd project-doc

Modify the app/modules/ConductRDocRendererModule.scala file to point to your PR change:

• For example we are looking at PR 405.
• This PR is based off the seglo repo, under the branch called licensing.
• Modify ConductRDocRendererProvider21 to point to the PR repo and branch:

@Singleton
class ConductRDocRendererProvider21 @Inject()(actorSystem: ActorSystem, wsClient: WSClient)
  extends ConductRDocRendererProvider(
    actorSystem,
    wsClient,
    new URI("https://github.com/seglo/conductr-doc/archive/licensing.zip"),
    "2.1.x",
    versions
  )

Once this is done, run the project-doc:

$ sbt run

You will need to restart the project-doc process to immediately see the changes pushed to the PR branch.

© Lightbend Inc., 2015-2016

conductr-doc's Issues

Document how to load a bundle with configuration

Currently it is not documented how to load a bundle with a bundle configuration.

Related PR in sbt-conductr (README) is: typesafehub/sbt-conductr#155

Documentation for DC/OS mode

The following sections need to be updated to document the instructions for the DC/OS mode:

• Consolidated logging
• Managing ConductR services

Developer quick start: Add docker section

Add a docker section to describe how to:

• Configure the docker environment (set enough memory for VirtualBox)
• Start the docker environment
• Verify if the docker client and server is up-to-date / List down the supported docker versions and clients
• Stop the docker enviroment

Document the new client lib

Document conductr-client-lib usage

Distinctive Prerequisite section per OS

This issue needs a change in conductr-doc and Typesafe.com

How to install docker and python3 is different on MacOS, Windows, Ubuntu, Debian, etc. Therefore, the prerequisite section should be separate for each OS. An example of what you might do is have separate sections for each, stacked vertically down the page, at the start you click to select which one you want and the page will scroll down to it (don't just use anchor links, use a smooth scroll so that the user can clearly see that they are being scrolled down), and then at the end of the OS specific instructions, have another link that scrolls you down to the rest of the instructions. The docker installation page has a good example of the scrolling I'm talking about: https://www.docker.com/products/docker

Doc Bundle Restarting

As someone evaluating the resiliency provided by ConductR, I want to know when bundles will be restarted and when they should not be restarted so that I can better understand how to build highly resilient solutions.

Describe ELB vs HAProxy

Describe the relationship of ELB and HAProxy; the former being about nodes and the latter being about services.

Instructions for custom endpoint names in Play app

The current documentation page http://conductr.lightbend.com/docs/1.1.x/BundleConfiguration#Endpoints describes endpoints and how ConductR creates certain environment variables. However, it is not very clear how these environment variables are used. In case of a Play application, it is not clear that the application.conf of the Play application need to be changed if a custom endpoint name is specified. In other words, the default logic of conductr-lib is not explained: https://github.com/typesafehub/conductr-lib/blob/master/play25-conductr-bundle-lib/src/main/resources/play/reference-overrides.conf#L16-L20

This can lead to errors when using custom endpoint names in a Play application. Example: https://support.lightbend.com/agent/case/7579

Release Notes

As a production user of ConductR, I would like release notes in the documentation so that I can better navigate release upgrades.

Document detached process scenarios

There are some scenarios that could warrant a bundle component becoming detached and having some script controlling its exit via trapping SIGTERM and then gracefully shutting down the detached. For example, a video streaming process should only shutdown when it detects that its existing connections have terminated. Reddis is also another example whereupon it seemingly likes to detach itself on receipt of a SIGTERM.

We should document how to handle such use cases. The BundleConfiguration.md may be a good place to do so.

Document ConductR properties

Conductr's application.conf should be available from the docs.

Move release notes to a separate group

Currently the release notes are listed under the Operations guide. We should move the release notes to a new group. This group should contain a migration guide and the release notes and should be the first group in the agenda (similar to the Play documentation).

Explain how to upgrade ConductR

Explain how a) we upgrade ConductR given patch releases and b) how we upgrade ConductR for Major/Minor releases.

Maintenance link

Need a link on the page to describe where to go to correct mistakes on the page, or to receiver further contributions.

Developer quick start: Add sbt as a prerequisite

Discuss auto recovery

Discuss how ConductR 1.1 automatically recovers from downing in here: https://conductr.typesafe.com/docs/1.1.x/ClusterTroubleshooting

Not clear that custom endpoints require that http.address is overrided

Trying to get my app working on conductr for the first time I ran in to an annoying-to-track-down problem because I had set:

BundleKeys.endpoints := Map(
"payments" -> Endpoint("http", services = Set(URI("http://:9000")))
)

But omitted:
javaOptions in Bundle ++= Seq("-Dhttp.address=$PAYMENTS_BIND_IP", "-Dhttp.port=$PAYMENTS_BIND_PORT")

I think this page http://conductr.lightbend.com/docs/1.1.x/BundleConfiguration could do with a big red warning if you use a custom endpoint or something. Even better would be if the wiring was done automagically.

Improve python cli install

From @rpokrywka:

instead of installing pip via easy_install, python3-pip package may be installed

I believe that easy_install is the correct way of doing things, but can you please provide your rationale @rpokrywka?

Add a License section to the first page of the docs

Just in case someone find the document and is interested in what license it is under.
Linking to the following page (& text) seems like the best option, as it will be possible to update those terms/restrictions in one place: https://www.typesafe.com/product/conductr/developer

Document bundle restart

Enhance documentation to cover off the following questions and answers:

I deployed our application to our 3-node ConductR cluster and stopped the conductr service running on the first node (the one where our application was running) with the expectation that our application should start on one of the remaining nodes where the conductr service is still running. Is this expectation correct?

Yes.

What are the prerequisites (configuration or code) for ConductR to be able to make an application failover to another available ConductR node?

If a bundle dies of its own accord then we do not attempt to restart it. We don't really know why it has stopped - it could be for a legitimate reason e.g. a short lived task. However if the ConductR service that was running on that node shuts down the one of the other nodes the cluster will re-schedule the bundles that were running for execution elsewhere.

1.1.x: Update load bundle control API

The load bundle documentation of the ConductR 1.1.x Control API is outdated. The documentation still contains the bundle attributes such as nrOfCpus and memory. The conductr code uses in V2 the following request attributes: https://github.com/typesafehub/conductr/blob/master/conductr/src/main/scala/com/typesafe/conductr/net/version2/BundlesRoute.scala#L265-L292

Change docker link to products page

This issue needs a change in conductr-doc and Typesafe.com

The instruction to install docker is referencing a link to http://docker.com. On http://docker.com it not obvious for a new user with what kind of product from Docker he should get started. Therefore we should link directly to the product page of the Docker tool: https://www.docker.com/products/docker

Here the user will find the download links for any OS.

Document new service locator APIs

google-code-prettify not downloaded

google-code-prettify cannot be downloaded because the webjars library that we are using is downloaded the files from a destination where the files are now hosted anymore. I've created the PR webjars/prettify#1 against https://github.com/webjars that solves this problem. Once a new webjar prettify version is released we need to upgrade conductr-doc to download the prettify sources correctly.

Document the check command and others

Summarise check and other commands, with links providing further details.

Document SSE

Server sent events require documentation

Document max content length

Include documentation on how to change the setting for increasing the amount of bytes that may be received for ConductR i.e. akka.http.server.parsing.max-content-length.

Current Version Notice

As a user following older links and URL's, I want to be notified if I am not reading the latest version doc so that I don't view old information unintentionally. For example, from Playframework.com:
"You are viewing the documentation for the 2.3.x release series. The latest stable release series is 2.5.x."

Control API doc please

As one who deploys ConductR, I want documentation for the Control Server so I can listen to bundles and members events in order to build management solutions for my deployment.

No anchor tags

Anchor links are not generated for pages like the control api and bundle api pages.

Default values for Endpoints

See sbt/sbt-bundle#36.

Document to create a “docker” group under Linux

This issue needs a change in conductr-doc and Typesafe.com

The Docker install instructions for Ubuntu contains a section “Optional configurations”: https://docs.docker.com/engine/installation/linux/ubuntulinux/#/optional-configurations

The only optional configuration for ConductR is to create the “docker” group (only for linux).

It is not obvious for the user which optional Docker configuration to follow. Therefore, we should be explicit about it in our Getting started guice, i.e. we should describe that the user need to add a “docker” group (only in a separate linux instruction).

Java installation improvement

From @rpokrywka:

Instead of update-alternatives --config java //(interactive)
one can use
update-alternatives --set java /usr/share/jre1.8.0_31/bin/java

@rpokrywka Can you please explain the rationale for this change and why it is better?

Describe cluster topologies

We need to document the usage of roles for 1.1 and 1.2 in particular, now that roles are mandatory. We should also mention how to turn role checking off.

Provide a search capability

site:http://conductr.typesafe.com

Provide Java examples alongside Scala

Bundling-Existing.html doesn't exist

There is a linkrot i.e. bundles

See the "legacy & third party bundles" section of the bundles document for more information on that

on page Dev-Quickstart.md

Firewall rules for bundles

Do we not need to document firewall rules for ports 9007 and 9008?

9007 is the status-server and 9008 is the service locator. Both are required to be accessed by bundles.

Here's the existing doc:

http://conductr.lightbend.com/docs/1.1.x/Install#Subnets-and-Security-Groups

No entry for $name_HOST_IP

While trying to wire up cassandra to my service following this doc: http://conductr.lightbend.com/docs/1.1.x/BundleEnvironmentVariables

I noticed that the HOST_IP doesn't appear as a stand alone variable, it's just referenced as part of name_HOST.

Reference Activator Templates

In order to facilitate the developer experience reference Reactive Maps and others possibly.

Provide a getting started one for Play. conductr-service-lookup could become the basis for this.

There should be a doc that brings these templates together.

Document Rolling Bundle Upgrades

As an operator I would like documentation on upgrading bundles so that I can upgrade without downtime.

Add section about Akka Cluster bundle

Describe the support for seed nodes.

Another use case (from customer) is to use ClusterClient to communicate between different clusters (bundles). ClusterClient needs contact points, which are Akka paths (with address). I think the service discovery can be used for this, and a specific example would be useful to illustrate how to do it.

Document name to environment variable transformation

From @jroper:

The ConductR docs state that endpoint configuration is exposed in environment variables using names such as name_BIND_PORT.

However, this is wrong, if I have an endpoint called helloservice, the environment variable is not helloservice_BIND_PORT, but HELLOSERVICE_BIND_PORT. Note that on Linux environment variables are case sensitive, so these are not the same. Furthermore, there are many other characters that are not permitted (or recommended) in environment variables, such as dashes, and given that the example docs recommend names like akka-remote, there must be some transformation going on that's more than just converting lower case to upper case. This transformation should be documented.

Document exit code usage

Document that process that fail should always exit with a non-zero exit code. The POSIX convention is to use exit code 1.

Exiting with a 0 will cause ConductR to view the exit as normal and therefore do nothing in terms of restarting it elsewhere.

It may also be helpful to provide some code around our Akka SBR downing of a self member should behave.

Remove mkdir from installation guide for 1.0.0-b3

Given https://github.com/typesafehub/typesafe-conductr/issues/570 and as a consequence of beta 3 being released, remove the mkdir statements from the installation doc for the purposes of tee-ing the conf. The application.ini files should already exist.

Configuring ConductR behind a proxy

We should document how to configure ConductR that operates behind a proxy. This will involve configuring a list of ConductR core instances as the backend, and the use of the "source" load balancing strategy.

@longmuir Since you have configured that already at a client project, are you able to document this?

Developer quick start: Document multiple applications

So far the developer quick start only describes how to setup a Play 2.4 application. This guide should be enhanced to a Scala, Akka and Java application. For this we should use the code snippet tabs from the Play documentation: https://playframework.com/documentation/2.4.x/ThreadPools#Application-class-loader

Document scaling and other popular features

Scaling is a topic in its own right, and a major drawcard for ConductR. We should have a page detailing how ConductR tackles scaling. Same goes for all of the features really:

• Automated seed node establishment
• Dynamic service discovery
• Scalable rolling updates
• DevOps-friendly immutable deployments
• Reactive Monitoring integration
• Load-balancing at scale
• Automated recovery for failed applications
• Automated node failure handling
• Automated network partition resolution
• Infrastructure agnostic
• Various packaging and container formats
• Developer sandboxing tools
• Consolidated events and logging

The intro doc would be a good place for this documentation.

Describe bundle publication

Bundle publication via bintray isn't documented. It should be.

Configuring conductr-haproxy in sandbox / development mode

If you follow along at https://conductr.lightbend.com/docs/2.1.x/DynamicProxyConfiguration#creating-and-deploying-custom-haproxy-configuration to deploy a custom HA proxy configuration on your sandbox, it will not work. It was not immediately clear to me that I would need to do something special to deploy the bundle on sandbox.

@markusjura pointed me to https://github.com/typesafehub/conductr/tree/master/conductr-haproxy/src/bundle-configuration/conductr-haproxy-dev-mode which was very helpful. This is not referenced in any way from the documentation though.

The dev-mode scripts are different (compare https://github.com/typesafehub/conductr/tree/master/conductr-haproxy/src/bundle-configuration/conductr-haproxy-dev-mode files with what is in the docs).

It would be great if a section somewhere can be added how to deploy a custom haproxy configuration on the sandbox, I think many users will find it useful to . (The relevant parts are to add a reloadHAProxy.sh script that kills the docker container and the runtime-config.sh that sets up sandbox / docker specifics)