microsoft / vsts-docker-preview

Preview repo for Docker extension

#Announcing VSTS Docker Extension Preview!

The world's largest enterprises rely on Docker to develop the world’s best applications. With the Docker extension you can now integrate your existing agile and DevOps workflows with Docker to easily manage distribution of your images.

##What is Docker? Docker allows you to package an application with all of its dependencies into a standardized unit for software development.

Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

##Visual Studio Team Services and Docker Take advantage of the Visual Studio Team Services (VSTS) cloud based build, release capabilities to build your Docker image, and upload that image to Docker Registry as part of your continuous integration or continuous deployment flow.

###Get started with Docker Extension

You can install the extension from the link

##What is in the box?

This extension installs the following components:

• A service endpoint for connecting to Docker Registry
• A service endpoint for connecting to Docker Container Host
• A Docker task: Set of pre-built Docker commands to manage the lifecycle of application images.
  • Build an image
  • Push an image or a repository to a Registry
  • Run an image in a Container
  • Run a Docker command
• A Docker Compose task: Docker Compose allows you to define your multi-container application with all of its dependencies in a single file, then spin your application up in a single command.
  • Run a Docker Compose command

###How tos?

####Create a Docker Registry Endpoint:

1. Open the Services page in your Visual Studio Team Services Control Panel

2. In the New Service Endpoint list, choose Registry

3. Enter the Name for your connection and your Docker Hub details to create the service endpoint

####Create a Docker Container Host Endpoint:

1. Locate your ca.pem, key.pem and cert.pem files used to secure your Docker host.
2. Open the Services page in your Visual Studio Team Services Control Panel.

3. In the New Service Endpoint list, choose Docker Host

4. Enter the Name for your connection and the URL to your Docker host 5. Copy and paste the entire contents of each file into the appropiate spaces

####Build your Docker image

1. Open your build definition and add the Docker task. The task can be found in the Build category of the Add Tasks dialog.
2. Enter the required parameter values for the Docker task:

• Docker Host Connection: Select your Docker Container Host connection.
• Docker Registry Connection: Select your Docker Registry
• Action: Select Build an image
• Docker File: Path to Dockerflle to use with the Docker Build action. For example **/Dockerfile.
• Context: The folder to upload to the Docker daemon to build the image.
• Working Directory: The folder where the certs folder will be created and the certs written into.

####Push your image to Docker Hub

1. Open your build definition and add the Docker task. The task can be found in the Build category of the Add Tasks dialog.
2. Enter the required parameter values for the Docker task:

• Docker Host Connection: Select your Docker Container Host connection.
• Docker Registry Connection: Select your Docker Registry
• Action: Select Push an image
• Image Name: The image you wish to create. You can tag your image by appending a : and the desired tag i.e fabrikam/webapp:$(Build.BuildId).
• Working Directory: The folder where the certs folder will be created and the certs written into.

####Run your image in a container

1. Open your build definition and add the Docker task. The task can be found in the Build category of the Add Tasks dialog.
2. Enter the required parameter values for the Docker task:

• Docker Host Connection: Select your Docker Container Host connection.
• Docker Registry Connection: Select your Docker Registry
• Action: Select Run an image
• Image Name: The image you wish to create. You can tag your image by appending a : and the desired tag i.e fabrikam/webapp:$(Build.BuildId).
• Container Name: Optional, Assign a name to the container
• Port: Optional, Port mappings to pass to the run comamnd in hostmachine-port:container-port format. For example, -p 80:8080 binds port 8080 of the container to port 80 of the host machine. Specify each port mapping in a new line.
• Environment Variables: Name value pairs to set as environment variables in the container. Specify each name=value pair in a new line.
• Command for container: Specify a command and its arguments, if any, for the container to run. For example, if the image contains a simple Python Flask web application you can specify 'python app.py' to launch the web application.
• Working Directory: The folder where the certs folder will be created and the certs written into.

####Run a Docker command

1. Open your build definition and add the Docker task. The task can be found in the Build category of the Add Tasks dialog.
2. Enter the required parameter values for the Docker task:

• Docker Host Connection: Select your Docker Container Host connection.
• Docker Registry Connection: Select your Docker Registry
• Action: Select Run a Docker command
• Command: Docker command with the arguments to execute. For example 'rmi -f image-name' sans quotes to remove an image
• Working Directory: The folder where the certs folder will be created and the certs written into.

####Run Docker Compose command

1. Open your build definition and add the Docker Compose task. The task can be found in the Build category of the Add Tasks dialog.
2. Enter the required parameter values for the Docker Compose task:

• Docker Host Connection: Select your Docker Container Host connection.
• Docker Registry Connection: Select the connection to your Docker Registry.
• Docker Compose File: Path to Docker compose file to use. Should be a '.yml' or '.yaml'
• Environment Variables: Arguments that need to be passed to docker-compose file. These will be set as environment variables. Specify each argument as name=value pair in a new line. For example, HTTP_PROXY=fabrikamweb.mydomain.com, HTTP_PORT=80
• Project Name: Specify an alternate project name.
• Docker Compose Command: Docker Compose command along with its arguments. for example, Using 'up -d' starts the containers in the background and leaves them running, other examples are 'down --rmi all -v' or 'restart'. Note: Recommended to use detached mode (-d) to avoid blocking the agent..
• Working Directory: The folder where the certs folder will be created and the certs written into.

###Feedback

For further information or to resolve issues, contact RM_Customer_Queries at Microsoft dot com or we're just a Tweet away.

###Trademarks

Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United States and/or other countries. Docker, Inc. and other parties may also have trademark rights in other terms used herein

vsts-docker-preview's Issues

RSA Private Key visible when using SSH

When executing the Deploy to ACS DCOS task on VSTS, in the build logs I see the private RSA key. I believe this is a security risk. Could this be removed or obfuscated?

incremental builds

Hi, I'm building my docker containers using the Hosted Linux Preview queue. Unfortunately, building the images takes some time (approximately 8 minutes). This is because the entire container is built from scratch, and no intermediary containers are stored.

Can I enable caching in any way?

Thanks,

Could not load X509 key pair: open '.dockercerts/cert.pem': no such file or directory. Make sure the key is not encrypted

Hi,

I am unable to run docker commands with the latest version of the plugin

Trying to run a very simple command "ps -a", I get this output:

******************************************************************************
Starting: Build
******************************************************************************
Current agent version: '2.109.2'
******************************************************************************
Starting: Get Sources
******************************************************************************
******************************************************************************
Starting: Run a Docker command
******************************************************************************
==============================================================================
Task         : Docker
Description  : Build, push or run Docker images, or run a Docker command
Version      : 0.3.1
Author       : Microsoft Corporation
Help         : [More Information](http://aka.ms/dockertoolsforvsts)
==============================================================================
8da9c0f3********** exists true
41660d8e********** exists true
/usr/bin/docker -H tcp://******:2376 --tls --tlscacert='.dockercerts/ca.pem' --tlscert='.dockercerts/cert.pem' --tlskey='.dockercerts/key.pem' login -u ******** -p ******** ********
Could not load X509 key pair: open '.dockercerts/cert.pem': no such file or directory. Make sure the key is not encrypted
/usr/bin/docker -H tcp://*******:2376 --tls --tlscacert='.dockercerts/ca.pem' --tlscert='.dockercerts/cert.pem' --tlskey='.dockercerts/key.pem' ps -a
Could not load X509 key pair: open '.dockercerts/cert.pem': no such file or directory. Make sure the key is not encrypted
Could not load X509 key pair: open '.dockercerts/cert.pem': no such file or directory. Make sure the key is not encrypted

Passwords are shown in clear text in build logs

The passwords should be shown as *******.

Currently, it's shown when a Docker task, for any action.

The docker compose task if fine though. The passwords are correctly hidden.

Example when using a build action with DockerHub:

##[Debug] 2016-07-10T17:33:20.612Z: /usr/bin/docker arg: login
##[Debug] 2016-07-10T17:33:20.612Z: /usr/bin/docker arg: -e [email protected]
##[Debug] 2016-07-10T17:33:20.612Z: /usr/bin/docker arg: -u myusername
##[Debug] 2016-07-10T17:33:20.612Z: /usr/bin/docker arg: -p mypassword
##[Debug] 2016-07-10T17:33:20.613Z: /usr/bin/docker arg: https://index.docker.io/v1/
##[Debug] 2016-07-10T17:33:20.632Z: exec tool: /usr/bin/docker
##[Debug] 2016-07-10T17:33:20.632Z: Arguments:
##[Debug] 2016-07-10T17:33:20.632Z:    --tls
##[Debug] 2016-07-10T17:33:20.633Z:    --tlscacert='.dockercerts/ca.pem'
##[Debug] 2016-07-10T17:33:20.633Z:    --tlscert='.dockercerts/cert.pem'
##[Debug] 2016-07-10T17:33:20.633Z:    --tlskey='.dockercerts/key.pem'
##[Debug] 2016-07-10T17:33:20.633Z:    -H
##[Debug] 2016-07-10T17:33:20.633Z:    tcp://1.2.3.4:2376/
##[Debug] 2016-07-10T17:33:20.633Z:    login
##[Debug] 2016-07-10T17:33:20.641Z:    -e
##[Debug] 2016-07-10T17:33:20.641Z:    [email protected]
##[Debug] 2016-07-10T17:33:20.642Z:    -u
##[Debug] 2016-07-10T17:33:20.642Z:    myusername
##[Debug] 2016-07-10T17:33:20.652Z:    -p
##[Debug] 2016-07-10T17:33:20.652Z:    mypassword
##[Debug] 2016-07-10T17:33:20.653Z:    https://index.docker.io/v1/
2016-07-10T17:33:20.653Z: [command]/usr/bin/docker --tls --tlscacert='.dockercerts/ca.pem' --tlscert='.dockercerts/cert.pem' --tlskey='.dockercerts/key.pem' -H tcp://1.2.3.4:2376/ login -e [email protected] -u myusername -p MyPassword https://index.docker.io/v1/

docker-compose.exe failed with return code: 4294967295

docker-compose was working previously for version 0.1.12. Starting from version 0.2.0 (which was deployed about 23 hours ago) it stoped working with the following error:

docker-compose.exe -H tcp://XX.XX.XX.XXX:2376/ --tls --tlscacert='.dockercerts\ca.pem' --tlscert='.dockercerts\cert.pem' --tlskey='.dockercerts\key.pem' -f D:/_work/13f36174b/project/scripts/docker-compose.yml -f .docker-compose.1476451184894.yml -p website up -d
Traceback (most recent call last):
File "", line 3, in
File "compose\cli\main.py", line 62, in main
File "compose\cli\main.py", line 111, in perform_command
File "compose\cli\command.py", line 34, in project_from_options
File "compose\cli\docker_client.py", line 39, in tls_config_from_options
File "site-packages\docker\tls.py", line 47, in init
docker.errors.TLSParameterError: Path to a certificate and key files must be provided through the client_config param. TLS configurations should map the Docker CLI client configurations. See https://docs.docker.com/engine/articles/https/ for API details.
docker-compose returned -1

Please advice.

Update: it does not work for version 0.2.1 as well.

docker-compose task in release can fail with strange error

If you don't set a Project Name in a docker-compose step as part of a release then when you attempt to run the build (normally it picks up the folder name) I get the following error in the logs

2016-10-17T12:49:03.9186980Z ##[error]Cannot read property 'toLowerCase' of null

setting a project name appears to fix this.

Repository names with uppercase letters break Docker tasks

Copied from microsoft/azure-pipelines-tasks#2988

If the name of the repository has an uppercase letter and the "Image Name" field in the task settings is left default as ($(Build.Repository.Name):$(Build.BuildId), the Docker task fails since Docker fails if the repository name is not all lowercase.
2016-10-31T21:32:10.1156180Z ##[section]Starting: Build an image
2016-10-31T21:32:10.4071990Z 4de0abcf-e253-4b34-b9c7-eb5cdfdd0c7a exists true
2016-10-31T21:32:10.4099210Z [command]/usr/bin/docker login -u ******** -p ******** ********
2016-10-31T21:32:11.0489340Z Login Succeeded
2016-10-31T21:32:11.0554530Z [command]/usr/bin/docker build -f /home/vstsbuild/vsts-agent/_work/1/s/Dockerfile -t OguzPastirmaci/vsts-dockerhub-dockercloud-azure:276 /home/vstsbuild/vsts-agent/_work/1/s
2016-10-31T21:32:11.0704490Z invalid argument "OguzPastirmaci/vsts-dockerhub-dockercloud-azure:276" for t: invalid reference format: repository name must be lowercase
2016-10-31T21:32:11.0712890Z See 'docker build --help'.
2016-10-31T21:32:11.0775540Z [command]/usr/bin/docker logout ********
2016-10-31T21:32:11.0882480Z Remove login credentials for ********
2016-10-31T21:32:11.0937030Z ##[error]/usr/bin/docker failed with return code: 1
2016-10-31T21:32:11.1322530Z ##[section]Finishing: Build an image

Make this extension OSS

It would be great if this extension were open source, as the community could contribute fixes/features :-)

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

[Question] docker run get more arguments

How Can I pass more arguments for the docker run command?
e.g:
-t
--cap-addd
--security-opt

Allow use from Azure Container Services

When using ACS we do not have a public secured Docker port, what we get is an ssh endpoint, and then we should forward port 2375. See ACS docs for more info.
So, to be able to use this extension with ACS you should provide another service configuration that uses SSH and port forwarding for a Docker Host.

Cannot set a non-DockerHub registry as Docker Registry endpoint

I am trying to register a custom Docker Registry with VSTS

The custom registry is hosted on ACS(Swarm), installed using these instructions
https://docs.docker.com/registry/
The container is registry:2
https://hub.docker.com/_/registry/
The registry port has been mapped in the ACS agent load balancer and I have tested it is available by telnet

I can save the service with a URL like this
tcp://DEMOagents.westeurope.cloudapp.azure.com:5000

However when I reopen the service configuration, the URL has been reset to:
https://index.docker.io/v1/

Is this plugin locked to Docker Hub?

issue with docker-compose task and custom registry

I am trying to use the docker-compose task and have run into a couple of issues

My VSTS config:

Docker Host Connection: custom linux VM in azure (this is working for docker tasks)
Docker Registry Connection: different custom linux VM in azure (this is working for docker tasks)
Docker Compose File: file referenced from VSTS Git
Docker Compose Command: up -d

Output from the task execution when run from a Release:

##[section]Starting: Docker Compose: up -d
5fff5eff-07b7-4763-a23d-XXXX exists true
ec098a78-e518-4224-94ff-XXXX exists true
[command]/usr/bin/docker login -e ***** -u ***** -p *****
Flag --email has been deprecated, will be removed in 1.13.
Error response from daemon: Get https://registry-1.docker.io/v2/: unauthorized: incorrect username or password
Not found docker-compose: null
##[section]Finishing: Docker Compose: up -d

The issues I see are:

• the task tries to login to Docker Hub rather than the custom registry I have configured in the task
• docker-compose command fails to run

This repo is missing a LICENSE file

This repository is currently missing a LICENSE file.

A license helps users understand how to use your project in a compliant manner. You can find the standard MIT license Microsoft uses at: https://github.com/microsoft/repo-templates/blob/main/shared/LICENSE.

If you would like to learn more about open source licenses, please visit the document at https://aka.ms/license (Microsoft-internal guidance).

docker-compose fails with ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)

I have created a release with a several docker tasks. I have an endpoint to a docker host machine. I am able to run docker commands successfully (like docker run) against the docker host. However, the docker-compose up commands fail with ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661). (In the task I'm using the "Run Services" action).

I can repro this on my client machine by running docker-compose ps against the same host - I get the cert error. However, if I specify the tls settings explicitly like this:

docker-compose -H $dockerHost --tls --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem ps

then the command succeeds. Is there some issue with the environment variables that the task is using to run docker-compose up?