cloudfoundry / buildpackapplifecycle

The bit that takes some bits, mashes them together with other bits, and produces a new bit

buildpackapplifecycle

Note: This repository should be imported as code.cloudfoundry.org/buildpackapplifecycle.

The buildpack lifecycle implements the traditional Cloud Foundry deployment strategy.

The Builder downloads buildpacks and app bits, and produces a droplet.

The Launcher runs the start command using a standard rootfs and environment.

Read about the app lifecycle spec here: https://github.com/cloudfoundry/diego-design-notes#app-lifecycles

Running tests

On linux or windows, please use ginkgo -r from this directory.

On a mac, you should use docker to run the tests on a linux machine

$ docker build -f Dockerfile.linux.test -t buildpackapplifecycle .
$ docker run -it -v .:/workdir buildpackapplifecycle
$> ginkgo -r

buildpackapplifecycle's Issues

`runner.contentsDir` fills up disk on Windows 2012R2

The directory is created here:

buildpackapplifecycle/buildpackrunner/runner.go

Line 206 in 1e1fa5a

runner.contentsDir, err = ioutil.TempDir("", "contents")

Note that it will end up in whatever the current temp directory is (e.g. c:\windows\temp) which will likely be outside the "container" filesystem.

However, it is never deleted. On Linux and Windows2016, the temp directory goes away when the container is destroyed. But since real containers don't exist on Windows2012R2, the temp directory grows over time, which can cause the disk to fill, taking down the cell

Warning about missing start command when Profile is missing

I am deploying a Python application without a Profile using the following manifest.yml:

applications:
- name: web-app
  random-route: true
  memory: 64M
  buildpacks:
  - python_buildpack
  command: python web-app.py

Although a command is specified and actually used (since the app starts and I can open it in the browser), I always see the following error when doing cf push:

...
Staging app and tracing logs...
   No start command specified by buildpack or via Procfile.
   App will not start unless a command is provided at runtime.
   Exit status 0
   Uploading droplet, build artifacts cache...
...

(irrelevant output omitted)

I think that the code that produces this error, only checks for start commands in a Profile but maybe I am missing something: https://github.com/cloudfoundry/buildpackapplifecycle/blob/master/buildpackrunner/runner.go#L109

I am seeing this on scf but looking at the code above this must be an upstream issue.

Here is the example app:

#!/usr/bin/python
from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer

PORT_NUMBER = 8080

#This class will handles any incoming request from
#the browser 
class myHandler(BaseHTTPRequestHandler):
	
	#Handler for the GET requests
	def do_GET(self):
		self.send_response(200)
		self.send_header('Content-type','text/html')
		self.end_headers()
		# Send the html message
		self.wfile.write("Welcome to the app!")
		return

try:
	#Create a web server and define the handler to manage the
	#incoming request
	server = HTTPServer(('', PORT_NUMBER), myHandler)
	print 'Started httpserver on port ' , PORT_NUMBER
	
	#Wait forever for incoming htto requests
	server.serve_forever()

except KeyboardInterrupt:
	print '^C received, shutting down the web server'
	server.socket.close()

Adding the following Procfile in the same directory makes the error message disappear:

web: python web-app.py

Failed to compress build artifacts: signal: broken pipe

After over a hundred successful deploys we are starting to see this error. It does not happen consistently - sometimes the deploy will work fine, and then sometimes it will fail, all with the exact some code. From doing research elsewhere, it seems that it may be a problem with the temp/ directory getting too large. None of the strategies we have heard suggested elsewhere seem to help.

Cannot clone git repositories when system buildpacks are present

Disclaimer: We do not think this repo is at fault here, but we were asked to place it here to start the conversations.

This issue is present with multiple buildpacks that contain both custom git buildpacks and system buildpacks.

When CloudController sends a staging task to Diego, we run the builder with the following arguments:

"run": {
  "path": "/tmp/lifecycle/builder",
  "args": [
    "-buildpackOrder=https://github.com/cloudfoundry/ruby-buildpack,a1d04795-b3c5-4e25-bdad-52d8d1375536_17d67a7a0ac5ddb34107bf1058e9e2b37a7146e1a54b32c551f13760fee91fe0",
    "-skipCertVerify=true",
    "-skipDetect=true",
    "-buildDir=/tmp/app",
    "-outputDroplet=/tmp/droplet",
    "-outputMetadata=/tmp/result.json",
    "-outputBuildArtifactsCache=/tmp/output-cache",
    "-buildpacksDir=/tmp/buildpacks",
    "-buildArtifactsCacheDir=/tmp/cache"
  ]
}

Here, the buildpackOrder contains a custom buildpack and a reference to a system buildpack. The UUID corresponds to the md5 of an asset downloaded via a different section of the staging task known as cached_dependencies. Here is an example of how this looks:

{
  "name": "go_buildpack",
  "from": "https://blobstore.service.cf.internal:4443/read/cc-buildpacks/a1/d0/a1d04795-b3c5-4e25-bdad-52d8d1375536_17d67a7a0ac5ddb34107bf1058e9e2b37a7146e1a54b32c551f13760fee91fe0?md5=gcf6IvatmS0wQO1Vh5dn6Q&expires=1500947864",
  "to": "/tmp/buildpacks/62cdbebce85c4a0f1960ec2c70e09009",
  "cache_key": "a1d04795-b3c5-4e25-bdad-52d8d1375536_17d67a7a0ac5ddb34107bf1058e9e2b37a7146e1a54b32c551f13760fee91fe0",
  "log_source": "",
  "checksum_algorithm": "sha256",
  "checksum_value": "17d67a7a0ac5ddb34107bf1058e9e2b37a7146e1a54b32c551f13760fee91fe0"
}

With this snippet, Diego will download that cached dependency to /tmp/buildpacks/62cdbebce85c4a0f1960ec2c70e09009. (Note: the md5 of a1d04795-b3c5-4e25-bdad-52d8d1375536_17d67a7a0ac5ddb34107bf1058e9e2b37a7146e1a54b32c551f13760fee91fe0 is 62cdbebce85c4a0f1960ec2c70e09009)

Our theory
In the process it created the /tmp/buildpacks in a way that disallows vcap write permissions. Therefore, when the builder attempts to clone to /tmp/buildpacks/<some-uuid> as vcap, it lacks the permissions necessary.

BAL emits confusing error if it is unable to download a zip buildpack

For example, if the url doesn't exist:

$ cf push myapp -b https://example.com/notarealurl.zip
Creating app myapp in org winc / space winc as admin...
OK

Creating route myapp.baboon.cf-app.com...
OK

Binding myapp.baboon.cf-app.com to myapp...
OK

Uploading myapp...
Uploading app files from: /Volumes/SSD/workspace/wats/assets/nora/NoraPublished
Uploading 21.5K, 23 files
Done uploading
OK

Starting app myapp in org winc / space winc as admin...
Creating container
Successfully created container
Downloading app package...
Downloaded app package (3.6M)
Staging...
Failed to run all supply scripts: Failed to read buildpack directory '/tmp/buildpackdownloads/adcea7bc74dff6aa78bed69055931f75' for buildpack 'https://example.com/notarealurl.zip'
Exit status 225
Staging failed: STG: Exited with status 225
Stopping instance 856d65c7-b851-42d0-8287-c4303c21fb2b
Destroying container
Successfully destroyed container

FAILED
Error restarting application: StagingError

Note that this occurs regardless of whether the stack is Linux or Windows

Provide a more descriptive README

Per http://pipr.co/cf-describe

Buildpack detection doesn't output negative results

When I push an app and use autodetection, I don't get information which buildpacks have failed to detect the app. This is very useful in case some of the buildpacks do output some errors during detection. The user then receives output with errors and failure to autodetect buildpack, but it is not obvious that all buildpacks have been tried and all failed. It is also not clear which buildpack the failure came from:

Downloading binary_buildpack...
Downloading ruby_buildpack...
Downloading java_buildpack...
Downloading staticfile_buildpack...
Downloading go_buildpack...
Downloaded ruby_buildpack
Downloading nodejs_buildpack...
Downloaded go_buildpack
Downloading python_buildpack...
Downloaded staticfile_buildpack
Downloading php_buildpack...
Downloaded binary_buildpack
Downloaded java_buildpack
Downloaded nodejs_buildpack
Downloaded php_buildpack
Downloaded python_buildpack
Creating container
Successfully created container
Downloading app package...
Downloaded app package (174B)
Staging...
/tmp/buildpacks/edeb088283706c4ca547c090a8a5df39/bin/detect: line 4: dirname /tmp/buildpacks/edeb088283706c4ca547c090a8a5df39/bin/detect: No such file or directory
/tmp/buildpacks/edeb088283706c4ca547c090a8a5df39/bin/detect: line 4: dirname : command not found
None of the buildpacks detected a compatible application
Exit status 222
Staging failed: Exited with status 222
Destroying container
Successfully destroyed container

FAILED
Error restarting application: NoAppDetectedError

The error comes from this line in nodejs buildpack. This output looks like some buildpack perhaps broke autodetection altogether. It's not possible which buildpack has output the error and that the autodetection then continued and tried the remaining buildpacks as well...

Large credhub interpolation value causes app to not stage or start

When there are many Credhub references, when they are interpolated to a large JSON (e.g. TLS certs) then VCAP_SERVICES can become too large and the app fails to stage/start. See error below.

2019-08-01T15:16:47.645-04:00 [STG/0] [OUT] Downloaded build artifacts cache (128B)
2019-08-01T15:16:48.699-04:00 [STG/0] [OUT] Downloaded app package (27.6M)
2019-08-01T15:16:48.959-04:00 [STG/0] [ERR] Failed to compile droplet: Failed to run finalize script: fork/exec /tmp/buildpacks/77bae9d93ed614882a3ab66e2c39ab8e/bin/finalize: argument list too long
2019-08-01T15:16:49.154-04:00 [STG/0] [OUT] Cell eda33d0c-120e-4025-991c-fd5f1c7d1e68 stopping instance 5195f998-e156-4646-b794-9dbd324ba565
2019-08-01T15:16:49.522-04:00 [STG/0] [OUT] Cell eda33d0c-120e-4025-991c-

Suggested solution is to check for CREDHUB_SKIP_INTERPOLATION environment variable and then bypass the interpolation step. This would allow for individual apps to opt out of interpolation, then the credentials could be retrieved manually by the app (e.g. use Spring Credhub).

credhub/credhub.go

func (c *Credhub) InterpolateServiceRefs(credhubURI string) error {
	if c.os.Getenv("CREDHUB_SKIP_INTERPOLATION") != "" {
		return nil
	}
	if !strings.Contains(c.os.Getenv("VCAP_SERVICES"), `"credhub-ref"`) {
		return nil
	}
	...
}

BAL launcher will emit a harmless error when a binary app is pushed to CF

for example running the following:

cf push binary-app -b binary_buildpack -c 'nc -kl 8080'
cf logs binary-app --recent

you will see the following error:

2017-08-03T14:52:53.28-0400 [APP/PROC/WEB/0] ERR /tmp/lifecycle/launcher: line 5: .profile.d/*: No such file or directory

I think this is due to this line not checking if the directory has files inside it or is empty.

Missing ENV_DIR argument to compile script to ensure heroku buildpack api compat

Seems the https://www.pivotaltracker.com/n/projects/966314/stories/70246426 was implemented for DEA but not backported to diego ?

I can't see the ENV_DIR argument to the compile script invocation nor related tests covering this case, or traces of ENV_DIR constant in the code base