vmware-archive / cf-redis-release Goto Github PK

BOSH release for a Cloud Foundry Redis service broker that supports shared-vm plans

Home Page: https://bosh.io/releases/github.com/pivotal-cf/cf-redis-release

License: Apache License 2.0

Shell 11.56% HTML 31.18% Ruby 57.26%

cf-redis-release's Introduction

Cloud Foundry Redis Release

This repository contains a BOSH release for a Cloud Foundry Redis service broker. It supports shared-vm plans. Dedicated-vm plans are being deprecated.

There is no active feature development for this repository. The repository will be supported until July 31st 2019, after which we will continue to support open source Redis deployments in the shared-redis-release repo. Note: Breaking change > version 434.3.6 - the redis.limits.maxclients property is now redis.maxclients

git clone https://github.com/pivotal-cf/cf-redis-release ~/workspace/cf-redis-release
cd ~/workspace/cf-redis-release
git submodule update --init --recursive

Deployment dependencies

BOSH CLI v2+ (you may use the old BOSH CLI but instructions will use the new one)
direnv (or set environment variables yourself)
a bosh director
a cloud foundry deployment

Deployment

Run the following steps:

fill out the following environment variables of the .envrc.template file and save as .envrc or export them. All or almost all the variables are required for tests but these are the minimum required for deploy:
- BOSH_ENVIRONMENT
- BOSH_CA_CERT
- BOSH_CLIENT
- BOSH_CLIENT_SECRET
- BOSH_DEPLOYMENT
if you're using the .envrc file
```
direnv allow
```

upload dependent releases

bosh upload-release http://bosh.io/d/github.com/cloudfoundry-incubator/cf-routing-release
bosh upload-release http://bosh.io/d/github.com/cloudfoundry/syslog-release
bosh upload-release http://bosh.io/d/github.com/cloudfoundry-incubator/bpm-release # required for routing 180+

Populate a vars file (using manifest/vars-lite.yml as a template), save it to secrets/vars.yml. You will need values from both your cloud-config and secrets from your cf-deployment.

To deploy:

bosh upload-stemcell https://s3.amazonaws.com/bosh-core-stemcells/warden/bosh-stemcell-97.3-warden-boshlite-ubuntu-xenial-go_agent.tgz
bosh create-release
bosh upload-release
bosh deploy --vars-file secrets/vars.yml manifest/deployment.yml

Network Configuration

The following ports and ranges are used in this service:

broker vm, port 12350: access to the broker from the cloud controllers
broker vm, ports 32768-61000: on the service broker from the Diego Cell and Diego Brain network(s). This is only required for the shared service plan
dedicated node, port 6379: access to all dedicated nodes from the Diego Cell and Diego Brain network(s)

Testing

install gem requirements
```
bundle install
```
run the tests
```
bundle exec rspec spec
```

Known Issues

Possible Data Leak when disabling Static IPs

In the past cf-redis-release expected to be deployed with static IPs for dedicated nodes specified in the manifest. It is often more desired to deploy without static IPs leaving BOSH to manage IP allocation.

There is a risk of data leak when transitioning from static to dynamic IPs. Consider the following scenario:

The operator:

has deployed cf-redis-release with static IPs
now decides to use dynamic IP allocation, and removes the static IPs from the manifest
then DOES NOT remove the static IPs from cloud config
re-deploys cf-redis-release with the new manifest

In the previous scenario BOSH will dynamically allocate IPs to the dedicated instances. BOSH will not try to re-use the previous IPs since those are still restricted in the cloud config. Since the IPs have changed, application bindings will break and the state in the broker will be out of sync with the new deployment. It is possible that previously allocated instances containing application data are erroneously re-allocated to another unrelated application, causing data to be leaked.

In order to avoid this scenario, the operator must remove the reserved static IPs in the cloud config at the same time as they are remove from the manifest. This will allow BOSH to keep the same IP addresses for the existing nodes.

To safely transition from static to dynamic IPs:

look up the static IPs that were specified in the manifest when deploying your dedicated nodes
ensure these IPs are no longer included in the static range of the network in your cloud config
remove the static IPs from the manifest
deploy using the manifest without static IPs

Errands

The Cf-Redis Service Broker offers the following BOSH errands:

broker-registrar

Communicates to the CloudFoundry Cloud Controller which maintains the database for the CF Marketplace. This enables the service plans on the CF Marketplace so that App Developers can create and delete service instances of those plans.

Note: As of v434.0.13, the broker-registrar will disable service access to the dedicated-vm service plan as the service plan is being deprecated.

broker-deregistrar

Communicates with the CF Cloud Controller to remove the cf-redis broker and service plans from the CF Marketplace as well as delete the broker deployment from BOSH.

Note: If any service instances exist, the errand will fail. This is to prevent orphan deployments which the CF Cloud Controller will lose record of but will continue to live as a BOSH deployment which would continue to incur costs.

If you wish to remove the service broker, perform a cf delete-service SERVICE_INSTANCE and remove all service instances associated with the broker and run the errand after all service instances have been removed.

smoke-tests

Runs smoke tests which tests the lifecycle and functionality of the both the dedicated-vm and shared-vm services. See the Redis documentation for more information.

remove-deprecated-functionality

Available as of v434.0.13. Communicates with the CF Cloud Controller to disable service access to the dedicated-vm service plan. This allows operators to remove the dedicated-vm service plan from the marketplace in order to prevent App Developers from creating new service instances in preparation for the deprecation of the dedicated-vm plan.

cleanup-metadata-if-dedicated-disabled

Available as of v434.0.15. Communicates with the CF Cloud Controller to purge references to dedicated-vm service instances if no dedicated-vms have been provisioned. The purpose for this errand is to provide a smoother experience for migrating to On-Demand Instances given the Ops Manager tile deployment flow. Intended as a temporary errand during deprecation of dedicated-vms.

cf-redis-release's People

Contributors

Stargazers

Watchers

cf-redis-release's Issues

redis can't bind ipv6 address on new bosh stemcells, fails to start

stemcell: bosh-stemcell-3363.20-openstack-kvm-ubuntu-trusty-go_agent
cf-redis-release: 429.0.0
routing-release: 0.152.0

appears as though redis processes fail to bind, due to attempting to bind to both ipv4 & ipv6, but I'm guessing ipv6 networking has been disabled as of https://github.com/cloudfoundry/bosh/releases/tag/stable-3363.19

because of this, deployment fails with Failed: 'dedicated-node/0 (ab055142-1184-4533-b26c-4c2065df3ae2)' is not running after update. Review logs for failed jobs: redis (00:03:49)

from dedicated-node/0's /var/vcap/sys/log/redis/redis.log

Creating Server TCP listening socket *:6379: unable to bind socket

nothing else is listening on this port:

dedicated-node/ab055142-1184-4533-b26c-4c2065df3ae2:/var/vcap/sys/log/redis# netstat -anp |grep LISTEN
tcp        0      0 127.0.0.1:54321         0.0.0.0:*               LISTEN      11952/agent
tcp        0      0 127.0.0.1:33331         0.0.0.0:*               LISTEN      851/bosh-agent
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1805/sshd
tcp        0      0 0.0.0.0:4443            0.0.0.0:*               LISTEN      12017/nginx.conf
tcp        0      0 0.0.0.0:55937           0.0.0.0:*               LISTEN      666/rpc.statd
tcp        0      0 127.0.0.1:2822          0.0.0.0:*               LISTEN      2258/monit
tcp        0      0 127.0.0.1:2825          0.0.0.0:*               LISTEN      851/bosh-agent
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      629/rpcbind

I believe the shared-node's redis processes had the same issue, I just didn't realize it til after I nuked my deployment and tried to recreate fresh... then encountered the same on dedicated-node as it was deployed first.

It doesn't seem like the redis bind address is configurable in the job params.

edit: confirmed that adding bind 0.0.0.0 to /var/vcap/store/redis/redis.conf allows redis to startup successfully

test gitbot (2)

cannot fix redis password

Hello
I would like to fix the redis auth password and manage it in credhub
Unfortunately even if i set it in my deployment by adding the following opsfile

- type: replace
  path: /properties/redis/security?/require_pass
  value: ((redis_password))

- type: replace
  path: /variables?/name=redis_password?
  value: 
    name: redis_password
    type: password

the provided password is well filled in /var/vcap/jobs/dedicated-node/config/redis.conf

...
# Warning: since Redis is pretty fast an outside user can try up to
# 150k passwords per second against a good box. This means that you should
# use a very strong password otherwise it will be very easy to break.
#

requirepass 1OujjY5MctbW9OZkfuT61rawh9Icxg

but it is overwritten with an uuid in /var/vcap/store/redis/redis.conf, which is the configuration file used by redis instance

aof-rewrite-incremental-fsync yes
requirepass b3c18089-ca3e-4c60-8f87-c42f27627253

Is there anything I missed ?

Tag the repo for each bosh release

deleted as unrelated

Dangerous silent purge service offering in deregistrer errand

The broker-deregistrar errand purges the service offering

cf purge-service-offering -f $BROKER_SERVICE_NAME

I understand its convenient for bosh release dev, but i believe its very dangerous to set this in the broker-deregistrer errand.
If production operator has to delete the a redis broker service with existing service instances, he should explicitly purge the service instances, before launching the errand

Merge option not there.

Migrating from redis 424 to 426 we checked that merge option has been removed from stubs and we have to manually put values in the sample stubs.
Can you please confirm us whether same process will be followed is future releases?

README.md addition

Hi there!

We wanted to add the following text to the README. We'd like to know what the proper procedure for submitting a commit like this to this project. Should this be a PR, or just a commit to a certain branch?

Thanks,

Jim && Jonathan

## Deploying

### Subnet ACL

Allow at least the following: 
- [ ] Destination port 443 access to the service broker from the cloud controllers
- [ ] All destination ports on the redis dedicated nodes from the DEA network(s).

### DNS

Configure redis-broker((meta.environment)) to point to the static IP assigned 
to the redis-broker.

test gitbot

Extend the broker to support multiple plans

Hi,

I was wondering if you would accept a contribution for broker plans portfolio extension (e.g. dedicated-small, dedicated-medium, dedicated-large)
Alternatively, would you accept that the broker could support some extensions (without being forked) ?

Thanks in advance

Eugenio

Is there plan to support shared-plan?

HI thanks to this repo I can deploy cf-redis-release

I wonder If I make shared plan, How can I do?
As mentioned below, Is it right limitation?
https://docs.pivotal.io/redis/overview.html#service
how to deploy multi redis_broker instance?
I can't see thing clearly expept multi dedicated-node deploying
+-------------------+---------+----------+-----------+
| VM | State | VM Type | IPs |
+-------------------+---------+----------+-----------+
| cf-redis-broker/0 | running | redis_z1 | 10.0.0.31 |
| dedicated-node/0 | running | redis_z1 | 10.0.0.32 |
+-------------------+---------+----------+-----------+
Is there no need to redis clustering?, such as p-mysql galera cluster
As i know redis backup data is stored in persistent disk
but when one of redis node occurs system failures, Is there any problem with the service?

Custom Space Names

Currently getting the following error running the redis smoke tests

Creating space cf-redis-smoke-tests-1-SPACE-09f61cc0310c2175 in org system as admin...
OK
Assigning role SpaceManager to user admin in org system / space cf-redis-smoke-tests-1-SPACE-09f61cc0310c2175 as admin...
The user exists in multiple origins. Specify an origin for the requested user from: 'uaa', 'ldap'
FAILED

To resolve am trying to set the space name to be a specific space name. However recently it seems this option has been made non-configurable - https://github.com/pivotal-cf/cf-redis-release/blob/2ac76cb681c41c5dfb326e22e75df9cd14e9b684/jobs/smoke-tests/templates/config.json.erb#L16

Can this be updated back to
'use_existing_space' => p('cf.space_name') ?

silent broker route-registrar failure

We recently had an incident on a p-redis deployment.
The broker route was not anymore published to go routers

Howerver, the job route-registrar status was ok.

+-----------------------------------------------------------+---------+-----+----------+----------------+
| Instance                                                  | State   | AZ  | VM Type  | IPs            |
+-----------------------------------------------------------+---------+-----+----------+----------------+
| cf-redis-broker/0 (8aa5ba07-dc48-43bc-a581-717c5b0655e0)* | running | n/a | redis_z1 | 192.168.30.190 |
+-----------------------------------------------------------+---------+-----+----------+----------------+

Inside the broker vm, i could see stale nats connection

==> route-registrar.stdout.log <==
[2016-06-02 09:42:56.309080796 +0000 UTC] - Adding NATS server 192.168.26.2:4222, for user nats.[2016-06-02 09:42:56.310282126 +0000 UTC] - Successfully connected to NATS.^C                                                                                                                                                                 
root@5031f7de-e954-4ec1-9c6a-3d17950624d8:/var/vcap/sys/log/route-registrar# ls
route-registrar.stderr.log  route-registrar.stdout.log                         
root@5031f7de-e954-4ec1-9c6a-3d17950624d8:/var/vcap/sys/log/route-registrar# netstat -na |grep 4222
tcp        0      0 192.168.30.190:37188    192.168.26.2:4222       CLOSE_WAIT 
tcp        0      0 192.168.30.190:49318    10.106.236.102:4222     ESTABLISHED

Connection to cf nats is CLOSE_WAIT. Restarting the route-registrar job fixed the issue.

Looks like issues we met with old nats clients. Maybe the route registrar blobs inside cf-redis-release needs an update ? or should include upstream route-registrer bosh release ?

upload syslog-migration link is broken

Forth step is incorrect
4.upload syslog-migration release 11
bosh upload-release https://github.com/pivotal-cf/syslog-migration-release/releases/download/v8/syslog-migration-11.tgz
Link https://github.com/pivotal-cf/syslog-migration-release/releases/download/v8/syslog-migration-11.tgz is broken. This release has been depricated in favor of syslog-release.

Also "ops-file adds a GCP specific vm_extension: public_ip" there is no public_ip properties for gcp vm_extention.
https://bosh.io/docs/google-cpi/#vm-types

Releases notes are lovely

If its possible, release notes for each tag/release would be awesome. Thanks!

How to monitor redis with prometheus ?

For example, we have deployment with:

Instance                                              Process State  AZ   IPs  
cf-redis-broker/84a0e577-4d14-4712-b376-7f2b3e13ba66  running        AZ1  10.1.1.1  
dedicated-node/0a2687b5-4a8d-4321-a36c-a2fd5052c18a   running        AZ2  10.1.1.2
dedicated-node/0dfd7963-c700-4f0a-a2fa-3d218949c3fc   running        AZ2  10.1.1.3
dedicated-node/0eb7ca0e-51c3-412c-a4ea-e1a8c3f8a2be   running        AZ3  10.1.1.4

What is the best practice of redis monitoring:

By the reason of https://github.com/pivotal-cf/cf-redis-release/blob/master/jobs/dedicated-node/spec:
redis.security.require_pass:
description: Require clients to issue AUTH PASSWORD before processing any other commands. This will be overwritten everytime the broker allocates it as in use.

redis pass is changing each time, end redis_exporter lost connection with:

time="2019-05-09T19:20:49Z" level=debug msg="Trying DialURL(): redis://127.0.0.1:6379"
time="2019-05-09T19:20:49Z" level=debug msg="DialURL() failed, err: ERR invalid password"

Help to resolve the issue,
Thanks

New Tile/BOSH Deployment installs fail

Because of process destroyer. It has impacted both PCF tile clients on the latest tile version as of today as well as a couple of OSS folks using this release have reported it.

In order to 'unstick' the deployment it is required to ssh onto the VM and manually stop process deployer and then deploy again.

Stopping Monitored Services: Stopping services '[process-destroyer]' errored

We are using cf-redis release 426.0.0. when we did small change like cf password change or stemcell version change and redeployed redis-service with chnages it gives error as follow:-

Action Failed get_task: Task 9567203f-dea9-416f-5632-e6a0b6bdc189 result: Stopping Monitored Services: Stopping services '[process-destroyer]' errored (00:01:15)

Error 450001: Action Failed get_task: Task 9567203f-dea9-416f-5632-e6a0b6bdc189 result: Stopping Monitored Services: Stopping services '[process-destroyer]' errored

We also tried to upgrade Redis to version 428.0.0. But getting the same error for
process-destroyer as above.

could you please advice.

New repo/home for this release?

Hello!

According to the README.md file, this repo "...will be supported until July 31st 2019." As it is now August 1st, 2019, would it be possible to add a link to the new home for this repo? :-)

The 'requirepass' property does not apply.

ISSUE

The requirepass property does not apply.

CONTEXT

I deployed cf-redis-release using the manifest with the requirepass attribute as shown below.

properties:
  redis:
    security:
      require_pass: "5a38c7c3-6cc2-434f-8713-f2344b77e47g"

However, this attribute did not apply to redis.

$ ./redis-cli -a 5a38c7c3-6cc2-434f-8713-f2344b77e47g
127.0.0.1:6379> info
NOAUTH Authentication required.

I checked two redis.conf files to analyze the cause.
First, the redis.conf configuration file in the /var/vcap/jobs/dedicated-node/config/ path was applied as shown below.

$ cat /var/vcap/jobs/dedicated-node/config/redis.conf | grep requirepass
requirepass 5a38c7c3-6cc2-434f-8713-f2344b77e47g

However, the redis.conf configuration file in the /var/vcap/store/redis/ path was not applied as shown below.

$ cat /var/vcap/store/redis/redis.conf | grep requirepass
requirepass 4a38c7c3-6cc2-434f-8713-f2344b77e47f

QUESTION

Why does this happen? Is this my mistake?
How is the redis.conf configuration file in the /var/vcap/store/redis/ path configuration file generated? (e.g. the redis.conf configuration file in the /var/vcap/jobs/dedicated-node/config/ path = cf-redis-release/jobs/dedicated-node/templates/redis.conf.erb)
How is the requirepass property generated? (Automatically generated values)

Redis::CannotConnectError: Timed out connecting to Redis

Sample redis application is not working, giving below error
Redis::CannotConnectError: Timed out connecting to Redis on --- broker-url:port---
Below is snap shot

Please suggest how to rectify the issue.

This is is not issue request for feature

hi thanks for your redis opensource, i wounder this release also supports high availability release ?.

bosh.io not showing latest releases available for download

The latest available release on bosh.io is v428.0.0. Later minor releases are not being shown:
http://bosh.io/releases/github.com/pivotal-cf/cf-redis-release?all=1

Colocated jobs creating package name conflicts

It looks like on July 4th the following changes were made:
e5a2eef

I believe this is the same issue descibed here:
#84

Colocating these jobs causes our deployment to fail just like it did before we updated the include the fix for the above issue. Here is the error we see:

Error: Package name collision detected in instance group 'cf-redis-broker': job 'cf-cli/cf-cli-6-linux' depends on package 'cf-cli/cf-cli-6-linux' with fingerprint 'af6517f138b3ed0584b263f6273f30f22f6e5c9d', job 'cf-redis/broker-registrar' depends on package 'cf-redis/cf-cli-6-linux' with fingerprint '0971d73c486be23f7d87702ddc75f4d6565942f5'. BOSH cannot currently collocate two packages with identical names and different fingerprints or dependencies.

Just thought i would give a heads up since my team happened to notice this.

Keenan

Connection Refused Error to the client

Hi Team!,

I followed the instructions and was able to successfully deploy redis as a service to my CF (bosh-light).

However when i try to use the instance via an app i get 'Connection Refused' Error

Via go client

dial tcp 10.244.3.46:40470: connection refused

as well as via ruby client when i try to use the example ruby app.

2015-08-12T18:28:13.57-0400 [RTR/0]      OUT redis-example-app.10.244.0.34.xip.io - [12/08/2015:22:28:03 +0000] "POST /foo HTTP/1.1" 404 8 18 "-" "curl/7.35.0" 10.0.2.15:57640 x_forwarded_for:"192.168.50.1, 10.0.2.15" vcap_request_id:cd0c4d45-242a-403c-518a-ad543539275d response_time:10.032475076 app_id:384c6268-9ee1-4a36-a4fd-e69081c58d0b
2015-08-12T18:34:38.18-0400 [App/0]      ERR Redis::CannotConnectError - Error connecting to Redis on 10.244.3.46:40470 (ECONNREFUSED):
2015-08-12T18:34:38.18-0400 [App/0]      ERR /home/vcap/app/vendor/bundle/ruby/2.2.0/gems/redis-3.1.0/lib/redis/client.rb:309:in `rescue in establish_connection'
2015-08-12T18:34:38.18-0400 [App/0]      ERR /home/vcap/app/vendor/bundle/ruby/2.2.0/gems/redis-3.1.0/lib/redis/client.rb:304:in `establish_connection'

does this work with opensource CF(210)? If yes how do i debug further?

Support for multiple nats addresses

Change nats.address property to nats.machines that takes a list of ips.

Bosh-Lite deploy of redis fails due to unresolved nodes

Bosh-lite install of redis fails on deploy after using the generate-deployment-manifest script in this repo

Steps to reproduce:

Pre-existing BOSH-Lite CF deployment
Run ./scripts/generate-deployment-manifest warden > ~/manifests/cf-redis.yml
bosh upload release https://bosh.io/d/github.com/cloudfoundry-incubator/cf-routing-release?v=0.143.0
Within cf-redis-release, bosh upload release releases/cf-redis/cf-redis-424.yml
bosh deployment ~/manifests/cf-redis.yml and bosh -n deploy

Received the following error upon deploy:

Deploying
---------
Are you sure you want to deploy? (type 'yes' to continue): yes

Director task 128
Deprecation: Ignoring cloud config. Manifest contains 'networks' section.

  Started preparing deployment > Preparing deployment. Done (00:00:01)

Error 100: Unable to render instance groups for deployment. Errors are:
   - Unable to render jobs for instance group 'cf-redis-broker'. Errors are:
     - Unable to render templates for job 'cf-redis-broker'. Errors are:
       - Error filling in template 'registrar_settings.yml.erb' (line 3: Can't find property '["cf.nats.port"]')

Was able to resolve the issue by editing templates/sample_stubs/meta-warden.yml with the following properties under meta.cf:

    nats:
      host: 10.244.0.6
      port: (( .meta.nats.port ))
      username: (( .meta.nats.user ))
      password: (( .meta.nats.password ))

Posting this as a temporary work around if this is indeed a bug.

Redis Manifest with Sentinel Job

Hi,

Does the latest release of cf-redis-release support sentinel feature of redis. If yes, could you please provide the sample manifest with sentinel Job sections.

Thanks,
Guru.

Imcomplete removal of CF CLI from release

Hi cf-redis-team,

We are getting the following error when trying to upgrade from 434.2.8 to 434.2.10.

Task 223563 | 08:32:54 | Preparing deployment: Preparing deployment (00:00:02)
                      L Error: Package name collision detected in instance group 'redis-broker': job 'cf-redis/broker-registrar' depends on package 'cf-redis/cf-cli-6-linux' with fingerprint '0971d73c486be23f7d87702ddc75f4d6565942f5', job 'cf-cli/cf-cli-6-linux' depends on package 'cf-cli/cf-cli-6-linux' with fingerprint 'b91c0df20f4cda678be03db1f00e0c40449bcd32'. BOSH cannot currently collocate two packages with identical names and different fingerprints or dependencies.
Task 223563 | 08:32:56 | Error: Package name collision detected in instance group 'redis-broker': job 'cf-redis/broker-registrar' depends on package 'cf-redis/cf-cli-6-linux' with fingerprint '0971d73c486be23f7d87702ddc75f4d6565942f5', job 'cf-cli/cf-cli-6-linux' depends on package 'cf-cli/cf-cli-6-linux' with fingerprint 'b91c0df20f4cda678be03db1f00e0c40449bcd32'. BOSH cannot currently collocate two packages with identical names and different fingerprints or dependencies.

I think this is because cf-redis-release 434.2.10 still contains the cf-cli-6-linux package even though the manifest uses the package from the cf-cli release. I tried to fix this by reverting back to the cf-cli-6-linux package that is still in the cf-redis-release using the following ops file

- type: replace
  path: /instance_groups/name=cf-redis-broker/jobs/name=cf-cli-6-linux/release
  value: cf-redis
- type: remove
  path: /releases/name=cf-cli

However I then got the error

Error: Job 'cf-cli-6-linux' not found in Template table

Which makes sense, as the job no longer exists in the new release.

kind regards,

Pete

Submodule update Fails

Problem

Running git submodule update --init --recursive on a freshly cloned repository fails with the following error:

Submodule path 'packages/ruby': checked out '39c9d158b81184f14d0c1ec1ab39702c816992e8'
Submodule path 'src/cf-redis-broker': checked out '8572eda575fe653a4fbff319b93c4184bead5ba7'
Submodule path 'src/cf-redis-smoke-tests': checked out '7aecad258297d30c5d0f06e3e5d272090ddcc4fb'
Submodule 'assets/cf-redis-example-app' ([email protected]:pivotal-cf/cf-redis-example-app) registered for path 'src/cf-redis-smoke-tests/assets/cf-redis-example-app'
Cloning into '/tmp/build/get/src/cf-redis-smoke-tests/assets/cf-redis-example-app'...
Host key verification failed.
fatal: Could not read from remote repository.
	
Please make sure you have the correct access rights and the repository exists.
fatal: clone of '[email protected]:pivotal-cf/cf-redis-example-app' into submodule path '/tmp/build/get/src/cf-redis-smoke-tests/assets/cf-redis-example-app' failed
Failed to clone 'assets/cf-redis-example-app'. Retry scheduled
Cloning into '/tmp/build/get/src/cf-redis-smoke-tests/assets/cf-redis-example-app'...
Host key verification failed.
fatal: Could not read from remote repository.
	
Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:pivotal-cf/cf-redis-example-app' into submodule path '/tmp/build/get/src/cf-redis-smoke-tests/assets/cf-redis-example-app' failed
Failed to clone 'assets/cf-redis-example-app' a second time, aborting
Failed to recurse into submodule path 'src/cf-redis-smoke-tests'

concourse github-resource says 'no versions available'

Hi friends,

I'm using concourse to get github release of pivotal-cf/cf-redis-release and deploy it to bosh.
But I got the message like below
'no versions available'

'https://api.github.com/repos/pivotal-cf/cf-redis-release/releases' returns just [ ] while I expected
lists of releases.

Is there anybody who using concourse github-release to get and deploy cf-redis-release??
Thank you for helping me in advance.

Spiff templates

Configurable shared instances ports

We would like to suggest and contribute a way to control a smaller preferred port range for shared redis instances.

Currently, when using a shared instance, ports are dynamically assigned for each service instance, taking the first available port see https://github.com/pivotal-cf/cf-redis-broker/blob/master/system/free_tcp_port.go#L8-L18 apparently randomly

The operational impact of having such a large port range is that firewall rules to allow consumption of shared service instances are wide and can't be restricted to a small port range.

This would be made available as a new property such as:

redis.broker.shared_min_port:
    description: The preferred lower port range to allocate for shared instances (e.g. "30000"). If no free port is available within this range, the service instance creation request will fail.
    default: 1024
redis.broker.shared_max_port:
    description: The preferred upper port range to allocate for shared instances (e.g. "40000"). If no free port is available within the preferred range, the service instance creation request will fail.
    default: 65535

This would complement the existing redis.broker.dedicated_port property which allows to control the port for dedicated instances:
https://github.com/pivotal-cf/cf-redis-release/blob/master/jobs/cf-redis-broker/spec#L170

If this suggestion seems of interest, we'll go ahead and submit PRs on the broker and release repos.
If this suggestion is not of interest and would not be merged, please comment so that we avoid working on such contrib.

/CC @ZiedZinelabidine @ealessandria-orange

Make submodules https for public access

I cannot sync the submodules to create dev releases with patches due to git@ private submodules. Could you please change to https:// submodules?

Please make sure you have the correct access rights
and the repository exists.
Clone of '[email protected]:pivotal-cf/cf-redis-broker.git' into submodule path 'src/cf-redis-broker' failed

Null bytes appended to AOF file during brutal VM shutdown

We recently encountered an issue where our NATS certificates expired and BOSH was not able to talk to its agents anymore. We resolved that issue by generating new NATS certs on the director and then running a bosh cck. Because the bosh director had no way to talk to the agents, it had no choice but to shoot the unresponsive VMs in the head and build new ones.

When the redis VM came back up, one of the redis instances on there failed to start up due to a corrupt AOF file. Examining the AOF file in an editor, it appeared that the every transaction had correctly been recorded in the AOF file, but there were 12 null 0x00 bytes appended to the file after the last transaction. Running the redis-check-aof --fix <file.aof> command quickly fixed the problem and the instance was then able to start.

The redis.conf already has the options enabled for loading truncated AOF file, but in this case the file was not truncated, but contained extra invalid bytes at the end.

A few days after we'd hit that problem, we were faced with the same situation again when the next set of NATS certs expired in a totally different CF deployment. Once again, when the vm was recreated by a bosh cck one of the redis AOF files had a few extra null bytes appended to the end and the instance was unable to start.... and once again, running the redis-check-aof file fixed things in a fraction of a second and got everything up and running again.

I've been contemplating raising a PR against cf-redis-release with a minor tweak to the prestart.erb template along the following lines...

fixtool=$(find /var/vcap/packages/redis/ -name redis-check-aof -print -quit)
find /var/vcap/store/cf-redis-broker/redis-data -name appendonly.aof | xargs -n 1 ${fixtool} --fix

Although it's a simple fix, it would ensure that all of the AOF files would be checked each time redis gets started.

I thought that rather than just raise the PR, I'd open an issue first to discuss the approach to the problem - I suspect that someone who knows Redis better than me might have a more elegant fix to offer than my shoddy bash script tweaks.

Would it be possible to update bosh.io/releases to latest release

Would it be possible to update bosh.io/releases to latest release. It looks to be a few versions behind this repo. We are seeing an issue with the upstream smoke-test sample app where the buildpack is now being specified, we get a random failure on the smoke-tests based on it trying to download all the versions of buildpacks.

Update redis to 3.2.4 to higher to address CVE and instability

https://raw.githubusercontent.com/antirez/redis/3.2/00-RELEASENOTES mentions

Redis 3.2.4 Released Mon Sep 26 08:58:21 CEST 2016

Upgrade urgency CRITICAL: Redis 3.2 and unstable contained a security
vulnerability fixed by this release.

AFFECTED VERSIONS:
All Redis 3.2.x versions are affected.

I understand cf-redis-release is running 3.2.1 vulnerable version (in the master branch and in the latest v436.3 tag

Is there plans to update to redis 3.2.4 or 3.2.5 ?

remove cf-redis/432.0.0 as it contains no-op job that is invalid

bosh.io tries to pick this version up, however, it cannot get the tarball built because job definition was invalid for no-op job.

Push Changes to hide CF_ADMIN_PASSWORD to bosh.io

Would it be possible to push the changes from d78fd58da4cc6accd2bad23cb0923d54cb9c9156 to hide secrets to bosh.io

Thanks,
Keenan