vmware-tanzu / astrolabe

Data protection framework for complex applications

Overview

Modern distributed applications are complex. They may consist of containers, virtual machines, databases, load balancers, Kubernetes namespaces and other objects. These applications need to be protected against data loss, service outages and disasters. Data protection applications have to contend with new types of object to protect and the complex topologies of the applications. Astrolabe is a framework for data protection applications to discover, backup/replicate data and restore complex applications. It provides a data protection-centric model of applications with APIs for snapshotting, data extraction, copying and restoration.

The Astrolabe project consists of:

• Kubernetes CRD API (in-progress)
• OpenAPI API specification
• Reference server implementation
• Reference Protected Entity implementations

astrolabe's Issues

Add maintainers file

We need a maintainers file that has details on ownership.

In the repositories, there needs to be an MAINTAINERS file that lists all maintainers of the project, and potential stakeholders (EXAMPLE). (From https://github.com/vmware-tanzu/community-engagement/blob/main/GUIDELINES.md)

Unable to specify target datastore for restore

First of all, we have a cluster without vSAN.
StorageClass is configured with single static datastore:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: example-vanilla-block-sc
  selfLink: /apis/storage.k8s.io/v1/storageclasses/example-vanilla-block-sc
  uid: e00d5987-c67b-46b1-b6ce-c73dcd28b4c8
  resourceVersion: '11659564'
  creationTimestamp: '2020-08-11T12:29:14Z'
  annotations:
    storageclass.kubernetes.io/is-default-class: 'true'
provisioner: csi.vsphere.vmware.com
parameters:
  DatastoreURL: 'ds:///vmfs/volumes/0c06e8ae-9ff93b29/'
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate

Backup works okay, I can see the volume in S3 bucket.
The restore procedure is failing for me with error:

time="2020-08-12T11:53:59Z" level=info msg="CreateCnsVolumeInCluster called with args, metadata: {{{} {{{} {{} 8c8893f8-fbc8-464c-ac24-7633b3ab8260} pvc-29775058-741d-4617-9a3e-ce18cc5bdead 2020-08-12 11:18:38.569 +0000 UTC 0xc000b78818 0xc000b7884c 0xc000b7885d 0xc000b7886d <nil> []} 300 [disk] [{{} 73e7ad42-ea37-4af8-94f6-90ccc0e1ae56}]}} Datastore:datastore-29 []}" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:390"
time="2020-08-12T11:54:10Z" level=info msg="Filtering out the retry upload request which comes in before next retry time" controller=upload generation=21 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/upload_controller.go:111" name=upload-fa118876-cefe-44ba-9239-57fa56077826 namespace=velero phase=UploadError
time="2020-08-12T11:54:10Z" level=info msg="Ignore retry upload request which comes in before next retry time, upload CR: upload-fa118876-cefe-44ba-9239-57fa56077826" controller=upload currentTime="2020-08-12 11:54:10.26083292 +0000 UTC m=+2658.530258347" generation=21 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/upload_controller.go:117" name=upload-fa118876-cefe-44ba-9239-57fa56077826 namespace=velero nextRetryTime="2020-08-12 12:08:03 +0000 UTC" phase=UploadError
time="2020-08-12T11:54:10Z" level=info msg="Filtering out the retry download request which comes in before next retry time" controller=download generation=14 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:105" name=download-ea989262-9017-4c97-aa83-773dc1530ca7-556eeab7-0184-4dd9-9669-b390b74f7681 namespace=velero phase=InProgress
time="2020-08-12T11:54:10Z" level=info msg="Ignore retry download request which comes in before next retry time, download CR: download-ea989262-9017-4c97-aa83-773dc1530ca7-556eeab7-0184-4dd9-9669-b390b74f7681" controller=download currentTime="2020-08-12 11:54:10.549658851 +0000 UTC m=+2658.819084243" generation=14 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:111" name=download-ea989262-9017-4c97-aa83-773dc1530ca7-556eeab7-0184-4dd9-9669-b390b74f7681 namespace=velero nextRetryTime="2020-08-12 11:58:59 +0000 UTC" phase=InProgress
time="2020-08-12T11:54:40Z" level=info msg="Filtering out the retry upload request which comes in before next retry time" controller=upload generation=21 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/upload_controller.go:111" name=upload-fa118876-cefe-44ba-9239-57fa56077826 namespace=velero phase=UploadError
time="2020-08-12T11:54:40Z" level=info msg="Ignore retry upload request which comes in before next retry time, upload CR: upload-fa118876-cefe-44ba-9239-57fa56077826" controller=upload currentTime="2020-08-12 11:54:40.260874529 +0000 UTC m=+2688.530299916" generation=21 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/upload_controller.go:117" name=upload-fa118876-cefe-44ba-9239-57fa56077826 namespace=velero nextRetryTime="2020-08-12 12:08:03 +0000 UTC" phase=UploadError
time="2020-08-12T11:54:40Z" level=info msg="Filtering out the retry download request which comes in before next retry time" controller=download generation=14 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:105" name=download-ea989262-9017-4c97-aa83-773dc1530ca7-556eeab7-0184-4dd9-9669-b390b74f7681 namespace=velero phase=InProgress
time="2020-08-12T11:54:40Z" level=info msg="Ignore retry download request which comes in before next retry time, download CR: download-ea989262-9017-4c97-aa83-773dc1530ca7-556eeab7-0184-4dd9-9669-b390b74f7681" controller=download currentTime="2020-08-12 11:54:40.549796808 +0000 UTC m=+2688.819222199" generation=14 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:111" name=download-ea989262-9017-4c97-aa83-773dc1530ca7-556eeab7-0184-4dd9-9669-b390b74f7681 namespace=velero nextRetryTime="2020-08-12 11:58:59 +0000 UTC" phase=InProgress
time="2020-08-12T11:55:00Z" level=info msg="CNS volume, 4c4d7872-6740-4837-84fa-8e8254934dbf, created" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:297"
time="2020-08-12T11:55:00Z" level=error msg="Failed to copy data from data source to newly-provisioned IVD protected entity" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager.go:340"
time="2020-08-12T11:55:00Z" level=error msg="Failed to copy from remote repository." Remote PEID="ivd:8c8893f8-fbc8-464c-ac24-7633b3ab8260:ea989262-9017-4c97-aa83-773dc1530ca7" error="copyInt failed: ServerFaultCode: NotFound" error.file="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager.go:217" error.function="github.com/vmware-tanzu/astrolabe/pkg/ivd.(*IVDProtectedEntityTypeManager).Copy" logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/dataMover/data_mover.go:118"
time="2020-08-12T11:55:00Z" level=info msg="Download status updated from InProgress to Retry" controller=download generation=14 logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/download_controller.go:348" name=download-ea989262-9017-4c97-aa83-773dc1530ca7-556eeab7-0184-4dd9-9669-b390b74f7681 namespace=velero phase=InProgress

I think that the problem lies in creating the Volume. The datamanager tries to create a volume on top of two datastores but this fails because we are not using vSAN, see this intercepted request:

<Envelope xmlns="http://schemas.xmlsoap.org/soap/envelope/"><Header xmlns="http://schemas.xmlsoap.org/soap/envelope/"><vcSessionCookie>349b5f29187321d6074726c28294ee865d972333</vcSessionCookie></Header><Body><CnsCreateVolume xmlns="urn:vsan"><_this type="CnsVolumeManager">cns-volume-manager</_this><createSpecs><name>pvc-29775058-741d-4617-9a3e-ce18cc5bdead</name><volumeType>BLOCK</volumeType><datastores type="Datastore">datastore-29</datastores><datastores type="Datastore">datastore-30</datastores><metadata><containerCluster><clusterType>KUBERNETES</clusterType><clusterId>***</clusterId><vSphereUser>[email protected]</vSphereUser></containerCluster><entityMetadata xmlns:XMLSchema-instance="http://www.w3.org/2001/XMLSchema-instance" XMLSchema-instance:type="CnsKubernetesEntityMetadata"><entityName>pvc-29775058-741d-4617-9a3e-ce18cc5bdead</entityName><entityType>PERSISTENT_VOLUME</entityType></entityMetadata></metadata><backingObjectDetails xmlns:XMLSchema-instance="http://www.w3.org/2001/XMLSchema-instance" XMLSchema-instance:type="CnsBlockBackingDetails"><capacityInMb>300</capacityInMb></backingObjectDetails></createSpecs></CnsCreateVolume></Body></Envelope>

This is the code responsible for the creation:

astrolabe/pkg/ivd/ivd_protected_entity_type_manager_util.go

Line 230 in cbccec5

dsList, err := findSharedDatastoresFromAllNodeVMs(ctx, client.Client, config, logger)

astrolabe/pkg/ivd/ivd_protected_entity_type_manager_util.go

Line 250 in cbccec5

Datastores: dsList,

Create issue and PR templates

These can likely be copied from the Velero project.

Run vale to check that Astrolabe uses inclusive terminology

Update README

The README was written when there was no code written. Additionally, we need to add more details on contribution guidelines (see https://tanzucommunityedition.io/docs/latest/contribute/contributing/ as an example).

From https://github.com/vmware-tanzu/community-engagement/blob/main/GUIDELINES.md:
The project README should be simple, focused, and mostly bullet item listed with steps to get up and running quickly. The README or landing page should include a project statement (what is does, why it’s important, and how it works), a list of features, screenshots (if a graphical UI exists), contribution guidelines, code of conduct, and links to relevant outside sources. (EXAMPLE).

Add Code of Conduct

Can be requested from OSPO.

Cannot access gvddk

Hello, I'd like to access gvddk imported in following file.

astrolabe/pkg/ivd/ivd_protected_entity.go

Line 29 in cbccec5

"github.com/vmware/gvddk/gDiskLib"

Can it be made available?

Unable to use S3 API with selfsigned cert

error message:

  status:
    currentBackOff: 60
    message: |-
      Failed to upload snapshot, ivd:0ae545bc-cdd7-4b1c-b02a-743e989e2839:fa118876-cefe-44ba-9239-57fa56077826, to durable object storage. Failed to delete peinfo from bucket "velero": Unable to delete object "plugins/vsphere-astrolabe-repo/ivd/peinfo/ivd:0ae545bc-cdd7-4b1c-b02a-743e989e2839:fa118876-cefe-44ba-9239-57fa56077826" from bucket "velero": RequestError: send request failed
      caused by: Get https://minio-velero.apps.<removed>/velero?prefix=plugins%2Fvsphere-astrolabe-repo%2Fivd%2Fpeinfo%2Fivd%3A0ae545bc-cdd7-4b1c-b02a-743e989e2839%3Afa118876-cefe-44ba-9239-57fa56077826: x509: certificate signed by unknown authority
    nextRetryTimestamp: "2020-08-12T11:07:54Z"
    phase: UploadError
    progress: {}
    retryCount: 8
    startTimestamp: "2020-08-12T09:22:30Z"

default backup storage location:

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  selfLink: /apis/velero.io/v1/namespaces/velero/backupstoragelocations/default
  resourceVersion: '12175000'
  name: default
  uid: 01783c7b-645c-4a5e-ac43-1dbcf9ebf15a
  creationTimestamp: '2020-08-07T07:18:29Z'
  generation: 7042
  namespace: velero
  labels:
    component: velero
spec:
  config:
    insecureSkipTLSVerify: 'true'
    region: minio
    s3ForcePathStyle: 'true'
    s3Url: 'https://minio-velero.apps.<removed>/'
  objectStorage:
    bucket: velero
  provider: aws
status:
  lastSyncedTime: '2020-08-12T10:51:55.094260285Z'

please notice the parameter insecureSkipTLSVerify: 'true' which works okay with velero.

Add trivy vulnerability scanner github action

add trivy vulnerability scanner github action: kubernetes-csi/external-snapshotter#708

Failed to find the list of all datastores in VC

Hi All
I face an issue "Failed to download snapshot, ivd:xxxxxxxxxxxxxxxxxxx, from durable object storage. copyInt failed: ServerFaultCode: InvalidArgument: is incorrectly formatted", when i restore the backup with command velero restore.

I also look into the pod logs, i find some error log as below. It seems like the pod try to get a datastore list from datacenter.But our policies don't allow this pod get whole datastore list from datacenter. Can i use any paramters to custom the datastore path for it ? Could you give me some advice?

time="2020-07-28T14:34:55Z" level=error msg="Failed to find the list of all datastores in VC" error="datastore '/xxx-xx01/datastore/*' not found" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:144"
time="2020-07-28T14:34:55Z" level=error msg="Failed to find any datastore in the underlying vSphere" error="datastore '/xxx-xx01/datastore/*' not found" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:232"
time="2020-07-28T14:34:55Z" level=error msg="Failed to call createCnsVolumeWithClusterConfig" error="datastore '/xxx-xx01/datastore/*' not found" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/ivd/ivd_protected_entity_type_manager_util.go:402"

Add getting started guide for both users and contributors

More info here https://github.com/vmware-tanzu/community-engagement/blob/main/GUIDELINES.md

Update Protobuf version

https://github.com/vmware-tanzu/astrolabe/security/dependabot/3
We need to update GoGo Protobuf to 1.3.2 or higher.

Clean up existing architecure docs

Both the SPEC.md and README.md (although the readme update is likely covered by #106)

Add DCO/CLA check

Look into if it's using DCO or CLA and add DCO/CLA check through vmwclabot

Fix linkage with old mongo DB driver

CVE-2021-20329 is an issue in the MongoDB driver earlier that 1.5.1. Astrolabe is indirectly linking with this thru the OpenAPI code (go-openapi/strfmt). Update to latest go-openapi/strfmt to resolve this.

Mechanism for letting clients know when they are not being used

It would be nice to be able to close connections to servers and release resources there when they are not needed. Consider mechanisms for hinting to the Protected Entities/Protected Entity Type Managers that there is no outstanding usage and they could release resources.

Plugin in mechanism for local Astrolabe

Astrolabe provides a remote API via REST and soon to be via CRDs, however for local services everything needs to be compiled together into the executable. We need a plugin mechanism that will let us compose an Astrolabe local process with multiple Protected Entity plugins that may not have been compiled together.

Need a storage system agnostic mechanism for finding PV PEs for PVCs

Currently the PVC PE will retrieve a PE for its PV, however this is hardcoded to work for vSphere and will not work for non vSphere systems.