openebs-archive / jiva

CAS Data Engine - iSCSI Distributed Block Storage Controller built-in Go

Jiva

OpenEBS Jiva can be used to dynamically provision highly available Kubernetes Persistent Volumes using local (ephemeral) storage available on the Kubernetes nodes.

Overview

Jiva provides containerized block storage controller.

Jiva Operator helps with dynamically provisioning Jiva Volumes and managing their lifecycle.

Usage

• Refer to the Jiva Operator Quickstart guide.

Supported Block Storage Features

• Thin Provisioning
• Enforce volume quota
• Synchronous replication
• High Availability
• Incremental/Full Snapshot
• Full Backup and Restore (using Velero)
• Supports OS/ARCH: linux/arm64, linux/amd64

How does High Availability work?

Jiva comprises of two components:

• A Target ( or a Storage Controller) that exposes iSCSI, while synchronously replicating the data to one or more Replicas.
• A set of Replicas that a Target uses to read/write data. Each of the replicas will be on a different node to ensure high availability against node or network failures. The Replicas save the data into sparse files on the host filesystem directories.

For ensuring that jiva volumes can sustain a node failure, the volumes must be configured with atleast 3 replicas. The target will serve the data as long as there are two healthy replicas. When a node with a replica fails and comes back online, the replica will start in a degraded mode and start rebuilding the missed data from the available healthy replicas.

If 2 of the 3 replicas go offline at the same time, then volume will be marked as read-only. This ensures that target can decided which of the replicas has the latest data.

How does Jiva Perform?

Jiva is optimized for consistency and availability and not performance. There is a significant degradation in performance compared to local storage due to the following design choices:

• Jiva uses "strong consistency" approach where target will make the write operation as completed only after the data is confirmed to be written to all the healthy replicas.
• Also Jiva does not implement any caching logic, to allow for the jiva target and replica containers to be ephemeral.
• Jiva engine is completely written in user space to be able to run on any platform and make upgrades seamless.

If you need low latency storage, please checkout other OpenEBS Engines like Mayastor and Local PVs.

Who uses Jiva?

Some of the organizations that have publicly shared the usage of Jiva.

• Arista
• CLEW Medical
• Clouds Sky GmbH
• CodeWave
• Verizon Media

You can see the full list of organizations/users using OpenEBS here.

If you are using Jiva, and would like yourself to be listed in this page as a an adopter, please raise an issue with the following details:

• Stateful Applications that you are running on OpenEBS
• Are you evaluating or already using in development, CI/CD, production
• Are you using it for home use or for your organization
• A brief description of the use case or details on how OpenEBS is helping your projects.

If you would like your name (as home user) or organization name to be added to the ADOPTERS.md, please provide a preferred contact handle like github id, twitter id, linkedin id, website etc.

Contributing

OpenEBS welcomes your feedback and contributions in any form possible.

• Join OpenEBS community on Kubernetes Slack
  • Already signed up? Head to our discussions at #openebs
• Want to raise an issue or help with fixes and features?
  • See open issues
  • See contributing guide
  • See Project Roadmap
  • Want to join our contributor community meetings, check this out.
• Join our OpenEBS CNCF Mailing lists
  • For OpenEBS project updates, subscribe to OpenEBS Announcements
  • For interacting with other OpenEBS users, subscribe to OpenEBS Users

Code of conduct

Please read the community code of conduct here.

Inspiration and Credit

OpenEBS Jiva uses the following two projects:

• Fork of the Longhorn engine, which helped with the significant portion of the code in this repo, helped us to validate that Storage controllers can be run as microservices and they can be coded in Go.
• The iSCSI functionality is provided by gostor/gotgt.

License

jiva's Issues

make build failded when reading vendor.conf

When I run the make build command. It would failded after the jiva image was build successfully.
The error showed

INFO[0000] Trash! Reading file: 'vendor.conf'
INFO[0000] Using 'github.com/openebs/jiva' as the project's root package (from vendor.conf)
INFO[0000] Preparing cache for 'github.com/Sirupsen/logrus'
INFO[0132] Checking out 'github.com/Sirupsen/logrus', commit: '446d1c146faa8ed3f4218f056fcd165f6bcfda81'
INFO[0132] Fetching latest commits from 'origin' for 'github.com/Sirupsen/logrus'
ERRO[0132] `git fetch -f -t origin` failed:
fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

FATA[0132] fetch failed                                  i={Package:github.com/Sirupsen/logrus Version:446d1c146faa8ed3f4218f056fcd165f6bcfda81 Repo: Update:false Options:{Transitive:false Staging:false}}
FATA[0135] exit status 1
make: *** [trash] Error 1

The solution from the community is Sirupsen/logrus repo has been renamed to sirupsen/logrus and we have not updated it yet. You can replace Sirupsen to sirupsen in the import paths in all the files in jiva repo and it will work.

But the repo 'github.com/aws/aws-sdk-go' still has problems.

INFO[0000] Trash! Reading file: 'vendor.conf'
INFO[0000] Using 'github.com/openebs/jiva' as the project's root package (from vendor.conf)
INFO[0000] Preparing cache for 'github.com/aws/aws-sdk-go'
INFO[0133] Checking out 'github.com/aws/aws-sdk-go', commit: 'c924893c38ecc04b18d7aab8a7aa561cb8b4c4cc'
INFO[0133] Fetching latest commits from 'origin' for 'github.com/aws/aws-sdk-go'
ERRO[0133] `git fetch -f -t origin` failed:
fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

FATA[0133] fetch failed                                  i={Package:github.com/aws/aws-sdk-go Version:c924893c38ecc04b18d7aab8a7aa561cb8b4c4cc Repo: Update:false Options:{Transitive:false Staging:false}}
FATA[0136] exit status 1
make: *** [trash] Error 1

Broken links in README.md

The 2 links mentioned in README.md file, referring to OpenEBS latest documentation aren't working. The url for the same has changed. I would like to work on this issue.

The Jiva iSCSI target returns a negative "target portal group tag" upon discovery

The discovery of a Jiva iSCSI target via the Ctrl Pod/Service IP is seen to return a negative value (-1) for the target portal group tag.

karthik@MayaHost01:/var/openebs$ sudo iscsiadm -m discovery -t st -p 10.36.0.1:3260
10.102.158.249:3260,-1 iqn.2016-09.com.openebs.jiva:pvc-2845cd53-c11f-11e7-9be8-000c298ff5fc

The ansible-based integration test attempting login/volume mount is seen to fail due to the above.

Some details about this key from iSCSI RFC :

TargetPortalGroupTag=<16-bit-binary-value>

Examples:
TargetPortalGroupTag=1

The target portal group tag is a 16-bit binary-value that uniquely
identifies a portal group within an iSCSI target node. This key
carries the value of the tag of the portal group that is servicing
the Login request. The iSCSI target returns this key to the
initiator in the Login Response PDU to the first Login Request PDU
that has the C bit set to 0 when TargetName is given by the
initiator.

For the complete usage expectations of this key see Section 5.3 Login
Phase

Test error details :

• Ansible uses an open-iscsi module to perform discovery, login etc.., It fails with following error :

TASK [Establish iSCSI session with volume] *****************************************************************************
task path: /home/ciuser/openebs/e2e/ansible/playbooks/feature/snapshots/test-snapshot.yml:67
fatal: [localhost -> None]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Shared connection to 20.10.49.12 closed.\r\n", "module_stdout": "\r\nTraceback (most recent call last):\r\n  File \"/tmp/ansible_8U8slY/ansible_module_open_iscsi.py\", line 390, in <module>\r\n    main()\r\n  File \"/tmp/ansible_8U8slY/ansible_module_open_iscsi.py\", line 333, in main\r\n    target = nodes[0]\r\nIndexError: list index out of range\r\n", "msg": "MODULE FAILURE", "rc": 0}

• In the ansible open-iscsi module, the discovery o/p is processed here :

def iscsi_get_cached_nodes(module, portal=None):

    cmd = '%s --mode node' % iscsiadm_cmd
    (rc, out, err) = module.run_command(cmd)

    if rc == 0:
        lines = out.splitlines()
        nodes = []
        for line in lines:
            # line format is "ip:port,target_portal_group_tag targetname"
            parts = line.split()
            if len(parts) > 2:
                module.fail_json(msg='error parsing output', cmd=cmd)
            target = parts[1]
            parts = parts[0].split(':')
            target_portal = parts[0]

            if portal is None or portal == target_portal:
                nodes.append(target)

Setup the CI pipeline for Jiva

The CI on Jiva should cover the following:

• Functional/Usecase Validation:
  • iSCSI Volumes
  • Data Protection via Snapshots
  • Backup to S3
• Non-functional Validation
  • Scale
  • Performance

The scope of this issue is to implement and document the CI framework for Jiva. The framework should provide example/guidelines for extending the CI framework with additional pipeline items under Functional and Non-Functional categories.

Dependency on uncommitted code

I haven't yet checked out the build script but trying to go get on linux made me have a WTF moment :P

This depends on port.Stats which exists only in this PR: https://github.com/openebs/gotgt/pull/9/files

Also, why is openebs/gotgt not vendored. /cc @kmova

Create dockerfile for jiva image

Jiva is meant to be an wrapper project that will pull in the different binaries from other projects like openebs/longhorn, openebs/gotgt etc., and generate an docker image named "jiva-"

Add snapshot related integration tests to jiva CI

Travis CI tests for openebs/jiva currently include only:

• volume provisioning
• data read/writes

The CI tests should include the tests for snapshot related functionality added as part of the OpenEBS v0.4.0 release.

Travis build failing in all branches

Travis build is failing with error

The command '/bin/sh -c wget -O - https://storage.googleapis.com/golang/go1.10.3.linux-${!GOLANG_ARCH}.tar.gz | tar -xzf - -C /usr/local && go get github.com/rancher/trash && go get github.com/golang/lint/golint && go get github.com/prometheus/client_golang/prometheus/promhttp' returned a non-zero code: 1

log.txt

Unable to mount jiva volume in IPv6 cluster

What steps did you take and what happened:
I am trying to use jiva volume with openebs hostpath backend on a IPv6 only kubernetes cluster.
Though I am able to create PVC/PV, pod is unable to mount the volume due to time out issues. Because jiva volume is stuck in Syncing phase which is supposed to be in Ready. Status shown as RO instead of RW.

Pod events,
kubelet MountVolume.MountDevice failed for volume "pvc-8173df52-68a8-4ca9-b4f3-9721479eb6c0" : rpc error: code = FailedPrecondition desc = Max retry count exceeded, volume: {pvc-8173df52-68a8-4ca9-b4f3-9721479eb6c0} is not ready

jiva volume status

[root@k8s-master01 debug]# kubectl -n openebs get jivavolume
NAME                                       REPLICACOUNT   PHASE     STATUS
pvc-8173df52-68a8-4ca9-b4f3-9721479eb6c0                  Syncing   RO
[root@k8s-master01 debug]# 

What did you expect to happen:
jiva volume to be in Ready phase and successfully attached to pod.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs <jiva controller pod name> -n openebs (Optional)

[root@k8s-master01 ~]# k -n openebs logs pvc-8173df52-68a8-4ca9-b4f3-9721479eb6c0-jiva-ctrl-5fd8c744kcg6
Defaulted container "jiva-controller" out of: jiva-controller, maya-volume-exporter
time="2023-04-11T09:10:07Z" level=info msg="RPC_READ_TIMEOUT env not set"
time="2023-04-11T09:10:07Z" level=info msg="RPC_WRITE_TIMEOUT env not set"
time="2023-04-11T09:10:07Z" level=info msg="REPLICATION_FACTOR: 3, RPC_READ_TIMEOUT: 0s, RPC_WRITE_TIMEOUT: 0s"
time="2023-04-11T09:10:07Z" level=info msg="Starting controller with frontendIP: , and clusterIP: fd00:10:1616:2222::6144"
time="2023-04-11T09:10:07Z" level=info msg="resetting controller"
time="2023-04-11T09:10:07Z" level=info msg="Listening on :9501"
time="2023-04-11T09:10:14Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2.038381423s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:14 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:14Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2.02715368s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:14 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:15Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2.029487934s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:15Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:15 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:19Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 7.037789788s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:19 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:19Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 7.026113632s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:19 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:20Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 7.027654537s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:20Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:20 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:24Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 12.036935653s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:24Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 12.026535002s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:25Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 12.028114649s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:25Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:25 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:29 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:29Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 17.038064107s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:10:29Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 17.025846103s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:29 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:30Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 17.0281659s State: closed Type: Backend RevisionCount: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:30 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:30Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:34 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:34Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 22.037787267s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:34Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:10:34Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 22.026016085s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:34Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:34 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:35Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 22.028069374s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:35Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:35 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:39 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:39Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 27.037710296s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:39Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:10:39Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 27.026159802s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:39Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:39 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:40 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:40Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 27.027734071s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:40Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:10:44Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 32.038739561s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:44Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:44 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:44Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 32.026159237s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:44Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:44 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:45Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 32.028531943s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:45Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:45 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:49Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 37.038299406s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:49Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:49 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:49Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 37.026251473s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:49Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:49 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:50Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 37.027753804s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:50Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:50 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:54 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:54Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 42.0376928s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:54Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:10:54Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 42.026162453s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:54Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:54 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:55Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 42.02781596s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:55Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:55 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:59Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 47.038438047s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:59Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:59 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:59 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:59Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 47.026486173s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:59Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:00 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:00Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 47.028286757s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:00Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:04 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:04Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 52.03781717s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:04Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:04Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 52.026367007s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:04Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:04 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:05Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 52.027807652s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:05Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:05 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:09Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 57.037809293s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:09Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:09 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:09Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 57.026394123s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:09Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:09 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:10 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:10Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 57.027881807s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:10Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:14Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m2.03841576s State: closed Type: Backend RevisionCount: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:14 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:14Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m2.026194299s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:14 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:15 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:15Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m2.028839823s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:15Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:19Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m7.037921763s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:19 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:19 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:19Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m7.026055585s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:20Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m7.027732459s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:20Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:20 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:24Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m12.037670403s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:24Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m12.025936605s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:25Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m12.028519605s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:25Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:25 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:29Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m17.037504881s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:29 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:29 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:29Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m17.026127066s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:30Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m17.02754257s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:30Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:30 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:34Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m22.037473733s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:34Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:34 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:34 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:34Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m22.026049639s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:34Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:35 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:35Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m22.027708656s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:35Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:39 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:39Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m27.037542166s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:39Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:39 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:39Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m27.026236535s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:39Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:40 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:40Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m27.027877113s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:40Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:44 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:44Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m32.037837041s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:44Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:44Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m32.026494435s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:44Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:44 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:45 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:45Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m32.027764564s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:45Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:49 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:49Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m37.037503691s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:49Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:49 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:49Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m37.026673576s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:49Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:50Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m37.031854737s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:50Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:50 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:54Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m42.037447622s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:54Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:54 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:54 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:54Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m42.026308224s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:54Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:55 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:55Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m42.028752873s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:55Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:59Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m47.037966348s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:59Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:59 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:59 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:59Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m47.026402208s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:59Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:00 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:00Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m47.029784276s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:00Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:04Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m52.037830946s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:04Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:04 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:04Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m52.027565843s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:04Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:04 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:05 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:05Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m52.029670778s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:05Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:09 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:09Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m57.03770945s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:09Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:09Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m57.026346501s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:09Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:09 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:10 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:10Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m57.027822187s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:10Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:14 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:14Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m2.038019684s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:14Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m2.026060834s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:14 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:15Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m2.028381957s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:15Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:15 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:19 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:19Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m7.03781419s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:19Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m7.025931586s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:19 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:20 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:20Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m7.028040384s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:20Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:24Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m12.03786774s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:24Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m12.026422844s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:25Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m12.028149798s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:25Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:25 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:29Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m17.038654957s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:29 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:29 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:29Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m17.026347298s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:30 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:30Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m17.029901277s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:30Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:34Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m22.038136584s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:34Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:34 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:34Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m22.026376421s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:34Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:34 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:35 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:35Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m22.028166348s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:35Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:39Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m27.038398724s State: closed Type: Backend RevisionCount: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:39 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:39Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:39Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m27.026729039s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:39Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:39 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:40Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m27.027991391s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:40Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:40 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:44Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m32.037518757s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:44Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:44 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:44Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m32.026780554s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:44Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:44 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:45Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m32.028121202s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:45Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:45 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:49Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m37.038071583s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:49Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:49 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:49Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m37.026273187s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:49Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:49 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:50Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m37.02777268s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:50Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:50 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:54 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:54Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m42.038401485s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:54Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:54Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m42.026322143s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:54Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:54 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:55 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:55Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m42.027968475s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:55Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:59Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m47.037906802s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:59Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:59 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:59Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m47.026308282s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:59Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:59 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:00Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m47.028735179s State: closed Type: Backend RevisionCount: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:13:00 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:00Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:13:04Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m52.037275711s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:04Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:13:04 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:04Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m52.026159242s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:04Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:13:04 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:13:05 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:05Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m52.028328942s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:05Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:13:09Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m57.037274849s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:09Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:13:09 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:13:09 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:09Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m57.026130557s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:09Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:13:10Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m57.027584541s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:10Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:13:10 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:14Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 3m2.038765509s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:13:14 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:13:14 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:14Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 3m2.026266502s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:13:15 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:15Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 3m2.031891604s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:15Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:13:19Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 3m7.037717213s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:13:19 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:13:19 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:19Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 3m7.026759958s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:13:20Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 3m7.027741008s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:20Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:13:20 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:24Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 3m12.0378223s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:13:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:24Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 3m12.026230865s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:13:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:25Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 3m12.02788935s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:25Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:13:25 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:29Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 3m17.03750985s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:13:29 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:13:29 +0000] "POST /v1/register HTTP/1.1" 200 0

• kubectl logs <jiva replica pod name> -n openebs (Optional)

[root@k8s-master01 ~]# k -n openebs logs pvc-8173df52-68a8-4ca9-b4f3-9721479eb6c0-jiva-rep-0
time="2023-04-11T09:17:48Z" level=info msg="MAX_CHAIN_LENGTH env not set, default value is 512"
time="2023-04-11T09:17:48Z" level=info msg="Configured logging with retentionPeriod: 180, maxLogFileSize: 100, maxBackups: 5"
time="2023-04-11T09:17:48Z" level=info msg="Creating volume openebs, size 5368709120/512"
time="2023-04-11T09:17:48Z" level=info msg="Starting replica having replicaType: , frontendIP: fd00:10:1616:2222::6144, size: 5368709120, dir: openebs"
time="2023-04-11T09:17:48Z" level=info msg="Setting replicaAddr: :9502, controlAddr: :9502, dataAddr: :9503, syncAddr: :9504"
time="2023-04-11T09:17:48Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:17:48Z" level=info msg="Listening on data :9503"
time="2023-04-11T09:17:48Z" level=info msg="Listening on control :9502"
time="2023-04-11T09:17:48Z" level=info msg="Listening on sync :9504 start: 9700 end: 9800"
time="2023-04-11T09:17:50Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:17:50Z" level=info msg="Closing replica"
time="2023-04-11T09:17:50Z" level=info msg="Skip closing replica, s.r not set"
time="2023-04-11T09:17:50Z" level=info msg="CheckAndResetFailedRebuild tcp://localhost:9502"
time="2023-04-11T09:17:50Z" level=info msg="Addreplica tcp://localhost:9502"
time="2023-04-11T09:17:50Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:17:50Z" level=info msg="Register replica at controller"
time="2023-04-11T09:17:52Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:17:54Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:17:55Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:17:55Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:17:55Z" level=info msg="Register replica at controller"
time="2023-04-11T09:17:56Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:17:58Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:00Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:00Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:00Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:00Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:02Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:04Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:05Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:05Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:05Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:06Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:08Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:10Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:10Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:10Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:10Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:12Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:14Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:15Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:15Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:15Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:16Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:18Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:20Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:20Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:20Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:20Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:22Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:24Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:25Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:25Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:25Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:26Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:28Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:30Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:30Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:30Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:30Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:32Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:34Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:35Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:35Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:35Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:36Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:38Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:40Z" level=info msg="Timed out waiting for response from controller, will retry"

• kubectl exec -it <jiva controller pod name> -n openebs -- jivactl ls

[root@k8s-master01 ~]# k -n openebs exec -it pvc-8173df52-68a8-4ca9-b4f3-9721479eb6c0-jiva-ctrl-5fd8c744kcg6 -- jivactl ls
Defaulted container "jiva-controller" out of: jiva-controller, maya-volume-exporter
ADDRESS MODE CHAIN
[root@k8s-master01 ~]# 

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Jiva version: 3.3.0
• OpenEBS version: 3.3.0
• Kubernetes version (use kubectl version): v1.24.9
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): CentOS Linux 7 (Core) 3.10.0-1160.81.1.el7.x86_64

Refactor Controller.Start(addresses...string)

I've selected Controller.Start(addresses...string) for refactoring, which is a unit of 74 lines of code and 17 branch points. Addressing this will make our codebase more maintainable and improve Better Code Hub's Write Simple Units of Code guideline rating! 👍

Here's the gist of this guideline:

• Definition 📖
  Limit the number of branch points (if, for, while, etc.) per unit to 4.
• Why❓
  Keeping the number of branch points low makes units easier to modify and test.
• How 🔧
  Split complex units with a high number of branch points into smaller and simpler ones.

You can find more info about this guideline in Building Maintainable Software. 📖

---

ℹ️ To know how many other refactoring candidates need addressing to get a guideline compliant, select some by clicking on the 🔲 next to them. The risk profile below the candidates signals (✅) when it's enough! 🏁

---

Good luck and happy coding! ✨ 💯

Run the libiscsi compliance test in jiva travis build

Run the libiscsi tool to run iSCSI compliance tests on the jiva iSCSI target in the travis build. Refer to the following to aid this task:

• Docker module in python : https://docker-py.readthedocs.io/en/stable/
• libiscsi test container : https://github.com/openebs/test-storage/tree/master/libiscsi
• Ansible test files to run libiscsi in openebs e2e : https://github.com/openebs/openebs/tree/master/e2e/ansible/playbooks/compliance/iscsi

chore: setup repo for the libiscsi test suite container

Describe the problem/challenge you have
Jiva uses libiscsi test suite as part of the CI. The tests are from a container called openebs/tests-libiscsi.

The source code for the container moved to a different project as part of the litmuschoas project spin-off and then it was deleted from the main repo. the code is available here:
https://github.com/litmuschaos/test-tools/tree/d72fe9078fa344e84113f455a6832558b9fcd53d/libiscsi

Describe the solution you'd like

• Setup a repo in openebs for creating the image for openebs/tests-libiscsi
• Check if the following folder https://github.com/openebs/jiva/tree/develop/libiscsi-testsuite is stil in use or can be merged with the above container.
• Check if this repository is still required: https://github.com/openebs/libiscsi

Broken filesystem after batch `jivactl snapshot rm`

I wanted to flatten jiva pvc as much as possible, as it consisted mostly out of empty snapshots. I entered pvc controller pod and executed jivactl snapshot rm $(jivactl snapshot ls | tail -1) a bunch of times in a loop, one after another. Initially everything looked good, but after some time mysql pod crashed (gralog logs attached).

Once I detected it I deleted the pod and rescaled it again, and automatic fsck fixed fs errors, but that ended with half of data missing. Recovery from sql backup fixed the issue.
mywql-graylog.csv.txt
mywql-graylog-pvc-ctrl.csv.txt
mywql-graylog-pvc-replica-kube2.csv.txt
mywql-graylog-pvc-replica-kube3.csv.txt

[FOSSA] Review jiva license issue

FOSSA has found issues in this repo:

• POLICY_FLAG: pip+directio$1.2

Generated by FOSSA on 03/26/2018

Reported by FOSSA user: muratkars

Replica error if underlying storagepool fs is ZFS

We're running Kubernetes 1.9.7 on ubuntu 16.04 with ZFS.

If the storagepool is a zfs volume, e.g. root/ebs with the following definition in k8s

apiVersion: openebs.io/v1alpha1
kind: StoragePool
metadata:
  name: ebs
  type: hostdir
spec:
  path: "/ebs"

the replica containers running

launch replica --frontendIP x.x.x.x --size 1G /openebs

fail with

time="2018-05-25T09:35:16Z" level=info msg="Creating volume /openebs, size 1073741824/512"
time="2018-05-25T09:35:16Z" level=fatal msg="Error running start replica command: open /openebs/revision.counter: invalid argument"

If we create an ext4 zvol based storagepool it works. Is this expected behaviour?

One replica fails to come up after running node failure tests

What steps did you take and what happened:
Running the node failure tests (openebs-archive/e2e-tests#564), where the kernel is made to crash leads to a metadata file issue as reported in one of the replica logs:
failed to read disk data, error while unmarshalling file: volume-snap-1db04a94-0aa0-4404-bbbb-3641392d587f.img.meta

What did you expect to happen:
The replicas should have come up and connected to the controller without any interruption.

Environment:

• Jiva version: v2.3.0
• OpenEBS version: v2.3.0
• Kubernetes version: v1.18.3
• OS : Ubuntu 18.04

[QUESTION] Read Write Many Support

Test modules/scripts for snapshot/revert functionality

Add endpoint "/metrics" in Controller and Replicas

Add endpoint /metrics in controller and replicas. This is required to monitor the controller and replicas by prometheus.
Add custom metrics to monitor all the endpoint response duration and request count.

Extend iSCSI e2e in Jiva travis build

This issue tracks the creation of a placeholder for Jiva specific test suite which will run post image build.

Currently, the jiva ci includes a vdbench test run on the iscsi target. This has to be extended & standardised to include further functional cases that can be run w/o kubernetes components (i.e., non-system level), which include, but are not limited to :

• iSCSI compliance (libiscsi)
• Replication & resync checks
• Basic resiliency such as lost replicas
• Data integrity
• Multi-session login/logouts
• CHAP support
• LUN attribute changes
• Space reclamation

Note that this should be separate in principle and wrt environment from the OpenEBS e2e which will consume Jiva and test it at a system-level (i.e., application use-cases, non-functional requirements like performance etc.,). This has been referenced in the issue #6

Run python script to mount iscsi volume and run I/O

Jiva CI fails during `make build`

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
GItHub Actions CI fails Jiva builds.

What did you expect to happen:
GitHub Actions CI passes Jiva builds.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)
https://gist.github.com/niladrih/a80e950b83090696c8935aba6df2d25d

• kubectl logs <jiva controller pod name> -n openebs (Optional)
• kubectl logs <jiva replica pod name> -n openebs (Optional)
• kubectl exec -it <jiva controller pod name> -n openebs -- jivactl ls

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Jiva version
• OpenEBS version
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

broken link in Jiva README

"JIVA OVERVIEW DIAGRAM" link https://github.com/openebs/openebs/blob/master/docs/images/JivaExample.png?raw=true is broken

Race condition detected while running race detector

WARNING: DATA RACE
Read at 0x00c4200fefc8 by goroutine 39:
  github.com/openebs/jiva/controller/rest.(*Server).RegisterReplica.func1()
      /go/src/github.com/openebs/jiva/controller/rest/replica.go:106 +0xfa
  github.com/openebs/jiva/controller/rest.(*Server).RegisterReplica()
      /go/src/github.com/openebs/jiva/controller/rest/replica.go:113 +0x42c
  github.com/openebs/jiva/controller/rest.(*Server).RegisterReplica-fm()
      /go/src/github.com/openebs/jiva/controller/rest/router.go:36 +0x63
  github.com/openebs/jiva/replica/rest.HandleError.func1()
      /go/src/github.com/openebs/jiva/replica/rest/router.go:15 +0x5e
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:1918 +0x51
  github.com/openebs/jiva/vendor/github.com/rancher/go-rancher/api.ApiHandler.func1()
      /go/src/github.com/openebs/jiva/vendor/github.com/rancher/go-rancher/api/handler.go:26 +0x177
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:1918 +0x51
  github.com/openebs/jiva/vendor/github.com/gorilla/context.ClearHandler.func1()
      /go/src/github.com/openebs/jiva/vendor/github.com/gorilla/context/context.go:141 +0xa2
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:1918 +0x51
  github.com/openebs/jiva/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
      /go/src/github.com/openebs/jiva/vendor/github.com/gorilla/mux/mux.go:103 +0x363
  github.com/openebs/jiva/vendor/github.com/gorilla/handlers.loggingHandler.ServeHTTP()
      /go/src/github.com/openebs/jiva/vendor/github.com/gorilla/handlers/handlers.go:69 +0x174
  github.com/openebs/jiva/vendor/github.com/gorilla/handlers.(*loggingHandler).ServeHTTP()
      <autogenerated>:1 +0xa1
  github.com/openebs/jiva/util.filteredLoggingHandler.ServeHTTP()
      /go/src/github.com/openebs/jiva/util/util.go:88 +0x91
  github.com/openebs/jiva/util.(*filteredLoggingHandler).ServeHTTP()
      <autogenerated>:1 +0xa2
  github.com/openebs/jiva/vendor/github.com/gorilla/handlers.ProxyHeaders.func1()
      /go/src/github.com/openebs/jiva/vendor/github.com/gorilla/handlers/proxy_headers.go:54 +0xc3
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:1918 +0x51
  net/http.serverHandler.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2619 +0xbc
  net/http.(*conn).serve()
      /usr/local/go/src/net/http/server.go:1801 +0x83b

Previous write at 0x00c4200fefc8 by goroutine 37:
  github.com/openebs/jiva/controller/rest.(*Server).RegisterReplica()
      /go/src/github.com/openebs/jiva/controller/rest/replica.go:84 +0x123
  github.com/openebs/jiva/controller/rest.(*Server).RegisterReplica-fm()
      /go/src/github.com/openebs/jiva/controller/rest/router.go:36 +0x63
  github.com/openebs/jiva/replica/rest.HandleError.func1()
      /go/src/github.com/openebs/jiva/replica/rest/router.go:15 +0x5e
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:1918 +0x51
  github.com/openebs/jiva/vendor/github.com/rancher/go-rancher/api.ApiHandler.func1()
      /go/src/github.com/openebs/jiva/vendor/github.com/rancher/go-rancher/api/handler.go:26 +0x177
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:1918 +0x51
  github.com/openebs/jiva/vendor/github.com/gorilla/context.ClearHandler.func1()
      /go/src/github.com/openebs/jiva/vendor/github.com/gorilla/context/context.go:141 +0xa2
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:1918 +0x51
  github.com/openebs/jiva/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
      /go/src/github.com/openebs/jiva/vendor/github.com/gorilla/mux/mux.go:103 +0x363
  github.com/openebs/jiva/vendor/github.com/gorilla/handlers.loggingHandler.ServeHTTP()
      /go/src/github.com/openebs/jiva/vendor/github.com/gorilla/handlers/handlers.go:69 +0x174
  github.com/openebs/jiva/vendor/github.com/gorilla/handlers.(*loggingHandler).ServeHTTP()
      <autogenerated>:1 +0xa1
  github.com/openebs/jiva/util.filteredLoggingHandler.ServeHTTP()
      /go/src/github.com/openebs/jiva/util/util.go:88 +0x91
  github.com/openebs/jiva/util.(*filteredLoggingHandler).ServeHTTP()
      <autogenerated>:1 +0xa2
  github.com/openebs/jiva/vendor/github.com/gorilla/handlers.ProxyHeaders.func1()
      /go/src/github.com/openebs/jiva/vendor/github.com/gorilla/handlers/proxy_headers.go:54 +0xc3
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:1918 +0x51
  net/http.serverHandler.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2619 +0xbc
  net/http.(*conn).serve()
      /usr/local/go/src/net/http/server.go:1801 +0x83b

Goroutine 39 (running) created at:
  net/http.(*Server).Serve()
      /usr/local/go/src/net/http/server.go:2720 +0x37c
  net/http.(*Server).ListenAndServe()
      /usr/local/go/src/net/http/server.go:2636 +0xc7
  net/http.ListenAndServe()
      /usr/local/go/src/net/http/server.go:2882 +0xfe
  github.com/openebs/jiva/app.startController()
      /go/src/github.com/openebs/jiva/app/controller.go:125 +0xc10
  github.com/openebs/jiva/app.ControllerCmd.func1()
      /go/src/github.com/openebs/jiva/app/controller.go:55 +0x3c
  github.com/openebs/jiva/vendor/github.com/codegangsta/cli.Command.Run()
      /go/src/github.com/openebs/jiva/vendor/github.com/codegangsta/cli/command.go:179 +0xcbb
  github.com/openebs/jiva/vendor/github.com/codegangsta/cli.(*App).Run()
      /go/src/github.com/openebs/jiva/vendor/github.com/codegangsta/cli/app.go:196 +0x1183
  main.longhornCli()
      /go/src/github.com/openebs/jiva/main.go:106 +0x822
  main.main()
      /go/src/github.com/openebs/jiva/main.go:27 +0xe4

Goroutine 37 (running) created at:
  net/http.(*Server).Serve()
      /usr/local/go/src/net/http/server.go:2720 +0x37c
  net/http.(*Server).ListenAndServe()
      /usr/local/go/src/net/http/server.go:2636 +0xc7
  net/http.ListenAndServe()
      /usr/local/go/src/net/http/server.go:2882 +0xfe
  github.com/openebs/jiva/app.startController()
      /go/src/github.com/openebs/jiva/app/controller.go:125 +0xc10
  github.com/openebs/jiva/app.ControllerCmd.func1()
      /go/src/github.com/openebs/jiva/app/controller.go:55 +0x3c
10.1.3.4 - - [28/Jun/2018:08:22:52 +0000] "POST /v1/register HTTP/1.1" 200 0
STATE = closed
10.1.4.4 - - [28/Jun/2018:08:22:52 +0000] "POST /v1/register HTTP/1.1" 200 0
  github.com/openebs/jiva/vendor/github.com/codegangsta/cli.Command.Run()
      /go/src/github.com/openebs/jiva/vendor/github.com/codegangsta/cli/command.go:179 +0xcbb
  github.com/openebs/jiva/vendor/github.com/codegangsta/cli.(*App).Run()
      /go/src/github.com/openebs/jiva/vendor/github.com/codegangsta/cli/app.go:196 +0x1183
  main.longhornCli()
      /go/src/github.com/openebs/jiva/main.go:106 +0x822
  main.main()
      /go/src/github.com/openebs/jiva/main.go:27 +0xe4

Can we make this work on a Mac?

/cc @kmova Any point investing efforts here?

➜  go get github.com/openebs/jiva
# github.com/openebs/jiva/vendor/github.com/frostschutz/go-fibmap
../gocode/src/github.com/openebs/jiva/vendor/github.com/frostschutz/go-fibmap/fibmap.go:216:9: undefined: syscall.Fallocate
../gocode/src/github.com/openebs/jiva/vendor/github.com/frostschutz/go-fibmap/fibmap.go:221:9: undefined: syscall.Fallocate
# github.com/openebs/jiva/vendor/github.com/rancher/sparse-tools/sparse
../gocode/src/github.com/openebs/jiva/vendor/github.com/rancher/sparse-tools/sparse/fiemap.go:110:9: undefined: syscall.Fallocate
../gocode/src/github.com/openebs/jiva/vendor/github.com/rancher/sparse-tools/sparse/fiemap.go:115:9: undefined: syscall.Fallocate
../gocode/src/github.com/openebs/jiva/vendor/github.com/rancher/sparse-tools/sparse/file.go:71:33: undefined: syscall.O_DIRECT
../gocode/src/github.com/openebs/jiva/vendor/github.com/rancher/sparse-tools/sparse/file.go:75:45: undefined: syscall.O_DIRECT
# github.com/openebs/jiva/vendor/github.com/yasker/nsfilelock
../gocode/src/github.com/openebs/jiva/vendor/github.com/yasker/nsfilelock/nsfilelock.go:72:50: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
# github.com/openebs/jiva/frontend/gotgt
../gocode/src/github.com/openebs/jiva/frontend/gotgt/frontend.go:37:15: undefined: port.Stats
# github.com/openebs/jiva/sync/agent
../gocode/src/github.com/openebs/jiva/sync/agent/process.go:141:12: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
../gocode/src/github.com/openebs/jiva/sync/agent/process.go:194:12: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
../gocode/src/github.com/openebs/jiva/sync/agent/process.go:246:12: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
../gocode/src/github.com/openebs/jiva/sync/agent/process.go:281:12: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
../gocode/src/github.com/openebs/jiva/sync/agent/process.go:316:12: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
../gocode/src/github.com/openebs/jiva/sync/agent/process.go:351:12: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
../gocode/src/github.com/openebs/jiva/sync/agent/process.go:390:12: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr

Data stored in a Jiva PVC doesn't replicate

What steps did you take and what happened:

I am using Dynamic NFS Provisioner on top of triple replicated Jiva volumes to make them ReadWriteMany.

After hard reboots, this seems to always happen:

It seems to me the data stored to the Jiva volumes do not go to the replicas, but instead go to the mounting pod local ephemeral storage.

I would expect the data to go to the replicas which are correctly set up, but instead Jiva stores the files to:

/var/snap/microk8s/common/var/lib/kubelet/pods/4ae2b03c-e03e-4088-92cb-dcf37ae2e118/volumes/kubernetes.io~csi/pvc-4f872aeb-385e-44e0-b0f1-9643a04ea4af/mount

Which doesn't seem to be mounted anywhere.

What did you expect to happen:

I would expect either an error in OpenEBS Jiva pvc pod start-up, or the Jiva replicas to be mounted to the mount directory using CSI/iSCSI so the data ends up on the replicas.

The output of the following commands will help us better understand what's going on:

# microk8s kubectl logs -n openebs pvc-4f872aeb-385e-44e0-b0f1-9643a04ea4af-jiva-ctrl-d74d78csgl2r
Defaulted container "jiva-controller" out of: jiva-controller, maya-volume-exportertime="2023-01-21T14:32:40Z" level=info msg="RPC_READ_TIMEOUT env not set"
time="2023-01-21T14:32:40Z" level=info msg="RPC_WRITE_TIMEOUT env not set"
time="2023-01-21T14:32:40Z" level=info msg="REPLICATION_FACTOR: 3, RPC_READ_TIMEOUT: 0s, RPC_WRITE_TIMEOUT: 0s"
time="2023-01-21T14:32:40Z" level=info msg="Starting controller with frontendIP: , and clusterIP: 10.152.183.28"
time="2023-01-21T14:32:40Z" level=info msg="resetting controller"
time="2023-01-21T14:32:40Z" level=info msg="Listening on :9501"
time="2023-01-21T14:34:24Z" level=info msg="Register Replica, Address: 10.1.109.199 UUID: 6f5a5f244254f12b437bb9cc8cd401407ee4ad92 Uptime: 25.199859908s State: dirty Type: Backend RevisionCount: 942"
time="2023-01-21T14:34:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
10.1.109.199 - - [21/Jan/2023:14:34:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-01-21T14:34:29Z" level=info msg="Register Replica, Address: 10.1.109.199 UUID: 6f5a5f244254f12b437bb9cc8cd401407ee4ad92 Uptime: 30.211059851s State: dirty Type: Backend RevisionCount: 942"
time="2023-01-21T14:34:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
10.1.109.199 - - [21/Jan/2023:14:34:29 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-01-21T14:34:34Z" level=info msg="Register Replica, Address: 10.1.109.199 UUID: 6f5a5f244254f12b437bb9cc8cd401407ee4ad92 Uptime: 35.131878985s State: dirty Type: Backend RevisionCount: 942"
time="2023-01-21T14:34:34Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
10.1.109.199 - - [21/Jan/2023:14:34:34 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-01-21T14:34:36Z" level=info msg="Register Replica, Address: 10.1.140.163 UUID: 2bf646cc281748b5ed9ac9ff00b429a6fb43ba1d Uptime: 15.814225ms State: dirty Type: Backend RevisionCount: 942"
time="2023-01-21T14:34:36Z" level=info msg="Replica 10.1.109.199 signalled to start, registered replicas: map[string]types.RegReplica{\"10.1.109.199\":types.RegReplica{Address:\"10.1.109.199\", UUID:\"6f5a5f244254f12b437bb9cc8cd401407ee4ad92\", UpTime:35131878985, RevCount:942, RepType:\"Backend\", RepState:\"dirty\"}, \"10.1.140.163\":types.RegReplica{Address:\"10.1.140.163\", UUID:\"2bf646cc281748b5ed9ac9ff00b429a6fb43ba1d\", UpTime:15814225, RevCount:942, RepType:\"Backend\", RepState:\"dirty\"}}"
10.1.140.163 - - [21/Jan/2023:14:34:36 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-01-21T14:34:36Z" level=info msg="resetting controller"
time="2023-01-21T14:34:36Z" level=info msg="Connecting to remote: 10.1.109.199:9502"
time="2023-01-21T14:34:36Z" level=info msg="Opening: 10.1.109.199:9502"
time="2023-01-21T14:34:36Z" level=info msg="check if replica tcp://10.1.109.199:9502 is already added"
time="2023-01-21T14:34:36Z" level=info msg="check if any WO replica available"
time="2023-01-21T14:34:36Z" level=info msg="Set replica mode of 10.1.109.199:9502 to : WO"
time="2023-01-21T14:34:36Z" level=info msg="Adding backend: tcp://10.1.109.199:9502"
time="2023-01-21T14:34:36Z" level=info msg="replicator reset false"
time="2023-01-21T14:34:36Z" level=info msg="buildreadwriters: prev: 0 0 cur: 1 0"
time="2023-01-21T14:34:36Z" level=info msg="Start monitoring tcp://10.1.109.199:9502"
time="2023-01-21T14:34:36Z" level=info msg="Get backend tcp://10.1.109.199:9502 clone status"
time="2023-01-21T14:34:36Z" level=info msg="Set replica mode of 10.1.109.199:9502 to : RW"
time="2023-01-21T14:34:36Z" level=info msg="Set backend tcp://10.1.109.199:9502 replica mode to RW"
time="2023-01-21T14:34:36Z" level=info msg="Set replica tcp://10.1.109.199:9502 to mode RW"
time="2023-01-21T14:34:36Z" level=info msg="addr tcp://10.1.109.199:9502 m: RW prev: WO in setmode"
time="2023-01-21T14:34:36Z" level=info msg="replicator reset false"
time="2023-01-21T14:34:36Z" level=info msg="buildreadwriters: prev: 0 0 cur: 1 1"
time="2023-01-21T14:34:36Z" level=info msg="Get backend tcp://10.1.109.199:9502 revision counter 942"
time="2023-01-21T14:34:36Z" level=info msg="sending add signal to 10.1.140.163"
time="2023-01-21T14:34:36Z" level=info msg="Create Replica for address tcp://10.1.140.163:9502"
time="2023-01-21T14:34:36Z" level=info msg="Update volume status"
time="2023-01-21T14:34:36Z" level=info msg="Previously Volume RO: true, Currently: true, Total Replicas: 1, RW replicas: 1, Total backends: 1"
time="2023-01-21T14:34:36Z" level=info msg="prevCheckpoint: , currCheckpoint: "
time="2023-01-21T14:34:36Z" level=info msg="Start SCSI target"
time="2023-01-21T14:34:36Z" level=info msg="SCSI device created"
time="2023-01-21T14:34:36Z" level=info msg="iSCSI service listening on: [::]:3260"
10.1.109.199 - - [21/Jan/2023:14:34:36 +0000] "POST /v1/volumes/cHZjLTRmODcyYWViLTM4NWUtNDRlMC1iMGYxLTk2NDNhMDRlYTRhZg==?action=start HTTP/1.1" 200 1038
time="2023-01-21T14:34:36Z" level=info msg="check if replica tcp://10.1.140.163:9502 is already added"
time="2023-01-21T14:34:36Z" level=info msg="check if any WO replica available"
time="2023-01-21T14:34:36Z" level=info msg="verify replication factor"
time="2023-01-21T14:34:36Z" level=info msg="Connecting to remote: 10.1.140.163:9502"
time="2023-01-21T14:34:36Z" level=info msg="Opening: 10.1.140.163:9502"
time="2023-01-21T14:34:36Z" level=info msg="check if replica tcp://10.1.140.163:9502 is already added"
time="2023-01-21T14:34:36Z" level=info msg="check if any WO replica available"
time="2023-01-21T14:34:36Z" level=info msg="Snapshot: 10.1.109.199:9502 23800da8-bbee-4636-8852-ead696f7d27e UserCreated false Created at 2023-01-21T14:34:36Z"
time="2023-01-21T14:34:36Z" level=info msg="successfully taken snapshots cnt 1"
time="2023-01-21T14:34:36Z" level=info msg="Snapshot: 10.1.140.163:9502 23800da8-bbee-4636-8852-ead696f7d27e UserCreated false Created at 2023-01-21T14:34:36Z"
time="2023-01-21T14:34:36Z" level=info msg="Set replica mode of 10.1.140.163:9502 to : WO"
10.1.140.163 - - [21/Jan/2023:14:34:36 +0000] "POST /v1/replicas HTTP/1.1" 200 431
time="2023-01-21T14:34:36Z" level=info msg="Adding backend: tcp://10.1.140.163:9502"
time="2023-01-21T14:34:36Z" level=info msg="replicator reset false"
time="2023-01-21T14:34:36Z" level=info msg="buildreadwriters: prev: 0 1 cur: 2 1"
time="2023-01-21T14:34:36Z" level=info msg="Previously Volume RO: true, Currently: true, Total Replicas: 2, RW replicas: 1, Total backends: 2"
time="2023-01-21T14:34:36Z" level=info msg="prevCheckpoint: , currCheckpoint: "
time="2023-01-21T14:34:36Z" level=info msg="Start monitoring tcp://10.1.140.163:9502"
10.1.140.163 - - [21/Jan/2023:14:34:36 +0000] "GET /v1/replicas/dGNwOi8vMTAuMS4xNDAuMTYzOjk1MDI= HTTP/1.1" 200 431
time="2023-01-21T14:34:36Z" level=info msg="Prepare Rebuild Replica for id tcp://10.1.140.163:9502"
time="2023-01-21T14:34:36Z" level=info msg="Synchronizing volume-head-020.img.meta@tcp://10.1.109.199:9502 to [email protected]:9700"
time="2023-01-21T14:34:37Z" level=info msg="Done synchronizing volume-head-020.img.meta to [email protected]:9700"
10.1.140.163 - - [21/Jan/2023:14:34:36 +0000] "POST /v1/replicas/dGNwOi8vMTAuMS4xNDAuMTYzOjk1MDI=?action=preparerebuild HTTP/1.1" 200 895
10.1.140.163 - - [21/Jan/2023:14:34:37 +0000] "GET /v1/replicas/dGNwOi8vMTAuMS4xNDAuMTYzOjk1MDI= HTTP/1.1" 200 431
time="2023-01-21T14:34:37Z" level=info msg="Verify Rebuild Replica for id tcp://10.1.140.163:9502"
time="2023-01-21T14:34:37Z" level=info msg="chain [volume-head-020.img volume-snap-23800da8-bbee-4636-8852-ead696f7d27e.img volume-snap-77bfa5d0-3a3a-4125-874d-64d981bf7cf7.img volume-snap-b2b1f0c5-b744-47ef-baae-b7697754898b.img volume-snap-668659c8-e7b8-4184-a216-46ef61c72420.img volume-snap-bc4a2971-3060-4d0d-9395-55a8d39167a6.img volume-snap-a9cd1a28-30e6-42e7-9175-dfc987262dc4.img volume-snap-0120e6c1-7962-4378-b090-006cf4305f50.img volume-snap-ccbd2dba-48f0-4964-abd2-bf29e30488f4.img volume-snap-b9397eee-69f0-41ca-808e-6fadba8c32c8.img volume-snap-088add84-75e5-4293-b06c-709a66220f9d.img volume-snap-8eede2b2-cb06-46d4-b862-882d3c4381bd.img volume-snap-93362d3c-51d0-4196-938a-a0455ffe463d.img volume-snap-f2d19ed0-7a0d-427f-bcba-315be4efb0f1.img] from rw replica tcp://10.1.109.199:9502, indx: 3"
time="2023-01-21T14:34:37Z" level=info msg="chain [volume-head-020.img volume-snap-23800da8-bbee-4636-8852-ead696f7d27e.img volume-snap-77bfa5d0-3a3a-4125-874d-64d981bf7cf7.img volume-snap-b2b1f0c5-b744-47ef-baae-b7697754898b.img volume-snap-668659c8-e7b8-4184-a216-46ef61c72420.img volume-snap-bc4a2971-3060-4d0d-9395-55a8d39167a6.img volume-snap-a9cd1a28-30e6-42e7-9175-dfc987262dc4.img volume-snap-0120e6c1-7962-4378-b090-006cf4305f50.img volume-snap-ccbd2dba-48f0-4964-abd2-bf29e30488f4.img volume-snap-b9397eee-69f0-41ca-808e-6fadba8c32c8.img volume-snap-088add84-75e5-4293-b06c-709a66220f9d.img volume-snap-8eede2b2-cb06-46d4-b862-882d3c4381bd.img volume-snap-93362d3c-51d0-4196-938a-a0455ffe463d.img volume-snap-f2d19ed0-7a0d-427f-bcba-315be4efb0f1.img] from wo replica tcp://10.1.140.163:9502, indx: 3"
time="2023-01-21T14:34:37Z" level=info msg="Get backend tcp://10.1.109.199:9502 revision counter 942"
time="2023-01-21T14:34:37Z" level=info msg="rw replica tcp://10.1.109.199:9502 revision counter 942"
time="2023-01-21T14:34:37Z" level=info msg="Set replica mode of 10.1.140.163:9502 to : RW"
time="2023-01-21T14:34:37Z" level=info msg="Set backend tcp://10.1.140.163:9502 replica mode to RW"
time="2023-01-21T14:34:37Z" level=info msg="Set revision counter of 10.1.140.163:9502 to : 942"
time="2023-01-21T14:34:37Z" level=info msg="Set backend tcp://10.1.140.163:9502 revision counter to 942"
time="2023-01-21T14:34:37Z" level=info msg="WO replica tcp://10.1.140.163:9502's chain verified, update replica mode to RW"
time="2023-01-21T14:34:37Z" level=info msg="Set replica tcp://10.1.140.163:9502 to mode RW"
time="2023-01-21T14:34:37Z" level=info msg="addr tcp://10.1.140.163:9502 m: RW prev: WO in setmode"
time="2023-01-21T14:34:37Z" level=info msg="replicator reset false"
time="2023-01-21T14:34:37Z" level=info msg="buildreadwriters: prev: 0 1 cur: 2 2"
time="2023-01-21T14:34:37Z" level=info msg="Previously Volume RO: true, Currently: false, Total Replicas: 2, RW replicas: 2, Total backends: 2"
time="2023-01-21T14:34:37Z" level=info msg="prevCheckpoint: , currCheckpoint: "
10.1.140.163 - - [21/Jan/2023:14:34:37 +0000] "POST /v1/replicas/dGNwOi8vMTAuMS4xNDAuMTYzOjk1MDI=?action=verifyrebuild HTTP/1.1" 200 431
time="2023-01-21T14:34:39Z" level=info msg="connection establishing at: 10.1.109.245:3260"
time="2023-01-21T14:34:39Z" level=info msg="Target is connected to initiator: 10.1.140.128:43935"
time="2023-01-21T14:34:39Z" level=error msg="read BHS failed:EOF"
time="2023-01-21T14:34:39Z" level=warning msg="iscsi connection[0] closed"
time="2023-01-21T14:34:39Z" level=info msg="connection establishing at: 10.1.109.245:3260"
time="2023-01-21T14:34:39Z" level=info msg="Target is connected to initiator: 10.1.140.128:3293"
time="2023-01-21T14:34:39Z" level=info msg="Discovery request received from initiator: iqn.2004-10.com.ubuntu:01:7e62554de3, Session type: Discovery, ISID: 0x23d000000"
time="2023-01-21T14:34:39Z" level=warning msg="unexpected connection state: full feature"
time="2023-01-21T14:34:39Z" level=error msg="read BHS failed:read tcp 10.1.109.245:3260->10.1.140.128:3293: read: connection reset by peer"
time="2023-01-21T14:34:39Z" level=warning msg="iscsi connection[0] closed"
time="2023-01-21T14:34:39Z" level=info msg="connection establishing at: 10.1.109.245:3260"
time="2023-01-21T14:34:39Z" level=info msg="Target is connected to initiator: 10.1.140.128:3075"
time="2023-01-21T14:34:40Z" level=info msg="Login request received from initiator: iqn.2004-10.com.ubuntu:01:7e62554de3, Session type: Normal, Target name:iqn.2016-09.com.openebs.jiva:pvc-4f872aeb-385e-44e0-b0f1-9643a04ea4af, ISID: 0x23d000001"
time="2023-01-21T14:34:40Z" level=error msg="rsa: 0h, sa:false not supported"
time="2023-01-21T14:34:40Z" level=warning msg="opcode: a3h err: check condition"
time="2023-01-21T14:35:23Z" level=info msg="Create Replica for address tcp://10.1.98.10:9502"
time="2023-01-21T14:35:23Z" level=info msg="check if replica tcp://10.1.98.10:9502 is already added"
time="2023-01-21T14:35:23Z" level=info msg="check if any WO replica available"
time="2023-01-21T14:35:23Z" level=info msg="verify replication factor"
time="2023-01-21T14:35:23Z" level=info msg="Connecting to remote: 10.1.98.10:9502"
time="2023-01-21T14:35:23Z" level=info msg="Opening: 10.1.98.10:9502"
time="2023-01-21T14:35:24Z" level=info msg="check if replica tcp://10.1.98.10:9502 is already added"
time="2023-01-21T14:35:24Z" level=info msg="check if any WO replica available"
time="2023-01-21T14:35:24Z" level=info msg="Snapshot: 10.1.140.163:9502 2c8fc776-68f3-4c3c-ac66-c34397b39c6e UserCreated false Created at 2023-01-21T14:35:24Z"
time="2023-01-21T14:35:24Z" level=info msg="Snapshot: 10.1.109.199:9502 2c8fc776-68f3-4c3c-ac66-c34397b39c6e UserCreated false Created at 2023-01-21T14:35:24Z"
time="2023-01-21T14:35:24Z" level=info msg="successfully taken snapshots cnt 2"
time="2023-01-21T14:35:24Z" level=info msg="Snapshot: 10.1.98.10:9502 2c8fc776-68f3-4c3c-ac66-c34397b39c6e UserCreated false Created at 2023-01-21T14:35:24Z"
time="2023-01-21T14:35:25Z" level=info msg="Set replica mode of 10.1.98.10:9502 to : WO"
time="2023-01-21T14:35:26Z" level=info msg="Adding backend: tcp://10.1.98.10:9502"
time="2023-01-21T14:35:26Z" level=info msg="replicator reset false"
time="2023-01-21T14:35:26Z" level=info msg="buildreadwriters: prev: 0 2 cur: 3 2"
time="2023-01-21T14:35:26Z" level=info msg="Previously Volume RO: false, Currently: false, Total Replicas: 3, RW replicas: 2, Total backends: 3"
time="2023-01-21T14:35:26Z" level=info msg="prevCheckpoint: , currCheckpoint: "
time="2023-01-21T14:35:26Z" level=info msg="Start monitoring tcp://10.1.98.10:9502"10.1.98.10 - - [21/Jan/2023:14:35:23 +0000] "POST /v1/replicas HTTP/1.1" 200 413
10.1.98.10 - - [21/Jan/2023:14:35:27 +0000] "GET /v1/replicas/dGNwOi8vMTAuMS45OC4xMDo5NTAy HTTP/1.1" 200 413
time="2023-01-21T14:35:27Z" level=info msg="Prepare Rebuild Replica for id tcp://10.1.98.10:9502"
time="2023-01-21T14:35:27Z" level=info msg="Synchronizing volume-head-021.img.meta@tcp://10.1.109.199:9502 to [email protected]:9700"
10.1.98.10 - - [21/Jan/2023:14:35:27 +0000] "POST /v1/replicas/dGNwOi8vMTAuMS45OC4xMDo5NTAy?action=preparerebuild HTTP/1.1" 200 946
time="2023-01-21T14:35:29Z" level=info msg="Done synchronizing volume-head-021.img.meta to [email protected]:9700"
10.1.109.199 - - [21/Jan/2023:14:35:36 +0000] "GET /v1/checkpoint HTTP/1.1" 200 10810.1.140.163 - - [21/Jan/2023:14:35:37 +0000] "GET /v1/checkpoint HTTP/1.1" 200 10810.1.98.10 - - [21/Jan/2023:14:36:10 +0000] "GET /v1/replicas/dGNwOi8vMTAuMS45OC4xMDo5NTAy HTTP/1.1" 200 413
time="2023-01-21T14:36:10Z" level=info msg="Verify Rebuild Replica for id tcp://10.1.98.10:9502"
time="2023-01-21T14:36:10Z" level=info msg="chain [volume-head-021.img volume-snap-2c8fc776-68f3-4c3c-ac66-c34397b39c6e.img volume-snap-23800da8-bbee-4636-8852-ead696f7d27e.img volume-snap-77bfa5d0-3a3a-4125-874d-64d981bf7cf7.img volume-snap-b2b1f0c5-b744-47ef-baae-b7697754898b.img volume-snap-668659c8-e7b8-4184-a216-46ef61c72420.img volume-snap-bc4a2971-3060-4d0d-9395-55a8d39167a6.img volume-snap-a9cd1a28-30e6-42e7-9175-dfc987262dc4.img volume-snap-0120e6c1-7962-4378-b090-006cf4305f50.img volume-snap-ccbd2dba-48f0-4964-abd2-bf29e30488f4.img volume-snap-b9397eee-69f0-41ca-808e-6fadba8c32c8.img volume-snap-088add84-75e5-4293-b06c-709a66220f9d.img volume-snap-8eede2b2-cb06-46d4-b862-882d3c4381bd.img volume-snap-93362d3c-51d0-4196-938a-a0455ffe463d.img volume-snap-f2d19ed0-7a0d-427f-bcba-315be4efb0f1.img] from rw replica tcp://10.1.109.199:9502, indx: 4"
time="2023-01-21T14:36:10Z" level=info msg="chain [volume-head-019.img volume-snap-2c8fc776-68f3-4c3c-ac66-c34397b39c6e.img volume-snap-23800da8-bbee-4636-8852-ead696f7d27e.img volume-snap-77bfa5d0-3a3a-4125-874d-64d981bf7cf7.img volume-snap-b2b1f0c5-b744-47ef-baae-b7697754898b.img volume-snap-668659c8-e7b8-4184-a216-46ef61c72420.img volume-snap-bc4a2971-3060-4d0d-9395-55a8d39167a6.img volume-snap-a9cd1a28-30e6-42e7-9175-dfc987262dc4.img volume-snap-0120e6c1-7962-4378-b090-006cf4305f50.img volume-snap-ccbd2dba-48f0-4964-abd2-bf29e30488f4.img volume-snap-b9397eee-69f0-41ca-808e-6fadba8c32c8.img volume-snap-088add84-75e5-4293-b06c-709a66220f9d.img volume-snap-8eede2b2-cb06-46d4-b862-882d3c4381bd.img volume-snap-93362d3c-51d0-4196-938a-a0455ffe463d.img volume-snap-f2d19ed0-7a0d-427f-bcba-315be4efb0f1.img] from wo replica tcp://10.1.98.10:9502, indx: 4"
time="2023-01-21T14:36:10Z" level=info msg="Get backend tcp://10.1.109.199:9502 revision counter 945"
time="2023-01-21T14:36:10Z" level=info msg="rw replica tcp://10.1.109.199:9502 revision counter 945"
time="2023-01-21T14:36:10Z" level=info msg="Set replica mode of 10.1.98.10:9502 to : RW"
time="2023-01-21T14:36:10Z" level=info msg="Set backend tcp://10.1.98.10:9502 replica mode to RW"
time="2023-01-21T14:36:10Z" level=info msg="Set revision counter of 10.1.98.10:9502 to : 945"
time="2023-01-21T14:36:10Z" level=info msg="Set backend tcp://10.1.98.10:9502 revision counter to 945"
time="2023-01-21T14:36:10Z" level=info msg="WO replica tcp://10.1.98.10:9502's chain verified, update replica mode to RW"
time="2023-01-21T14:36:10Z" level=info msg="Set replica tcp://10.1.98.10:9502 to mode RW"
time="2023-01-21T14:36:10Z" level=info msg="addr tcp://10.1.98.10:9502 m: RW prev: WO in setmode"
time="2023-01-21T14:36:10Z" level=info msg="replicator reset false"
time="2023-01-21T14:36:10Z" level=info msg="buildreadwriters: prev: 0 2 cur: 3 3"
time="2023-01-21T14:36:10Z" level=info msg="Previously Volume RO: false, Currently: false, Total Replicas: 3, RW replicas: 3, Total backends: 3"
10.1.98.10 - - [21/Jan/2023:14:36:10 +0000] "POST /v1/replicas/dGNwOi8vMTAuMS45OC4xMDo5NTAy?action=verifyrebuild HTTP/1.1" 200 413
time="2023-01-21T14:36:10Z" level=info msg="successfully set checkpoint cnt 3"
time="2023-01-21T14:36:10Z" level=info msg="prevCheckpoint: , currCheckpoint: volume-snap-2c8fc776-68f3-4c3c-ac66-c34397b39c6e.img"
10.1.109.199 - - [21/Jan/2023:14:36:36 +0000] "GET /v1/checkpoint HTTP/1.1" 200 16010.1.140.163 - - [21/Jan/2023:14:36:37 +0000] "GET /v1/checkpoint HTTP/1.1" 200 16010.1.98.10 - - [21/Jan/2023:14:37:11 +0000] "GET /v1/checkpoint HTTP/1.1" 200 160

# microk8s kubectl exec -it -n openebs pvc-4f872aeb-385e-44e0-b0f1-9643a04ea4af-jiva-ctrl-d74d78csgl2r -- jivactl ls
Defaulted container "jiva-controller" out of: jiva-controller, maya-volume-exporterADDRESS                 MODE CHAIN
tcp://10.1.109.199:9502 RW   [volume-head-021.img volume-snap-2c8fc776-68f3-4c3c-ac66-c34397b39c6e.img volume-snap-23800da8-bbee-4636-8852-ead696f7d27e.img volume-snap-77bfa5d0-3a3a-4125-874d-64d981bf7cf7.img volume-snap-b2b1f0c5-b744-47ef-baae-b7697754898b.img volume-snap-668659c8-e7b8-4184-a216-46ef61c72420.img volume-snap-bc4a2971-3060-4d0d-9395-55a8d39167a6.img volume-snap-a9cd1a28-30e6-42e7-9175-dfc987262dc4.img volume-snap-0120e6c1-7962-4378-b090-006cf4305f50.img volume-snap-b9397eee-69f0-41ca-808e-6fadba8c32c8.img volume-snap-93362d3c-51d0-4196-938a-a0455ffe463d.img volume-snap-f2d19ed0-7a0d-427f-bcba-315be4efb0f1.img]
tcp://10.1.140.163:9502 RW   [volume-head-021.img volume-snap-2c8fc776-68f3-4c3c-ac66-c34397b39c6e.img volume-snap-23800da8-bbee-4636-8852-ead696f7d27e.img volume-snap-77bfa5d0-3a3a-4125-874d-64d981bf7cf7.img volume-snap-b2b1f0c5-b744-47ef-baae-b7697754898b.img volume-snap-668659c8-e7b8-4184-a216-46ef61c72420.img volume-snap-bc4a2971-3060-4d0d-9395-55a8d39167a6.img volume-snap-a9cd1a28-30e6-42e7-9175-dfc987262dc4.img volume-snap-0120e6c1-7962-4378-b090-006cf4305f50.img volume-snap-b9397eee-69f0-41ca-808e-6fadba8c32c8.img volume-snap-93362d3c-51d0-4196-938a-a0455ffe463d.img volume-snap-f2d19ed0-7a0d-427f-bcba-315be4efb0f1.img]
tcp://10.1.98.10:9502   RW   [volume-head-019.img volume-snap-2c8fc776-68f3-4c3c-ac66-c34397b39c6e.img volume-snap-23800da8-bbee-4636-8852-ead696f7d27e.img volume-snap-77bfa5d0-3a3a-4125-874d-64d981bf7cf7.img volume-snap-b2b1f0c5-b744-47ef-baae-b7697754898b.img volume-snap-668659c8-e7b8-4184-a216-46ef61c72420.img volume-snap-bc4a2971-3060-4d0d-9395-55a8d39167a6.img volume-snap-a9cd1a28-30e6-42e7-9175-dfc987262dc4.img volume-snap-0120e6c1-7962-4378-b090-006cf4305f50.img volume-snap-b9397eee-69f0-41ca-808e-6fadba8c32c8.img volume-snap-93362d3c-51d0-4196-938a-a0455ffe463d.img volume-snap-f2d19ed0-7a0d-427f-bcba-315be4efb0f1.img]

Environment:

• Jiva version 3.3.0
• OpenEBS version 3.3.0
• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.5", GitCommit:"804d6167111f6858541cef440ccc53887fbbc96a", GitTreeState:"clean", BuildDate:"2022-12-19T15:26:36Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.5", GitCommit:"804d6167111f6858541cef440ccc53887fbbc96a", GitTreeState:"clean", BuildDate:"2022-12-19T15:27:17Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}

Using MicroK8S v1.25.5

• OS (e.g. from /etc/os-release):
  Ubuntu 22.04.1 LTS

jivavolume keep pending

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)

Use helm install openebs-3.3.1

helm install openebs openebs-3.3.1.tgz --set jiva.enabled=true -n openebs

install success

kubectl get pod -n openebs

NAME                                           READY   STATUS    RESTARTS   AGE
openebs-jiva-csi-controller-0                  5/5     Running   0          116s
openebs-jiva-csi-node-5zv5g                    3/3     Running   0          116s
openebs-jiva-csi-node-ccps8                    3/3     Running   0          116s
openebs-jiva-csi-node-fwtzf                    3/3     Running   0          116s
openebs-jiva-operator-f994f6868-hftwx          1/1     Running   0          116s
openebs-localpv-provisioner-77c9bcfd96-n2nf8   1/1     Running   0          116s
openebs-ndm-8dbzp                              1/1     Running   0          116s
openebs-ndm-operator-5d55748dfd-cmv6n          1/1     Running   0          116s
openebs-ndm-rvcr8                              1/1     Running   0          116s
openebs-ndm-ztwxn                              1/1     Running   0          116s

Then I use this to test

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: example-jiva-csi-pvc
spec:
  storageClassName: openebs-jiva-csi-default
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
  labels:
    app: busybox
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - resources:
          limits:
            cpu: 0.5
        name: busybox
        image: busybox
        command: ['sh', '-c', 'echo Container 1 is Running ; sleep 3600']
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 3306
          name: busybox
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: demo-vol1
      volumes:
      - name: demo-vol1
        persistentVolumeClaim:
          claimName: example-jiva-csi-pvc

pod about pvc are running

pvc-7c04c4c0-8f23-44b2-b7d9-e5cb6a371922-jiva-ctrl-6c66f965hq5x   2/2     Running   0          17m
pvc-7c04c4c0-8f23-44b2-b7d9-e5cb6a371922-jiva-rep-0               1/1     Running   0          17m
pvc-7c04c4c0-8f23-44b2-b7d9-e5cb6a371922-jiva-rep-1               1/1     Running   1          17m
pvc-7c04c4c0-8f23-44b2-b7d9-e5cb6a371922-jiva-rep-2               1/1     Running   0          17m

the pvc create successfully

but pod cant't attach the volume

The jivavolume is pending not Ready

kubectl get jivavolume -n openebs

NAME                                       REPLICACOUNT   PHASE     STATUS
pvc-7c04c4c0-8f23-44b2-b7d9-e5cb6a371922                  Pending   

And I can't find any error log in jiva-ctl and jiva-rep

What did you expect to happen:

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

jiva-log.tar.gz

this contain jiva-ctrl and jiva-rep log

• kubectl logs <jiva controller pod name> -n openebs (Optional)
• kubectl logs <jiva replica pod name> -n openebs (Optional)
• kubectl exec -it <jiva controller pod name> -n openebs -- jivactl ls

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Jiva version 3.3.0
• OpenEBS version 3.3.1
• Kubernetes version (use kubectl version):
• Kubernetes installer & version: 1.19.16
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): CentOS Linux 7 (Core)

How to use the pprof function ?

https://github.com/openebs/jiva/blob/06ae965ec74df6b92101024e32d36732c29bc22b/main.go#L64

I want to use the pprof and go-torch to generate the flamegraph of jiva. Where should I set the env PPROFILE to enable this function ?

Test modules/scripts for creating longhorn controller/replica's

The command or script should provide details on the parameters and bring out any dependencies on the infrastructure like:

• arguments required for docker to launch controller or replica
• network dependencies (communication between controller and replica)
• storage dependencies ( local dir, size etc..,)
• types of error returned in case of failed dependencies

JIVA does not clean stale mounts

What steps did you take and what happened:
It appears that when an issue occurs, JIVA remounts the volume, but does not remove a stale mount point, resulting in high IO latency.

What did you expect to happen:

Jiva to cleanup stale mount points periodically.

Anything else you would like to add:

Here are a list of mount points which are not cleaned up causing high load averages; and terrible performance for this storage solution.

Environment: K3s

• Jiva version: v3.1.0
• OpenEBS version: v3.1.0
• Kubernetes version (use kubectl version): v1.23.4+k3s1
• Kubernetes installer & version: v1.23.4+k3s1
• Cloud provider or hardware configuration: AMD Ryzen servers, 64 to 128GB ram, 3 nodes
• OS (e.g. from /etc/os-release): Ubuntu

Modularize the jiva CI integration test script (start_init_test.sh)

The jiva CI integration tests are currently placed in a single bash script that is executed as part of the travis build process. This script needs to be modularized (separate out the individual test functions) to aid easier management and adhere to BCH standards. Increased comments/test-descriptions to specify test intent also will help.

https://github.com/openebs/jiva/blob/master/ci/start_init_test.sh

Support for read-only volumes

Handle cases like:

• mounting a read-only snapshot
• allowing access from multiple clients (write from one, read-many)

Provide a mechanism to check the health of a controller and replica

It is possible that that container that is launching the controller/replica may be up and running, but either the communication to the replica's or from replica to underlying storage could be having issues. The client will detect such issues as IO errors. However, there needs to be a mechanism where maya should be able to check the health of VSM (controller or replica) and take necessary actions like restart, etc.,

This can be as simple as an script or docker exec command.

Stats() returns incorrect ReadIOPS/WriteIOPS values

I got statistics via Jiva's stats API.

{
  "Name": "pvc-30266c71-21b1-11e8-953c-060010b04334",
  "ReadIOPS": "327066",
  "ReplicaCounter": 0,
  "RevisionCounter": 0,
  "SCSIIOCount": {
    "0": 6640,
    "158": 454,
    "160": 2,
    "163": 2,
    "18": 3388,
    "26": 908,
    "40": 327066,
    "42": 258086
  },
  "SectorSize": "4096",
  "Size": "5368709120",
  "TotalReadBlockCount": "17549045760",
  "TotalReadTime": "307301413134",
  "TotalWriteTime": "215794984886",
  "TotatWriteBlockCount": "5395550208",
  "UpTime": 81611.82397587,
  "UsedBlocks": "57861",
  "UsedLogicalBlocks": "29075",
  "WriteIOPS": "244996",
  "actions": {},
  "links": {
    "self": "http://192.168.1.11:9501/v1/stats"
  },
  "type": "stats"
}

I think that these ReadIOPS/WriteIOPS values is TotalIOCount rather than IOPS.
I would like gotgt to return correct IOPS values.

Setup automated jiva image generation