[Bug]: Can't connect to DB when deploying in Kubernetes about helm-charts

[harshitsinghvi22]

Hi @MaxTeiger

I just tried the documentation steps on GCP and everything seems to be working just fine.

Neo4j Browser is loading as it should be . However , i did notice a bug in the logs which will be fixed in the upcoming release but that should not be the reason for your issue.

I have used the .tgz of 5.5.0 from the github releases section

Thanks,
Harshit

[MaxTeiger]

Hi @MaxTeiger

I just tried the documentation steps on GCP and everything seems to be working just fine.

Neo4j Browser is loading as it should be . However , i did notice a bug in the logs which will be fixed in the upcoming release but that should not be the reason for your issue.

I have used the .tgz of 5.5.0 from the github releases section

Thanks,

Harshit

And did you manage to connect to the database from the UI using localhost port forward ?

[harshitsinghvi22]

yes, i was able to connect to the database from the UI using port-forward

[MaxTeiger]

Ok thanks, I'll give it another shot this afternoon.

Did you use 0.5cpu & 1Gb memory for the resources of the pod ?

[harshitsinghvi22]

@MaxTeiger you can use the below config to install the enterprise edition of neo4j helm chart which will by default install a LB for you and you would not need to do the port forwarding

neo4j:
  name: my-standalone
  resources:
    cpu: "0.5"
    memory: "2Gi"

  # Uncomment to set the initial password
  password: "my-initial-password"

  # Uncomment to use enterprise edition
  edition: "enterprise"
  acceptLicenseAgreement: "yes"

volumes:
  data:
    mode: "dynamic"
    dynamic:
      # In GKE;
      # * premium-rwo provisions SSD disks (recommended)
      # * standard-rwo provisions balanced SSD-backed disks
      # * standard provisions HDD disks
      storageClassName: premium-rwo

yes, in my previous reply i have tried the config as mentioned in the documentation

neo4j:
  name: my-standalone
  resources:
    cpu: "0.5"
    memory: "2Gi"

  # Uncomment to set the initial password
  #password: "my-initial-password"

  # Uncomment to use enterprise edition
  #edition: "enterprise"
  #acceptLicenseAgreement: "yes"

volumes:
  data:
    mode: "dynamic"
    dynamic:
      # In GKE;
      # * premium-rwo provisions SSD disks (recommended)
      # * standard-rwo provisions balanced SSD-backed disks
      # * standard provisions HDD disks
      storageClassName: premium-rwo

[ojhughes]

It looks like the LoadBalancer service is not being created? I suspect this is an issue because ArgoCD uses helm template to install the k8s resources. You might need to create the LoadBalancer service manually in this case

[MaxTeiger]

As I use the community version, isn't it normal that the load balancer service isn't created ?

[ojhughes]

I just tried and definitely works for me to port forward

kubectl port-forward services/standalone tcp-bolt tcp-http

[ojhughes]

Actually I did see the same error as you after a minute and had to restart the port forward. I would recommend just creating a separate service for the DB

[MaxTeiger]

Thank you for the tip. I just created another service targeting port 7867 (tcp-bolt)

Here is it's definition in YAML :

apiVersion: v1
kind: Service
metadata:
  labels:
    app: neo4j
    argocd.argoproj.io/instance: staging-neo4j
    helm.neo4j.com/instance: neo4j
    helm.neo4j.com/neo4j.name: neo4j
    helm.neo4j.com/service: default
  name: neo4j-bolt
  namespace: default
spec:
  ports:
    - name: tcp-bolt
      port: 7687
      protocol: TCP
      targetPort: 7687
  publishNotReadyAddresses: false
  selector:
    app: neo4j
    helm.neo4j.com/instance: neo4j
  type: ClusterIP

I then run the port forwards (in different terminal ofc) and get the following outputs :

❯ kubectl port-forward svc/neo4j 7475:7474
Forwarding from 127.0.0.1:7475 -> 7474
Forwarding from [::1]:7475 -> 7474
Handling connection for 7475
Handling connection for 7475
Handling connection for 7475
Handling connection for 7475
...
❯ kubectl port-forward svc/neo4j-bolt 7687:7687
Forwarding from 127.0.0.1:7687 -> 7687
Forwarding from [::1]:7687 -> 7687
Handling connection for 7687
Handling connection for 7687
Handling connection for 7687
Handling connection for 7687
Handling connection for 7687
Handling connection for 7687
Handling connection for 7687
E0329 12:10:06.512509   26712 portforward.go:406] an error occurred forwarding 7687 -> 7687: error forwarding port 7687 to pod 95d188b7f0e1f9332e4a7cf00707cb743e32fd57b7f8f43de39cd9d827122d88, uid : failed to execute portforward in network namespace "/var/run/netns/cni-4d7497c8-5e94-bc6b-2bf2-5605987ad804": read tcp4 127.0.0.1:41318->127.0.0.1:7687: read: connection reset by peer
Handling connection for 7687
E0329 12:10:06.513054   26712 portforward.go:346] error creating error stream for port 7687 -> 7687: use of closed network connection
E0329 12:10:06.513391   26712 portforward.go:234] lost connection to pod

After 1 or 2 minutes the port forward stops and I am disconnected from the DB.

But now I have logs on the pod (I had the time to run :clear & :help command on the DB before it crashes)

ERROR [bolt-357] Terminating connection due to unexpected error
org.neo4j.bolt.protocol.error.streaming.BoltStreamingWriteException: Failed to finalize batch: Cannot write result response
at org.neo4j.bolt.protocol.common.transaction.result.ResultHandler.onFinish(ResultHandler.java:116) ~[neo4j-bolt-5.5.0.jar:5.5.0]
at org.neo4j.bolt.protocol.common.fsm.AbstractStateMachine.after(AbstractStateMachine.java:126) ~[neo4j-bolt-5.5.0.jar:5.5.0]
at org.neo4j.bolt.protocol.common.fsm.AbstractStateMachine.process(AbstractStateMachine.java:99) ~[neo4j-bolt-5.5.0.jar:5.5.0]
at org.neo4j.bolt.protocol.common.connector.connection.AtomicSchedulingConnection.lambda$submit$4(AtomicSchedulingConnection.java:112) ~[neo4j-bolt-5.5.0.jar:5.5.0]
at org.neo4j.bolt.protocol.common.connector.connection.AtomicSchedulingConnection.executeJob(AtomicSchedulingConnection.java:335) ~[neo4j-bolt-5.5.0.jar:5.5.0]
at org.neo4j.bolt.protocol.common.connector.connection.AtomicSchedulingConnection.doExecuteJobs(AtomicSchedulingConnection.java:269) ~[neo4j-bolt-5.5.0.jar:5.5.0]
at org.neo4j.bolt.protocol.common.connector.connection.AtomicSchedulingConnection.executeJobs(AtomicSchedulingConnection.java:211) ~[neo4j-bolt-5.5.0.jar:5.5.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.netty.channel.StacklessClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
Caused by: io.netty.channel.unix.Errors$NativeIoException: writevAddresses(..) failed: Connection reset by peer

Do you have any clue ? Do not hesitate if you need more information
Thank you 🙂

[ojhughes]

Hi @MaxTeiger this seems to be an issue that others have encountered

Out of interest, are using M1 Mac?

[ojhughes]

This is strange.. try entering http://127.0.0.1:7474/browser/ instead of http://localhost:7474/browser/. That seems to work for me Nope it still crashes after some time

[MaxTeiger]

Using 127.0.0.1 seems to work for me, after 5mn still no crash.

I am using Mac Intel i9

I created two ingress, one for the db service I created and one for the http service. Once I tested it and seting up the firewall correctly for my LB, I keep you inform if this solves the problem 🙂

Thank you

[harshitsinghvi22]

@MaxTeiger we have introduced the loadbalancer for community as well now with the following PR
#167

We will update the documentation once the above is released this week.

thanks for bringing this to our notice. Hope the above PR resolves your issue and allows you to connect to Neo4j

[harshitsinghvi22]

closing this now as the documentation is updated.

[luchillo17]

Can't find anything on the docs about running that load balancer behind an Ingress, for example, if I want my db exposed at <hostname>/neo4j or <hostname>/db, I can use the rewrite/redirect feature of nginx to properly render the neo4j browser, but when connecting the issue is the server's advertised bolt address uses localhost without any path, idk what to do here:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: neo4j-ingress-http
  namespace: neo4j
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    # nginx.ingress.kubernetes.io/configuration-snippet: |
    #   rewrite ^(/neo4j)$ $1/ redirect;
spec:
  rules:
    - http:
        paths:
          - pathType: Prefix
            path: /neo4j/http(/|$)(.*)
            backend:
              service:
                name: neo4j-db-lb-neo4j
                port:
                  name: http
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: neo4j-ingress-bolt
  namespace: neo4j
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite ^(/neo4j.7687)$ $1/ redirect;
spec:
  rules:
    - http:
        paths:
          - pathType: Prefix
            path: /neo4j.7687(/|$)(.*)
            backend:
              service:
                name: neo4j-db-lb-neo4j
                port:
                  name: tcp-bolt

To be precise the bolt port does connect properly over HTTP, but it then advertises a web socket URL under ws://localhost:7687 which is both missing the path & adding a port, regardless of what I do in Ingress, so I can't find a way to connect to it...