Solr registers with incorrect port 80 in Zookeeper about solr-operator

[alewis001]

Apologies, it looks like the issue was due to another piece of software that had Solr embedded and the two "instances" we conflicting. Also, sorry it took so long to get back to this.

[whitleykeith]

I'm having the same problem, also on AWS EKS, K8 1.14. I've tried with Solr 8.4.1 and still have the same issue.

[HoustonPutman]

Thanks for reporting this!

The reason it registers itself as port 80 is that it's going through the headless service, which maps port 80 -> 8983. That way, solr listens on port 8983 but all requests to it go through port 80.

I've seen this before, as have others (#71). After a lot of digging, I think I've finally figured it out. In this issue, kubernetes/kubernetes#20488, someone mentions that targetPort and port must be the same for headless services. I haven't seen this documented anywhere, and I have been scouring the kubernetes docs.

So we have two options to fix this:

• Use individual SolrNode services always, no matter if an Ingress is being used or not. This will eliminate our need for Headless Services.
• Use Port 8983 for non-ingress clouds, and keep the headless service.

We are trying to make addressability a lot more configurable in #68. I think in the short term, we should likely use port 8983 instead of 80 and keep the headless service. But that means that we should definitely not create the headless service when using the ingress. Once this is fixed, we can work on expanding the addressability options.

This should be a relatively straightforward change.

[gmaiztegi]

The thing with headless services is that they are just hostnames that resolve to all pods, so there is no port translation from port to targetPort.

I see that there is a regular/proxy service being created for each one of the pods (matching-solrcloud-0 in my case), so I guess that you could use that as hostname to make it work on port 80.

By the way, thank you so much for the quick response.

[HoustonPutman]

I see that there is a regular/proxy service being created for each one of the pods (matching-solrcloud-0 in my case), so I guess that you could use that as hostname to make it work on port 80.

The reason for the proxy services is so that an ingress rule can be matched to each individual Solr pod. The solr-operator will set up each solr cloud node with ingress-urls if the ingress-base-domain flag is passed in on startup, or with headless-service urls if the flag is not used. Without the ingress being created, there's no need to create the individual services for every pod so that step is skipped.

So in the PR that I created, the proxy services and common cloud service still use port 80. The headless service is the only one that exposes 8983.

And no worries, I'm glad the fix was quick!

[gmaiztegi]

That makes a lot of sense. Thank you so much. Looking forward to see it merged.

[HoustonPutman]

This will go out in release v0.2.3.

If possible, would you mind testing it out on your clusters to make sure it works as intended before I cut the release? It works on mine, but the more confidence the better.

[gmaiztegi]

I tested the PR by building it myself and it worked perfectly. The only issue was that the chart does not have a way to specify image pull secrets, so I might submit a PR for that.

[HoustonPutman]

Thanks so much for testing it!

And that'd be amazing. Ping me here on a new issue or on the slack channel if you need help or want to get it reviewed.

[HoustonPutman]

v0.2.3 has been released, so you should be able to use the docker hub image now!

[alewis001]

Hi there. I'm seeing what appears to be exactly this issue on 0.3.0. I followed the instructions on this page https://apache.github.io/solr-operator/docs/local_tutorial but using minikube rather than the K8s in Docker Desktop. I followed the instructions down to and including this section: https://apache.github.io/solr-operator/docs/local_tutorial#start-an-example-solr-cloud-cluster

At this point I see the NPE as per this issue. Also, I cannot successfully add a collection.

Using helm list it outputs:
solr-operator fender 1 2021-06-18 09:48:29.97069 +0100 BST deployed solr-operator-0.3.0 v0.3.0
Which would suggest I've deployed the right version.

Can you please confirm whether this issue has regressed since v0.2.3 or whether I may have done something wrong. The only deviation from the instructions was that I set the replica count to 1 from 3 (and that I'm using minikube but I don't think that should have any affect AFAIK but I can't discount it).

[HoustonPutman]

So can you post some more information? Such as your SolrCloud yaml as well as your Solr logs?

There should not be a headless service exposing port 80 anymore, since that just does not work. I'd be very concerned if the same issue was popping up in v0.3.0.

[alewis001]

Hi, apologies for not getting back sooner.

I'll re-run through the instructions that I linked to and I'll then get all resources that are deployed to k8s. Hopefully I'll be able to get back to this in the next day or so.

[HoustonPutman]

Glad to hear things are working now!