zalf-rdm / geonode-k8s Goto Github PK

Task Description

The first two releases used a version scheme which aligned with GeoNode. Since #66 we have separated the Chart release version from the GeoNode release version which makes sense for several reason.

However, as there are tags which are now conflicting with our current version scheme (version 4.0.x and 4.0.2). From the current scheme perspective, these would lay in the future. I would propose to either re-tag those and document this fact in the README, or to document these two tags to be obsoleted and skip them, we could even skip them on our next release, using version 5.0.0 and document that fact. Then we can leave that version hassle behind.

What do you think?

Additional Information

Any additional information or context that may be helpful in completing the task.

Task Description

test run the new geonode release with geonode-k8s.

Additional Information

Any additional information or context that may be helpful in completing the task.

enable configurations to attached geonode installation onto a already running postgresql Database server via values.yaml configuration

Task Description

the current tasks.py distributed by geonode-k8s, I basically the tasks.py from the geonode project with some minor changes. The original tasks.py designed for docker compose uses the python docker SDK to gather information about the network setup. As kubernetes can use different container engines this must be changed at some point.

Additional Information

Any additional information or context that may be helpful in completing the task.

with the release of geonode 4.1.x this helm chart should support the new version

There is a new patroni version available. Bump charts.yaml to newest version: https://github.com/zalando/patroni/blob/master/docs/releases.rst#version-302

when using a vanilla geonode image build like:

git clone [email protected]:GeoNode/geonode.git
cd geonode
docker build . -t mwall2bitflow/geonode:4.0.x

using this image within the chart leads to an import error related to ldap import in local.settings.py.

Bug Description

when uploading a dataset using the default resources definition from the values.yaml the celery container exceeded the memory limits and went into OOMKilled state.

Additional Information

Any additional information or context that may be helpful in resolving the bug.

Is your feature request related to a problem? Please describe. | Gehört diese neue Funktion zu einem bestehenden Problem? Welches?
This feature should allow a configuration which loads the local geonode code into minikube environment and allows to develop geonode inside of minikube and kubernetes.

Task Description

Generate readthedocs page from docs folder

combine the helm deployment with the geonode unittest to test the helm chart with a spectrum of different configurations.

Task Description

Some parameters in values.yaml are missing a description. Specially external_postgres, rabbitmq, postgres-operator-ui

Additional Information

Any additional information or context that may be helpful in completing the task.

Bug Description

when deploying geonode-k8s the celery container continously crashes, e.g. using minikube deployment

Additional Information

Somehow env var: IS_CELERY is not set anymore in current main branch

Is your feature request related to a problem? Please describe. | Gehört diese neue Funktion zu einem bestehenden Problem? Welches?

currently no celery container is started in the helm chart. Add the container to the deployment for harvesting and other jobs ...

bump dependencies to latest versions

The idea here is to add a thesaurus configuration like:

• Name|identifier
• Url - Url where to download catalog from
• title
• date
• description
• slug
• about
• card min
• card max
• facet
• order

There should be a commented list of available example configurations for e.g. inspire and agrovoc inside the values.yaml. The idea of implementation would be seperate sidecar container which starts if thesauris are enabled. This container then downloads the rdf file from the given url and adds it to the database. Furhter the database itself must be manipulated regarding the extra parameters which
python manage.py load_thesaurus --file ...
cannot handle. Maybe its possible to make this available inside the script as well. Current docs: https://docs.geonode.org/en/master/admin/thesaurus/index.html

Additionally the settings.py must be edit regarding the thesauri entries.

Bug Description

Running a default installation like the minikube, geoserver has an issue connecting to the geoserver database in some cases e.g. when trying to click on some of the references in metadata overview of a dataset. e.g. like excel. This returns in the following error shown in the browser:

 com.google.common.util.concurrent.UncheckedExecutionException: org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection org.hibernate.exception.GenericJDBCException: Cannot open connection Cannot open connection Cannot create PoolableConnectionFactory (FATAL: password authentication failed for user "geonode") FATAL: password authentication failed for user "geonode" 

Reproduction Steps

best to provide your values.yaml, a brief description of your cluster and version of geonode-k8s here.

Behavior

A description of what you expected to happen and what actually happened.

Additional Information

Any additional information or context that may be helpful in resolving the bug.

providing small templates for pull requests, and issues

The geonode/geoserver provides a / templates/geofence directory containing a j2 template for geofence datasource properties. Since 2.23.1 the data connection is a hard wired URL: jdbc:postgresql://db:5432/{{ GEONODE_GEODATABASE }}.

Version 2.23.0 instead has a configurable URL: jdbc:postgresql://{{ DATABASE_HOST }}:{{ DATABASE_PORT }}/{{ GEONODE_GEODATABASE }}

You can check this by either running

docker run --rm geonode/geoserver:2.23.1 cat /templates/geofence/geofence-datasource-ovr.properties.j2

and

docker run --rm geonode/geoserver:2.23.0 cat /templates/geofence/geofence-datasource-ovr.properties.j2

We would have to overload the template until the geonode/geoserver has been updated.

Bug Description

running an image build after geonode release 4.1.2 breaks the geonode container init process. This is related to changes in tasks.py and the deletion of some variables inside the default settings.py. This leads to a stop at entrypoint step to run set the geoserver admin password and further this breaks all furhter steps as they do not get executed.

Error output:

****************************geoserver********************************
GeoServer is available for HTTP calls!
waitforgeoserver tasks done
********************geoserver fixture********************************
Setting GeoServer Admin Password...
sentry enabled ...
https://[email protected]/4504473572016128
Traceback (most recent call last):
  File "/usr/local/bin/invoke", line 8, in <module>
    sys.exit(program.run())
  File "/usr/local/lib/python3.10/dist-packages/invoke/program.py", line 398, in run
    self.execute()
  File "/usr/local/lib/python3.10/dist-packages/invoke/program.py", line 583, in execute
    executor.execute(*self.tasks)
  File "/usr/local/lib/python3.10/dist-packages/invoke/executor.py", line 140, in execute
    result = call.task(*args, **call.kwargs)
  File "/usr/local/lib/python3.10/dist-packages/invoke/tasks.py", line 138, in __call__
    result = self.body(*args, **kwargs)
  File "/usr/src/geonode/tasks.py", line 388, in geoserverfixture
    _geoserver_info_provision(f"{os.environ['GEOSERVER_LOCATION']}rest/")
  File "/usr/src/geonode/tasks.py", line 499, in _geoserver_info_provision
    cat = Catalog(url, username=settings.OGC_SERVER_DEFAULT_USER, password=settings.OGC_SERVER_DEFAULT_PASSWORD)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 83, in __getattr__
    val = getattr(self._wrapped, name)
AttributeError: 'Settings' object has no attribute 'OGC_SERVER_DEFAULT_USER'

Additional Information

Any additional information or context that may be helpful in resolving the bug.

Task Description

the geonode-deploy.yaml contains a merge artifact:

Additional Information

Any additional information or context that may be helpful in completing the task.

Bug Description

when adding a multiple line command into .Values.geonode.tasks_pre_script or .Values.geonode.tasks_post_script the idention inside of the tasks.py file isn't correct anymore. see: https://stackoverflow.com/questions/50951124/multiline-string-to-a-variable-in-a-helm-template

Additional Information

Any additional information or context that may be helpful in resolving the bug.

Task Description

Currently, pod_name is being used for deployed containers. In k8s context pods are volatile and could be replicated as well. We should use the service name for a given deployment. This would reduce the configuration pod_name boilerplate at several locations.

Also, IMO there is no need to make the name configurable. Is there a reason I do not see?

Bug Description

when enabling ldap support in values.yaml. This error occurs in the geonode log:

Traceback (most recent call last):
  File "/usr/src/geonode/./geonode/wsgi.py", line 29, in <module>
    application = get_wsgi_application()
  File "/usr/local/lib/python3.10/dist-packages/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/usr/local/lib/python3.10/dist-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/src/geonode/./geonode/settings.py", line 2395, in <module>
    LDAP_ENABLED = strtobool(os.getenv("LDAP_ENABLED", "False"))
NameError: name 'strtobool' is not defined
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. GAME OVER ***

This also might only occur if also sentry is enabled. I guess the geonode-k8s-settings.py is just misses to import from distutils.util import strtobool

Reproduction Steps

best to provide your values.yaml, a brief description of your cluster and version of geonode-k8s here.

Behavior

A description of what you expected to happen and what actually happened.

Additional Information

Any additional information or context that may be helpful in resolving the bug.

Task Description

geonode.ingress.externalPort had to be set as mandatory parameter. This was due to mistakenly usage as ingress target port. However, now it can be optional.

Additional Information

• Add a check for nil here:
  

  geonode-k8s/deployment/geonode/templates/_helpers.tpl

  Line 85 in bde327c

  {{- if or (eq (toString .Values.geonode.ingress.externalPort) "80") (eq (toString .Values.geonode.ingress.externalPort) "443") -}}

• Check for Oauth config

implement resources into pod deployments

Bug Description

For version 4.1.0 the Helm chart refers to a non final image version 4.1.x. This is suboptimal, as it will not preserve using the same image in the future.

@mwallschlaeger what do you think? Shall we consider to re-release the Chart?

Task Description

this requires changes to the charts.yaml e.g. like the one for jenkins: https://github.com/jenkinsci/helm-charts/blob/main/charts/jenkins/Chart.yaml

Additional Information

Any additional information or context that may be helpful in completing the task.

Task Description

remove the deployment folder from from project structure, so that the geonode folder is in the root of this repository

Additional Information

Any additional information or context that may be helpful in completing the task.

Bug Description

After uploading an dataset to a deployed geonode edit styles leads to a authentication dialog. See issue on github: GeoNode/geonode#11282

Task Description

It would be nice to have a manifest which includes fixtures which are configurable via values.yaml.

Additional Information

• Use {{ toYaml .Values.geonodeFixtures | nindent 2 }} to include the fixtures.

Not sure about auto loading the fixtures via python manage.py loaddata <yourfixturehere>. For now this would have be done manually. What is your opinion on this @mwallschlaeger ?

pycsw is currently used as a library inside of GeoNode. To reduce the complexity of the code and improve microservice design pattern, to improve scalability and resource management pycsw setup as a single deployment running pycsw. GeoNode already supports external pycsw service by settings.py parameters

Task Description

use official geosolutions docker image in geonode-k8s as default image

Is your feature request related to a problem? Please describe. | Gehört diese neue Funktion zu einem bestehenden Problem? Welches?

as described here: https://lists.osgeo.org/pipermail/geonode-users/2015-June/000533.html

csw CATALOGUEs for harvesting can be defined in the settings.py. This must be configurable via values.yaml

Describe the solution you'd like | Beschreibe deinen Lösungsansatz

implement a list of objects which get parsed into settings.py

Bug Description

when deploying geonode-k8s its missing external database credentials in the environment variables inside geonode and pycsw containers

Additional Information

Any additional information or context that may be helpful in resolving the bug.

Feature Description

Currently, I have to configure confidential values in the 'values.yaml'. So this is also true for local values.yaml file for which I want to version configuration settings but not the secret ones.

Proposed Solution

Provide a secret sample including demo setting (from the current values.yaml) and make the name configurable. Then, we can reference to secrets in the env section of each deployment.

In case of overriding, one would have to create a custom secret file within a dedicated configure and configure that secret name. The secret can be safely ignored by version management while all other configuration can be put under version control.

after deploying the helm chart it sometimes randomly orrures that the geonode container can not find the database or does not have access to it. You see the issue in the geonode container logs like:

MONITORING_SERVICE_NAME=geonode-monitoring
MONITORING_DATA_TTL=365
/usr/lib/python3/dist-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
**************************databases*******************************
psql: error: connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  password authentication failed for user "postgres"
connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.244.0.254", user "postgres", database "postgres", no encryption
Postgres is unavailable - sleeping
psql: error: connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  password authentication failed for user "postgres"
connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.244.0.254", user "postgres", database "postgres", no encryption
Postgres is unavailable - sleeping
psql: error: connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  password authentication failed for user "postgres"
connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.244.0.254", user "postgres", database "postgres", no encryption
Postgres is unavailable - sleeping
psql: error: connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  password authentication failed for user "postgres"
connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.244.0.254", user "postgres", database "postgres", no encryption
Postgres is unavailable - sleeping
psql: error: connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  password authentication failed for user "postgres"
connection to server at "geonode-postgresql" (10.104.197.204), port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "10.244.0.254", user "postgres", database "postgres", no encryption
Postgres is unavailable - sleeping

if the database is found once it will be found forever but sometimes within the initial deploy its not working

Bug Description

When running geonode-k8s on a cluster for some days. I noticed that the container isn't responding to web traffic anymore. The logs just reinitializing workers all the time:

worker 10 lifetime reached, it was running for 3601 second(s)
Respawned uWSGI worker 10 (new pid: 1324)
Respawned uWSGI worker 9 (new pid: 1325)
worker 10 lifetime reached, it was running for 3601 second(s)
worker 9 lifetime reached, it was running for 3601 second(s)
Respawned uWSGI worker 10 (new pid: 1331)
Respawned uWSGI worker 9 (new pid: 1332)
worker 9 lifetime reached, it was running for 3601 second(s)
worker 10 lifetime reached, it was running for 3601 second(s)
Respawned uWSGI worker 10 (new pid: 1347)
Respawned uWSGI worker 9 (new pid: 1348)
worker 9 lifetime reached, it was running for 3601 second(s)
worker 10 lifetime reached, it was running for 3601 second(s)
Respawned uWSGI worker 10 (new pid: 1354)
Respawned uWSGI worker 9 (new pid: 1355)
worker 9 lifetime reached, it was running for 3601 second(s)
worker 10 lifetime reached, it was running for 3601 second(s)
Respawned uWSGI worker 10 (new pid: 1361)
Respawned uWSGI worker 9 (new pid: 1362)
worker 9 lifetime reached, it was running for 3601 second(s)
worker 10 lifetime reached, it was running for 3601 second(s)
Respawned uWSGI worker 10 (new pid: 1400)
Respawned uWSGI worker 9 (new pid: 1401)
worker 10 lifetime reached, it was running for 3601 second(s)
Respawned uWSGI worker 10 (new pid: 1413)
worker 9 lifetime reached, it was running for 3601 second(s)
Respawned uWSGI worker 9 (new pid: 1415)
worker 10 lifetime reached, it was running for 3601 second(s)
Respawned uWSGI worker 10 (new pid: 1420)
worker 9 lifetime reached, it was running for 3601 second(s)
Respawned uWSGI worker 9 (new pid: 1422)
worker 10 lifetime reached, it was running for 3601 second(s)
worker 9 lifetime reached, it was running for 3601 second(s)
Respawned uWSGI worker 10 (new pid: 1428)
Respawned uWSGI worker 9 (new pid: 1429)
worker 10 lifetime reached, it was running for 3601 second(s)
worker 9 lifetime reached, it was running for 3601 second(s)
Respawned uWSGI worker 10 (new pid: 1435)
Respawned uWSGI worker 9 (new pid: 1436)
worker 9 lifetime reached, it was running for 3601 second(s)
worker 10 lifetime reached, it was running for 3601 second(s)
Respawned uWSGI worker 10 (new pid: 1442)
Respawned uWSGI worker 9 (new pid: 1443)
worker 9 lifetime reached, it was running for 3601 second(s)
worker 10 lifetime reached, it was running for 3601 second(s)

Maye it's useful here to increase the .Values.geonode.uswgi.max_worker_lifetime default to something higher that 3600 seconds. Also it would be good to have Liveness probes for all the geonode service.

Reproduction Steps

best to provide your values.yaml, a brief description of your cluster and version of geonode-k8s here.

Behavior

A description of what you expected to happen and what actually happened.

Additional Information

Any additional information or context that may be helpful in resolving the bug.

Helm provides the capability to allow reading e.g. configmap content from files. This could be used to allow users to bring their own custom configuration files into this deployment.

I can imagine this can be helpful for several files like: settings.py, geoserver configuration, tasks.py, entrypoint.sh, nginx-configuration and so on.

Implementing this into the helm chart is a doable task.

allow the user to change static database entries like:

• maximum number of parallel downloads

to be edited via the values.yaml.

Therefore is maybe an additional management script required to edit database entries via cmd. This could be executed at starttime

including https://github.com/camptocamp/helm-geoserver-cloud could make the whole deployment scalable.

Bug Description

Error while fetching latest Helm release: Error: [@octokit/auth-action] `GITHUB_TOKEN` variable is not set. It must be set on either `env:` or `with:`. See https://github.com/octokit/auth-action.js#createactionauth. Using default version v3.9.0

The geonode's ingress uses the .Values.geonode.ingress.externalPort to target the geonode nginx service. This is wrong as the service stays on port 80.

This results in an HTTP 503 error when ingress is configured to use TLS (port 443).

Bug Description

PR #49 has an issue as it configures a wrong FIXTURE_DIR. See here

https://github.com/zalf-rdm/geonode-k8s/pull/50/files#r1288372585

Reproduction Steps

Apply Helm Config and see geonode fails to start with

*** Operational MODE: preforking ***
Traceback (most recent call last):
  File "/usr/src/geonode/./geonode/wsgi.py", line 29, in <module>
    application = get_wsgi_application()
  File "/usr/local/lib/python3.10/dist-packages/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/usr/local/lib/python3.10/dist-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/src/geonode/./geonode/settings.py", line 2333
    FIXTURE_DIRS: [ /usr/src/geonode/fixtures ]
                    ^
SyntaxError: invalid syntax
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. GAME OVER ***
2023/08/09 11:59:17 Command exited with error: exit status 22

Behavior

GeoNode starts without an issue

Additional Information

Any additional information or context that may be helpful in resolving the bug.

Task Description

PR #47 missed to update Helm documentation

Additional Information

Any additional information or context that may be helpful in completing the task.

Hi Zalf,
I like your helm chart geonode version.
I am trying to installing on minikube based on your suggestions

But I noticed the pod/geonode-geonode-0 is never been up,
I also tried with

I am wondering is it the lastest geonode image zalf/geonode is tested or if not cloud you suggest which version should I try?
Thanks

Task Description

For more customization, it would be good to have DJANGO_SETTINGS_MODULE configurable.

Additional Information

Any additional information or context that may be helpful in completing the task.

Hey @mwallschlaeger,
Would it be possible for you to add a licence to this repository? I am interested in using this Helm-Chart in the course of my bachelor thesis, but this is not possible without a suitable licence.

For me an MIT license would be ideal, but also the GNU Affero General Public License v3.0 which you use in your other repositories should fit.

I would be very grateful if this could be arranged.

Best regards
Lucas Nickel

Task Description

Currently, externalDomain and externalScheme are part of the ingress configuration. However, this config is optional and one could choose to disable it. It is confusing/surprising, that (at least) the mentioned settings are still required within the setup to construct important configuration, like the GEONODE_LB_HOST_IP.

We should also wrap the ingress manifest with {{ if (eq .Values.geonode.ingress.enabled true) }}. It raises errors in cases where ingress is turned in favor of a different ingress setup and where externalDomain is configured (to resolve the described issue):

Error: UPGRADE FAILED: cannot patch "geonode-nginx-ingress" with kind Ingress: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "<your-domain>" and path "/" is already defined in ingress <namespace/ingress-name>

Task Description

Add a pre-commit hook configuration to ensure helm-docs has been run after updating values.yaml.

Additional Information

Put a contribution section into the README.md.