Comments (6)
billimek
commented on August 20, 2024
Current configuration:
➜ k -n logs get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-elasticsearch-data-0 Bound pvc-880f2c76-1d23-11e9-86f8-36db0084bcac 30Gi RWO rbd 56d
data-elasticsearch-data-1 Bound pvc-ec6878d8-1d23-11e9-86f8-36db0084bcac 30Gi RWO rbd 56d
data-elasticsearch-master-0 Bound pvc-881d660e-1d23-11e9-86f8-36db0084bcac 4Gi RWO rbd 56d
data-elasticsearch-master-1 Bound pvc-c0ed5031-1d23-11e9-86f8-36db0084bcac 4Gi RWO rbd 56d
data-elasticsearch-master-2 Bound pvc-de0ff848-1d23-11e9-86f8-36db0084bcac 4Gi RWO rbd 56d
fluentd-syslog Bound pvc-48321811-48d4-11e9-bc07-febc8c45caab 10Gi RWO rbd 19h
nfs-backup-restic-pv-10g-logs Bound nfs-backup-restic-pv-10g-logs 1Mi RWX data-restic-logs 67d
The data pvcs in particular:
➜ for i in $(kubectl get pv | grep rbd | awk '{print $1}')
do
echo "$i => $(kubectl describe pv "$i" | grep RBDImage | awk '{print $2}')"
done| grep pvc-880f2c76-1d23-11e9-86f8-36db0084bcac
pvc-880f2c76-1d23-11e9-86f8-36db0084bcac => kubernetes-dynamic-pvc-8dc13bbc-1d23-11e9-b4f5-524dbb781fe7
➜ for i in $(kubectl get pv | grep rbd | awk '{print $1}')
do
echo "$i => $(kubectl describe pv "$i" | grep RBDImage | awk '{print $2}')"
done| grep pvc-ec6878d8-1d23-11e9-86f8-36db0084bcac
pvc-ec6878d8-1d23-11e9-86f8-36db0084bcac => kubernetes-dynamic-pvc-f01af949-1d23-11e9-b4f5-524dbb781fe7
The ceph pool is currently utilizing 88.9GB of storage.
from k8s-gitops.
billimek
commented on August 20, 2024
Exported kibana saved searches:
[
{
"_id": "13d08380-47d1-11e9-bc0f-ed19e2d2c26e",
"_type": "search",
"_source": {
"title": "flux-helm-operator filtered",
"description": "",
"hits": 0,
"columns": [
"log"
],
"sort": [
"@timestamp",
"desc"
],
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"\",\"language\":\"lucene\"},\"filter\":[{\"meta\":{\"negate\":false,\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"type\":\"phrase\",\"key\":\"kubernetes.labels.app\",\"value\":\"flux-helm-operator\",\"params\":{\"query\":\"flux-helm-operator\",\"type\":\"phrase\"},\"disabled\":false,\"alias\":null},\"query\":{\"match\":{\"kubernetes.labels.app\":{\"query\":\"flux-helm-operator\",\"type\":\"phrase\"}}},\"$state\":{\"store\":\"appState\"}},{\"meta\":{\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"negate\":true,\"disabled\":false,\"alias\":null,\"type\":\"phrase\",\"key\":\"log\",\"value\":\"\\\"End of releasesync\\\"\",\"params\":{\"query\":\"\\\"End of releasesync\\\"\",\"type\":\"phrase\"}},\"query\":{\"match\":{\"log\":{\"query\":\"\\\"End of releasesync\\\"\",\"type\":\"phrase\"}}},\"$state\":{\"store\":\"appState\"}},{\"meta\":{\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"negate\":true,\"disabled\":false,\"alias\":null,\"type\":\"phrase\",\"key\":\"log\",\"value\":\"\\\"processing release\",\"params\":{\"query\":\"\\\"processing release\",\"type\":\"phrase\"}},\"query\":{\"match\":{\"log\":{\"query\":\"\\\"processing release\",\"type\":\"phrase\"}}},\"$state\":{\"store\":\"appState\"}},{\"meta\":{\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"negate\":true,\"disabled\":false,\"alias\":null,\"type\":\"phrase\",\"key\":\"log\",\"value\":\"\\\"Start of releasesync\\\"\",\"params\":{\"query\":\"\\\"Start of releasesync\\\"\",\"type\":\"phrase\"}},\"query\":{\"match\":{\"log\":{\"query\":\"\\\"Start of releasesync\\\"\",\"type\":\"phrase\"}}},\"$state\":{\"store\":\"appState\"}}]}"
}
}
},
{
"_id": "aac38e60-41cb-11e9-9538-5765be0961f2",
"_type": "search",
"_source": {
"title": "Home Assistant logs",
"description": "",
"hits": 0,
"columns": [
"log"
],
"sort": [
"@timestamp",
"desc"
],
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"\",\"language\":\"lucene\"},\"filter\":[{\"meta\":{\"negate\":false,\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"type\":\"phrase\",\"key\":\"kubernetes.labels.app\",\"value\":\"home-assistant\",\"params\":{\"query\":\"home-assistant\",\"type\":\"phrase\"},\"disabled\":false,\"alias\":null},\"query\":{\"match\":{\"kubernetes.labels.app\":{\"query\":\"home-assistant\",\"type\":\"phrase\"}}},\"$state\":{\"store\":\"appState\"}},{\"meta\":{\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"negate\":true,\"disabled\":false,\"alias\":null,\"type\":\"phrase\",\"key\":\"log\",\"value\":\"Serving / to\",\"params\":{\"query\":\"Serving / to\",\"type\":\"phrase\"}},\"query\":{\"match\":{\"log\":{\"query\":\"Serving / to\",\"type\":\"phrase\"}}},\"$state\":{\"store\":\"appState\"}},{\"meta\":{\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"negate\":true,\"disabled\":false,\"alias\":null,\"type\":\"phrase\",\"key\":\"log\",\"value\":\"python3 /config/shell\",\"params\":{\"query\":\"python3 /config/shell\",\"type\":\"phrase\"}},\"query\":{\"match\":{\"log\":{\"query\":\"python3 /config/shell\",\"type\":\"phrase\"}}},\"$state\":{\"store\":\"appState\"}}]}"
}
}
},
{
"_id": "1c34ae70-1d29-11e9-8611-35fa412a1238",
"_type": "search",
"_source": {
"title": "descheduler evicted",
"description": "",
"hits": 0,
"columns": [
"kubernetes.labels.app",
"log"
],
"sort": [
"@timestamp",
"desc"
],
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"evicted\",\"language\":\"lucene\"},\"filter\":[{\"meta\":{\"negate\":false,\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"type\":\"phrase\",\"key\":\"kubernetes.container_name\",\"value\":\"descheduler\",\"params\":{\"query\":\"descheduler\",\"type\":\"phrase\"},\"disabled\":false,\"alias\":null},\"query\":{\"match\":{\"kubernetes.container_name\":{\"query\":\"descheduler\",\"type\":\"phrase\"}}},\"$state\":{\"store\":\"appState\"}},{\"meta\":{\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"negate\":true,\"disabled\":false,\"alias\":null,\"type\":\"phrase\",\"key\":\"log\",\"value\":\"Evicted 0 pods\",\"params\":{\"query\":\"Evicted 0 pods\",\"type\":\"phrase\"}},\"query\":{\"match\":{\"log\":{\"query\":\"Evicted 0 pods\",\"type\":\"phrase\"}}},\"$state\":{\"store\":\"appState\"}}]}"
}
}
},
{
"_id": "9cc64be0-2ecf-11e9-961b-3bbae06572c4",
"_type": "search",
"_source": {
"title": "flux-helm-operator errors",
"description": "",
"hits": 0,
"columns": [
"kubernetes.labels.app",
"log"
],
"sort": [
"@timestamp",
"desc"
],
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"error\",\"language\":\"lucene\"},\"filter\":[{\"meta\":{\"negate\":false,\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"type\":\"phrase\",\"key\":\"kubernetes.labels.app\",\"value\":\"flux-helm-operator\",\"params\":{\"query\":\"flux-helm-operator\",\"type\":\"phrase\"},\"disabled\":false,\"alias\":null},\"query\":{\"match\":{\"kubernetes.labels.app\":{\"query\":\"flux-helm-operator\",\"type\":\"phrase\"}}},\"$state\":{\"store\":\"appState\"}}]}"
}
}
},
{
"_id": "e606e660-1d28-11e9-8611-35fa412a1238",
"_type": "search",
"_source": {
"title": "default app view",
"description": "",
"hits": 0,
"columns": [
"kubernetes.labels.app",
"log"
],
"sort": [
"@timestamp",
"desc"
],
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"d6a5d3c0-1d28-11e9-8611-35fa412a1238\",\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"\",\"language\":\"lucene\"},\"filter\":[]}"
}
}
},
{
"_id": "a6b3a1b0-48ed-11e9-a56f-713f68ab4e3b",
"_type": "search",
"_source": {
"title": "syslog",
"description": "",
"hits": 0,
"columns": [
"host",
"ident",
"message"
],
"sort": [
"@timestamp",
"desc"
],
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"e0d9ea90-48e1-11e9-a56f-713f68ab4e3b\",\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"\",\"language\":\"lucene\"},\"filter\":[]}"
}
}
}
]
from k8s-gitops.
billimek
commented on August 20, 2024
Deleting elasticsearch the helmrelease:
➜ k -n logs delete helmrelease/elasticsearch
helmrelease.flux.weave.works "elasticsearch" deleted
... This will take a while as the data pods take a really long time to delete.
from k8s-gitops.
billimek
commented on August 20, 2024
... It took about 60 mins to delete the data pods:
➜ k -n logs delete pvc/data-elasticsearch-data-0 pvc/data-elasticsearch-data-1
persistentvolumeclaim "data-elasticsearch-data-0" deleted
persistentvolumeclaim "data-elasticsearch-data-1" deleted
jeff in patch-pve at home on master at ☸️ k8s took 58m 49s
... After deleting the pods, the ceph pool was consuming only 32GB of storage.
from k8s-gitops.
billimek
commented on August 20, 2024
Forgot that I needed to also delete the master pvcs too:
➜ k -n logs delete pvc/data-elasticsearch-data-0 pvc/data-elasticsearch-data-1 pvc/data-elasticsearch-master-0 pvc/data-elasticsearch-master-1 pvc/data-elasticsearch-master-2
persistentvolumeclaim "data-elasticsearch-data-0" deleted
persistentvolumeclaim "data-elasticsearch-data-1" deleted
persistentvolumeclaim "data-elasticsearch-master-0" deleted
persistentvolumeclaim "data-elasticsearch-master-1" deleted
persistentvolumeclaim "data-elasticsearch-master-2" deleted
jeff in patch-pve at home on master at ☸️ k8s took 50m 21s
from k8s-gitops.
billimek
commented on August 20, 2024
After re-creating everything, the 60G storage is in-place:
➜ k -n logs exec -ti elasticsearch-data-0 -c elasticsearch -- /usr/bin/df -h /usr/share/elasticsearch/data
Filesystem Size Used Avail Use% Mounted on
/dev/rbd7 59G 285M 59G 1% /usr/share/elasticsearch/data
from k8s-gitops.
Related Issues (20)
- yeet k8s-at-home charts HOT 1
- implement 1Password connect HOT 1
- leverage 1Password for secrets
- experiment with talos or kairos HOT 1
- Enable rook/ceph object storage
- Migrate loki to ceph object storage
- Consider leveraging vector instead of promtail
- Migrate thanos to ceph object storage
- Revisit gitops flux directory structure HOT 3
- Explore draino HOT 3
- rook/ceph OSD repaving incident on 2023-02-22 HOT 1
- Renovate (Mend) Dashboard
- Incorrect ingress in echo-server
- experiment with NixOS HOT 6
- Running homeassistant as non root user HOT 1
- upgrade to k8s 1.20 HOT 2
- cert-manager crds race condition HOT 5
- Migrate from minio to minio-operator HOT 1
- Migrate from zwave2mqtt to zwavejs2mqtt HOT 1
- explore sops HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k8s-gitops.