Comments (13)
Hello @surekha3
Could you provide the following information:
- which version of VictoriaMetrics are you running?
- are there any errors in VictoriaMetrics logs?
- could you check the network tab in browser to get the full error text and share the result?
from victoriametrics.
Hi @zekker6 ,
- We're running victoriametrics app version 1.95.1
- In victoriametrics select logs, I have seen attached error log. Also after sometime, I observed there is drastic consumption of CPU and RAM by victoriametrics storage pods. Is this because of reason of VMUI?
- It's working sometimes but when I changed the date on top right it's throwing attached error page
![victoriaselectlogs](https://private-user-images.githubusercontent.com/49813610/337123711-d2934940-6a33-47cd-a99b-0ef274f7ff3b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTkwMzE3NzgsIm5iZiI6MTcxOTAzMTQ3OCwicGF0aCI6Ii80OTgxMzYxMC8zMzcxMjM3MTEtZDI5MzQ5NDAtNmEzMy00N2NkLWE5OWItMGVmMjc0ZjdmZjNiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjIyVDA0NDQzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTcxMmI4ZDVkZTZiNzk2NDA2MDFmZmUwYjViMTMwZjE0MzQyNDBkNjAzYTA4NmYyYjc1MGI1MmNiNGYzZjAzOTkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.D9ffYgTxKzkWZVtaNDWC9Mm4OTdXKydR0-LPnTJS7iE)
![victoriauierror](https://private-user-images.githubusercontent.com/49813610/337123788-d9f71f84-682c-494e-9566-6e130398fc7d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTkwMzE3NzgsIm5iZiI6MTcxOTAzMTQ3OCwicGF0aCI6Ii80OTgxMzYxMC8zMzcxMjM3ODgtZDlmNzFmODQtNjgyYy00OTRlLTk1NjYtNmUxMzAzOThmYzdkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjIyVDA0NDQzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVlY2Q4NDVmOGM2N2JmNzU5MTUxNGQzYjYxMzRhYjg3Nzk5YmQ4OWEyYmVmYWIyYWE4NTU3NGMzNDE4YWMwMTcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.shOPtIuGydiYtY7X2qmdDHgIFgPbp_LaD1Jl-bt53I0)
![victoriaselectlogs](https://private-user-images.githubusercontent.com/49813610/337124006-aadbb897-68b9-4acf-82a2-f29c82b3c755.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTkwMzE3NzgsIm5iZiI6MTcxOTAzMTQ3OCwicGF0aCI6Ii80OTgxMzYxMC8zMzcxMjQwMDYtYWFkYmI4OTctNjhiOS00YWNmLTgyYTItZjI5YzgyYjNjNzU1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjIyVDA0NDQzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTQ5MmZjNjRjYzk2ZjU5ZmVmNDdhMTRmNWVlYmY3MzM4YTdjMWI1ODRlMTk3MWU4OTk0MDUyN2Y4NDM1YmI3M2MmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.V9bbUwe9mVaApKWntNUfVY5JFsHxAsN2FdGf_U88yKA)
from victoriametrics.
@surekha3 Based on the vmselect error messages it seems like vmstorage is not able to keep up with the requests rate and thus vmselect returns an error.
I would suggest to set up a monitoring for VictoriaMetrics cluster and a dashboard. By using this dashboard you will be able to see if vmstorage hits the resources limits.
from victoriametrics.
@zekker6 You mean vmstorage pod resource limits or -search.maxconcurrentRequests parametes in victoriametrics config?
from victoriametrics.
@surekha3 In this case it is search.maxconcurrentRequests
. Default value for this flag is calculated as number of CPUs * 2
in this case. Usually this default allows to effectively utilize resources available to vmstorage node so it is likely that vmstorage is hitting resource limits as well.
from victoriametrics.
okay when I load VMUI explore cardinality for today's date. it's working fine. But when I load for yesterday's date it's throwing above attached 422 error. So in this case, when I increase maxconcurrentRequests parameter, will it work fine?
Also after using this VMUI, cpu and memory utilisation increased drastically. Is it because of this parameter only?
from victoriametrics.
okay when I load VMUI explore cardinality for today's date. it's working fine. But when I load for yesterday's date it's throwing above attached 422 error. So in this case, when I increase maxconcurrentRequests parameter, will it work fine?
Based on the info above it should help. However, screenshot of browser requests lacks an actual request which returned 422 error. It would be great if you could open "network" tab and run a request once again just to confirm that request does return this specific error.
Also after using this VMUI, cpu and memory utilisation increased drastically. Is it because of this parameter only?
It's hard to tell without cluster monitoring as mentioned above. Using Grafana dashboard for the cluster monitoring will allow to see which requests were taking place at this time and correlate these requests with resource usage spikes.
from victoriametrics.
okay
Can you please guide why there are so many yellow coloured vertical lines in the attached screenshot of one cluster. But I'm not seeing them in other cluster. Is this some issue with first cluster
from victoriametrics.
@surekha3 You can hover the cursor over an arrow in the bottom of the graph and see the title of this vertical line. Yellow ones are usually representing a component restart. It's better to investigate the root cause of the restart in order to make sure the cluster is stable.
from victoriametrics.
@zekker6 you're correct. It states getting restarted. Is that because of leaving default value for -search.maxconcurrentRequests value as 2. Will heavy load causes these restarts?
from victoriametrics.
@surekha3 Heavy load can cause restarts only in case OOM is involved. -search.maxConcurrentRequests
limits amount of requests vmstorage node will process at a time in order to prevent running out of resources, so this should not cause restarts.
Could you check the reason of pods restart in k8s events?
from victoriametrics.
@zekker6 Though it's showing as pod restarts in Grafana dashboard, when I ran kubectl events command, it's not showing as pod get restarted which seems strange
from victoriametrics.
@surekha3 Restarts annotation uses the following query: sum(changes(vm_app_start_timestamp)) by(job)
Could you run this query and check the value of vm_app_start_timestamp
?
from victoriametrics.
Related Issues (20)
- kube-scheduler and kube-controller-manager scrape tls issue on victoria-metrics-k8s-stack HOT 3
- Global WITH templates in vmselect / vmsingle
- vmauth cannot start with empty configuration. HOT 1
- vmauth: default httpserver incorrectly exposed
- VMAlert oauth2 error HOT 1
- vmagent using k8s endpoint role for pod role jobs HOT 6
- Add type parameter to prometheus metrics in vmagent HOT 2
- GetRowsDiff func takes a lot of time when vmagent pushes kube-state-metrics data HOT 3
- Using Query from the beginning of the day HOT 1
- Allow VMAlert to replay alerts till the current time HOT 2
- Different authorization for different targets, via relabel
- IndexDB data consuming more space after pod restarts HOT 3
- Indexdb add scroll mode when vmstorage -retentionPeriod parameter set very long time
- vmalert-tool: process won't exit when no rule is defined under group HOT 1
- Duplicate metrics in a scrape break Prometheus scraping HOT 1
- Victorialogs can't parse syslog timestamps HOT 4
- victorialogs vmui: unresponsive after query execution HOT 1
- Random inconsistent results in query operations from the same instance and exporter HOT 3
- vmalert: replay exit with status 0 when generated samples are not successfully pushed HOT 1
- How to optimize the cross AZ traffic cost of vmagent in AWS eks clusters HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from victoriametrics.