Comments (15)
popduke
commented on August 11, 2024
你的问题跟这两个指标关系不大。大消息体在rpc过程中对cpu的消耗不能忽视，目前为减少内存copy，mqtt payload会在direct memory上分配，并且再完成dist后才会被间接释放。用OpenJDK的话，可以尝试通过JVM Arg -XX:ZCollectionInterval=N提升gc的频率来加速间接回收。另外，你的生产环境只有2C 4G？你期望的性能目标是什么？
from bifromq.
masterOcean
commented on August 11, 2024
你的问题跟这两个指标关系不大。大消息体在rpc过程中对cpu的消耗不能忽视，目前为减少内存copy，mqtt payload会在direct memory上分配，并且再完成dist后才会被间接释放。用OpenJDK的话，可以尝试通过JVM Arg
-XX:ZCollectionInterval=N提升gc的频率来加速间接回收。另外，你的生产环境只有2C 4G？你期望的性能目标是什么？
- 我们其实关注的是理论上 qos 0 消息不会持久化的，为什么 dist 模块设置的 dataEninge 设置为 rocksdb 时比 memory 差这么多；
- 我们是在测试环境验证时发现了这个问题，生产环境肯定不是这个配置了
from bifromq.
popduke
commented on August 11, 2024
cleansession=true，消息并没有持久化，你说的性能差指哪方面，吞吐，latency？有数字吗？还是仅是rocksdb下，direct memory OOM了？
from bifromq.
masterOcean
commented on August 11, 2024
cleansession=true，消息并没有持久化，你说的性能差指哪方面，吞吐，latency？有数字吗？还是仅是rocksdb下，direct memory OOM了？
rocksdb 下, direct memory OOM 了，然后吞吐 比 dataEngineType = memory 差很多
from bifromq.
popduke
commented on August 11, 2024
很远是多少，用的什么压测工具，提供下详细的压测参数？如果是OOM了，吞吐必然受影响，你最好先优化下gc参数，保证不OOM的情况下，在对比结果
from bifromq.
masterOcean
commented on August 11, 2024
很远是多少，用的什么压测工具，提供下详细的压测参数？如果是OOM了，吞吐必然受影响，你最好先优化下gc参数，保证不OOM的情况下，在对比结果
- 压测工具是测试同学 golang 开发的。测试参数： 4w cleanSession=true 的连接，客户端每隔 2S 发送 4K 的消息（所有客户端不是一起发送的，没有峰值压力），发送topic 为 xxx/${client_id}/pub，没有客户端订阅这些消息。
- memory 性能大概是 rocksdb 的 4倍（memory 能支持 16k 消息， rocksdb 下只有 3.8k ）
- gc 参数我们调整过，没有效果。通过 arthas 工具观察 memory 情况，heap 内存很正常
from bifromq.
popduke
commented on August 11, 2024
检查下你rocksdb和memory运行的gc日志里gc运行的频率差异。
3个2C 4GNode Cluster下用emqtt_bench测rocksdb + 16KB payload并没有出现你说的问题，Direct Buffer稳定在800MB左右
cmd: ./emqtt_bench pub -V 4 -h <NODE1_ADDR,NODE2_ADDR,NODE3_ADDR> -c 40000 -I 2000 -i 1 -t bench/%i -s 16384 -u DevOnly/16KPayload
JVM Args:
2024-05-09 17:05:52.950 INFO [main] --- c.b.bifromq.starter.StandaloneStarter [StandaloneStarter.java:611] JVM arguments:
-Xms2G
-Xmx2G
-XX:MetaspaceSize=128m
-XX:MaxMetaspaceSize=500m
-XX:MaxDirectMemorySize=2G
-XX:ZCollectionInterval=5
-XX:MaxInlineLevel=15
-Djava.awt.headless=true
-XX:+UnlockExperimentalVMOptions
-XX:+UnlockDiagnosticVMOptions
-XX:+UseZGC
-XX:ZAllocationSpikeTolerance=5
from bifromq.
mefly2012
commented on August 11, 2024
@popduke 谢谢你的耐心，我的压测模型里每个客户端还订阅了3个clientid相关的topic，而且每个client都往同一个topic（也可以是随机往3个固定的topic）里发送，不知道跟这个是否有关系，我空了再试试具体是哪个原因引起的。
from bifromq.
mefly2012
commented on August 11, 2024
启动参数：
java -Xms1106m -Xmx2212m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=200m -XX:MaxDirectMemorySize=773234680 -server -XX:MaxInlineLevel=15 -Djav.awt.headless=true -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+UseZGC -XX:ZAllocationSpikeTolerance=5 -Xlog:async -Xlog:gc:file=/data2/bifromq/bifromq-2.1.1/bin/../logs/gc-%t.log:time,tid,tags:filecount=5,filesize=50m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data2/bifromq/bifromq-2.1.1/bin/../logs -Ddist_worker_topic_match_expiry_seconds=600 -Ddist_worker_match_parallelism=8 -cp /data2/bifromq/bifromq-2.1.1/bin/../lib/* -DLOG_DIR=/data2/bifromq/bifromq-2.1.1/bin/../logs -DCONF_DIR=/data2/bifromq/bifromq-2.1.1/bin/../conf -DDATA_DIR=/data2/bifromq/bifromq-2.1.1/bin/../data -DBIND_ADDR=10.89.144.20 -Dlogback.configurationFile=/data2/bifromq/bifromq-2.1.1/bin/../conf/logback.xml -Dpf4j.pluginsDir=/data2/bifromq/bifromq-2.1.1/bin/../plugins com.baidu.bifromq.starter.StandaloneStarter -c /data2/bifromq/bifromq-2.1.1/bin/../conf/standalone.yml
参数配置：
bootstrap: true
clusterConfig:
env: "Test"
host: "blabla"
port: 8899
seedEndpoints: "blabla"
mqttServerConfig:
connTimeoutSec: 20
maxConnPerSec: 2000
maxDisconnPerSec: 1000
maxMsgByteSize: 262144
maxResendTimes: 5
maxConnBandwidth: 524288
defaultKeepAliveSec: 300
qos2ConfirmWindowSec: 5
bossELGThreads: 1
workerELGThreads: 2
tcpListener:
enable: true
host: "0.0.0.0"
port: 1883
tlsListener:
enable: true
host: "0.0.0.0"
port: 8883
sslConfig:
certFile: "server.crt"
keyFile: "server_pkcs8.key"
trustCertsFile: "root.crt"
clientAuth: "REQUIRE"
wsListener:
enable: true
host: "0.0.0.0"
port: 8080
wsPath: "/mqtt"
wssListener:
enable: false
host: "0.0.0.0"
port: 8443
wsPath: "/mqtt"
rpcClientConfig:
workerThreads: 4
rpcServerConfig:
host: "blabla"
port: 0
workerThreads: 4
baseKVRpcServerConfig:
port: 0
stateStoreConfig:
queryThreads: 2
tickerThreads: 1
bgWorkerThreads: 1
distWorkerConfig:
queryPipelinePerStore: 1000
compactWALThreshold: 2500
dataEngineConfig:
type: "rocksdb"
dataPathRoot: ""
manualCompaction: false
compactMinTombstoneKeys: 200000
compactMinTombstoneRanges: 100000
compactTombstoneRatio: 0.3
asyncWALFlush: true
fsyncWAL: false
walEngineConfig:
type: "rocksdb"
dataPathRoot: ""
manualCompaction: false
compactMinTombstoneKeys: 200000
compactMinTombstoneRanges: 100000
compactTombstoneRatio: 0.3
asyncWALFlush: true
fsyncWAL: false
balanceConfig:
scheduleIntervalInMs: 5000
balancers:
- "com.baidu.bifromq.dist.worker.balance.RangeLeaderBalancerFactory"
- "com.baidu.bifromq.dist.worker.balance.ReplicaCntBalancerFactory"
inboxStoreConfig:
queryPipelinePerStore: 100
compactWALThreshold: 2500
gcIntervalSeconds: 600
purgeDelaySeconds: 180
dataEngineConfig:
type: "rocksdb"
dataPathRoot: ""
manualCompaction: false
compactMinTombstoneKeys: 200000
compactMinTombstoneRanges: 100000
compactTombstoneRatio: 0.3
asyncWALFlush: true
fsyncWAL: false
walEngineConfig:
type: "rocksdb"
dataPathRoot: ""
manualCompaction: true
compactMinTombstoneKeys: 2500
compactMinTombstoneRanges: 2
compactTombstoneRatio: 0.3
asyncWALFlush: true
fsyncWAL: false
balanceConfig:
scheduleIntervalInMs: 5000
balancers:
- "com.baidu.bifromq.inbox.store.balance.ReplicaCntBalancerFactory"
- "com.baidu.bifromq.inbox.store.balance.RangeSplitBalancerFactory"
- "com.baidu.bifromq.inbox.store.balance.RangeLeaderBalancerFactory"
retainStoreConfig:
queryPipelinePerStore: 100
compactWALThreshold: 2500
gcIntervalSeconds: 600
dataEngineConfig:
type: "rocksdb"
dataPathRoot: ""
manualCompaction: false
compactMinTombstoneKeys: 200000
compactMinTombstoneRanges: 100000
compactTombstoneRatio: 0.3
asyncWALFlush: true
fsyncWAL: false
walEngineConfig:
type: "rocksdb"
dataPathRoot: ""
manualCompaction: true
compactMinTombstoneKeys: 5000
compactMinTombstoneRanges: 2
compactTombstoneRatio: 0.3
asyncWALFlush: true
fsyncWAL: false
balanceConfig:
scheduleIntervalInMs: 5000
balancers:
- "com.baidu.bifromq.retain.store.balance.ReplicaCntBalancerFactory"
apiServerConfig:
enable: true
httpPort: 8091
apiBossThreads: 1
apiWorkerThreads: 2
httpsListenerConfig:
enable: false
port: 8090
我用emqtt-bench的工具也会将服务压挂，
指令如下：docker run -it emqx/emqtt-bench pub -c 40000 -i 10 -V 4 -I 2000 -t benchtopic -h ip1,ip2,ip3 -u abc/def -s 4096
报错信息如下图
集群某个服务上会有大量的1883链接无法释放，cpu占用居高不下，要很久才会恢复。
如果将包的大小改成-s 3500，就没问题
from bifromq.
mefly2012
commented on August 11, 2024
@popduke 谢谢你的耐心，我的压测模型里每个客户端还订阅了3个clientid相关的topic，而且每个client都往同一个topic（也可以是随机往3个固定的topic）里发送，不知道跟这个是否有关系，我空了再试试具体是哪个原因引起的。
后来我改了我的压测代码，client不订阅，或者往多个topic里发送数据，效果是一样的，也会挂。改为使用emqtt-bench测试。
from bifromq.
popduke
commented on August 11, 2024
-XX:ZCollectionInterval选项试过了？- 贴上来你的gc日志
from bifromq.
mefly2012
commented on August 11, 2024
主节点# cat gc-2024-05-10_14-15-28.log
[2024-05-10T14:15:28.136+0800][7821][gc] Using The Z Garbage Collector
[2024-05-10T14:15:31.736+0800][7825][gc] GC(0) Garbage Collection (Warmup) 230M(10%)->26M(1%)
[2024-05-10T14:17:57.320+0800][7825][gc] GC(1) Garbage Collection (Warmup) 444M(20%)->146M(7%)
[2024-05-10T14:18:32.836+0800][7825][gc] GC(2) Garbage Collection (Warmup) 664M(30%)->194M(9%)
[2024-05-10T14:19:15.205+0800][7825][gc] GC(3) Garbage Collection (Proactive) 1482M(67%)->282M(13%)
[2024-05-10T14:19:50.076+0800][7825][gc] GC(4) Garbage Collection (Allocation Rate) 1906M(86%)->344M(16%)
[2024-05-10T14:19:50.677+0800][7825][gc] GC(5) Garbage Collection (Allocation Rate) 408M(18%)->226M(10%)
[2024-05-10T14:20:18.468+0800][7825][gc] GC(6) Garbage Collection (Allocation Rate) 1888M(85%)->440M(20%)
[2024-05-10T14:20:19.130+0800][7825][gc] GC(7) Garbage Collection (Allocation Rate) 470M(21%)->276M(12%)
[2024-05-10T14:20:19.527+0800][7825][gc] GC(8) Garbage Collection (Allocation Rate) 276M(12%)->272M(12%)
[2024-05-10T14:20:42.422+0800][7825][gc] GC(9) Garbage Collection (Allocation Rate) 1882M(85%)->402M(18%)
[2024-05-10T14:20:43.096+0800][7825][gc] GC(10) Garbage Collection (Allocation Rate) 476M(22%)->298M(13%)
[2024-05-10T14:21:03.033+0800][7825][gc] GC(11) Garbage Collection (Allocation Rate) 1854M(84%)->448M(20%)
[2024-05-10T14:21:03.639+0800][7825][gc] GC(12) Garbage Collection (Allocation Rate) 490M(22%)->306M(14%)
[2024-05-10T14:21:04.219+0800][7825][gc] GC(13) Garbage Collection (Allocation Rate) 312M(14%)->306M(14%)
[2024-05-10T14:21:22.013+0800][7825][gc] GC(14) Garbage Collection (Allocation Rate) 1818M(82%)->480M(22%)
[2024-05-10T14:21:22.653+0800][7825][gc] GC(15) Garbage Collection (Allocation Rate) 506M(23%)->322M(15%)
[2024-05-10T14:21:23.234+0800][7825][gc] GC(16) Garbage Collection (Allocation Rate) 328M(15%)->318M(14%)
[2024-05-10T14:21:39.385+0800][7825][gc] GC(17) Garbage Collection (Allocation Rate) 1788M(81%)->504M(23%)
[2024-05-10T14:21:40.007+0800][7825][gc] GC(18) Garbage Collection (Allocation Rate) 510M(23%)->340M(15%)
[2024-05-10T14:21:40.561+0800][7825][gc] GC(19) Garbage Collection (Allocation Rate) 344M(16%)->332M(15%)
[2024-05-10T14:21:54.940+0800][7825][gc] GC(20) Garbage Collection (Allocation Rate) 1728M(78%)->462M(21%)
[2024-05-10T14:21:55.583+0800][7825][gc] GC(21) Garbage Collection (Allocation Rate) 518M(23%)->332M(15%)
[2024-05-10T14:21:56.190+0800][7825][gc] GC(22) Garbage Collection (Allocation Rate) 336M(15%)->336M(15%)
[2024-05-10T14:22:09.655+0800][7825][gc] GC(23) Garbage Collection (Allocation Rate) 1726M(78%)->472M(21%)
[2024-05-10T14:22:10.360+0800][7825][gc] GC(24) Garbage Collection (Allocation Rate) 528M(24%)->366M(17%)
[2024-05-10T14:22:11.047+0800][7825][gc] GC(25) Garbage Collection (Allocation Rate) 372M(17%)->350M(16%)
[2024-05-10T14:22:23.301+0800][7825][gc] GC(26) Garbage Collection (Allocation Rate) 1692M(76%)->450M(20%)
[2024-05-10T14:22:24.069+0800][7825][gc] GC(27) Garbage Collection (Allocation Rate) 540M(24%)->368M(17%)
[2024-05-10T14:22:24.712+0800][7825][gc] GC(28) Garbage Collection (Allocation Rate) 376M(17%)->354M(16%)
[2024-05-10T14:22:35.717+0800][7825][gc] GC(29) Garbage Collection (Allocation Rate) 1620M(73%)->478M(22%)
[2024-05-10T14:22:36.492+0800][7825][gc] GC(30) Garbage Collection (Allocation Rate) 536M(24%)->378M(17%)
[2024-05-10T14:22:37.136+0800][7825][gc] GC(31) Garbage Collection (Allocation Rate) 382M(17%)->368M(17%)
[2024-05-10T14:22:47.331+0800][7825][gc] GC(32) Garbage Collection (Allocation Rate) 1592M(72%)->448M(20%)
[2024-05-10T14:22:48.247+0800][7825][gc] GC(33) Garbage Collection (Allocation Rate) 554M(25%)->392M(18%)
[2024-05-10T14:22:57.773+0800][7825][gc] GC(34) Garbage Collection (Allocation Rate) 1560M(71%)->498M(23%)
[2024-05-10T14:22:58.670+0800][7825][gc] GC(35) Garbage Collection (Allocation Rate) 552M(25%)->406M(18%)
[2024-05-10T14:23:07.505+0800][7825][gc] GC(36) Garbage Collection (Allocation Rate) 1462M(66%)->586M(26%)
[2024-05-10T14:23:08.300+0800][7825][gc] GC(37) Garbage Collection (Allocation Rate) 590M(27%)->418M(19%)
[2024-05-10T14:23:09.063+0800][7825][gc] GC(38) Garbage Collection (Allocation Rate) 424M(19%)->406M(18%)
[2024-05-10T14:23:16.163+0800][7825][gc] GC(39) Garbage Collection (Allocation Rate) 1298M(59%)->486M(22%)
[2024-05-10T14:23:17.158+0800][7825][gc] GC(40) Garbage Collection (Allocation Rate) 546M(25%)->442M(20%)
[2024-05-10T14:23:24.823+0800][7825][gc] GC(41) Garbage Collection (Allocation Rate) 1420M(64%)->502M(23%)
[2024-05-10T14:23:25.696+0800][7825][gc] GC(42) Garbage Collection (Allocation Rate) 558M(25%)->424M(19%)
[2024-05-10T14:23:32.821+0800][7825][gc] GC(43) Garbage Collection (Allocation Rate) 1354M(61%)->486M(22%)
[2024-05-10T14:23:33.746+0800][7825][gc] GC(44) Garbage Collection (Allocation Rate) 560M(25%)->436M(20%)
[2024-05-10T14:23:40.544+0800][7825][gc] GC(45) Garbage Collection (Allocation Rate) 1334M(60%)->560M(25%)
[2024-05-10T14:23:41.476+0800][7825][gc] GC(46) Garbage Collection (Allocation Rate) 628M(28%)->456M(21%)
[2024-05-10T14:23:42.328+0800][7825][gc] GC(47) Garbage Collection (Allocation Rate) 464M(21%)->434M(20%)
[2024-05-10T14:23:49.404+0800][7825][gc] GC(48) Garbage Collection (Allocation Rate) 1384M(63%)->534M(24%)
[2024-05-10T14:23:50.339+0800][7825][gc] GC(49) Garbage Collection (Allocation Rate) 570M(26%)->458M(21%)
[2024-05-10T14:23:51.311+0800][7825][gc] GC(50) Garbage Collection (Allocation Rate) 472M(21%)->456M(21%)
[2024-05-10T14:23:57.612+0800][7825][gc] GC(51) Garbage Collection (Allocation Rate) 1294M(58%)->532M(24%)
[2024-05-10T14:23:58.637+0800][7825][gc] GC(52) Garbage Collection (Allocation Rate) 586M(26%)->472M(21%)
[2024-05-10T14:23:59.627+0800][7825][gc] GC(53) Garbage Collection (Allocation Rate) 486M(22%)->464M(21%)
[2024-05-10T14:24:05.862+0800][7825][gc] GC(54) Garbage Collection (Allocation Rate) 1286M(58%)->630M(28%)
[2024-05-10T14:24:06.846+0800][7825][gc] GC(55) Garbage Collection (Allocation Rate) 640M(29%)->478M(22%)
[2024-05-10T14:24:07.806+0800][7825][gc] GC(56) Garbage Collection (Allocation Rate) 494M(22%)->472M(21%)
[2024-05-10T14:24:12.895+0800][7825][gc] GC(57) Garbage Collection (Allocation Rate) 1160M(52%)->506M(23%)
[2024-05-10T14:24:13.996+0800][7825][gc] GC(58) Garbage Collection (Allocation Rate) 572M(26%)->496M(22%)
[2024-05-10T14:24:19.251+0800][7825][gc] GC(59) Garbage Collection (Allocation Rate) 1208M(55%)->538M(24%)
[2024-05-10T14:24:20.234+0800][7825][gc] GC(60) Garbage Collection (Allocation Rate) 588M(27%)->508M(23%)
[2024-05-10T14:24:21.376+0800][7825][gc] GC(61) Garbage Collection (Allocation Rate) 522M(24%)->540M(24%)
[2024-05-10T14:24:23.269+0800][7825][gc] GC(62) Garbage Collection (Allocation Rate) 700M(32%)->496M(22%)
[2024-05-10T14:24:27.815+0800][7825][gc] GC(63) Garbage Collection (Allocation Rate) 1110M(50%)->566M(26%)
[2024-05-10T14:24:28.852+0800][7825][gc] GC(64) Garbage Collection (Allocation Rate) 586M(26%)->526M(24%)
[2024-05-10T14:24:29.906+0800][7825][gc] GC(65) Garbage Collection (Allocation Rate) 544M(25%)->544M(25%)
[2024-05-10T14:24:34.484+0800][7825][gc] GC(66) Garbage Collection (Allocation Rate) 1154M(52%)->560M(25%)
[2024-05-10T14:24:35.557+0800][7825][gc] GC(67) Garbage Collection (Allocation Rate) 590M(27%)->520M(24%)
[2024-05-10T14:24:36.584+0800][7825][gc] GC(68) Garbage Collection (Allocation Rate) 536M(24%)->504M(23%)
[2024-05-10T14:24:41.336+0800][7825][gc] GC(69) Garbage Collection (Allocation Rate) 1134M(51%)->582M(26%)
[2024-05-10T14:24:42.390+0800][7825][gc] GC(70) Garbage Collection (Allocation Rate) 616M(28%)->490M(22%)
[2024-05-10T14:24:43.388+0800][7825][gc] GC(71) Garbage Collection (Allocation Rate) 502M(23%)->498M(23%)
[2024-05-10T14:24:47.920+0800][7825][gc] GC(72) Garbage Collection (Allocation Rate) 1114M(50%)->516M(23%)
[2024-05-10T14:24:49.195+0800][7825][gc] GC(73) Garbage Collection (Allocation Rate) 606M(27%)->516M(23%)
[2024-05-10T14:24:50.146+0800][7825][gc] GC(74) Garbage Collection (Allocation Rate) 520M(24%)->534M(24%)
[2024-05-10T14:24:51.121+0800][7825][gc] GC(75) Garbage Collection (Allocation Rate) 548M(25%)->516M(23%)
[2024-05-10T14:24:55.927+0800][7825][gc] GC(76) Garbage Collection (Allocation Rate) 1168M(53%)->542M(25%)
[2024-05-10T14:24:56.991+0800][7825][gc] GC(77) Garbage Collection (Allocation Rate) 594M(27%)->498M(23%)
[2024-05-10T14:25:02.024+0800][7825][gc] GC(78) Garbage Collection (Allocation Rate) 1194M(54%)->548M(25%)
[2024-05-10T14:25:03.066+0800][7825][gc] GC(79) Garbage Collection (Allocation Rate) 614M(28%)->494M(22%)
[2024-05-10T14:25:04.057+0800][7825][gc] GC(80) Garbage Collection (Allocation Rate) 504M(23%)->488M(22%)
[2024-05-10T14:25:09.390+0800][7825][gc] GC(81) Garbage Collection (Allocation Rate) 1232M(56%)->564M(25%)
[2024-05-10T14:25:10.392+0800][7825][gc] GC(82) Garbage Collection (Allocation Rate) 608M(27%)->506M(23%)
[2024-05-10T14:25:11.285+0800][7825][gc] GC(83) Garbage Collection (Allocation Rate) 514M(23%)->504M(23%)
[2024-05-10T14:25:16.070+0800][7825][gc] GC(84) Garbage Collection (Allocation Rate) 1152M(52%)->558M(25%)
[2024-05-10T14:25:17.096+0800][7825][gc] GC(85) Garbage Collection (Allocation Rate) 588M(27%)->518M(23%)
[2024-05-10T14:25:18.075+0800][7825][gc] GC(86) Garbage Collection (Allocation Rate) 524M(24%)->496M(22%)
[2024-05-10T14:25:23.379+0800][7825][gc] GC(87) Garbage Collection (Allocation Rate) 1218M(55%)->554M(25%)
[2024-05-10T14:25:24.577+0800][7825][gc] GC(88) Garbage Collection (Allocation Rate) 598M(27%)->564M(25%)
[2024-05-10T14:25:25.556+0800][7825][gc] GC(89) Garbage Collection (Allocation Rate) 574M(26%)->502M(23%)
[2024-05-10T14:25:26.690+0800][7825][gc] GC(90) Garbage Collection (Allocation Rate) 532M(24%)->500M(23%)
[2024-05-10T14:25:31.084+0800][7825][gc] GC(91) Garbage Collection (Allocation Rate) 1084M(49%)->548M(25%)
[2024-05-10T14:25:32.135+0800][7825][gc] GC(92) Garbage Collection (Allocation Rate) 584M(26%)->518M(23%)
[2024-05-10T14:25:33.058+0800][7825][gc] GC(93) Garbage Collection (Allocation Rate) 534M(24%)->498M(23%)
[2024-05-10T14:25:35.141+0800][7825][gc] GC(94) Garbage Collection (Allocation Rate) 686M(31%)->520M(24%)
[2024-05-10T14:25:36.408+0800][7825][gc] GC(95) Garbage Collection (Allocation Rate) 578M(26%)->506M(23%)
[2024-05-10T14:25:40.863+0800][7825][gc] GC(96) Garbage Collection (Allocation Rate) 1102M(50%)->542M(25%)
[2024-05-10T14:25:41.901+0800][7825][gc] GC(97) Garbage Collection (Allocation Rate) 582M(26%)->494M(22%)
[2024-05-10T14:25:43.182+0800][7825][gc] GC(98) Garbage Collection (Allocation Rate) 536M(24%)->498M(23%)
[2024-05-10T14:25:47.665+0800][7825][gc] GC(99) Garbage Collection (Allocation Rate) 1094M(49%)->540M(24%)
[2024-05-10T14:25:48.803+0800][7825][gc] GC(100) Garbage Collection (Allocation Rate) 596M(27%)->502M(23%)
[2024-05-10T14:25:49.789+0800][7825][gc] GC(101) Garbage Collection (Allocation Rate) 506M(23%)->498M(23%)
[2024-05-10T14:25:54.856+0800][7825][gc] GC(102) Garbage Collection (Allocation Rate) 1190M(54%)->552M(25%)
[2024-05-10T14:25:55.894+0800][7825][gc] GC(103) Garbage Collection (Allocation Rate) 592M(27%)->500M(23%)
[2024-05-10T14:25:56.902+0800][7825][gc] GC(104) Garbage Collection (Allocation Rate) 506M(23%)->490M(22%)
[2024-05-10T14:26:02.141+0800][7825][gc] GC(105) Garbage Collection (Allocation Rate) 1218M(55%)->540M(24%)
[2024-05-10T14:26:03.204+0800][7825][gc] GC(106) Garbage Collection (Allocation Rate) 596M(27%)->502M(23%)
[2024-05-10T14:26:04.274+0800][7825][gc] GC(107) Garbage Collection (Allocation Rate) 522M(24%)->484M(22%)
[2024-05-10T14:26:09.517+0800][7825][gc] GC(108) Garbage Collection (Allocation Rate) 1204M(54%)->576M(26%)
[2024-05-10T14:26:10.538+0800][7825][gc] GC(109) Garbage Collection (Allocation Rate) 596M(27%)->560M(25%)
[2024-05-10T14:26:11.545+0800][7825][gc] GC(110) Garbage Collection (Allocation Rate) 576M(26%)->498M(23%)
[2024-05-10T14:26:15.748+0800][7825][gc] GC(111) Garbage Collection (Allocation Rate) 1060M(48%)->524M(24%)
[2024-05-10T14:26:16.924+0800][7825][gc] GC(112) Garbage Collection (Allocation Rate) 596M(27%)->508M(23%)
[2024-05-10T14:26:21.615+0800][7825][gc] GC(113) Garbage Collection (Allocation Rate) 1152M(52%)->560M(25%)
[2024-05-10T14:26:22.622+0800][7825][gc] GC(114) Garbage Collection (Allocation Rate) 584M(26%)->526M(24%)
[2024-05-10T14:26:23.530+0800][7825][gc] GC(115) Garbage Collection (Allocation Rate) 528M(24%)->482M(22%)
[2024-05-10T14:27:16.193+0800][7825][gc] GC(116) Garbage Collection (Proactive) 1054M(48%)->362M(16%)
[2024-05-10T14:31:40.084+0800][7825][gc] GC(117) Garbage Collection (Proactive) 584M(26%)->300M(14%)
[2024-05-10T14:33:06.636+0800][7825][gc] GC(118) Garbage Collection (Allocation Rate) 576M(26%)->374M(17%)
[2024-05-10T14:33:07.268+0800][7825][gc] GC(119) Garbage Collection (Allocation Rate) 376M(17%)->296M(13%)
[2024-05-10T14:33:07.914+0800][7825][gc] GC(120) Garbage Collection (Allocation Rate) 350M(16%)->502M(23%)
[2024-05-10T14:33:08.443+0800][7825][gc] GC(121) Garbage Collection (Allocation Rate) 502M(23%)->338M(15%)
[2024-05-10T14:37:42.477+0800][7825][gc] GC(122) Garbage Collection (Proactive) 560M(25%)->246M(11%)
[2024-05-10T14:42:27.004+0800][7825][gc] GC(123) Garbage Collection (Proactive) 468M(21%)->212M(10%)
[2024-05-10T14:47:11.975+0800][7825][gc] GC(124) Garbage Collection (Proactive) 434M(20%)->176M(8%)
[2024-05-10T14:52:12.260+0800][7825][gc] GC(125) Garbage Collection (Proactive) 394M(18%)->176M(8%)
[2024-05-10T14:57:03.754+0800][7825][gc] GC(126) Garbage Collection (Proactive) 398M(18%)->178M(8%)
[2024-05-10T15:02:04.043+0800][7825][gc] GC(127) Garbage Collection (Proactive) 394M(18%)->178M(8%)
[2024-05-10T15:07:04.353+0800][7825][gc] GC(128) Garbage Collection (Proactive) 394M(18%)->176M(8%)
[2024-05-10T15:12:02.947+0800][7825][gc] GC(129) Garbage Collection (Proactive) 398M(18%)->178M(8%)
从1节点
[2024-05-10T14:15:28.184+0800][30659][gc] Using The Z Garbage Collector
[2024-05-10T14:15:31.445+0800][30663][gc] GC(0) Garbage Collection (Warmup) 226M(10%)->22M(1%)
[2024-05-10T14:15:38.660+0800][30663][gc] GC(1) Garbage Collection (Warmup) 456M(21%)->192M(9%)
[2024-05-10T14:17:36.897+0800][30663][gc] GC(2) Garbage Collection (Allocation Rate) 478M(22%)->136M(6%)
[2024-05-10T14:18:23.822+0800][30663][gc] GC(3) Garbage Collection (Warmup) 666M(30%)->180M(8%)
[2024-05-10T14:18:53.157+0800][30663][gc] GC(4) Garbage Collection (Proactive) 870M(39%)->208M(9%)
[2024-05-10T14:19:20.911+0800][30663][gc] GC(5) Garbage Collection (Proactive) 1214M(55%)->272M(12%)
[2024-05-10T14:19:50.290+0800][30663][gc] GC(6) Garbage Collection (Proactive) 1670M(75%)->294M(13%)
[2024-05-10T14:20:16.993+0800][30663][gc] GC(7) Garbage Collection (Allocation Rate) 1910M(86%)->380M(17%)
[2024-05-10T14:20:17.601+0800][30663][gc] GC(8) Garbage Collection (Allocation Rate) 420M(19%)->238M(11%)
[2024-05-10T14:20:41.133+0800][30663][gc] GC(9) Garbage Collection (Allocation Rate) 1852M(84%)->362M(16%)
[2024-05-10T14:20:41.804+0800][30663][gc] GC(10) Garbage Collection (Allocation Rate) 442M(20%)->252M(11%)
[2024-05-10T14:21:02.710+0800][30663][gc] GC(11) Garbage Collection (Allocation Rate) 1848M(84%)->398M(18%)
[2024-05-10T14:21:03.450+0800][30663][gc] GC(12) Garbage Collection (Allocation Rate) 466M(21%)->268M(12%)
[2024-05-10T14:21:21.776+0800][30663][gc] GC(13) Garbage Collection (Allocation Rate) 1788M(81%)->410M(19%)
[2024-05-10T14:21:22.468+0800][30663][gc] GC(14) Garbage Collection (Allocation Rate) 476M(22%)->284M(13%)
[2024-05-10T14:21:23.051+0800][30663][gc] GC(15) Garbage Collection (Allocation Rate) 292M(13%)->274M(12%)
[2024-05-10T14:21:39.845+0800][30663][gc] GC(16) Garbage Collection (Allocation Rate) 1782M(81%)->406M(18%)
[2024-05-10T14:21:40.492+0800][30663][gc] GC(17) Garbage Collection (Allocation Rate) 486M(22%)->288M(13%)
[2024-05-10T14:21:41.083+0800][30663][gc] GC(18) Garbage Collection (Allocation Rate) 292M(13%)->294M(13%)
[2024-05-10T14:21:56.307+0800][30663][gc] GC(19) Garbage Collection (Allocation Rate) 1760M(80%)->416M(19%)
[2024-05-10T14:21:57.035+0800][30663][gc] GC(20) Garbage Collection (Allocation Rate) 486M(22%)->310M(14%)
[2024-05-10T14:21:57.621+0800][30663][gc] GC(21) Garbage Collection (Allocation Rate) 312M(14%)->314M(14%)
[2024-05-10T14:22:11.093+0800][30663][gc] GC(22) Garbage Collection (Allocation Rate) 1704M(77%)->426M(19%)
[2024-05-10T14:22:11.846+0800][30663][gc] GC(23) Garbage Collection (Allocation Rate) 490M(22%)->318M(14%)
[2024-05-10T14:22:12.475+0800][30663][gc] GC(24) Garbage Collection (Allocation Rate) 318M(14%)->326M(15%)
[2024-05-10T14:22:24.934+0800][30663][gc] GC(25) Garbage Collection (Allocation Rate) 1676M(76%)->446M(20%)
[2024-05-10T14:22:25.688+0800][30663][gc] GC(26) Garbage Collection (Allocation Rate) 502M(23%)->326M(15%)
[2024-05-10T14:22:36.717+0800][30663][gc] GC(27) Garbage Collection (Allocation Rate) 1578M(71%)->428M(19%)
[2024-05-10T14:22:37.536+0800][30663][gc] GC(28) Garbage Collection (Allocation Rate) 494M(22%)->338M(15%)
[2024-05-10T14:22:47.938+0800][30663][gc] GC(29) Garbage Collection (Allocation Rate) 1570M(71%)->464M(21%)
[2024-05-10T14:22:48.789+0800][30663][gc] GC(30) Garbage Collection (Allocation Rate) 524M(24%)->354M(16%)
[2024-05-10T14:22:58.499+0800][30663][gc] GC(31) Garbage Collection (Allocation Rate) 1524M(69%)->442M(20%)
[2024-05-10T14:22:59.366+0800][30663][gc] GC(32) Garbage Collection (Allocation Rate) 514M(23%)->364M(16%)
[2024-05-10T14:23:08.417+0800][30663][gc] GC(33) Garbage Collection (Allocation Rate) 1472M(67%)->462M(21%)
[2024-05-10T14:23:09.294+0800][30663][gc] GC(34) Garbage Collection (Allocation Rate) 516M(23%)->370M(17%)
[2024-05-10T14:23:17.667+0800][30663][gc] GC(35) Garbage Collection (Allocation Rate) 1438M(65%)->458M(21%)
[2024-05-10T14:23:18.624+0800][30663][gc] GC(36) Garbage Collection (Allocation Rate) 522M(24%)->378M(17%)
[2024-05-10T14:23:27.041+0800][30663][gc] GC(37) Garbage Collection (Allocation Rate) 1470M(66%)->446M(20%)
[2024-05-10T14:23:27.912+0800][30663][gc] GC(38) Garbage Collection (Allocation Rate) 526M(24%)->386M(17%)
[2024-05-10T14:23:29.085+0800][30663][gc] GC(39) Garbage Collection (Allocation Rate) 454M(21%)->386M(17%)
[2024-05-10T14:23:37.085+0800][30663][gc] GC(40) Garbage Collection (Allocation Rate) 1438M(65%)->474M(21%)
[2024-05-10T14:23:37.986+0800][30663][gc] GC(41) Garbage Collection (Allocation Rate) 532M(24%)->396M(18%)
[2024-05-10T14:23:39.075+0800][30663][gc] GC(42) Garbage Collection (Allocation Rate) 438M(20%)->392M(18%)
[2024-05-10T14:23:46.449+0800][30663][gc] GC(43) Garbage Collection (Allocation Rate) 1380M(62%)->476M(22%)
[2024-05-10T14:23:47.390+0800][30663][gc] GC(44) Garbage Collection (Allocation Rate) 536M(24%)->406M(18%)
[2024-05-10T14:23:54.530+0800][30663][gc] GC(45) Garbage Collection (Allocation Rate) 1372M(62%)->486M(22%)
[2024-05-10T14:23:55.424+0800][30663][gc] GC(46) Garbage Collection (Allocation Rate) 516M(23%)->452M(20%)
[2024-05-10T14:23:56.402+0800][30663][gc] GC(47) Garbage Collection (Allocation Rate) 474M(21%)->406M(18%)
[2024-05-10T14:24:03.239+0800][30663][gc] GC(48) Garbage Collection (Allocation Rate) 1348M(61%)->498M(23%)
[2024-05-10T14:24:04.269+0800][30663][gc] GC(49) Garbage Collection (Allocation Rate) 548M(25%)->428M(19%)
[2024-05-10T14:24:05.239+0800][30663][gc] GC(50) Garbage Collection (Allocation Rate) 442M(20%)->456M(21%)
[2024-05-10T14:24:11.056+0800][30663][gc] GC(51) Garbage Collection (Allocation Rate) 1248M(56%)->536M(24%)
[2024-05-10T14:24:12.098+0800][30663][gc] GC(52) Garbage Collection (Allocation Rate) 574M(26%)->464M(21%)
[2024-05-10T14:24:13.108+0800][30663][gc] GC(53) Garbage Collection (Allocation Rate) 488M(22%)->462M(21%)
[2024-05-10T14:24:18.463+0800][30663][gc] GC(54) Garbage Collection (Allocation Rate) 1208M(55%)->530M(24%)
[2024-05-10T14:24:19.526+0800][30663][gc] GC(55) Garbage Collection (Allocation Rate) 584M(26%)->504M(23%)
[2024-05-10T14:24:20.497+0800][30663][gc] GC(56) Garbage Collection (Allocation Rate) 508M(23%)->476M(22%)
[2024-05-10T14:24:25.876+0800][30663][gc] GC(57) Garbage Collection (Allocation Rate) 1238M(56%)->522M(24%)
[2024-05-10T14:24:26.913+0800][30663][gc] GC(58) Garbage Collection (Allocation Rate) 576M(26%)->484M(22%)
[2024-05-10T14:24:27.990+0800][30663][gc] GC(59) Garbage Collection (Allocation Rate) 510M(23%)->480M(22%)
[2024-05-10T14:24:33.370+0800][30663][gc] GC(60) Garbage Collection (Allocation Rate) 1238M(56%)->530M(24%)
[2024-05-10T14:24:34.492+0800][30663][gc] GC(61) Garbage Collection (Allocation Rate) 594M(27%)->516M(23%)
[2024-05-10T14:24:35.510+0800][30663][gc] GC(62) Garbage Collection (Allocation Rate) 524M(24%)->500M(23%)
[2024-05-10T14:24:40.369+0800][30663][gc] GC(63) Garbage Collection (Allocation Rate) 1154M(52%)->522M(24%)
[2024-05-10T14:24:41.403+0800][30663][gc] GC(64) Garbage Collection (Allocation Rate) 578M(26%)->494M(22%)
[2024-05-10T14:24:42.448+0800][30663][gc] GC(65) Garbage Collection (Allocation Rate) 500M(23%)->478M(22%)
[2024-05-10T14:24:47.151+0800][30663][gc] GC(66) Garbage Collection (Allocation Rate) 1126M(51%)->512M(23%)
[2024-05-10T14:24:48.220+0800][30663][gc] GC(67) Garbage Collection (Allocation Rate) 594M(27%)->490M(22%)
[2024-05-10T14:24:49.274+0800][30663][gc] GC(68) Garbage Collection (Allocation Rate) 512M(23%)->490M(22%)
[2024-05-10T14:24:54.469+0800][30663][gc] GC(69) Garbage Collection (Allocation Rate) 1212M(55%)->520M(24%)
[2024-05-10T14:24:55.524+0800][30663][gc] GC(70) Garbage Collection (Allocation Rate) 580M(26%)->488M(22%)
[2024-05-10T14:24:56.485+0800][30663][gc] GC(71) Garbage Collection (Allocation Rate) 494M(22%)->480M(22%)
[2024-05-10T14:25:01.613+0800][30663][gc] GC(72) Garbage Collection (Allocation Rate) 1190M(54%)->560M(25%)
[2024-05-10T14:25:02.518+0800][30663][gc] GC(73) Garbage Collection (Allocation Rate) 586M(26%)->498M(23%)
[2024-05-10T14:25:03.439+0800][30663][gc] GC(74) Garbage Collection (Allocation Rate) 502M(23%)->480M(22%)
[2024-05-10T14:25:08.327+0800][30663][gc] GC(75) Garbage Collection (Allocation Rate) 1184M(54%)->530M(24%)
[2024-05-10T14:25:09.460+0800][30663][gc] GC(76) Garbage Collection (Allocation Rate) 582M(26%)->496M(22%)
[2024-05-10T14:25:10.582+0800][30663][gc] GC(77) Garbage Collection (Allocation Rate) 512M(23%)->486M(22%)
[2024-05-10T14:25:15.241+0800][30663][gc] GC(78) Garbage Collection (Allocation Rate) 1100M(50%)->562M(25%)
[2024-05-10T14:25:16.231+0800][30663][gc] GC(79) Garbage Collection (Allocation Rate) 582M(26%)->508M(23%)
[2024-05-10T14:25:17.198+0800][30663][gc] GC(80) Garbage Collection (Allocation Rate) 512M(23%)->478M(22%)
[2024-05-10T14:25:22.109+0800][30663][gc] GC(81) Garbage Collection (Allocation Rate) 1152M(52%)->576M(26%)
[2024-05-10T14:25:23.058+0800][30663][gc] GC(82) Garbage Collection (Allocation Rate) 582M(26%)->484M(22%)
[2024-05-10T14:25:24.092+0800][30663][gc] GC(83) Garbage Collection (Allocation Rate) 508M(23%)->482M(22%)
[2024-05-10T14:25:28.947+0800][30663][gc] GC(84) Garbage Collection (Allocation Rate) 1158M(52%)->516M(23%)
[2024-05-10T14:25:30.034+0800][30663][gc] GC(85) Garbage Collection (Allocation Rate) 580M(26%)->492M(22%)
[2024-05-10T14:25:31.081+0800][30663][gc] GC(86) Garbage Collection (Allocation Rate) 514M(23%)->480M(22%)
[2024-05-10T14:25:36.108+0800][30663][gc] GC(87) Garbage Collection (Allocation Rate) 1172M(53%)->546M(25%)
[2024-05-10T14:25:37.123+0800][30663][gc] GC(88) Garbage Collection (Allocation Rate) 586M(26%)->484M(22%)
[2024-05-10T14:25:38.052+0800][30663][gc] GC(89) Garbage Collection (Allocation Rate) 488M(22%)->484M(22%)
[2024-05-10T14:25:42.873+0800][30663][gc] GC(90) Garbage Collection (Allocation Rate) 1168M(53%)->536M(24%)
[2024-05-10T14:25:43.901+0800][30663][gc] GC(91) Garbage Collection (Allocation Rate) 584M(26%)->486M(22%)
[2024-05-10T14:25:44.906+0800][30663][gc] GC(92) Garbage Collection (Allocation Rate) 496M(22%)->480M(22%)
[2024-05-10T14:25:50.188+0800][30663][gc] GC(93) Garbage Collection (Allocation Rate) 1226M(55%)->560M(25%)
[2024-05-10T14:25:51.120+0800][30663][gc] GC(94) Garbage Collection (Allocation Rate) 614M(28%)->480M(22%)
[2024-05-10T14:25:52.084+0800][30663][gc] GC(95) Garbage Collection (Allocation Rate) 486M(22%)->478M(22%)
[2024-05-10T14:25:57.084+0800][30663][gc] GC(96) Garbage Collection (Allocation Rate) 1168M(53%)->550M(25%)
[2024-05-10T14:25:58.067+0800][30663][gc] GC(97) Garbage Collection (Allocation Rate) 580M(26%)->484M(22%)
[2024-05-10T14:25:59.064+0800][30663][gc] GC(98) Garbage Collection (Allocation Rate) 502M(23%)->482M(22%)
[2024-05-10T14:26:04.321+0800][30663][gc] GC(99) Garbage Collection (Allocation Rate) 1222M(55%)->542M(25%)
[2024-05-10T14:26:05.326+0800][30663][gc] GC(100) Garbage Collection (Allocation Rate) 590M(27%)->494M(22%)
[2024-05-10T14:26:06.271+0800][30663][gc] GC(101) Garbage Collection (Allocation Rate) 500M(23%)->482M(22%)
[2024-05-10T14:26:11.257+0800][30663][gc] GC(102) Garbage Collection (Allocation Rate) 1178M(53%)->520M(24%)
[2024-05-10T14:26:12.287+0800][30663][gc] GC(103) Garbage Collection (Allocation Rate) 582M(26%)->490M(22%)
[2024-05-10T14:26:13.360+0800][30663][gc] GC(104) Garbage Collection (Allocation Rate) 516M(23%)->478M(22%)
[2024-05-10T14:26:18.472+0800][30663][gc] GC(105) Garbage Collection (Allocation Rate) 1200M(54%)->526M(24%)
[2024-05-10T14:26:19.460+0800][30663][gc] GC(106) Garbage Collection (Allocation Rate) 582M(26%)->480M(22%)
[2024-05-10T14:26:20.495+0800][30663][gc] GC(107) Garbage Collection (Allocation Rate) 496M(22%)->482M(22%)
[2024-05-10T14:26:25.550+0800][30663][gc] GC(108) Garbage Collection (Allocation Rate) 1190M(54%)->526M(24%)
[2024-05-10T14:26:27.296+0800][30663][gc] GC(109) Garbage Collection (Allocation Rate) 586M(26%)->484M(22%)
[2024-05-10T14:26:27.874+0800][30663][gc] GC(110) Garbage Collection (Allocation Rate) 486M(22%)->304M(14%)
[2024-05-10T14:30:45.977+0800][30663][gc] GC(111) Garbage Collection (Proactive) 526M(24%)->294M(13%)
[2024-05-10T14:33:06.056+0800][30663][gc] GC(112) Garbage Collection (Allocation Rate) 576M(26%)->360M(16%)
[2024-05-10T14:33:06.551+0800][30663][gc] GC(113) Garbage Collection (Allocation Rate) 360M(16%)->290M(13%)
[2024-05-10T14:33:08.189+0800][30663][gc] GC(114) Garbage Collection (Allocation Rate) 438M(20%)->312M(14%)
[2024-05-10T14:37:21.899+0800][30663][gc] GC(115) Garbage Collection (Proactive) 534M(24%)->250M(11%)
[2024-05-10T14:41:40.408+0800][30663][gc] GC(116) Garbage Collection (Proactive) 472M(21%)->208M(9%)
[2024-05-10T14:46:07.282+0800][30663][gc] GC(117) Garbage Collection (Proactive) 430M(19%)->172M(8%)
[2024-05-10T14:50:41.379+0800][30663][gc] GC(118) Garbage Collection (Proactive) 394M(18%)->174M(8%)
[2024-05-10T14:55:08.474+0800][30663][gc] GC(119) Garbage Collection (Proactive) 396M(18%)->174M(8%)
[2024-05-10T14:59:42.171+0800][30663][gc] GC(120) Garbage Collection (Proactive) 396M(18%)->174M(8%)
[2024-05-10T15:04:25.960+0800][30663][gc] GC(121) Garbage Collection (Proactive) 396M(18%)->172M(8%)
[2024-05-10T15:08:56.575+0800][30663][gc] GC(122) Garbage Collection (Proactive) 394M(18%)->172M(8%)
从2节点
[2024-05-10T14:15:28.018+0800][13495][gc] Using The Z Garbage Collector
[2024-05-10T14:15:30.445+0800][13499][gc] GC(0) Garbage Collection (Warmup) 224M(10%)->22M(1%)
[2024-05-10T14:15:38.927+0800][13499][gc] GC(1) Garbage Collection (Warmup) 502M(23%)->240M(11%)
[2024-05-10T14:15:39.456+0800][13499][gc] GC(2) Garbage Collection (Allocation Rate) 240M(11%)->218M(10%)
[2024-05-10T14:17:37.096+0800][13499][gc] GC(3) Garbage Collection (Allocation Rate) 544M(25%)->238M(11%)
[2024-05-10T14:17:37.723+0800][13499][gc] GC(4) Garbage Collection (Allocation Rate) 238M(11%)->218M(10%)
[2024-05-10T14:17:38.920+0800][13499][gc] GC(5) Garbage Collection (Allocation Rate) 448M(20%)->214M(10%)
[2024-05-10T14:18:28.139+0800][13499][gc] GC(6) Garbage Collection (Warmup) 666M(30%)->186M(8%)
[2024-05-10T14:19:06.644+0800][13499][gc] GC(7) Garbage Collection (Proactive) 1198M(54%)->238M(11%)
[2024-05-10T14:19:42.392+0800][13499][gc] GC(8) Garbage Collection (Proactive) 1722M(78%)->376M(17%)
[2024-05-10T14:19:42.977+0800][13499][gc] GC(9) Garbage Collection (Allocation Rate) 390M(18%)->216M(10%)
[2024-05-10T14:20:13.839+0800][13499][gc] GC(10) Garbage Collection (Allocation Rate) 1868M(84%)->366M(17%)
[2024-05-10T14:20:14.475+0800][13499][gc] GC(11) Garbage Collection (Allocation Rate) 436M(20%)->234M(11%)
[2024-05-10T14:20:14.995+0800][13499][gc] GC(12) Garbage Collection (Allocation Rate) 238M(11%)->238M(11%)
[2024-05-10T14:20:39.804+0800][13499][gc] GC(13) Garbage Collection (Allocation Rate) 1864M(84%)->374M(17%)
[2024-05-10T14:20:40.448+0800][13499][gc] GC(14) Garbage Collection (Allocation Rate) 452M(20%)->250M(11%)
[2024-05-10T14:20:41.012+0800][13499][gc] GC(15) Garbage Collection (Allocation Rate) 256M(12%)->240M(11%)
[2024-05-10T14:21:03.041+0800][13499][gc] GC(16) Garbage Collection (Allocation Rate) 1870M(85%)->394M(18%)
[2024-05-10T14:21:03.692+0800][13499][gc] GC(17) Garbage Collection (Allocation Rate) 470M(21%)->264M(12%)
[2024-05-10T14:21:04.249+0800][13499][gc] GC(18) Garbage Collection (Allocation Rate) 268M(12%)->262M(12%)
[2024-05-10T14:21:23.453+0800][13499][gc] GC(19) Garbage Collection (Allocation Rate) 1822M(82%)->392M(18%)
[2024-05-10T14:21:24.152+0800][13499][gc] GC(20) Garbage Collection (Allocation Rate) 478M(22%)->280M(13%)
[2024-05-10T14:21:24.830+0800][13499][gc] GC(21) Garbage Collection (Allocation Rate) 286M(13%)->280M(13%)
[2024-05-10T14:21:41.902+0800][13499][gc] GC(22) Garbage Collection (Allocation Rate) 1784M(81%)->422M(19%)
[2024-05-10T14:21:42.632+0800][13499][gc] GC(23) Garbage Collection (Allocation Rate) 494M(22%)->300M(14%)
[2024-05-10T14:21:43.264+0800][13499][gc] GC(24) Garbage Collection (Allocation Rate) 302M(14%)->300M(14%)
[2024-05-10T14:21:58.416+0800][13499][gc] GC(25) Garbage Collection (Allocation Rate) 1726M(78%)->418M(19%)
[2024-05-10T14:21:59.184+0800][13499][gc] GC(26) Garbage Collection (Allocation Rate) 500M(23%)->320M(14%)
[2024-05-10T14:21:59.849+0800][13499][gc] GC(27) Garbage Collection (Allocation Rate) 324M(15%)->306M(14%)
[2024-05-10T14:22:13.593+0800][13499][gc] GC(28) Garbage Collection (Allocation Rate) 1678M(76%)->440M(20%)
[2024-05-10T14:22:14.309+0800][13499][gc] GC(29) Garbage Collection (Allocation Rate) 500M(23%)->328M(15%)
[2024-05-10T14:22:14.994+0800][13499][gc] GC(30) Garbage Collection (Allocation Rate) 330M(15%)->314M(14%)
[2024-05-10T14:22:27.497+0800][13499][gc] GC(31) Garbage Collection (Allocation Rate) 1628M(74%)->440M(20%)
[2024-05-10T14:22:28.342+0800][13499][gc] GC(32) Garbage Collection (Allocation Rate) 508M(23%)->332M(15%)
[2024-05-10T14:22:39.222+0800][13499][gc] GC(33) Garbage Collection (Allocation Rate) 1528M(69%)->434M(20%)
[2024-05-10T14:22:39.995+0800][13499][gc] GC(34) Garbage Collection (Allocation Rate) 508M(23%)->340M(15%)
[2024-05-10T14:22:40.737+0800][13499][gc] GC(35) Garbage Collection (Allocation Rate) 346M(16%)->342M(15%)
[2024-05-10T14:22:51.221+0800][13499][gc] GC(36) Garbage Collection (Allocation Rate) 1542M(70%)->450M(20%)
[2024-05-10T14:22:52.104+0800][13499][gc] GC(37) Garbage Collection (Allocation Rate) 510M(23%)->356M(16%)
[2024-05-10T14:23:01.574+0800][13499][gc] GC(38) Garbage Collection (Allocation Rate) 1466M(66%)->452M(20%)
[2024-05-10T14:23:02.429+0800][13499][gc] GC(39) Garbage Collection (Allocation Rate) 508M(23%)->374M(17%)
[2024-05-10T14:23:03.213+0800][13499][gc] GC(40) Garbage Collection (Allocation Rate) 374M(17%)->374M(17%)
[2024-05-10T14:23:12.347+0800][13499][gc] GC(41) Garbage Collection (Allocation Rate) 1470M(66%)->464M(21%)
[2024-05-10T14:23:13.268+0800][13499][gc] GC(42) Garbage Collection (Allocation Rate) 524M(24%)->380M(17%)
[2024-05-10T14:23:21.791+0800][13499][gc] GC(43) Garbage Collection (Allocation Rate) 1412M(64%)->514M(23%)
[2024-05-10T14:23:22.622+0800][13499][gc] GC(44) Garbage Collection (Allocation Rate) 528M(24%)->382M(17%)
[2024-05-10T14:23:23.460+0800][13499][gc] GC(45) Garbage Collection (Allocation Rate) 384M(17%)->384M(17%)
[2024-05-10T14:23:31.243+0800][13499][gc] GC(46) Garbage Collection (Allocation Rate) 1346M(61%)->478M(22%)
[2024-05-10T14:23:32.200+0800][13499][gc] GC(47) Garbage Collection (Allocation Rate) 530M(24%)->396M(18%)
[2024-05-10T14:23:33.183+0800][13499][gc] GC(48) Garbage Collection (Allocation Rate) 416M(19%)->396M(18%)
[2024-05-10T14:23:40.643+0800][13499][gc] GC(49) Garbage Collection (Allocation Rate) 1338M(60%)->484M(22%)
[2024-05-10T14:23:41.634+0800][13499][gc] GC(50) Garbage Collection (Allocation Rate) 534M(24%)->428M(19%)
[2024-05-10T14:23:42.564+0800][13499][gc] GC(51) Garbage Collection (Allocation Rate) 442M(20%)->426M(19%)
[2024-05-10T14:23:49.633+0800][13499][gc] GC(52) Garbage Collection (Allocation Rate) 1320M(60%)->514M(23%)
[2024-05-10T14:23:50.763+0800][13499][gc] GC(53) Garbage Collection (Allocation Rate) 566M(26%)->438M(20%)
[2024-05-10T14:23:51.729+0800][13499][gc] GC(54) Garbage Collection (Allocation Rate) 446M(20%)->406M(18%)
[2024-05-10T14:23:57.635+0800][13499][gc] GC(55) Garbage Collection (Allocation Rate) 1162M(53%)->456M(21%)
[2024-05-10T14:23:58.780+0800][13499][gc] GC(56) Garbage Collection (Allocation Rate) 542M(25%)->440M(20%)
[2024-05-10T14:24:04.526+0800][13499][gc] GC(57) Garbage Collection (Allocation Rate) 1196M(54%)->530M(24%)
[2024-05-10T14:24:05.608+0800][13499][gc] GC(58) Garbage Collection (Allocation Rate) 578M(26%)->508M(23%)
[2024-05-10T14:24:06.549+0800][13499][gc] GC(59) Garbage Collection (Allocation Rate) 512M(23%)->470M(21%)
[2024-05-10T14:24:12.348+0800][13499][gc] GC(60) Garbage Collection (Allocation Rate) 1218M(55%)->528M(24%)
[2024-05-10T14:24:13.433+0800][13499][gc] GC(61) Garbage Collection (Allocation Rate) 586M(26%)->486M(22%)
[2024-05-10T14:24:14.598+0800][13499][gc] GC(62) Garbage Collection (Allocation Rate) 486M(22%)->558M(25%)
[2024-05-10T14:24:15.715+0800][13499][gc] GC(63) Garbage Collection (Allocation Rate) 568M(26%)->530M(24%)
[2024-05-10T14:24:16.675+0800][13499][gc] GC(64) Garbage Collection (Allocation Rate) 532M(24%)->472M(21%)
[2024-05-10T14:24:21.313+0800][13499][gc] GC(65) Garbage Collection (Allocation Rate) 1054M(48%)->534M(24%)
[2024-05-10T14:24:22.440+0800][13499][gc] GC(66) Garbage Collection (Allocation Rate) 572M(26%)->496M(22%)
[2024-05-10T14:24:23.550+0800][13499][gc] GC(67) Garbage Collection (Allocation Rate) 512M(23%)->488M(22%)
[2024-05-10T14:24:29.004+0800][13499][gc] GC(68) Garbage Collection (Allocation Rate) 1192M(54%)->566M(26%)
[2024-05-10T14:24:30.206+0800][13499][gc] GC(69) Garbage Collection (Allocation Rate) 602M(27%)->524M(24%)
[2024-05-10T14:24:31.283+0800][13499][gc] GC(70) Garbage Collection (Allocation Rate) 528M(24%)->530M(24%)
[2024-05-10T14:24:32.289+0800][13499][gc] GC(71) Garbage Collection (Allocation Rate) 540M(24%)->490M(22%)
[2024-05-10T14:24:36.810+0800][13499][gc] GC(72) Garbage Collection (Allocation Rate) 1060M(48%)->546M(25%)
[2024-05-10T14:24:37.917+0800][13499][gc] GC(73) Garbage Collection (Allocation Rate) 578M(26%)->504M(23%)
[2024-05-10T14:24:38.965+0800][13499][gc] GC(74) Garbage Collection (Allocation Rate) 506M(23%)->500M(23%)
[2024-05-10T14:24:43.541+0800][13499][gc] GC(75) Garbage Collection (Allocation Rate) 1088M(49%)->530M(24%)
[2024-05-10T14:24:44.706+0800][13499][gc] GC(76) Garbage Collection (Allocation Rate) 588M(27%)->510M(23%)
[2024-05-10T14:24:45.759+0800][13499][gc] GC(77) Garbage Collection (Allocation Rate) 516M(23%)->496M(22%)
[2024-05-10T14:24:50.558+0800][13499][gc] GC(78) Garbage Collection (Allocation Rate) 1122M(51%)->538M(24%)
[2024-05-10T14:24:51.697+0800][13499][gc] GC(79) Garbage Collection (Allocation Rate) 582M(26%)->498M(23%)
[2024-05-10T14:24:52.698+0800][13499][gc] GC(80) Garbage Collection (Allocation Rate) 504M(23%)->516M(23%)
[2024-05-10T14:24:57.421+0800][13499][gc] GC(81) Garbage Collection (Allocation Rate) 1124M(51%)->550M(25%)
[2024-05-10T14:24:58.603+0800][13499][gc] GC(82) Garbage Collection (Allocation Rate) 602M(27%)->510M(23%)
[2024-05-10T14:24:59.721+0800][13499][gc] GC(83) Garbage Collection (Allocation Rate) 516M(23%)->490M(22%)
[2024-05-10T14:25:04.660+0800][13499][gc] GC(84) Garbage Collection (Allocation Rate) 1112M(50%)->570M(26%)
[2024-05-10T14:25:05.777+0800][13499][gc] GC(85) Garbage Collection (Allocation Rate) 618M(28%)->516M(23%)
[2024-05-10T14:25:06.870+0800][13499][gc] GC(86) Garbage Collection (Allocation Rate) 524M(24%)->492M(22%)
[2024-05-10T14:25:11.385+0800][13499][gc] GC(87) Garbage Collection (Allocation Rate) 1086M(49%)->540M(24%)
[2024-05-10T14:25:12.427+0800][13499][gc] GC(88) Garbage Collection (Allocation Rate) 590M(27%)->496M(22%)
[2024-05-10T14:25:13.475+0800][13499][gc] GC(89) Garbage Collection (Allocation Rate) 496M(22%)->504M(23%)
[2024-05-10T14:25:18.454+0800][13499][gc] GC(90) Garbage Collection (Allocation Rate) 1152M(52%)->558M(25%)
[2024-05-10T14:25:19.566+0800][13499][gc] GC(91) Garbage Collection (Allocation Rate) 612M(28%)->498M(23%)
[2024-05-10T14:25:20.643+0800][13499][gc] GC(92) Garbage Collection (Allocation Rate) 508M(23%)->480M(22%)
[2024-05-10T14:25:25.683+0800][13499][gc] GC(93) Garbage Collection (Allocation Rate) 1158M(52%)->522M(24%)
[2024-05-10T14:25:26.844+0800][13499][gc] GC(94) Garbage Collection (Allocation Rate) 604M(27%)->504M(23%)
[2024-05-10T14:25:27.947+0800][13499][gc] GC(95) Garbage Collection (Allocation Rate) 516M(23%)->490M(22%)
[2024-05-10T14:25:33.230+0800][13499][gc] GC(96) Garbage Collection (Allocation Rate) 1200M(54%)->554M(25%)
[2024-05-10T14:25:34.372+0800][13499][gc] GC(97) Garbage Collection (Allocation Rate) 612M(28%)->510M(23%)
[2024-05-10T14:25:35.399+0800][13499][gc] GC(98) Garbage Collection (Allocation Rate) 518M(23%)->498M(23%)
[2024-05-10T14:25:40.268+0800][13499][gc] GC(99) Garbage Collection (Allocation Rate) 1130M(51%)->546M(25%)
[2024-05-10T14:25:41.413+0800][13499][gc] GC(100) Garbage Collection (Allocation Rate) 600M(27%)->510M(23%)
[2024-05-10T14:25:42.478+0800][13499][gc] GC(101) Garbage Collection (Allocation Rate) 512M(23%)->500M(23%)
[2024-05-10T14:25:47.230+0800][13499][gc] GC(102) Garbage Collection (Allocation Rate) 1122M(51%)->540M(24%)
[2024-05-10T14:25:48.370+0800][13499][gc] GC(103) Garbage Collection (Allocation Rate) 596M(27%)->508M(23%)
[2024-05-10T14:25:49.473+0800][13499][gc] GC(104) Garbage Collection (Allocation Rate) 518M(23%)->496M(22%)
[2024-05-10T14:25:54.485+0800][13499][gc] GC(105) Garbage Collection (Allocation Rate) 1154M(52%)->548M(25%)
[2024-05-10T14:25:55.581+0800][13499][gc] GC(106) Garbage Collection (Allocation Rate) 596M(27%)->508M(23%)
[2024-05-10T14:25:56.620+0800][13499][gc] GC(107) Garbage Collection (Allocation Rate) 518M(23%)->516M(23%)
[2024-05-10T14:26:01.266+0800][13499][gc] GC(108) Garbage Collection (Allocation Rate) 1104M(50%)->540M(24%)
[2024-05-10T14:26:02.311+0800][13499][gc] GC(109) Garbage Collection (Allocation Rate) 592M(27%)->494M(22%)
[2024-05-10T14:26:03.298+0800][13499][gc] GC(110) Garbage Collection (Allocation Rate) 496M(22%)->490M(22%)
[2024-05-10T14:26:08.271+0800][13499][gc] GC(111) Garbage Collection (Allocation Rate) 1136M(51%)->562M(25%)
[2024-05-10T14:26:09.370+0800][13499][gc] GC(112) Garbage Collection (Allocation Rate) 596M(27%)->506M(23%)
[2024-05-10T14:26:10.487+0800][13499][gc] GC(113) Garbage Collection (Allocation Rate) 520M(24%)->504M(23%)
[2024-05-10T14:26:15.155+0800][13499][gc] GC(114) Garbage Collection (Allocation Rate) 1106M(50%)->528M(24%)
[2024-05-10T14:26:16.289+0800][13499][gc] GC(115) Garbage Collection (Allocation Rate) 588M(27%)->510M(23%)
[2024-05-10T14:26:17.341+0800][13499][gc] GC(116) Garbage Collection (Allocation Rate) 518M(23%)->496M(22%)
[2024-05-10T14:26:22.183+0800][13499][gc] GC(117) Garbage Collection (Allocation Rate) 1130M(51%)->540M(24%)
[2024-05-10T14:26:23.285+0800][13499][gc] GC(118) Garbage Collection (Allocation Rate) 602M(27%)->506M(23%)
[2024-05-10T14:26:24.354+0800][13499][gc] GC(119) Garbage Collection (Allocation Rate) 514M(23%)->510M(23%)
[2024-05-10T14:27:17.912+0800][13499][gc] GC(120) Garbage Collection (Proactive) 938M(42%)->344M(16%)
[2024-05-10T14:31:23.330+0800][13499][gc] GC(121) Garbage Collection (Proactive) 566M(26%)->290M(13%)
[2024-05-10T14:35:43.249+0800][13499][gc] GC(122) Garbage Collection (Proactive) 512M(23%)->266M(12%)
[2024-05-10T14:40:01.194+0800][13499][gc] GC(123) Garbage Collection (Proactive) 488M(22%)->220M(10%)
[2024-05-10T14:44:32.694+0800][13499][gc] GC(124) Garbage Collection (Proactive) 442M(20%)->206M(9%)
[2024-05-10T14:48:46.262+0800][13499][gc] GC(125) Garbage Collection (Proactive) 428M(19%)->172M(8%)
[2024-05-10T14:53:23.993+0800][13499][gc] GC(126) Garbage Collection (Proactive) 394M(18%)->174M(8%)
[2024-05-10T14:57:44.363+0800][13499][gc] GC(127) Garbage Collection (Proactive) 396M(18%)->174M(8%)
[2024-05-10T15:02:19.951+0800][13499][gc] GC(128) Garbage Collection (Proactive) 396M(18%)->174M(8%)
[2024-05-10T15:06:50.856+0800][13499][gc] GC(129) Garbage Collection (Proactive) 396M(18%)->172M(8%)
[2024-05-10T15:11:13.852+0800][13499][gc] GC(130) Garbage Collection (Proactive) 394M(18%)->174M(8%)
from bifromq.
popduke
commented on August 11, 2024
检查下你rocksdb和memory运行的gc日志里gc运行的频率差异。
3个
2C 4GNode Cluster下用emqtt_bench测rocksdb + 16KB payload并没有出现你说的问题，Direct Buffer稳定在800MB左右 cmd:./emqtt_bench pub -V 4 -h <NODE1_ADDR,NODE2_ADDR,NODE3_ADDR> -c 40000 -I 2000 -i 1 -t bench/%i -s 16384 -u DevOnly/16KPayloadJVM Args:2024-05-09 17:05:52.950 INFO [main] --- c.b.bifromq.starter.StandaloneStarter [StandaloneStarter.java:611] JVM arguments: -Xms2G -Xmx2G -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=500m -XX:MaxDirectMemorySize=2G -XX:ZCollectionInterval=5 -XX:MaxInlineLevel=15 -Djava.awt.headless=true -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+UseZGC -XX:ZAllocationSpikeTolerance=5
同样的2C4G环境+默认配置，运行你的测试，没问题，一样可以通过。
[2024-05-10T15:54:23.773+0800][23183][gc] GC(279) Garbage Collection (Timer) 946M(46%)->460M(22%)
[2024-05-10T15:54:24.568+0800][23183][gc] GC(280) Garbage Collection (Allocation Rate) 528M(26%)->416M(20%)
[2024-05-10T15:54:30.480+0800][23183][gc] GC(281) Garbage Collection (Timer) 942M(46%)->522M(25%)
[2024-05-10T15:54:31.275+0800][23183][gc] GC(282) Garbage Collection (Allocation Rate) 554M(27%)->418M(20%)
[2024-05-10T15:54:37.141+0800][23183][gc] GC(283) Garbage Collection (Timer) 940M(46%)->482M(24%)
[2024-05-10T15:54:37.974+0800][23183][gc] GC(284) Garbage Collection (Allocation Rate) 550M(27%)->420M(21%)
[2024-05-10T15:54:43.878+0800][23183][gc] GC(285) Garbage Collection (Timer) 948M(46%)->476M(23%)
[2024-05-10T15:54:44.691+0800][23183][gc] GC(286) Garbage Collection (Allocation Rate) 532M(26%)->418M(20%)
[2024-05-10T15:54:50.570+0800][23183][gc] GC(287) Garbage Collection (Timer) 940M(46%)->476M(23%)
[2024-05-10T15:54:51.377+0800][23183][gc] GC(288) Garbage Collection (Allocation Rate) 534M(26%)->422M(21%)
[2024-05-10T15:54:57.302+0800][23183][gc] GC(289) Garbage Collection (Timer) 938M(46%)->516M(25%)
[2024-05-10T15:54:58.078+0800][23183][gc] GC(290) Garbage Collection (Allocation Rate) 556M(27%)->442M(22%)
[2024-05-10T15:54:58.868+0800][23183][gc] GC(291) Garbage Collection (Allocation Rate) 450M(22%)->414M(20%)
[2024-05-10T15:55:04.818+0800][23183][gc] GC(292) Garbage Collection (Timer) 942M(46%)->516M(25%)
[2024-05-10T15:55:05.621+0800][23183][gc] GC(293) Garbage Collection (Allocation Rate) 554M(27%)->480M(23%)
[2024-05-10T15:55:06.423+0800][23183][gc] GC(294) Garbage Collection (Allocation Rate) 482M(24%)->442M(22%)
[2024-05-10T15:55:12.257+0800][23183][gc] GC(295) Garbage Collection (Timer) 1000M(49%)->476M(23%)
[2024-05-10T15:55:13.055+0800][23183][gc] GC(296) Garbage Collection (Allocation Rate) 550M(27%)->428M(21%)
[2024-05-10T15:55:13.737+0800][23183][gc] GC(297) Garbage Collection (Allocation Rate) 428M(21%)->404M(20%)
[2024-05-10T15:55:19.542+0800][23183][gc] GC(298) Garbage Collection (Timer) 924M(45%)->456M(22%)
[2024-05-10T15:55:20.467+0800][23183][gc] GC(299) Garbage Collection (Allocation Rate) 572M(28%)->434M(21%)
[2024-05-10T15:55:26.353+0800][23183][gc] GC(300) Garbage Collection (Timer) 954M(47%)->462M(23%)
[2024-05-10T15:55:27.279+0800][23183][gc] GC(301) Garbage Collection (Allocation Rate) 542M(26%)->416M(20%)
[2024-05-10T15:55:33.176+0800][23183][gc] GC(302) Garbage Collection (Timer) 926M(45%)->510M(25%)
from bifromq.
masterOcean
commented on August 11, 2024
检查下你rocksdb和memory运行的gc日志里gc运行的频率差异。
3个2C 4GNode Cluster下用emqtt_bench测rocksdb + 16KB payload并没有出现你说的问题，Direct Buffer稳定在800MB左右 cmd:./emqtt_bench pub -V 4 -h <NODE1_ADDR,NODE2_ADDR,NODE3_ADDR> -c 40000 -I 2000 -i 1 -t bench/%i -s 16384 -u DevOnly/16KPayloadJVM Args:2024-05-09 17:05:52.950 INFO [main] --- c.b.bifromq.starter.StandaloneStarter [StandaloneStarter.java:611] JVM arguments: -Xms2G -Xmx2G -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=500m -XX:MaxDirectMemorySize=2G -XX:ZCollectionInterval=5 -XX:MaxInlineLevel=15 -Djava.awt.headless=true -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+UseZGC -XX:ZAllocationSpikeTolerance=5同样的2C4G环境+默认配置，运行你的测试，没问题，一样可以通过。
[2024-05-10T15:54:23.773+0800][23183][gc] GC(279) Garbage Collection (Timer) 946M(46%)->460M(22%) [2024-05-10T15:54:24.568+0800][23183][gc] GC(280) Garbage Collection (Allocation Rate) 528M(26%)->416M(20%) [2024-05-10T15:54:30.480+0800][23183][gc] GC(281) Garbage Collection (Timer) 942M(46%)->522M(25%) [2024-05-10T15:54:31.275+0800][23183][gc] GC(282) Garbage Collection (Allocation Rate) 554M(27%)->418M(20%) [2024-05-10T15:54:37.141+0800][23183][gc] GC(283) Garbage Collection (Timer) 940M(46%)->482M(24%) [2024-05-10T15:54:37.974+0800][23183][gc] GC(284) Garbage Collection (Allocation Rate) 550M(27%)->420M(21%) [2024-05-10T15:54:43.878+0800][23183][gc] GC(285) Garbage Collection (Timer) 948M(46%)->476M(23%) [2024-05-10T15:54:44.691+0800][23183][gc] GC(286) Garbage Collection (Allocation Rate) 532M(26%)->418M(20%) [2024-05-10T15:54:50.570+0800][23183][gc] GC(287) Garbage Collection (Timer) 940M(46%)->476M(23%) [2024-05-10T15:54:51.377+0800][23183][gc] GC(288) Garbage Collection (Allocation Rate) 534M(26%)->422M(21%) [2024-05-10T15:54:57.302+0800][23183][gc] GC(289) Garbage Collection (Timer) 938M(46%)->516M(25%) [2024-05-10T15:54:58.078+0800][23183][gc] GC(290) Garbage Collection (Allocation Rate) 556M(27%)->442M(22%) [2024-05-10T15:54:58.868+0800][23183][gc] GC(291) Garbage Collection (Allocation Rate) 450M(22%)->414M(20%) [2024-05-10T15:55:04.818+0800][23183][gc] GC(292) Garbage Collection (Timer) 942M(46%)->516M(25%) [2024-05-10T15:55:05.621+0800][23183][gc] GC(293) Garbage Collection (Allocation Rate) 554M(27%)->480M(23%) [2024-05-10T15:55:06.423+0800][23183][gc] GC(294) Garbage Collection (Allocation Rate) 482M(24%)->442M(22%) [2024-05-10T15:55:12.257+0800][23183][gc] GC(295) Garbage Collection (Timer) 1000M(49%)->476M(23%) [2024-05-10T15:55:13.055+0800][23183][gc] GC(296) Garbage Collection (Allocation Rate) 550M(27%)->428M(21%) [2024-05-10T15:55:13.737+0800][23183][gc] GC(297) Garbage Collection (Allocation Rate) 428M(21%)->404M(20%) [2024-05-10T15:55:19.542+0800][23183][gc] GC(298) Garbage Collection (Timer) 924M(45%)->456M(22%) [2024-05-10T15:55:20.467+0800][23183][gc] GC(299) Garbage Collection (Allocation Rate) 572M(28%)->434M(21%) [2024-05-10T15:55:26.353+0800][23183][gc] GC(300) Garbage Collection (Timer) 954M(47%)->462M(23%) [2024-05-10T15:55:27.279+0800][23183][gc] GC(301) Garbage Collection (Allocation Rate) 542M(26%)->416M(20%) [2024-05-10T15:55:33.176+0800][23183][gc] GC(302) Garbage Collection (Timer) 926M(45%)->510M(25%)
调整了 参数 MaxDirectMemory ，由 737m 改成 2G 后就正常了，看来还是内存不足导致的。
from bifromq.
popduke
commented on August 11, 2024
v3.1.0包含一些内存改进，可以试用一下，有问题再开Issue。
from bifromq.
Related Issues (20)
- bifromq 管理控制台有没有规划,可以实现监控,配置等功能 HOT 1
- 单Pub单Sub 单Topic情况下的无法保障topic顺序消费 HOT 1
- LWT doesn't work when the client process is killed, in v2.1 release
- authPlugin无法进入auth(MQTT5AuthData authData) 方法
- clean session HOT 5
- Setting Provider plugin value invalid HOT 1
- 长时间运行后api-server出现java.lang.StackOverflowError in
- docker bifromq:3.0.0版本客户端无法正常接收推送消息 HOT 2
- 有否正式商用的同行嘛？ HOT 1
- bifromq 性能测试系统可能阻塞在DistWorkerCoProcFactory的 ForkJoinPool HOT 1
- docker bifromq:3.0.2版本集群搭建，集群模式不可用！ HOT 11
- 建议在POM文件中升级lombok版本以便支持Java21的编译 HOT 1
- 集群cpu很高 HOT 12
- Bifromq Bridge HOT 1
- mqtt 集群压测导致内存溢出然后宕机 HOT 1
- 客户端订阅了2个共享组，取消订阅其中一个，竟然收不到另一个topic的消息了
- 集群运行一段时间后，warn.log 就会打印 There is no storeDescriptor for local store HOT 1
- 长时间压测后，线程wal-raft-executor-112680774442680320_0 和 basekv-range-mutator CPU高，一直降不下来 HOT 3
- 集群启动时，域名解析类型转换错误
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bifromq.