Very High Idle CPU Usage when API key is invalid about syncthing-macos

[jleyser]

Just discovered: this only happens when launching Syncthing via the OS X Application.

If I run syncthing from the command line, like:
/Applications/Syncthing.app/Contents/Resources/syncthing/syncthing -no-browser

I do not get the high CPU usage

[virusman]

Which process is consuming CPU? If it’s syncthing (lowecase), please file a ticket for Syncthing. Syncthing-macosx (this project) is only a thin UI wrapper around it.

[virusman]

Interesting. Event streaming may be at fault.

[jleyser]

Ya, it's this one (from 'ps auxww'):
jeff 23190 83.0 0.2 2558600 38868 ?? R 7:51PM 0:23.74 /Applications/Syncthing.app/Contents/MacOS/Syncthing

[xor-gate]

You should analyze like this, as you can see the syncthing subprocess is listed as a separate entry (on my system with v0.14.47):

 > ps auxww | grep -i syncthing
USER               PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
jerry              712   0.0  0.3 558532780  52084   ??  SN   Wed01AM  32:30.10 /Applications/Syncthing.app/Contents/Resources/syncthing/syncthing -no-browser
jerry              707   0.0  0.0 558490620    592   ??  S    Wed01AM   0:00.57 /Applications/Syncthing.app/Contents/Resources/syncthing/syncthing -no-browser
jerry              694   0.0  0.0  4414984   7264   ??  S    Wed01AM   0:31.36 /Applications/Syncthing.app/Contents/MacOS/Syncthing

[jleyser]

Yes, that's what I did. I only showed the wrapper before, but here is all of Syncthing:

jeff             98034  93.4  0.2  2545248  19968   ??  R     9:03AM   0:12.62 /Applications/Syncthing.app/Contents/MacOS/Syncthing
jeff             98036  11.8  0.5 556661992  44188   ??  SN    9:03AM   0:04.48 /Applications/Syncthing.app/Contents/Resources/syncthing/syncthing -no-browser
jeff             98035   0.0  0.1 556665604   8368   ??  S     9:03AM   0:00.06 /Applications/Syncthing.app/Contents/Resources/syncthing/syncthing -no-browser

And here is the highest few items in top -u

Processes: 324 total, 3 running, 11 stuck, 310 sleeping, 1801 threads                                          09:04:51
Load Avg: 3.15, 2.59, 2.40  CPU usage: 32.61% user, 28.53% sys, 38.84% idle
SharedLibs: 140M resident, 13M data, 5672K linkedit. MemRegions: 99111 total, 2824M resident, 73M private, 656M shared.
PhysMem: 7888M used (1999M wired), 303M unused.
VM: 1952G vsize, 533M framework vsize, 7550551(0) swapins, 8288548(0) swapouts.
Networks: packets: 452040294/133G in, 453022312/156G out. Disks: 19999651/981G read, 11044267/370G written.

PID    COMMAND      %CPU TIME     #TH   #WQ  #PORTS  MEM    PURG   CMPRS  PGRP  PPID  STATE    BOOSTS            %CPU_ME
98034  Syncthing    91.8 01:26.33 12/3  5    197     8112K  128K   0B     98034 1     running  *0[18]            9.68097
73503  nsurlstorage 14.2 62:45.20 5     3/1  156     3040K  0B     4444K  73503 1     sleeping *0[14505738+]     0.00000
97428  iTerm2       13.4 00:56.92 12    6    249-    53M+   256K   9064K  97428 1     sleeping *0-[215]          0.00000

As you can see, the wrapper has very high CPU usage. As more evidence that this is an issue with the wrapper, notice CPU Utilization in the Syncthing Web UI:

Again, happy to provide any logs you think might be helpful.

[jleyser]

Looking with dtruss, it looks like the wrapper is caught in a loop polling syncthing:

write(0xF, "GET /rest/events?limit=1 HTTP/1.1\r\nHost: localhost:8384\r\nX-API-Key: \r\nAccept: */*\r\nUser-Agent: Syncthing/144601 CFNetwork/760.9 Darwin/15.6.0 (x86_64)\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nConnection: keep-alive\r\n\r\n\0", 0xEA)		 = 234 0
select(0x10, 0x0, 0x7000004AA0B0, 0x0, 0x7000004AA0A0)		 = 1 0
select(0x20, 0x7F9C0B54FB10, 0x7F9C0B558560, 0x0, 0x0)		 = 1 0
read(0xF, "HTTP/1.1 403 Forbidden\r\nContent-Type: text/plain; charset=utf-8\r\nX-Content-Type-Options: nosniff\r\nX-Frame-Options: SAMEORIGIN\r\nX-Syncthing-Id: Y23W4SD-DBLXB2X-C26Z7TF-ZSGDDCR-SWXS2NV-SERKJEQ-XBSBAIY-RPISMA7\r\nX-Syncthing-Version: v0.14.47\r\nX-Xss-Protection:", 0x1000)

That appears over and over, at a very high rate. Is the 403 not expected?

[vincentardern]

I am also having this issue (I believe the wrapper is at fault), and am happy to help although I don't have much experience with debugging things from the terminal unfortunately.

[xor-gate]

Hi guys, I think the HTTP 403 error is indeed not as expected and causes a storm of requests which results in high CPU usage. Maybe @virusman could also agree on this. The line https://github.com/xor-gate/syncthing-macosx/blob/d23dbf5a7c463dc428afef0e40fccf0e0a900297/syncthing/STStatusMonitor.m#L39 is fired and returns a 403, the longPoll function runs in the background and returns. Then it is probably restarted without delay https://github.com/xor-gate/syncthing-macosx/blob/d23dbf5a7c463dc428afef0e40fccf0e0a900297/syncthing/STStatusMonitor.m#L69-L70.

@jleyser credits for your debugging with dtruss, this is very usefull. You are probably right about the unexpected 403.

[virusman]

Yeah, it shouldn’t return 403, seems like the API key doesn’t match, or Syncthing changed something in the API. Either way, there should be a delay on reconnect in the case of an error.

[ctueck]

I had the same issue here. It was caused by the API key, after copying it over from the syncthing native web UI into the preferences dialogue of the tray app and restarting (!), CPU usage is down.

Might have missed some changes between releases, but previously I never entered or copied the API key manually (but never had CPU issues either). API key field in the tray app was basically empty, and I seem to remember it had a greyed out default value in the past (supposed this was the API key gathered from the syncthing config?).

[jleyser]

Yup, that did the trick for me, too. Thanks @ctueck!

[xor-gate]

Between two versions there is indeed a little change that the API key is auto loaded from the xml file if it exists and loaded into the application data. Still we should have a fix it doesn't spawn at infinite speed when an http request error occurs. @virusman i'm not very familiar with objective c api's could you have a look at this?

[jogu]

I had the same issue on a new install - sorting out the API key fixed it. I had at some point clicked the 'regenerate API key' button in the web interface (due to not really being familiar with syncthing) so that may have been what triggered it, but it persisted across several restarts of the app.

[xor-gate]

Reopen because we also need some kind of mechanism to notify the user about a possible misconfiguration or auto-reconfigure the API key/uri from the xml.

[calmh]

I think the CPU usage is no longer an issue and the remaining problem is covered by #59?

[xor-gate]

@calmh yes you are right. We have covered it. In the next binary release this issue is not present anymore with fix in PR #47.