Sqlite3 reporting database is locked? about find3

[schollz]

@victorhooi I've never had that issue... I'm using a separate locking file for SQLite access so it seems that that ran into some race condition (possibly deleting a file that was created by another process so it thought it was okay to write). I think I need to do some load testing to see if I can reproduce

[victorhooi]

Will it help if I share with you my sqlite db files? Or the logfiles?

Or I can give you private access to the system?

[schollz]

@victorhooi Maybe the db. But first - did you ever get it to unlock?

[victorhooi]

I checked just now, and I'm not seeing the database is locked lines. However, I did shutdown 2 of the 5 Raspberry PI passive trackers.

My SQLite3 database file is around 800 MB...file size limit here is 25MB.

Any other thoughts on best way to get it across to you?

UPDATE - bzip2 managed to get it down to 78 MB.

[victorhooi]

@schollz Uploaded to Slack for you.

[schollz]

This should be fixed with c787ee8

[victorhooi]

I'm still seeing this a lot - particularly when I try to access the dashboard:

2018-04-28 22:32:34 [DEBUG] [PID-12] db.go AddSensor:350 [gcc] inserted sensor data, 49.05879ms
2018-04-28 22:32:35 [DEBUG] [PID-12] server.go 1:140 [gcc] getting device counts
2018-04-28 22:32:40 [WARN] [PID-12] server.go parseRollingData:890 [gcc] problem saving: problem getting device ID: AddName: database is locked
2018-04-28 22:32:40 [DEBUG] [PID-12] server.go parseRollingData:892 [gcc] saved reverse sensor data for wifi-c2:23:f8:40:10:8b
2018-04-28 22:32:40 [DEBUG] [PID-12] server.go parseRollingData:881 [gcc] reverse sensor data: {Timestamp:1524954750413 Family:gcc Device:wifi-00:26:08:fa:d2:ef Location: Sensors:map[wifi:map[entrance-wifi:-69]] GPS:{Latitude:0 Longitude:0 Altitude:0}}
2018-04-28 22:32:50 [WARN] [PID-12] server.go parseRollingData:890 [gcc] problem saving: AddSensor: database is locked
2018-04-28 22:32:50 [DEBUG] [PID-12] server.go parseRollingData:892 [gcc] saved reverse sensor data for wifi-00:26:08:fa:d2:ef
2018-04-28 22:32:50 [DEBUG] [PID-12] server.go parseRollingData:881 [gcc] reverse sensor data: {Timestamp:1524954750465 Family:gcc Device:wifi-e2:b9:e5:ac:5e:5e Location: Sensors:map[wifi:map[mainhallleft-wifi:-73 kidsroom-wifi:-75 entrance-wifi:-75 mainhallright-wifi:-77]] GPS:{Latitude:0 Longitude:0 Altitude:0}}
2018-04-28 22:32:50 [WARN] [PID-12] analysis.go AnalyzeSensorData:117 [gcc] nb1 classify: problem preparing SQL: database is locked
2018-04-28 22:32:55 [WARN] [PID-12] server.go parseRollingData:890 [gcc] problem saving: AddSensor: database is locked
2018-04-28 22:32:55 [DEBUG] [PID-12] server.go parseRollingData:892 [gcc] saved reverse sensor data for wifi-e2:b9:e5:ac:5e:5e
2018-04-28 22:32:55 [DEBUG] [PID-12] server.go parseRollingData:881 [gcc] reverse sensor data: {Timestamp:1524954750702 Family:gcc Device:wifi-08:3e:5d:13:48:46 Location: Sensors:map[wifi:map[entrance-wifi:-75 cafetv-wifi:-73]] GPS:{Latitude:0 Longitude:0 Altitude:0}}
2018-04-28 22:32:57 [WARN] [PID-12] server.go handlerReverse:819 Set: database is locked
2018-04-28 22:32:57 [INFO] [PID-12] server.go func1:978 172.27.177.141:60532 POST /passive 10.049664648s
2018-04-28 22:33:00 [WARN] [PID-12] server.go handlerReverse:819 Set: database is locked
2018-04-28 22:33:00 [INFO] [PID-12] server.go func1:978 172.27.198.243:55060 POST /passive 16.998434206s
2018-04-28 22:33:00 [WARN] [PID-12] server.go parseRollingData:890 [gcc] problem saving: AddSensor: database is locked
2018-04-28 22:33:00 [DEBUG] [PID-12] server.go parseRollingData:892 [gcc] saved reverse sensor data for wifi-08:3e:5d:13:48:46
2018-04-28 22:33:00 [DEBUG] [PID-12] server.go parseRollingData:881 [gcc] reverse sensor data: {Timestamp:1524954750733 Family:gcc Device:wifi-f0:9f:c2:7d:76:56 Location: Sensors:map[wifi:map[mainhallleft-wifi:-63 cafetv-wifi:-51 kidsroom-wifi:-27 entrance-wifi:-47 mainhallright-wifi:-55]] GPS:{Latitude:0 Longitude:0 Altitude:0}}
2018-04-28 22:33:05 [WARN] [PID-12] server.go parseRollingData:890 [gcc] problem saving: AddSensor: database is locked
2018-04-28 22:33:05 [DEBUG] [PID-12] server.go parseRollingData:892 [gcc] saved reverse sensor data for wifi-f0:9f:c2:7d:76:56
2018-04-28 22:33:05 [DEBUG] [PID-12] server.go parseRollingData:881 [gcc] reverse sensor data: {Timestamp:1524954750991 Family:gcc Device:wifi-ec:8c:a2:27:b8:38 Location: Sensors:map[wifi:map[mainhallleft-wifi:-57]] GPS:{Latitude:0 Longitude:0 Altitude:0}}
2018-04-28 22:33:10 [WARN] [PID-12] server.go handlerReverse:819 Set: database is locked
2018-04-28 22:33:10 [INFO] [PID-12] server.go func1:978 172.27.207.39:56890 POST /passive 15.949608689s
2018-04-28 22:33:10 [WARN] [PID-12] server.go parseRollingData:890 [gcc] problem saving: AddSensor: database is locked
2018-04-28 22:33:10 [DEBUG] [PID-12] server.go parseRollingData:892 [gcc] saved reverse sensor data for wifi-ec:8c:a2:27:b8:38
2018-04-28 22:33:10 [DEBUG] [PID-12] server.go parseRollingData:881 [gcc] reverse sensor data: {Timestamp:1524954751053 Family:gcc Device:wifi-2c:76:8a:c5:9c:be Location: Sensors:map[wifi:map[kidsroom-wifi:-75]] GPS:{Latitude:0 Longitude:0 Altitude:0}}
2018-04-28 22:33:15 [WARN] [PID-12] server.go parseRollingData:890 [gcc] problem saving: AddSensor: database is locked
2018-04-28 22:33:15 [DEBUG] [PID-12] server.go parseRollingData:892 [gcc] saved reverse sensor data for wifi-2c:76:8a:c5:9c:be
2018-04-28 22:33:15 [DEBUG] [PID-12] server.go parseRollingData:881 [gcc] reverse sensor data: {Timestamp:1524954750630 Family:gcc Device:wifi-34:8a:ae:e9:9b:2f Location: Sensors:map[wifi:map[entrance-wif

Stopping and starting the docker container for find3 seems to help.

[victorhooi]

Also, when hitting the locations endpoint (api/v1/by_location/<family>?history=5), I'm still seeing frequent timeouts (30 seconds).