GithubHelp home page GithubHelp logo

Comments (44)

mckay27 avatar mckay27 commented on August 16, 2024

I can second this problem. Everything sounds identical. Any updates/information would be appreciated.

from weewx-belchertown.

41south avatar 41south commented on August 16, 2024

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

Just thinking.... do I have to open both ports (MQTT and Websockets) on the router?

from weewx-belchertown.

41south avatar 41south commented on August 16, 2024

Well this guide from HiveMQ suggests that you could be right ;

https://www.hivemq.com/blog/mqtt-security-fundamentals-securing-mqtt-systems

......
Firewall
Every connection to an MQTT broker should pass at least one firewall that implements sophisticated rules for accessing downstream components. If your are able to block attackers at the firewall level, they won’t be able to access other systems in your infrastructure. Unfortunately, there is no universally applicable solution. Firewall rules need to be configured individually for each use case. Many commercial and open source firewall solutions are currently available.

A solid rule of thumb is: only forward expected traffic to downstream systems. Block any traffic that you don’t expect in your downstream applications. For an MQTT broker the following traffic is probably worth blocking:

UDP: MQTT uses TCP, you can block all UDP datagram packets.
ICMP: While it may not be the smartest idea to block all ICMP traffic, ICMP traffic, ping and traceroute ICMP packets could be worth investigating as candidates to block.
It’s also a good idea to block traffic to any ports that are not needed for your MQTT system.

Don’t block the following standard MQTT ports with your firewall:

1883: This is the default MQTT port. 1883 is defined at IANA as MQTT over TCP.
8883: This is the default MQTT port for MQTT over TLS. It’s registered at IANA for Secure MQTT.
If you are in full control of your MQTT system and know the IPs of your MQTT clients, only allow traffic for the needed IP ranges. This precaution locks out any clients that are not in the defined IP ranges.

from weewx-belchertown.

mckay27 avatar mckay27 commented on August 16, 2024

I think I have solved my problem. I don't know if this is related, as I didn't realize before that you were going through a router.

I've got weewx and mosquitto running on the same server all local. My problem was that in the weewx.conf file, I had the server listed as "localhost" which seems to be an ipv6 address. This works fine for port 1883 as mosquitto supports ipv4 and ipv6 for publishing, but for websockets only ipv4 is supported (or that's what it seems like due to to this post. So I just changed the server to the local FQDN using ipv4 and it's working perfectly.

Sorry if this isn't related to your problem, I just figured I would post this here so it might help someone in the future.

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

Thanks for all your suggestions. I replaced localhost with the IP address in the weewx.conf and it did NOT solve my issue. Since the station is remote, the changes in the router (forward port 1883) have to wait until I have access to the router (hopefully on the week-end). I will keep you posted.

from weewx-belchertown.

41south avatar 41south commented on August 16, 2024

Right, well all I opened in my home firewall was TCP 9001 and it's working fine off my home mqtt broker @ www.41south.net.nz.

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

Oh no, you are killing me... that was my best hope to solve the issue :(

from weewx-belchertown.

41south avatar 41south commented on August 16, 2024

If you want to test your skin config you could subscribe to my simulated weewx data. Just point your skin.conf mqtt at 41south.net.nz:9001 (ssl = 0) - the topic is weather/41south/loop (or #)

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

Thanks, using an online websocket client I can connect to your broker and susbscribe to your data. However, using the same websocket client, I can connect to my system but I can't subscribe. It 'feels' like the connection is not fully established. I changed my mosquitto conf file to log everything and I see the following:

1539820643: New client connected from 199.19.95.109 as clientId-53vcqTiEW6 (c1, k60).
1539820643: Sending CONNACK to clientId-53vcqTiEW6 (0, 0)
1539820694: Socket error on client clientId-53vcqTiEW6, disconnecting.

At least I have now an error message to investigate :)

from weewx-belchertown.

poblabs avatar poblabs commented on August 16, 2024

How's it going? Any progress?

from weewx-belchertown.

poblabs avatar poblabs commented on August 16, 2024

@mash55 I'm going to close this. Let me know if you need more help though

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

No progress... thanks for asking. I continue to get SOCKET ERROR message when trying to access data via websockets. Tried different OS version (Jessie and Wheezy) without any luck. Not sure whether I should try and compile mosquito with websocket support from scratch on a raspberry pi. Anyhow, it has to wait until I am back up in the mountain. Any other suggestions would be appreciated.

from weewx-belchertown.

poblabs avatar poblabs commented on August 16, 2024

Were you able to reach the MQTT data using an MQTT client?

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

Nope :(
Same errorin log file

from weewx-belchertown.

41south avatar 41south commented on August 16, 2024

from weewx-belchertown.

poblabs avatar poblabs commented on August 16, 2024

I would agree. From the same host that has MQTT installed on it, try to connect locally. Firewall shouldn't be in play at that point.

I also wrote a blog post that may be helpful if you haven't seen it yet.

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

Well, I have both ports open on the router/firewall. A local MQTT client supporting Websockets might be an option. Will go up to the cabin on the weekend and will see to find a client which supports websockets on windows or android so I can test within the local area network. Any suggestions for such a client?
Thanks...

from weewx-belchertown.

poblabs avatar poblabs commented on August 16, 2024

Edit: Updating some details for clarity

At this point I think it might be important to just have MQTT working. Once you can verify that weewx is sending data to MQTT and your mosquitto_sub client can see that data, then I think you can look into websockets. See if local websockets work, then try externally.

Remember that websockets is a layer on-top of the MQTT broker. So it's important to have MQTT itself working.

  1. I'd try mosquitto_sub on the Linux host first (see my blog post for details - linked above)

  2. Then I'd try your phone to connect to the broker, subscribe to the topic and check for data. Your weewx topic should be the loop topic. For example mashNET/loop. You can subscribe to mashNET/# for testing but only use that in testing.

  • You mention Android, so on Android I use MQTT Client. This does MQTT and websockets.
  1. So in the app add a new broker and do not select websockets. Fill in the host, port and client ID then save. Connect to the broker and subscribe to mashNET/# and see if you see anything. If so great, move on. If not, stop and re-check your broker settings, your firewall, and make sure mosquitto is running on the server.

  2. Add a 2nd broker in the app and this time tick the websocket checkbox, and make sure you define your websockets port. Fill out the host and client ID and subscribe to the topic and you should see everything. If you do, it's a matter of configuring the skin settings and restarting weewx.

Hope this helps! Do read my blog post, I know it has helped others in past who were stuck.

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

MQTT is running just fine. weewx is publishing and I can subscribe just fine (LAN and WAN). Problem is related to websockets. I read your blog (almost memorized it by now ;) )
Thanks for the android client... will test during the week-end.

from weewx-belchertown.

poblabs avatar poblabs commented on August 16, 2024

OK cool.

Yeah I saw your initial post - but when I get stuck I always start back at the basics. So I wanted to lay out what I'd try first.

So if your apps are connecting fine, then it's something with the skin settings or firewall ports on the websocket server or router or even the port forwarding from router to the websocket server. So many elements to double check.

from weewx-belchertown.

41south avatar 41south commented on August 16, 2024

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

Thanks guys, will report back after the week-end.

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

Her the latest update:
Installed MQTTBox on Windows 10 machine.
MQTTBox and mosquitto broker on same subnet.
Connected (to IP address) and subscribed via mqtt/tcp protocol
Worked

Connected (to IP adddress) and subscribed via ws protocol
Client shows a connection but collapses after a few seconds

Entry in mosquitto log-file:

1542480562: New client connected from 10.66.78.136 as mash_test_windows1542479922590 (c1, k10).
1542480562: Sending CONNACK to mash_test_windows1542479922590 (0, 0)
1542480577: Client mash_test_windows1542479922590 has exceeded timeout, disconnecting.
1542480590: Socket error on client mash_test_windows1542479922590, disconnecting.

I think its safe to say that my issue is NOT related to the router/firewall but to the mosquitto version/setup/config.

Not sure how to continue from here.

from weewx-belchertown.

poblabs avatar poblabs commented on August 16, 2024

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

Still here :)

mosquitto conf file:

# Place your local configuration in /etc/mosquitto/conf.d/
#
# A full description of the configuration file is at
# /usr/share/doc/mosquitto/examples/mosquitto.conf.example

pid_file /var/run/mosquitto.pid

persistence true
persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d

Thats the original file. I am overwriting with my own config file (/etc/mosquitto/conf.d/mystart.conf):

persistence false
log_type websockets error
# log_type all
allow_anonymous true

listener 1883
protocol mqtt

listener 9001
protocol websockets

hostname: cabin.schnizer.com
ports: 1883 (mqqt) and 9001 (ws) respectively.

Appreciate your help...

from weewx-belchertown.

41south avatar 41south commented on August 16, 2024

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

mashNET/loop

from weewx-belchertown.

41south avatar 41south commented on August 16, 2024

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

you could connect to 1883, right? Just websockets is a nogo?
Thanks for trying...

from weewx-belchertown.

41south avatar 41south commented on August 16, 2024

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

Dropout might have been due to heavy snowfall (wireless link up here)...

from weewx-belchertown.

41south avatar 41south commented on August 16, 2024

from weewx-belchertown.

poblabs avatar poblabs commented on August 16, 2024

from weewx-belchertown.

poblabs avatar poblabs commented on August 16, 2024

I also was able to connect to your MQTT broker without any issue and subsribed to mashNET/loop too.

I added your MQTT details to my lab server and was getting the following errors. It connects to your websocket, but doesn't seem to subscribe correctly or it's constantly being disconnected and reconnecting or something. I'm a little confused by it. It's retrying many times per millisecond - so something's gotta be off somewhere.

Edit: re-read you said you installed it from another repo. I wonder if that repo's install has something wrong with the websockets portion?

I haven't installed the broker on a Pi yet, but I have a couple laying around. I might have to try it out tomorrow.

image

from weewx-belchertown.

poblabs avatar poblabs commented on August 16, 2024

I'm looking at the copy/paste you did of your /etc/mosquitto/conf.d/mystart.conf and I don't know if it's GitHub or not, but it looks like you have extra spaces at the end of each config line. Can you confirm if they are there on the system? If so, remove them, then restart and see if that helps.

The MQTT broker is super picky about its config (like white spaces).

It might be a long-shot, but worth investigating.

image

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

The broker drops the connection (via ws) even without subscribing.... it takes a few seconds (~25) until the error and retry.
I checked the start-up file for white spaces but they were introduced during copy/paste.
The reason for using a different repo is the fact that the default repo version of mosquitto is not compiled with WS support. Installed it with 'apt-get' from 'repo.mosquitto.org/debian/mosquitto-jessie.list' on a 'Raspberry Pi 3 Model B Rev 1.2' using 'Raspbian GNU/Linux 8 (jessie)'. Also tried the broker on a 'Raspbian GNU/Linux 7 (wheezy)' version with the same result.
Let me know what you find out on raspberry platform if you have the time.
Many thanks

from weewx-belchertown.

poblabs avatar poblabs commented on August 16, 2024

I did some testing today with good results.

  • tl;dr Raspberry Pi Stretch and doing sudo apt-get install mosquitto mosquitto-clients works with websockets and the Belchertown skin just fine. Read on for more details.

My Raspberry Pi I'm using has Stretch installed

pi@garagepi:~ $ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
pi@garagepi:~ $

I then ran sudo apt update and sudo apt-get install mosquitto mosquitto-clients from the standard apt repository.

pi@garagepi:~ $ sudo apt-get install mosquitto mosquitto-clients
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  lxkeymap python-cairo python-gobject python-gobject-2 python-gtk2 python-xklavier
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libc-ares2 libev4 libmosquitto1 libwebsockets8
Suggested packages:
  apparmor
The following NEW packages will be installed:
  libc-ares2 libev4 libmosquitto1 libwebsockets8 mosquitto mosquitto-clients
0 upgraded, 6 newly installed, 0 to remove and 107 not upgraded.
Need to get 418 kB of archives.
After this operation, 897 kB of additional disk space will be used.
Do you want to continue? [Y/n] y

I can see from the initial command that it installed libwebsockets8 so I just continued on.

Once it was installed, I added a quick config with sudo nano /etc/mosquitto/conf.d/myconfig.conf and added a very basic, non-secured config:

persistence false

allow_anonymous true

listener 1883
protocol mqtt

# websockets
listener 9001
protocol websockets

Then I tried to run this config with

pi@garagepi:~ $ sudo mosquitto -c /etc/mosquitto/mosquitto.conf
pi@garagepi:~ $

But it failed. It didn't do anything. So I wanted to see if the port was open or not and did sudo netstat -tulpn and it returned this:

pi@garagepi:~ $ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      516/mysqld
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      406/sshd
tcp        0      0 0.0.0.0:1883            0.0.0.0:*               LISTEN      4715/mosquitto
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      382/blink1-tiny-ser
tcp6       0      0 :::22                   :::*                    LISTEN      406/sshd
tcp6       0      0 :::1883                 :::*                    LISTEN      4715/mosquitto
udp        0      0 0.0.0.0:42445           0.0.0.0:*                           284/avahi-daemon: r
udp        0      0 0.0.0.0:68              0.0.0.0:*                           334/dhcpcd
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           284/avahi-daemon: r
udp6       0      0 :::546                  :::*                                334/dhcpcd
udp6       0      0 :::41606                :::*                                284/avahi-daemon: r
udp6       0      0 :::5353                 :::*                                284/avahi-daemon: r

So if my config didn't work, but port 1883 is open then something else is running it. So I did a ps aux and found that it was running under a new user mosquitto that I didn't know about.

pi@garagepi:~ $ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.6  27112  6116 ?        Ss   Oct28   0:27 /sbin/init splash
<TRUNCATED OUTPUT>
mosquit+  4715  0.0  0.4   6332  3992 ?        S    17:05   0:00 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

So I force killed that process with sudo kill -9 4715 and re-ran sudo mosquitto -c /etc/mosquitto/mosquitto.conf. This time it didn't exit so quickly, it sat there (looking like it's done nothing). Checking the ports again I can see it's opened 1883 and 9001:

pi@garagepi:~ $ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:9001            0.0.0.0:*               LISTEN      5267/mosquitto
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      516/mysqld
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      406/sshd
tcp        0      0 0.0.0.0:1883            0.0.0.0:*               LISTEN      5267/mosquitto
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      382/blink1-tiny-ser
tcp6       0      0 :::22                   :::*                    LISTEN      406/sshd
tcp6       0      0 :::1883                 :::*                    LISTEN      5267/mosquitto
udp        0      0 0.0.0.0:42445           0.0.0.0:*                           284/avahi-daemon: r
udp        0      0 0.0.0.0:68              0.0.0.0:*                           334/dhcpcd
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           284/avahi-daemon: r
udp6       0      0 :::546                  :::*                                334/dhcpcd
udp6       0      0 :::41606                :::*                                284/avahi-daemon: r
udp6       0      0 :::5353                 :::*                                284/avahi-daemon: r

I opened 2 more SSH connections to the Pi.

On new connection 1 I ran mosquitto_sub -h localhost -t weather/# to subscribe to the topic.

On new connection 2 I ran mosquitto_pub -h localhost -t "weather/test" -m "hello" to publish to that topic.

So far so good - connection 1 can see the published update.

pi@garagepi:~ $ mosquitto_sub -h localhost -t weather/#
hello

Then on another server I have, I ran this websocket test utility that I found on GitHub (just download as zip, add to your webserver then browse to it with http://where-you-installed-it). They also have it hosted on the internet which may be easier - http://hobbyquaker.github.io/mqtt-admin but just know that GitHub will promote this to HTTPS and your broker is not using Secure Websockets it won't connect and will give you a white page. This is why I opted to host it myself on a non-https server. If you get stuck at a white screen you need to clear your cookies.

Anyways, once past that hurdle, I added my Pi and port:

image

Which connected me no problem. I switched to the subscribe tab and subscribed to weather/#

image

Then back into SSH connection 2 I re-published a couple of messages:

image

(MQTT saw them too, so it all appears to be working so far)

pi@garagepi:~ $ mosquitto_sub -h localhost -t weather/#
hello
hello
hello websockets?
hello websockets!

Finally I got around to weewx. I updated my weewx.conf with

    [[Belchertown]]
        skin = Belchertown
        HTML_ROOT = /var/www/html/btownwx
        [[[Extras]]]
            mqtt_enabled = 1
            mqtt_host = 192.168.0.34
            mqtt_port = 9001
            mqtt_ssl = 0
            mqtt_topic = weather/weewx/loop

Restarted weewx and forced a skin update: root@weewxlab:~# service weewx restart; wee_reports; tail -f /var/log/syslog -n 50.

I don't have any real data going to this new Pi broker, so I submitted some fake data:

pi@garagepi:~ $ mosquitto_pub -h localhost -t "weather/weewx/loop" -m '{"UV": "0.68", "barometer_inHg": "30.4306066431", "outHumidity": "45.8", "maxSolarRad": "399.067737313", "dewpoint_F": "17.6078693897", "cloudbase_foot": "4723.66604779", "heatindex_F": "36.5", "altimeter_inHg": "30.3995019668", "dayRain_in": "0.0", "outTemp_F": "36.5", "appTemp_F": "31.2879264251", "windchill_F": "36.5", "hourRain_in": "0.0", "humidex_F": "36.5", "rainRate_inch_per_hour": "0.0", "pressure_inHg": "29.9401590264", "usUnits": "1.0", "txBatteryStatus": "0.0", "windSpeed_mph": "1.0", "rain24_in": "0.0", "dateTime": "1542561977", "radiation_Wpm2": "124.813456"}'

and the page updated

image

So, I think if you can upgrade or install Raspbian Stretch, and just do a normal install of mosquitto you'll be in good shape? I'm not sure how you can uninstall the old mosquitto install - so maybe a fresh install of Raspbian if possible?

Let me know your thoughts.

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

Thank you soooo much for all the work and infos. I was thinking about STRRECH before but as far as I know there is no supported uprade path and installing from scratch with all the other services on this box is a lot of work. I guess I have to look into ugrading again and make a decision how to get to STRETCH.

from weewx-belchertown.

poblabs avatar poblabs commented on August 16, 2024

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

http://schnizer.net/ => :)
Decided to install from scratch on a virgin STRETCH release et voila...
Only issue right now is the 'disconnect_live_visitor' parameter. Would like to disconnect users after 5 minutes but no matter what I am using it disconnects after the default 30 min - puzzling
Thanks again for all your help (incl. Colin)... best from Colorado

from weewx-belchertown.

poblabs avatar poblabs commented on August 16, 2024

Looks great! Working for me here!

Ah, you may have found a bug introduced between versions! I changed the config name for that and will update in 0.8 from disconnect_live_visitor, to disconnect_live_website_visitor. If you want to get a head-start on that setting, try using disconnect_live_website_visitor in your weewx.conf overrides.

Don't forget to restart weewx after you've updated it.

from weewx-belchertown.

41south avatar 41south commented on August 16, 2024

from weewx-belchertown.

mash55 avatar mash55 commented on August 16, 2024

disconnect_live_website_visitor worked perfectly - thanks

from weewx-belchertown.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.