Comments (15)
Why not using ZeroMQ?
ZeroMQ can add the ability to have an interprocess communication through pipe or network.
That can help where there's a need to do a fleet of machines in clusters not only some processes.
from spark-server.
That is definitely something we can look into for going server-to-server. I was just planning on using Redis but if you have some good reasons why we should use that instead I'd be open to it.
For now we are trying to keep things simple so anyone using spark-server only has to install spark-server to run it. I don't want them to have to spin up a MongoDB and ZeroMQ if they don't really need the extra services.
from spark-server.
Usually i use redis to share sessions cross web servers in php and store sensors data for time series.
ZeroMQ is a communication framework. In my vision some processes outside the device side (coap) can speak each others through ZeroMQ. Some processes cpu intensive can be written in c/c++ for realtime tasks
from spark-server.
Alright, I did a bit of digging ZeroMQ it is. Thanks for the suggestion.
In order to implement this we need to make sure that the spark-server will work with or without a ZeroMQ service running in the background.
from spark-server.
In our solution we use RabbitMQ as a Messaging Server, especially to provide a "Bumper" between our different Components. The Spark-API es a good example. You can use the API Perfect for commands, but it's problematic for events, cause if you update your Component you will lose Events as long as your connection to the SparkServer is not reestablished.
One Good (first) Scenario would be a configuration where all Events go directly into a (one or more) configurable messageQueues (in addition to the default way). In Docker you manage this over ENVIROMENT Variables, so you can activate such an interface by detecting this enviroments.
I can provide a sample and documentation for this. I could also implement such a solution but i need a little help ;) to know at which place to implement such an extension.
I Think there is no need to replace the complete API with a message queue.
I Prefer RabbitMQ cause administration and production management is very good, and there are also very good Images to directly build as solution using Docker. https://hub.docker.com/_/rabbitmq/
To store data between Instances REDIS is a good and stable solution (i already implement such solutions).
from spark-server.
Did anyone ever get things working with a message bus? I can’t remember.
from spark-server.
Yep. We are using our fork, also for production.
https://github.com/keatec/spark-rabbit
from spark-server.
Awesome, I had someone ask about multi-server scaling.
How many devices are you running?
from spark-server.
We are currently testing in our own enviroment (40-50 Devices). Real Live is planned starting next year 10.000 -> 100.000). We are running different spark_servers (managing 10-20 Devices in our demo) and a single RabbitMQ.
Our libary also includes (in events) informations about the messagequeue to communicate with the device producing the event, and a directory service (not part of the fork) to store informations about the sparkserver for a device (REDIS).
Devices are connecting using a LoadBalancer which is configured to send a sourceIP and Port always to the same spark server (so if a particle reconnects to a different spark-server, the informations are updated in the directory, and processes answering to events can use the ActionQueue directly from the incomming event)
from spark-server.
Me no,
i'm busy on the customization of the firmware.
from spark-server.
@haeferer - check out https://github.com/Brewskey/particle-collider if you'd like to test the scaling.
So you always reconnect the device to the same server? Aren't you worried about a case where you need to take a server down for maintenance?
from spark-server.
The loadbalancer will handle this Szenario.
Not perfect, but works. One Sparkserver goes down, Particle is send to next server and fails (cause of wrong handshake ) then reconnects ... fine
from spark-server.
@jlkalberer Just a shy question - is clustering on a roadmap? We run the server with 400 devices (roughly half online) and sometimes it eats up all server CPU, slowing down the response. Possibility to spread the load over more processors/cores would be great.. Just asking, I know you are busy..
from spark-server.
No, we aren't looking to add it. I tried to get it working a while back and ran into issues with the way we're dispatching events. We didn't change this much from the original spark-server
so there are problems :/
I don't think you should be seeing issues with only 400 devices. There are other people using a single server with close to 1000.
Are you calling a lot of functions/variables/webhooks?
from spark-server.
from spark-server.
Related Issues (20)
- .env not working if spark-server is used as dependency HOT 5
- Error on spark-server update-firmware HOT 4
- Response timeout for event: spark-server/call_device_function HOT 5
- ERROR: DeviceServer.js for local server HOT 2
- firmware device counter increases with failed flash attempts HOT 2
- Cannot upload device IDs with non-numeric characters
- Hook for device status change? HOT 3
- Is there a way to pass the contents of the publish event directly to MQ? HOT 7
- Memory Leak
- update-firmware requests failing HOT 3
- support for mesh devices HOT 1
- Function call returns error when firmware function returns zero HOT 6
- Electron session end after every 30 seconds HOT 5
- Throws error on starting spark-server HOT 1
- device names not accepted as deviceId - breaks particle-cli HOT 15
- Device.js/14528 on hostName: Device disconnected (cache_key=_5, deviceID=deviceid, duration=46.791, disconnectCounter=1) HOT 1
- Error with handshake process HOT 32
- Claiming / provisioning a devices without an deviceID HOT 5
- Does this still work? HOT 15
- Is there a renamed implementation of Particle.subscribe(particle/device/name); ? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark-server.