samizdapp / herakles Goto Github PK
View Code? Open in Web Editor NEWslaying the hydra of IP/NAT/DNS/TLS/AppStore
slaying the hydra of IP/NAT/DNS/TLS/AppStore
There is a pathway of using the WiFi hotspot that doesn't require the installation of the app for Android.
For the past several days, connections from client to box over relays have generally not been working.
#59 Complicates understanding this issue fully.
The browser attempts to update the service worker by fetching the script url. This url will be the original url of the script at the time of registration.
The fetch request will not be handled by the service worker:
Set request’s service-workers mode to "none".
https://w3c.github.io/ServiceWorker/#update-algorithm
This means that unless the browser can directly access the script via the original URL, no update will happen.
Create a bootloader service worker that gets updated rarely. It fetches the full worker script and loads and executes it.
eval()
probably won't work with such a large bundleFunction
constructorSome of the work in the planned status update concerning worker versions should be implemented as a part of this solution (assuming this gets done before the status update).
The steps for updating the service worker are more or less the same as the steps for installing the client.
These steps will likely still apply assuming we go with the above solution. They will also have to be done in order to migrate over to the above solution.
The worker should automatically update following normal worker update rules.
#unsafely-treat-insecure-origin-as-secure
flag, you'll need to set it again.The worker should automatically update following normal worker update rules.
Use the Shortcut again?
Metadata:
Original Estimate:
Priority: Low
Epic: SAM-3 Improve logging
Uncaught (in promise) TypeError: Failed to execute 'put' on 'Cache': Partial response (status code 206) is unsupported
at eval (worker-app.js:2:931542)
#59 Reports 4 big inefficiencies with our libp2p network. Problem number 1) is that libp2p attempts to connect to a bunch of addresses that don't exist rather than attempting to connect to the addresses in the bootstrap list that it is given.
This problem seems sporadic, but I think I have tracked it down to each libp2p node announcing these no-op local addresses to the network. The client receives these announced addresses from each node in the network and attempts to use them to connect to the node.
Nodes should not announce addresses that they can't be reached by. I believe the way to prevent this is to specify an announceFilter
: libp2p/js-libp2p#769 (specifying the announce
config only adds addresses, not subtracts them).
Currently, if the URL doesn't exist, Pleroma will redirect to a 404.
Sometimes, when attempting to upload a file to Pleroma, the upload will not happen (stays pending with no progress).
At this point, the client's connection to the box will also fail (home page will show offline). One time, it stayed offline for awhile, the other time it recovered quickly.
This has not been debugged yet.
If http://samizdapp.local
is not added to the insecure origin browser flag (or if the flag is disabled), then the service worker will fail to install with no error either in the UI or in the browser console. (the loading gif just keeps spinning).
Ideally, we'd be able to detect when the origin isn't being treated as secure and broadcast a message in the UI.
At the very least, we should log the status of the service worker (whether it is installed or not) and log an error if it doesn't get installed by the browser.
A minimum of 0 disables autodialing.
Currently, if something is wrong with the box and the client can't connect, it won't keep retrying. i.e. once the box has recovered, the worker will need to be restarted.
The libp2p client should retry failed connections on an interval until a connection can be reestablished.
In the soapbox UI, when a link is posted, an attempt is made to show a preview of it, but the loading skeleton ultimately goes away and no preview is shown.
For users who run a lot of UPnP applications (i.e. video games), it will be common for the UPnP table in their router to be full.
When this happens, a 402 will often be returned.
Handle 402 errors by checking for many UPnP entries (past some threshold?) and removing either some or all of them.
This Perl script deletes multiple entries by iterating over the list and deleting each one individually: https://www.howtoforge.com/administrating-your-gateway-device-via-upnp#igdctlpl
I couldn't find a best practice on this; it is possible that clearing the entire table is the preferred method so that any applications that lose their entries are more likely to recover.
Metadata:
Original Estimate:
Priority: Medium
Epic: SAM-5 Setup CI/CD, Deployment, and Automated testing
There should be 3 environments:
feature - dev
dev - dev
prod - alpha
Feature and prod will require approval with environment protections. Dev will have no protections.
The new build-deploy workflow will use the configured environments to run deploy.
Metadata:
Original Estimate:
Priority: High
Epic: SAM-98 WebRTC Transport & Secure Bootstrapping
Metadata:
Original Estimate:
Priority: High
Epic: SAM-98 WebRTC Transport & Secure Bootstrapping
Create data structure in PWA to hold new app manifests (app manifests should have a list of components with a type
and a specification
. The existing manifest.manifest
property would be converted into a manifest.components[].type === "docker-compose" && manifest.components[].specification === '{version: "1.0".... }'
.
The PWA data structure wouldn’t need to include types for the docker-compose component, but would need to type a new ui
component. The ui
component specification
property should be some HTML that gets rendered.
A new /pwa/apps
path should render the app launcher (TODO - not a part of this ticket), and a parameterized /pwa/apps/:id
path should just render the app with the given id (the app launcher will launch an app by just redirecting to this path).
Hardcode the manifest for pleroma in a sensible place. It should render an iframe that points to /timelines/fediverse
(we can use this method: [https://reactjs.org/docs/react-dom-server.html#rendertostaticmarkup|https://reactjs.org/docs/react-dom-server.html#rendertostaticmarkup|smart-link] to specify the hardcoded iframe using JSX).
Metadata:
Original Estimate: 2h
Priority: Medium
Epic: SAM-88 Status Overhaul
New relays are added to libp2p.relays
when received from the p2p_proxy; however, old addresses are never removed from the list.
Somehow, on my dev box, the networking service added a bad yggdrasil config pee:
"Peers": [
"tls://51.38.64.12:28395",
"tcp://172.114.149.250:5008",
"tcp://98.53.136.200:5000",
"tcp://98.53.136.200:5001",
"tcp://24.212.191.150:5000",
"tcp://100.64.153.14:5000",
"tcp://73.181.212.115:5000",
"tcp://67.161.135.149:5000",
"tcp://45.18.74.227:5000",
"tcp://118.208.141.64:5000",
"tcp://;; UDP setup with 1.0.0.1#53(1.0.0.1) for whoami.cloudflare failed: network unreachable.\n;; UDP setup with 1.0.0.1#53(1.0.0.1) for whoami.cloudflare failed: network unreachable.\n;; UDP setup with 1.0.0.1#53(1.0.0.1) for whoami.cloudflare failed: network unreachable.:5005"
],
Manually editing the yggdrasil config file would not resolve the issue as the networking service would simply put the bad value back in.
It isn't clear whether this was a bad value stuck in memory, or whether the networking service was making the fetch node info call every time and getting the same error. Regardless, I don't know why this bad value wasn't sanitized; my only guess is that some sort of race condition was happening, but I don't see a place in the code where a race condition could occur. I couldn't reproduce this locally by manually inserting the error into the yggdrasil config (it would correctly get sanitized).
The network error itself happens here: https://github.com/samizdapp/athena/blob/2a10367cc7124e3983c908a3384432450089e942/packages/networking-service/src/yggdrasil/manager.ts#L36
to make for a better relay experience, we currently have a fork of pleroma that fetches some history every time a box follows another box. The changes are pretty minimal, but do to the fact that we were originally forking the repo wholesale, we've diverged from main substantially.
As we'd like to keep parity with the main pleroma project, we should do some housekeeping here.
Two options:
1 (preferred) submit our changes to pleroma as a PR and get them accepted. The substantive change is something that used to be in pleroma but was removed after running into infinite recursion. Our implementation is constrained in scope so as to remove this problem.
2 rebase against pleroma master branch, keeping only the 20ish lines of code we need, and periodically merge upstream changes.
Currently, we are unable to open websocket connections to the box because the proxy only supports HTTP. This causes many features in Pleroma to fail.
The gateway_client image is over a gigabyte, but it should be much smaller (under 100 Mb). Try deleting node_modules/
at the end of the Dockerfile
to see if that takes away most of the size.
I've tried all combinations of the 4 NPM scripts, and I always get this error when opening the app locally:
A bad HTTP response code (404) was received when fetching the script.
localhost/:1 Uncaught (in promise) TypeError: Failed to register a ServiceWorker for scope ('http://localhost:8001/pwa/') with script ('http://localhost:8001/pwa/sw.js'): A bad HTTP response code (404) was received when fetching the script.
Metadata:
Original Estimate:
Priority: Medium
Epic: SAM-98 WebRTC Transport & Secure Bootstrapping
I forget exactly how that feature is implemented, but we only want to navigate in the frame, not the entire page. Figure out how to implement it so that only the frame navigation is affected, with an eye towards modularizing pleroma stuff out of the worker.
Metadata:
Original Estimate:
Priority: Medium
Epic: SAM-98 WebRTC Transport & Secure Bootstrapping
There is no longer a need for the debug mode gesture because there will be a toolbar with a status link in it that can take the user to the status dashboard at any time.
The gesture is now unwanted, because if it does get triggered, it will redirect the pleroma iframe to the status dashboard page.
Inside Pleroma, links to Pleroma posts are broken, they are absolute links that use the domain name of the pleroma server, instead of pointing to the current domain (samizdapp.local) or being relative.
If it is easy to configure this, that could be a fix, but the more robust solution is to probably implement service worker logic that handles these urls.
When the PWA is installed via Chrome or Brave on Windows, it is opened automatically, and it is redirected to /
.
However, it the PWA is then closed and then reopened, it will stay on /pwa
and not be redirected.
/pwa
should always redirect to /
when the PWA is started.
Scheduled posts in Pleroma never send.
After the scheduled time, they'll still show up under "Scheduled Posts" perpetually with a time left of "Moments remaining"
It is possible for the box to be in a state where it becomes defederated from the SamizdApp network.
This is most evident when photos and videos that other people post fail to load (Pleroma returns 404s) - also when people's cover photos and avatars fail to load. Presumably, posts and likes from you won't be seen by others? But this wasn't tested. I'm assuming that the reason a post with an image will appear in the timeline, but the image fails to load is that Pleroma saved the post to the box but doesn't attempt to load the image until requested. Or maybe posts get copied to the box, but media never gets copied (is always loaded from the source box).
Restarting the box fixed this, but because I had to wait for the Yggdrasil crawl to happen (about 10-20 minutes), I'm not sure if only restarting Yggdrasil or Pleroma would have also fixed it (I didn't wait long enough after restarting the services before restarting the box).
It has been observed that even if there is a UPnP port open on the router, traffic will not be able to go through it. While restarting the router does fix this problem, restarting the proxy service also fixes the problem, suggesting that there is something that can be done on our side.
Come up with a way to detect stale UPnP ports and recover from them.
When I grant permission to the PWA to show me operating system notifications, none are generated.
This is because we need to implement service worker logic to handle sending out notifications for the PWA.
Metadata:
Original Estimate:
Priority: High
Epic: SAM-98 WebRTC Transport & Secure Bootstrapping
When I refresh the browser on my laptop, the entire page refreshes, not just the frame.
In most cases, this is actually what we want to likely have happen. However, this presumably also happens when the users swipes down on mobile. Assuming this is the case, we probably want to override that gesture somehow and allow apps to register a onRefreshGesture
handler. In the iframe case, it should just reload the iframe.
When setting up the client by navigating to http://samizdapp.local/pwa
, the docs instruct the user to look for a message saying "the origin is trusted!". After seeing that message, they can click the install button to install the PWA.
However, no such message is shown; instead, a spinner is shown under the message: "please install PWA to continue reloading false: no error"
Metadata:
Original Estimate: 0.5h
Priority: Medium
Epic: SAM-88 Status Overhaul
Metadata:
Original Estimate: 3h
Priority: Low
Epic: SAM-88 Status Overhaul
Should give status of both ethernet and wifi. Should probably use metadata for this ([https://joshuacwebdeveloper.atlassian.net/browse/SAM-30|https://joshuacwebdeveloper.atlassian.net/browse/SAM-30|smart-link]).
Metadata:
Original Estimate:
Priority: Medium
Epic: SAM-88 Status Overhaul
Sometimes, it takes awhile for my browser to give me the option (or popup message) of installing the PWA (i.e. it allows me to add a shortcut for my desktop instead of a PWA, but will then show me the notification and PWA install option later).
I believe what is happening is that the manifest.json file is being downloaded via the service worker, which may take awhile if the service worker hasn’t connected yet.
Show the Installing… message until the manifest.json has been fully loaded.
Currently the title is "Next.JS Progressive Web App"
Metadata:
Original Estimate: 3h
Priority: Medium
Epic: SAM-88 Status Overhaul
Maybe animate the graphic in a loop.
Additionally, we previously wanted to show which relay connection is currently being attempted in the worker status box. However, with the recent worker refactor, that doesn’t make as much sense because p2p will attempt connections more or less simultaneously (i.e. if a connection fails or reties, all addresses will have already been tried).
Think of some other useful connection attempt status that can be shown an include it.
Quite often, attempts to load Pleroma will timeout due to requests from the browser receiving no responses from Caddy.
Sometimes, this means that requests to http://setup.local
timeout. When this happens, no errors (or logs from anywhere) are generally observed.
Other times, requests to http://setup.local
will respond successfully, but requests to the Pleroma API timeout.
Most recently, requests to the Pleroma API timing out were accompanied by these errors:
yggdrasil Traceback (most recent call last):
yggdrasil File "/crawler/example.py", line 12, in <module>
yggdrasil ygg = YggdrasilConnection.fromServer()
yggdrasil File "/crawler/yggdrasil_iface.py", line 56, in fromServer
yggdrasil return cls(s.AF_INET, (host, port))
yggdrasil File "/crawler/yggdrasil_iface.py", line 40, in __init__
yggdrasil self.socket.connect(address)
yggdrasil ConnectionRefusedError: [Errno 111] Connection refused
daemon_proxy caught error TypeError: Cannot read properties of null (reading 'arrayBuffer')
daemon_proxy at PocketProxy.handleEvent (/proxy/dist/pocket_proxy.js:168:29)
daemon_proxy at runMicrotasks (<anonymous>)
daemon_proxy at processTicksAndRejections (node:internal/process/task_queues:96:5)
Usually, this issue will go away after awhile and requests will start responding again. A restart of the box/services doesn't necessarily help.
The overall result of all of this is that the box is not highly available and access to Pleroma is not reliable.
Requests made to the box should result in a response. Both Caddy and the Proxy should be able to handle the failure of services they forward requests to.
When uploading a file, the p2p connection is severely degraded for the duration of the upload. This results in requests taking 10x - 20x longer to complete. This also results in operations with timeouts (such as ping) to fail. Ping failing will cause the client to reconnect, thereby killing the upload. This effectively means that any file that takes longer than 20 seconds (2 rounds of ping with a 10 second timeout) to upload can never be successfully uploaded
Even if the client is able to connect to the box through the LAN, it can sometimes take over a minute for that connection to actually happen (during which, the user will be shown the online message).
This is due to a few observed inefficiencies:
There a couple of issues that come up frequently with the new ReadableStreamDefaultController
that the NativeRequestStream
uses.
The first is that release()
attempts to call close()
on a stream that sometimes has already errored out:
native-request.ts:269 Uncaught (in promise) TypeError: Failed to execute 'close' on 'ReadableStreamDefaultController': Cannot close an errored readable stream
at NativeRequestStream.release (native-request.ts:269:38)
at NativeRequestStream.receiveResponseEnd (native-request.ts:260:14)
at NativeRequestStream.receiveChunk (native-request.ts:175:22)
at NativeRequestStream.initInbox (native-request.ts:156:18)
The second is that a response is attempted to be read into a stream after the stream has closed:
index.ts:54 ERROR [worker/p2p/streams/native-request] receiveResponseBody TypeError: Failed to execute 'enqueue' on 'ReadableStreamDefaultController': Cannot enqueue a chunk into a closed readable stream
at NativeRequestStream.receiveResponseBody (native-request.ts:251:41)
at NativeRequestStream.receiveChunk (native-request.ts:171:22)
at NativeRequestStream.initInbox (native-request.ts:156:18)
Metadata:
Original Estimate:
Priority: Medium
Epic: SAM-62 Refactor service worker
As a part of the [Connection Manager Overhaul|], libp2p’s tendency to have connections time out and fail to deliver messages is discussed.
This could be what is causing some of the issues on iOS. Attempt to reproduce this, and if successful, patch it.
Metadata:
Original Estimate:
Priority: High
Epic: SAM-98 WebRTC Transport & Secure Bootstrapping
See if being loaded in a frame or being injected into multiple active pages will cause any issues with any of our injectors.
Also, review how we’re using the worker clients to see if controlling multiple clients at the same time will cause any issues. Also, quickly just check and see if there are any unexpected caveats with controlling multiple clients.
Metadata:
Original Estimate: 2h
Priority: Medium
Epic: SAM-88 Status Overhaul
Currently the logic for limiting the number of logs doesn’t work well enough, eventually yggdrasil will start logging dozens of logs.
For example, if the service worker fails to load, the success message is still shown. Don't show it until we're show the user has the install button available.
When I attempt to connect to the wifi hotspot by entering the ssid and passphrase into the UI, the server crashes with the following error:
[Logs] [11/1/2022, 8:48:59 AM] [service_wifi-connect] Deleting existing WiFi connection to the same network: "SamizdApp"
[Logs] [11/1/2022, 8:48:59 AM] [service_wifi-connect] Stopping access point 'SamizdApp'...
[Logs] [11/1/2022, 8:48:59 AM] [service_wifi-connect] [network_manager::dbus_api:ERROR] org.freedesktop.NetworkManager.Settings.Connection::Delete method call failed on /org/freedesktop/NetworkManager/Settings/15
[Logs] [11/1/2022, 8:48:59 AM] [service_wifi-connect] Stopping access point 'SamizdApp'...
[Logs] [11/1/2022, 8:48:59 AM] [service_wifi-connect] [network_manager::dbus_api:ERROR] org.freedesktop.NetworkManager.Settings.Connection::Delete method call failed on /org/freedesktop/NetworkManager/Settings/15
[Logs] [11/1/2022, 8:48:59 AM] [service_wifi-connect] Error: Stopping the access point failed
[Logs] [11/1/2022, 8:48:59 AM] [service_wifi-connect] caused by: D-Bus failure: org.freedesktop.NetworkManager.Settings.Connection::Delete method call failed on /org/freedesktop/NetworkManager/Settings/15
[Logs] [11/1/2022, 8:48:59 AM] [service_wifi-connect] caused by: "No such interface âorg.freedesktop.NetworkManager.Settings.Connectionâ on object at path /org/freedesktop/NetworkManager/Settings/15"
All subsequent attempts to access the server fail, but the container stays running.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.