FR: Add Port Forwarding options about xpipe

[crschnick]

Yes that makes a lot of sense, I will see what I can come up with.

[crschnick]

Yes I think the proposed hierarchy of @Lockszmith-GH makes more sense, so I will quickly change that.

About the command-line options of @ashleysommer, I experimented a lot with different options.
The reason it doesn't use -N and -f is that this would make it much harder to detect when the tunnel is ready and to exit the connection, so I didn't go for that.
It already uses -T, because we don't want a tty when the connection is running in the background.
I will have to look into the -C option though and whether I could be used.

[Lockszmith-GH]

Wasn't critiquing - abslolutly agree. The license goes as far as litigation will take it, and most of FOSS maintainers are not litigious at all. We can't all be Stallman (nor do we want to).

The fact that you interact so activley here is amazing in my mind, and I appreciate it greatly (I mean it sincerely and emphatically - I understand how hard it is to maintain a project, I always look at active maintainers in awe of their acomplishment).

[crschnick]

Alright, I think this can be closed now.

@Lockszmith-GH the things you listed that are still to do here are added to my todo list, but I will open separate issues for these.

@ashleysommer now that xpipe supports sourcing from your ssh config, you can freely configure your connections with arbitrary options and then use them in xpipe that way.

[crschnick]

Were you thinking of these options on a global basis or also support to configure that for individual connections?

[Lockszmith-GH]

Similar to the way each connection has 'sub-options', configuring these could be as part of that

Something like:

+ SSH Connection Name
   + Dynamic Tunnel on localhost:1080 [Toggle on/off switch]
   + Port tunnel localhost:8443 to localhost:443

This should make configuring easy, and also should make inter-bastion tunneling straight forward.
I would also recommend adding an option to 'auto start' these tunnels.

I hope that makes sense.

[Lockszmith-GH]

I do this today with a PowerShell script on Windows.

The script runs in a terminal tab, it's a loop, tests for a TCP connection, if the TCP port is open, it runs ssh with the port-forwarding cli params - I acutally run htop for that session, instead of a shell, so I'll have a 'reason' to keep the terminal tab open, and not accidentally close it.

When it disconnects (or fails to find an open TCP port) it waits for a number of seconds (30 by default) I've set it, and tries again.

With xpipe, there is no need for a terminal at all, as ssh has the -N option (more details) - that's why I thought about this FR.

[crschnick]

So I did a little bit of implementation and am currently deciding whether there is a use case for people to also allow tunneling from one remote system to another. With XPipe, you are not limited to establishing tunnels from or to your local machine, but I'm not sure whether people would actually do that.

[crschnick]

I came up with this for now:

[Lockszmith-GH]

... whether there is a use case for people to also allow tunneling from one remote system to another.

I think, for now Local forwarding makes more sense, if someone comes up with a clear scenario requiring Remote forwarding that would be the time to consider it.

I personally use Local forwarding and Dynamic forwarding, and have yet to find situations where I need Remote forwarding, or if I did, I setup Wireguard - which makes much more sense to me.

Then again - That's just me.

This looks awesome BTW.

[crschnick]

Alright can you try this staging version: https://github.com/xpipe-io/xpipe_staging/releases/tag/1.4.0

[Lockszmith-GH]

I just took it for a spin, and it looks as though the functionality is exactly what I wasking for.

I do have a note about the UI workflow. There is something unintuitive in the setup (at least to me).

I'm going to use the following diagram and attempt to explain my concern:

flowchart
    subgraph flowchart["flowchart -- <a href='https://mermaid.live/edit#pako:eNqdVVt3okgQ_isc92Feho0goPHM5hxBIwp4CUQD6x4Ol0Zagcbm7sz89wHjaLJJdna3X5qu_uqrrur66K8tB7mg1W95ASoc38LpJiLqkWT2FluxT1zsf26uGIIkiS8W4WPg_fHJT9M46d_chACHFnR_D2AOboAL099ia4_6YHZwV7PbnK4GSzNwlnvWegIpV82qTKZEfRF5Ot4fclp3mWHi9Ib8QzKaj7rrW7GXjxRvpK1KiikemTzPmG53txsAAAfT43aPEjpuqwYeGTQ3ttmH8n7a1ubdMTUoelU-VBgJbXNZ83m0cx9k8d6CUjKfuAadTUQAZ1Mhe9DmRibqfqgYQUSK3lZiodCpT4C43NbH7fZBHmKGHMHp7RGYSWcmzajD4HY9OOr0zOAnEIM9m9DRYZzDh0hGk_CodB4PY7fDJ92lPlpHU189zJQuMCsVyJWNJV2L-DyTZ_Ij2h_gnp2OotF2Nx2h2Bl2s4nm8VsOOmmoyu3YfHqi5u3HsXRcPIntCukhFzyOwthe-Wu4g2Gplx4jcIEGinLqqn7gyZhR1tNeeRvj3F9hV37iFENSqdg0JL497XR65sFhebNidh1SIhWRz8ZZ91hqR6COS9OsI2mrSpDQLrIMzZ_ZR6k37xY9EKXLhVAyuyOr5R5t6KbM2bKzrxxn0eVczVAekgOQ2XRJAZrnHE5mUhcqwaKtdxaWjCU4uk_XOrsTFXfNjidov2WoIDNTk4VDBx9Jx6BYGyxo_57WzK1A0bke-5RNVRKPNcXxhYlIp65ZiEt_OBQyIdcnJLdjBpFnhnl7L7IFWzCGYGhzlR6veErAw7VHIcdalpO0s54_coqVDYRZd7Lld8xRx9m2wBbMJrY45QwbHdCnOyeAzp7wAQZEioimgb_cWHeb1l_PimiGW9-4k0IUEbx2tWYJwNfVRTlBHT6oVSM3M6FWSQrCV2zNKGMYAzJJ_PfMaRZFILjugMi9Lt6J6KOkkalYT8AlFoGVegi_jXnB29YJ30x1Tm9wzWj2yMRvxG8lfc8iAytOUUyoPgiCjz3i8uphYccvrIpQ54KkEguMyuqN4ymz1yb_lMWVpoCRiwoytEoYwiMgzlmugU0M4vhmsJi8Yv0b4yXnNcL71WL2QUWKercO2YCIlfJufg3kv1Xk5PG_KvLS8HrlZKer-6A2p_P_y8pcOrD-qd8R3zYtVRU3rW8_b_7XwHNBXuvhhHmnuS97jf8p_VpM_X6HbtN0LbsvNr65O9-sCnAOHdCEeF8OH3FRL7iEuk6ofpk-JRcFfkT30kogshmoodZOlvpAKAf4RHpSdtOfz-e-VCsu_4Ht11zUmevcL1euM_nPXIkG9CyPK-bs9ArTNMkzor7z1ufW-YWuX_uvjXnTSn0Q1gXu15-uhfeb1ib6XuOsLEVqFTmtfooz8LmVxa6VgiG0apGEz8bvPwBVXZJC'>click here to edit</a>"]
        direction BT
        user
        subgraph local["Local System"]
            xpipe-ssh
            xpipe-tunnel
        end
        
        subgraph host["Hosted Platform"]
            subgraph bast["bastion"]
                bast-sh["fas:fa-laptop Shell"]
                bast-px["fas:fa-archway SOCKS Proxy"]
            end

            hosted["fas:fa-window-maximize Hosted Web App/API"]
        end

        subgraph WorkVPN
            subgraph work["Work VM"]
                work-sh["fas:fa-laptop Shell"]
                work-px["fas:fa-archway SOCKS Proxy"]
            end
            
            cust["fas:fa-window-maximize Work Web App/API"]
        end

        xpipe-ssh --> |"SSH"| bast-sh
        xpipe-ssh --> |"SSH"| work-sh
        user --> xpipe-ssh
        user --> |"SOCKS on ::32022 to<br/>Hosted Service"| xpipe-tunnel
        user --> |"SOCKS on ::31022 to<br/>Customer's System"| xpipe-tunnel
        xpipe-tunnel o-----o |"Tunneled over<br/>localhost:32022"| bast-px
        xpipe-tunnel o---o |"Tunneled over<br/>localhost:31022"| work-px
        bast-px --> |" "| hosted
        work-px --> |" "| cust
    end

In the current implementation the window looks like:

Hosted Platform (click for SSH)
WorkVM (click for SSH)
Local Machine
    Hosted Platform bastion ON/OFF
    WorkVPN bastion ON/OFF

I was envisioning this as

Hosted Platform (click for SSH)
    Dynamic tunnel ON/OFF
WorkVM (click for SSH)
    Dynamic tunnel ON/OFF
Local Machine

While I undertand that the tunnel is opened 'from the localhost', so are the SSH connections, but they are not presented as sub-objects of Local Machine.

I noticed you DO have an option to select 'from where' this is opened, but that just indicates whether an SSH connection needs to be established to run the instantiation command.

Is this something that makes sense to you as well?

[ashleysommer]

I came here looking for this functionality too. Glad to see it is already implemented, and I'll test out the staging version.

[ashleysommer]

Working great as far as I can tell.

I don't have an opinion whether the entry for the tunnel should appear under the Local Machine tab, or on the remote machine tab. I see the logic behind both options, and both could be considered correct, so it doesn't much matter.

An implementation note, when I normally establish port-forward connections on the commandline, I use SSH with the N, C, T, and f options, eg:

*> ssh -i ~/.ssh/id_rsa -NCTf -L "127.0.0.1:7443:172.11.12.13:6443" [email protected]

For explanation, N causes SSH to not execute anything on the remote system (doesn't try to run sh)
C enables compression of the datastream. You could add this as a toggleable option when setting up the tunnel.
T doesn't allocate a pseudo terminal, so stdin/stdout/stderr are not linked to the SSH connection (this might not be compatible with your xpipe shell-ssh execution model)
-f this causes SSH to be sent to the background after establishing the connection, (this also not be compatible with or required with your xpipe shell-ssh execution model)
-f also implies -n that connects SSH's stdin to /dev/null, this is required to hold the connection open when SSH is sent to the background.

[crschnick]

About custom options, you can already specify them in your ssh config files for specific hosts or wildcards. They will get applied automatically as they would normally do when you would just call the ssh client.

But I should be able to emphasize this somehow in the connection creation dialog window, so let me think about that.

[crschnick]

This feature is now properly released in 1.4.0 with the proposed hierarchy changes included. In your case, when you load up your configuration from the staging version there, the tunnel connections might float around without a parent because you created them earlier. For newly created tunnels, it should work properly.

[Lockszmith-GH]

Thanks, I'll take this for a spin

[Lockszmith-GH]

Absolutley fabulous, this is a real quality-of-life improvement for me.

I do have an FR and I found a bug (let me know if you want me to open it separately):

• BUG: If the listening port is already in use on my system, there is no indication of failure, switching the tunnel on returns "Ready". It should probably provide an error (doesn't need to be a pop, that will be annoying, but the switch turning off, and the error printed in place of Ready would be very helpful.

• FR: I love that swithcing it on, is remembered when the app is restarted - absolutly great - but... Can we have this optional - for those connections we need temporarily? Maybe even add a time-limit?

Quesiton: What would happen if the connection is broken? will it retry? Should this something configurable?

[crschnick]

I have to look into what ssh outputs if a port is already occupied.

It is only switched on automatically on startup when it was also enabled when you exited, to somehow provide a continuous workflow. I guess this could be made configurable.

If the connection is broken, nothing really is displayed or done for the simple reason that it does not poll the current state of the process. I guess we could check every couple of seconds if the process is still alive and if not throw an error.

[Lockszmith-GH]

I'll take a look at your code later, havn't really touched Java in a long time, but maybe there is a way to execute the process asynchronously and wait on a separate thread for it to 'exit', then test the exit code.
Do you have any 'architectural' map of your code? to make my exploration easier?

[crschnick]

There's is definitely an easy way to wait for the exit, that is not the issue, I just need to implement and test it properly.

The thing about the codebase is that not all parts can be found in this repository. I decided to not open source select few parts of the codebase to simply prevent the possibility of some company coming in, forking the code, and selling it as their own product. (That happened to one project I contributed once, so I'm kinda wary of that)
Originally this only concerned the process handling implementation, but due to necessary subclassing of that implementation, that kinda evolved also to other implementations like some connections, including ssh tunnels. I always wanted to refactor that aspect and decouple it such that these parts can be included in this repository, but never got around to it.
The vast majority of the code is included here, just these few things are not.

[Lockszmith-GH]

Fair enough, sorry you got hurt like that. In anyway, I'm greatful for the tool.

[crschnick]

I am open for suggestions on how to make this project more open while still achieving the goal of keeping it away from company takeovers.

[Lockszmith-GH]

That's one of the toughest aspect of operating open-source in a commercial world. Other than making sure the code itself ALWAYS remains open - like GPL (I think v3), without taking anyone to court it's next to impossible to provide this promise.

I am happy though, that you have not lost your faith in the users of your software. :)

[crschnick]

I don't believe sharing all code publicly with GPL would prevent the issue, they could just violate it. Now at least, such a takeover is also technically difficult.

The current solution still allows anyone interested in contributing to still do so as this Apache licensed code is included in what is distributed. And apart from the connection implementation, which most people probably would have a hard time understanding anyways, you will find most of the code here.

Some parts are also purposefully designed to be extended easily, such as the file browser and external application integration.

[crschnick]

FR: I love that swithcing it on, is remembered when the app is restarted - absolutly great - but... Can we have this optional - for those connections we need temporarily? Maybe even add a time-limit?

This feature is now implemented in the 1.7 PTB release.

[Lockszmith-GH]

This is very good. Thank you for taking the time and implementing this.

Since no error reporting exists right now, I would suggest you at least test before initiating the SSH tunnel, that the listening port is free on the system. And maybe monitor that post later on and make sure it's tied to the process id of the ssh conneciton you opened.
On an unstable connection, the lack of visiblity makes it harder to diagnose.

[crschnick]

I can just hardcode the output of the ssh executable that is printed out when a port is occupied or the connection dies. To make this easy, you could just tell me what strings to look out for in the output.