yob / graval

An experimental go FTP server framework

graval

An experimental FTP server framework. By providing a simple driver class that responds to a handful of methods you can have a complete FTP server.

Some sample use cases include persisting data to:

• an Amazon S3 bucket
• a relational database
• redis
• memory

There is a sample in-memory driver available - see the usage instructions below for the steps to use it.

Full documentation for the package is available on godoc

Installation

go get github.com/yob/graval

Usage

To boot an FTP server you will need to provide a driver that speaks to your persistence layer - the required driver contract is listed below.

There is a sample in-memory driver available as a demo. You can build it with this command:

go install github.com/yob/graval/graval-mem

Then run it:

./bin/graval-mem

And finally, connect to the server with any FTP client and the following details:

host: 127.0.0.1
username: test
password: 1234

The Driver Contract

Your driver MUST implement a number of simple methods. You can view the required contract in the package docs on godoc

Contributors

• James Healy [email protected] http://www.yob.id.au

Warning

FTP is an incredibly insecure protocol. Be careful about forcing users to authenticate with a username or password that are important.

License

This library is distributed under the terms of the MIT License. See the included file for more detail.

Contributing

All suggestions and patches welcome, preferably via a git repository I can pull from. If this library proves useful to you, please let me know.

Further Reading

There are a range of RFCs that together specify the FTP protocol. In chronological order, the more useful ones are:

• http://tools.ietf.org/rfc/rfc959.txt
• http://tools.ietf.org/rfc/rfc1123.txt
• http://tools.ietf.org/rfc/rfc2228.txt
• http://tools.ietf.org/rfc/rfc2389.txt
• http://tools.ietf.org/rfc/rfc2428.txt
• http://tools.ietf.org/rfc/rfc3659.txt
• http://tools.ietf.org/rfc/rfc4217.txt

For an english summary that's somewhat more legible than the RFCs, and provides some commentary on what features are actually useful or relevant 24 years after RFC959 was published:

• http://cr.yp.to/ftp.html

For a history lesson, check out Appendix III of RCF959. It lists the preceding (obsolete) RFC documents that relate to file transfers, including the ye old RFC114 from 1971, "A File Transfer Protocol"

This library is heavily based on em-ftpd, an FTPd framework with similar design goals within the ruby and EventMachine ecosystems. It worked well enough, but you know, callbacks and event loops make me something something.

graval's Issues

Security: when a passive data socket is opened, check the remote IP matches the control socket

This is a sensible security precaution to ensure the data and control sockets are coming from the same host

Need a way to close the FTP Server

FTPServer.Close() error

Unable to control it, I need to close it.

Explore this fork for commits we can cherry-pick

This fork has progressed a long way past this project.

https://github.com/koofr/graval

It'd be good to take a look through it and cherry-pick the bug fixes and features with broad appeal.

Does the project support PASV mode?

Hi,
Does the project support PASV mode? I use this project with PASV mode but it seems not to support this mode?
Thanks,
zhangkai

Security: before opening an active data socket, check the requested remote IP matches the control socket

this is a sensible security precaution

CLOSE not sending OK when sent from new modem

We have a system that has been using some older modems for years now with no issues. Recently, we switched to some new modems and now everything is working except that when it issues a ftp close our service doesn't send an "OK" back which is breaking our knowing if the request was process successfully.
What could be different that would break this and how might we be able to fix it?

Move ignoring of list and nlst flags into command structs

We're not ready to handle these flags yet, and there's already some basic ignore logic in buildPath().

That seems like a weird place for it though. It's probably better to move the ignoing logic up into the command structs, and then expand it to include other flags that might appear (like here koofr@4de02f5)

Ask for why should sleep before close dataconn

graval/ftpconn.go

Line 205 in 20ac9ad

// Chrome dies on localhost if we close connection to soon

I don't understand what is the meaning here, if not sleeping, the client will receive less data?

Implement MLST and MLSD commands

These commands are alternatives to LIST and NLST. They return a machine readable format though, and modern FTP clients often prefer to use them.

RFC 3659 has the details: https://tools.ietf.org/html/rfc3659#section-7

Stops accepting connections after a time.

We have a VERY simple ftp server that is based on the MemDriver example. All the server is trying to do is accept PUTs from an authorized user and then putting the content of that file onto a RabbitMQ. It seems to do perfectly at first and then over time (and after a lot of people/bots trying to poke it) it will get to the point that it wont accept any connects and doesn't out put anything to the terminal until I hit a key and then all at once the terminal is filled with the log below

For context the server name is "customFTP to RabbitMQ":

<...>
2021/04/21 19:59:55 04ddd99e3e71e101dd15 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 eb4bea5f0671a6b9422e < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 a59ed5afe828bd1a7e35 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 c051f443cf41e1951c17 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 c97bbf855411d0e2519c < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 1ea95c9b919f3fbb4ea2 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 4d6ae8213285278b6d40 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 bdaf9bc67ebc4b2a2192 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 7b9bc71471b5473cded1 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 519f3e86acee326fdfd1 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 84880d8355473fd48b9d < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 c8d93c4d73b1c9dafefd < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 69d79e6b9f6afdc4dc4a < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 1b14fed69e3d235d29c2 < 220 customFTP to RabbitMQ
2021/04/21 19:59:55 97db1e07f44308a7774b   Connection Terminated
2021/04/21 19:59:55 7c764a2d542b9ca7718d   Connection Terminated
2021/04/21 19:59:55 f64d2a10de15d43124bf   Connection Terminated
2021/04/21 19:59:55 18bbe7ab30353b8eb572   Connection Terminated
2021/04/21 19:59:55 38d1c35e88c49e9e8f89   Connection Terminated
2021/04/21 19:59:55 6a609514292e5325fc8a   Connection Terminated
2021/04/21 19:59:55 7ea2e9b97f7534697307   Connection Terminated
2021/04/21 19:59:55 99a39e0cd4a95cdf9dc8   Connection Terminated
2021/04/21 19:59:55 410928f066cd0d6d8d47   Connection Terminated
2021/04/21 19:59:55 41188211fa8be2ce8777   Connection Terminated
2021/04/21 19:59:55 f6f3ae623ca6d77fd010   Connection Terminated
2021/04/21 19:59:55 3e3a75de4d35c19bff56   Connection Terminated
2021/04/21 19:59:55 25b7ed6e6b381fd4b315   Connection Terminated
2021/04/21 19:59:55 9146fdda835b3ef85e6f   Connection Terminated
<...>

Would love to hear any ideas! Happy to add more info on request, but I checked and we seem to be handling all the Close() calls for the various connections and the rest of the system seems stable, so I don't think it's a memory leak. For other context this is running on a windows server, not sure if that makes a difference! Thanks!

Hosting Graval based FTP server in a Docker container on MacOS

So I've made a custom FTP server using graval and it works great on it's own, but I need it to run in a docker container. I have set the pasvMin/MaxPorts and then exposed those both in the image and using docker compose, however, the server/client keep trying to connect on ports outside of that range when I try to STOR a file from a standard ftp client. I have debugged enough to know that it is creating an active socket instead of a passive one for this type of command. Perhaps, it's my lack of understanding of FTP, but any ideas how I can get the server to force the use of a certain set of ports so I can make this work in a container/behind a firewall? I'm happy to make changes and a pull request, but would like guidance as to where/what the change would need to be made to make it work well for everyone, not just me. Thanks!

For reference, here is my server set up code:

	opts := &graval.FTPServerOpts{
		Factory:     factory,
		ServerName:  "customFTP",
		PasvMinPort: 60000,
		PasvMaxPort: 60100,
		Port:        21,
		Hostname:    "0.0.0.0",
	}
	ftpServer := graval.NewFTPServer(opts)

And here is my docker-compose file:

version: "3.1"

services:
  ftp:
    image: myreg/testing/go/customftp:latest
    environment:
      FTP_USER: test
      FTP_PASS: 1234
    ports:
      - "21:21"
      - "20:20"
      - "60000-60100:60000-60100"

Support TLS

Old school FTP is completely unencrypted, which isn't particularly acceptable in 2019.

There are extensions to the protocol that enable TLS encryption, and it might be nice to support them.

Two forks with TLS support that might have some useful ideas:

1. https://github.com/koofr/graval
2. https://github.com/goftp/server

Unable to connect to FTP

$ ./graval-mem

$ ftp 127.0.0.1
ftp: connect: Connection refused

Did I do something wrong?

File downloading should support streaming to avoid loading full files into memory

The driver contract currently includes the following method:

// returns - true if data was successfully sent
GetFile(string) (string, error)

The string return value is the contents of a file that should be sent to the client. This means the driver must load the full file into memory and then return it.

It'd be much nicer if we had a method signature like:

GetFile(string, io.Writer) bool

... or maybe:

WriteFile(string, io.Writer) bool

The driver would be responsible for writing the file to the client (via the io.Writer). When it's done, it returns a boolean to indicate success or error.

Here's a fork that's implemented this idea: pzduniak@cbd1943

This fork has implemented an alternative approach: https://github.com/koofr/graval/blob/master/ftpdriver.go#L63