drakkan / sftpgo

Full-featured and highly configurable SFTP, HTTP/S, FTP/S and WebDAV server - S3, Google Cloud Storage, Azure Blob

SFTPGo

Full-featured and highly configurable event-driven file transfer solution. Server protocols: SFTP, HTTP/S, FTP/S, WebDAV. Storage backends: local filesystem, encrypted local filesystem, S3 (compatible) Object Storage, Google Cloud Storage, Azure Blob Storage, other SFTP servers.

With SFTPGo you can leverage local and cloud storage backends for exchanging and storing files internally or with business partners using the same tools and processes you are already familiar with.

The WebAdmin UI allows to easily create and manage your users, folders, groups and other resources.

The WebClient UI allows end users to change their credentials, browse and manage their files in the browser and setup two-factor authentication which works with Microsoft Authenticator, Google Authenticator, Authy and other compatible apps.

Sponsors

We strongly believe in Open Source software model, so we decided to make SFTPGo available to everyone, but maintaining and evolving SFTPGo takes a lot of time and work. To make development and maintenance sustainable you should consider to support the project with a sponsorship.

The open source license grant you freedom but not assurance of help. So why would you rely on free software without support or any guarantee it will stay healthy and maintained for the upcoming years?

Supporting the project benefit businesses and the community because if the project is financially sustainable, using this business model, we don't have to restrict features and/or switch to an Open-core model. The technology stays truly open source. Everyone wins.

It is important to understand that you should support SFTPGo and any other Open Source project you rely on for ongoing maintenance, even if you don't have any questions or need new features, to mitigate the business risk of a project you depend on going unmaintained, with its security and development velocity implications.

We also provide commercial support.

Thank you to our sponsors

Platinum sponsors

Silver sponsors

Bronze sponsors

Support policy

You can use SFTPGo for free, respecting the obligations of the Open Source license, but please do not ask or expect free support as well.

Use discussions to ask questions and get support from the community.

If you report an invalid issue and/or ask for step-by-step support, your issue will be closed as invalid without further explanation and/or the "support request" label will be added. Invalid bug reports may confuse other users. Thanks for understanding.

Documentation

You can read more about supported features and documentation at docs.sftpgo.com.

Release Cadence

SFTPGo releases are feature-driven, we don't have a fixed time based schedule. As a rough estimate, you can expect 1 or 2 new major releases per year and several bug fix releases.

Acknowledgements

SFTPGo makes use of the third party libraries listed inside go.mod.

We are very grateful to all the people who contributed with ideas and/or pull requests.

Thank you to ysura for granting us stable access to a test AWS S3 account.

Thank you to KeenThemes for granting us a custom license to use their amazing Mega Bundle for SFTPGo UI.

Thank you to Crowdin for granting us an Open Source License.

Thank you to Incode for helping us to improve the UI/UX.

License

SFTPGo source code is licensed under the GNU AGPL-3.0-only.

The theme used in WebAdmin and WebClient user interfaces is proprietary, this means:

• KeenThemes HTML/CSS/JS components are allowed for use only within the SFTPGo product and restricted to be used in a resealable HTML template that can compete with KeenThemes products anyhow.
• The SFTPGo WebAdmin and WebClient user interfaces (HTML, CSS and JS components) based on this theme are allowed for use only within the SFTPGo product and therefore cannot be used in derivative works/products without an explicit grant from the SFTPGo Team.

More information about compliance.

Copyright

Copyright (C) 2019 Nicola Murino

sftpgo's Issues

API authentication

Is it possible to force authentication for the API endpoints?

error if trailing backslash in config-dir (Windows)

SFTPGo 0.9.5-2be3721-2020-01-12T13:56:20Z

In Windows, I'd expect to type paths with backslashes:
sftpgo.exe service install --config-dir="C:\ProgramData\SFTPGO\"
but it writes an error to C:\Program Files\SFTPGo\sftpgo.log unless I use forward slashes:
sftpgo.exe service install --config-dir="C:/ProgramData/SFTPGO/"

{"level":"info","time":"2020-03-01T21:57:32.547","sender":"service","connection_id":"","message":"starting SFTPGo 0.9.5-2be3721-2020-01-12T13:56:20Z, config dir: C:\\ProgramData\\SFTPGO\", config file: sftpgo, log max size: 10 log max backups: 5 log max age: 28 log verbose: true, log compress: false"}
{"level":"warn","time":"2020-03-01T21:57:32.550","sender":"config","connection_id":"","message":"error loading configuration file: Config File \"sftpgo\" Not Found in \"[C:\\\\ProgramData\\\\SFTPGO\\\" C:\\\\Program Files\\\\SFTPGo]\". Default configuration will be used: {SFTPD:{Banner:SFTPGo_0.9.5 BindPort:2022 BindAddress: IdleTimeout:15 MaxAuthTries:0 Umask:0022 UploadMode:0 Actions:{ExecuteOn:[] Command: HTTPNotificationURL:} Keys:[] IsSCPEnabled:false KexAlgorithms:[] Ciphers:[] MACs:[] LoginBannerFile: SetstatMode:0 EnabledSSHCommands:[md5sum sha1sum cd pwd]} ProviderConf:{Driver:sqlite Name:sftpgo.db Host: Port:5432 Username: Password:[redacted] SSLMode:0 ConnectionString: UsersTable:users ManageUsers:1 TrackQuota:1 PoolSize:0 UsersBaseDir: Actions:{ExecuteOn:[] Command: HTTPNotificationURL:} ExternalAuthProgram: ExternalAuthScope:0} HTTPDConfig:{BindPort:8080 BindAddress:127.0.0.1 TemplatesPath:templates StaticFilesPath:static BackupsPath:backups}}"}
{"level":"warn","time":"2020-03-01T21:57:32.551","sender":"sqlite","connection_id":"","message":"sqlite database file does not exists, please be sure to create and initialize a database before starting sftpgo"}
{"level":"error","time":"2020-03-01T21:57:32.551","sender":"service","connection_id":"","message":"error initializing data provider: CreateFile C:\\ProgramData\\SFTPGO\"\\sftpgo.db: The filename, directory name, or volume label syntax is incorrect."}

You'll also notice that the log file has way too many slashes in it. Looks like you're trying to escape characters using slashes, but that ends up printing them incorrectly to the log:
Config File \"sftpgo\" Not Found in \"[C:\\\\ProgramData\\\\SFTPGO\\\" C:\\\\Program Files\\\\SFTPGo]\".

Suggestion: add dependencies in vendor dir

It seems to be a common practice now in Goland to add your dependencies in the vendor/ dir. This would mean I (as a contributor) would not need to get my own copy of your dependencies, since they live inside your git tree.

Just a suggestion.

Take action on uploaded file

Hi there,

Thanks for this great library! Having something like this would have saved my team and I a bunch of time had it been around a year ago :)

One thing that we do that doesn't seem to be supported in this library is taking immediate action on a file that is added to the server. So when upload.csv gets added to a user's folder, we immediately pick it up, ship it off to a different server via an API, and delete the file (via a script that is pretty fragile).

Is something like that in scope for this project? I think it would be an amazing addition that could really pull this together as a fully managed solution.

Regardless of if it is possible or not, we plan to switch to this project in the near future :) Thank you for your work!!

initprovider missing in Windows build

SFTPGo 0.9.5-2be3721-2020-01-12T13:56:20Z

New user, I see this error in the log:

{"level":"warn","time":"2020-03-01T21:59:58.969","sender":"sqlite","connection_id":"","message":"sqlite database file does not exists, please be sure to create and initialize a database before starting sftpgo"}
{"level":"error","time":"2020-03-01T21:59:58.969","sender":"service","connection_id":"","message":"error initializing data provider: CreateFile C:\\ProgramData\\SFTPGO\\sftpgo.db: The system cannot find the file specified."}

The readme says "For SQLite provider the database file will be auto created if missing" so I thought I wouldn't need to run initprovider. Seeing the error though, I ran initprovider as the readme instructs, but the executable does not recognize initprovider.

C:\Program Files\SFTPGo>sftpgo initprovider
Error: unknown command "initprovider" for "sftpgo"
Run 'sftpgo --help' for usage.
unknown command "initprovider" for "sftpgo"

C:\Program Files\SFTPGo>sftpgo --help
Full featured and highly configurable SFTP server

Usage:
  sftpgo [command]

Available Commands:
  help        Help about any command
  portable    Serve a single directory
  serve       Start the SFTP Server
  service     Install, Uninstall, Start, Stop and retrieve status for SFTPGo Windows Service

Flags:
  -h, --help      help for sftpgo
  -v, --version

Use "sftpgo [command] --help" for more information about a command.

I can't create the database so I'm stuck for now.

Question: Authenticating via external script command?

Hello!

I'm trying to integrate SFTP access into an existing application which has its own user accounts and roles defined internally. Because the main application is in PHP and uses the built-in argon2i password hashing method, its passwords are hashed in a slightly different format than the one sftpgo expects, so I can't just point this app at the same database that the web application is using.

What I'm curious to find out is whether authentication via invoking a script is possible in the current version of this application. I see a number of built-in locations where hooks can be called to either programs or external URLs, including one in the data_providers section, but it doesn't appear that this hook can actually handle authentication.

Given this app already has its own internal REST API, my current plan is just to use either that REST API or a small SQLite database to sync up the web application's user accounts and the sftpgo accounts periodically.

Is that the current best solution for this kind of setup? Thanks in advance for your assistance, and a massive kudos to the team who built this wonderful application, as it solves a number of problems I've been having with my existing FTP implementation.

Fail2ban integration: Adding the source IP address in logs

Hello Drakkan,

I was thinking about fail2ban integration with sftpgo but unfortunatly, they is no the source IP address in logs.
I notify two cases :

When user does not exist:
{"level":"warn","sender":"sqlite","connection_id":"","time":"2019-11-10T20:09.25.756","message":"error authenticating user: toto, error: Not found: sql: no rows in result set"}

When user exist but password failed:
{"level":"warn","sender":"sftpd","connection_id":"","time":"2019-11-10T20:10.08.775","message":"failed to accept an incoming connection: [ssh: no auth passed yet, could not validate credentials, could not validate credentials, could not validate credentials, could not validate credentials, could not validate credentials]"}
In this second case, user is not mentioned in the log. Is it normal ?

I'm not developer, but if someone add IP in logs, i'll make the fail2ban configuration :)
Regards,

Feature request: add "Denied" to sub-directory permission

Maybe I've overlooked something, but it seems like there is no way to block a user form accessing a sub-directory.

Currently I'm using only listing as permission to prevent download, but user can still see the content.

portable/User Mode

Hi,
it would be cool, if your sftpgo server would have the following feature, a portable mode.
a user can start sftpgo, perhaps with --portable as parameter, but has at least to provide the parameter directory, where he has access.
at most he has to deliver the parameters

• password (could be generated by sftpgo as well, so optional)
• user (could be optional)
• port (has to be a non privileged port)

and this should stop when the user stops/exits the program.

Additionally (especially for mobile users) you could output an qrcode which has the sftp-url with user, port, ip/name as qr-code, perhaps as mode even with password, so mobile users could just use the qrcode to create an connection.

i would be willing to try helping with it, but i am a total newbie in go, though i wrote programs in python and perl back in the day.

quota_files crashes sftpgo

When I create a user with the quota_files parameter set to 5 for example and then try to upload any file after the quota is reached it will crash sftpgo.

Here is the crash log from syslog:

Jul 26 11:50:38 raspberrypi systemd[1]: Started SFTPGo sftp server.
Jul 26 11:50:53 raspberrypi sftpgo[12844]: panic: runtime error: invalid memory address or nil pointer dereference
Jul 26 11:50:53 raspberrypi sftpgo[12844]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x43be40]
Jul 26 11:50:53 raspberrypi sftpgo[12844]: goroutine 86 [running]:
Jul 26 11:50:53 raspberrypi sftpgo[12844]: github.com/drakkan/sftpgo/dataprovider.GetUsedQuota(...)
Jul 26 11:50:53 raspberrypi sftpgo[12844]: #011/root/go/src/github.com/drakkan/sftpgo/dataprovider/dataprovider.go:132
Jul 26 11:50:53 raspberrypi sftpgo[12844]: github.com/drakkan/sftpgo/sftpd.Connection.hasSpace(0x286e540, 0x40, 0x8, 0x0, 0x28885d8, 0x4, 0x29ca2a0, 0x61, 0x0, 0x0, ...)
Jul 26 11:50:53 raspberrypi sftpgo[12844]: #011/root/go/src/github.com/drakkan/sftpgo/sftpd/handler.go:404 +0x380
Jul 26 11:50:53 raspberrypi sftpgo[12844]: github.com/drakkan/sftpgo/sftpd.Connection.Filewrite(0x286e540, 0x40, 0x8, 0x0, 0x28885d8, 0x4, 0x29ca2a0, 0x61, 0x0, 0x0, ...)
Jul 26 11:50:53 raspberrypi sftpgo[12844]: #011/root/go/src/github.com/drakkan/sftpgo/sftpd/handler.go:98 +0x984
Jul 26 11:50:53 raspberrypi sftpgo[12844]: github.com/pkg/sftp.(*Request).open(0x2932060, 0x66f540, 0x29b0240, 0x66f558, 0x29b0300, 0x66f510, 0x29b03c0, 0x66f528, 0x29b0480, 0x6715e0, ...)
Jul 26 11:50:53 raspberrypi sftpgo[12844]: #011/root/go/src/github.com/pkg/sftp/request.go:180 +0x114
Jul 26 11:50:53 raspberrypi sftpgo[12844]: github.com/pkg/sftp.(*RequestServer).packetWorker(0x2942500, 0x673520, 0x2902a60, 0x286e5c0, 0x0, 0x0)
Jul 26 11:50:53 raspberrypi sftpgo[12844]: #011/root/go/src/github.com/pkg/sftp/request-server.go:167 +0x4b8
Jul 26 11:50:53 raspberrypi sftpgo[12844]: github.com/pkg/sftp.(*RequestServer).Serve.func1.1(0x2888630, 0x2942500, 0x673520, 0x2902a60, 0x286e5c0)
Jul 26 11:50:53 raspberrypi sftpgo[12844]: #011/root/go/src/github.com/pkg/sftp/request-server.go:98 +0x58
Jul 26 11:50:53 raspberrypi sftpgo[12844]: created by github.com/pkg/sftp.(*RequestServer).Serve.func1
Jul 26 11:50:53 raspberrypi sftpgo[12844]: #011/root/go/src/github.com/pkg/sftp/request-server.go:96 +0x7c
Jul 26 11:50:53 raspberrypi systemd[1]: sftpgo.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 26 11:50:53 raspberrypi systemd[1]: sftpgo.service: Failed with result 'exit-code'.

Switch to kingpin?

For argument parsing in my golang projects, I use kingpin. Would you be interested in a PR for switching to this library?

zombie processes

I'm using the actions configuration to run a callback upon uploads and it appears the process is not being properly cleaned up. I was running it as pid 1, but switched to tini to see if that would help, but it doesn't.

I'm running this in docker and/or kubernetes inside an alpine image.

Here is an example of ps after I upload two files. The tester is the command that is run in the action. You can see the stat is set to Z.

~ $ ps -o pid,vsz,rss,tty,stat,time,ruser,args
PID   VSZ  RSS  TT     STAT TIME  RUSER    COMMAND
    1  768    4 ?      S     0:00 sftp     /sbin/tini -- /app/sftpgo serve
    6 248m 145m ?      S     0:01 sftp     /app/sftpgo serve
   13 1628  992 136,0  S     0:00 sftp     ash
   27    0    0 ?      Z     0:00 sftp     [tester]
   29    0    0 ?      Z     0:00 sftp     [tester]
   30 1556    4 136,0  R     0:00 sftp     ps -o pid,vsz,rss,tty,stat,time,ruser,args

Here is the Dockerfile

FROM golang:1.12-alpine3.10 as sftpgo
RUN apk add --no-cache git gcc g++ ca-certificates \
  && go get -u github.com/drakkan/sftpgo

FROM alpine:3.10
RUN  apk add --no-cache openssh-keygen curl tini \
  && adduser sftp -h /app -D \
  && mkdir -p /data /app/log \
  && ln -sf /dev/stdout /app/log/sftpgo.log \
  && ssh-keygen -t rsa -N "" -f /app/id_rsa \
  && chown sftp /data /app/id_rsa /app/id_rsa.pub /app/log
COPY --from=sftpgo /go/bin/sftpgo /app
COPY . /app
RUN chown sftp /app/sftpgo.db
WORKDIR /app
USER sftp
EXPOSE 2022
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["/app/sftpgo","serve"]

config

{
  "sftpd": {
    "bind_port": 2022,
    "bind_address": "",
    "idle_timeout": 15,
    "max_auth_tries": 0,
    "umask": "0022",
    "upload_mode": 1,
    "banner": "tester sftp",
    "actions": {
      "execute_on": ["upload"],
      "command": "/app/tester",
      "http_notification_url": ""
    },
    "keys": []
  },
  "data_provider": {
    "driver": "sqlite",
    "name": "sftpgo.db",
    "host": "",
    "port": 5432,
    "username": "",
    "password": "",
    "sslmode": 0,
    "connection_string": "",
    "users_table": "users",
    "manage_users": 1,
    "track_quota": 0
  },
  "httpd": {
    "bind_port": 8080,
    "bind_address": "127.0.0.1"
  }
}

action script

#!/bin/ash

ACTION=$1
USERNAME=$2
PATH=$3
TARGET_PATH=$4

echo "GOT $1 for $2 @ $3"

I haven't done much in golang to be useful here for submitting a PR.

However, I did fine some related links.

https://stackoverflow.com/questions/36050503/golang-child-processes-become-zombies
https://github.com/ramr/go-reaper

Let me know if you want some help in this and I can give it a shot.

Occasional Malfunction of Connection

Occasionally (about once every 12-18 hours) the server will stop accepting the correct username and password. It requires a service reboot to fix. This is the closest set of logs to the latest attempt to login while in this error state:

{"level":"debug","sender":"sftpd","connection_id":"","time":"2019-09-11T03:27.05.832","message":"idle connections check ticker 2019-09-11 03:27:05.832582724 +0000 UTC m=+44100.001673267"}
{"level":"info","sender":"sftpd","connection_id":"6508f97ceb9b67f8e7fc4928cd9b0cd10615a47024a8279611e9e8f3f6ef24b4","time":"2019-09-11T03:27.05.832","message":"close idle connection, idle time: 6h27m4.548987039s"}
{"level":"warn","sender":"sftpd","connection_id":"6508f97ceb9b67f8e7fc4928cd9b0cd10615a47024a8279611e9e8f3f6ef24b4","time":"2019-09-11T03:27.05.832","message":"idle connection close failed: close tcp 172.31.18.230:22->148.59.44.16:64168: use of closed network connection"}
{"level":"info","sender":"sftpd","connection_id":"8fe250f6489ada67fcb4161871743f4358892324ce8841dcbe4903eabfcc2a59","time":"2019-09-11T03:27.05.832","message":"close idle connection, idle time: 6h27m4.535060735s"}
{"level":"warn","sender":"sftpd","connection_id":"8fe250f6489ada67fcb4161871743f4358892324ce8841dcbe4903eabfcc2a59","time":"2019-09-11T03:27.05.832","message":"idle connection close failed: close tcp 172.31.18.230:22->148.59.44.16:64169: use of closed network connection"}
{"level":"debug","sender":"sftpd","connection_id":"","time":"2019-09-11T03:27.05.832","message":"check idle connections ended"}

Any ideas? Is there any way to stop it trying to close idle connections? Perhaps that's the problem?

Switch to viper?

Similar as my kingpin suggestion, viper is a great tool for configuration loading. It supports multiple formats out of the box, can go through multiple loading locations (eg. [$HOME, "/etc/sftpgo/"]) and has numerous interesting features.

I would also suggest to use sftpgo.json instead of sftpgo.conf for the json-formatted configuration file - my editor would be grateful :-)

Support simple config file as data provider (for users definition)

Hi,
You have started a very nice project ^_^
I would like to run SFTPGo in a higly secure environment or at least ask security team for the permission ^^'

I'm looking for a file server (protocol can be sftp, or else) with virtual user / non unix user, with everything in immutable configuration file.
Config can be generated by a config manager like chef, ansible watever.
You already have conf key to disable web interface etc.
So, I suggest a config file as data provider.

Have a nice day !

Feature suggest: Public key in file

Hi

When giving the user a uid/guid it would also be nice to have the option to use a file as public key repository.

Just like normal users have it.

Console messages

sftpgo should output to console some informational messages about what it is doing. Errors such as not being able to load config file should be output to console aswell as log file.

httpd ignoring conf file?

Hello,

My conf file is very similar to your example, except I wanted to test out the httpd API and made the changes below.

   "httpd":{
       "bind_port":8000,
           "bind_address": ""
   }

1. Even though the port has changed to 8000, nothing answers there and still wants to answer on 8080. Tested with nc on the localhost and remote host.

2. Even though I have to listening on all available addresses, nothing answers on remote hosts. Tested with a webbrowser and nc on two separate hosts.

Am I doing something wrong? A missing comma, period, quotes?

Thanks!

portable mode sftpgo.db error

I build sftpgo docker image with alpine instructions.

I am trying to start server with sftpgo portable, but it throws ERR error initializing data provider: stat sftpgo.db: no such file or directory error all the time.

I investigated the code and portable mode selects memory data provider. Unfortunately I could not find the bug.

crash when creating private key

SFTPGo 0.9.5-2be3721-2020-01-12T13:56:20Z
Windows 10

I found a db file at C:\Program Files\SFTPGo\sftpgo.db, so I copied it to my config-dir and tried to start the windows service but the logs show the service keeps restarting when it gets to "creating new private key for server"

{"level":"info","time":"2020-03-01T22:18:11.303","sender":"service","connection_id":"","message":"starting SFTPGo 0.9.5-2be3721-2020-01-12T13:56:20Z, config dir: C:/ProgramData/SFTPGO/, config file: sftpgo, log max size: 10 log max backups: 5 log max age: 28 log verbose: true, log compress: false"}
{"level":"debug","time":"2020-03-01T22:18:11.305","sender":"config","connection_id":"","message":"config file used: 'C:\\ProgramData\\SFTPGO\\sftpgo.json', config loaded: {SFTPD:{Banner:Private Server BindPort:2022 BindAddress: IdleTimeout:15 MaxAuthTries:5 Umask:0022 UploadMode:1 Actions:{ExecuteOn:[] Command: HTTPNotificationURL:} Keys:[] IsSCPEnabled:false KexAlgorithms:[] Ciphers:[] MACs:[] LoginBannerFile: SetstatMode:0 EnabledSSHCommands:[md5sum sha1sum cd pwd]} ProviderConf:{Driver:sqlite Name:sftpgo.db Host: Port:999 Username: Password:[redacted] SSLMode:0 ConnectionString: UsersTable:users ManageUsers:1 TrackQuota:0 PoolSize:0 UsersBaseDir: Actions:{ExecuteOn:[] Command: HTTPNotificationURL:} ExternalAuthProgram: ExternalAuthScope:0} HTTPDConfig:{BindPort:8081 BindAddress: TemplatesPath:templates StaticFilesPath:static BackupsPath:backups}}"}
{"level":"debug","time":"2020-03-01T22:18:11.305","sender":"sqlite","connection_id":"","message":"sqlite database handle created, connection string: \"file:C:\\\\ProgramData\\\\SFTPGO\\\\sftpgo.db?cache=shared\""}
{"level":"debug","time":"2020-03-01T22:18:11.306","sender":"httpd","connection_id":"","message":"initializing HTTP server with config {BindPort:8081 BindAddress: TemplatesPath:templates StaticFilesPath:static BackupsPath:backups}"}
{"level":"debug","time":"2020-03-01T22:18:11.306","sender":"service","connection_id":"","message":"initializing SFTP server with config {Banner:Private Server BindPort:2022 BindAddress: IdleTimeout:15 MaxAuthTries:5 Umask:0022 UploadMode:1 Actions:{ExecuteOn:[] Command: HTTPNotificationURL:} Keys:[] IsSCPEnabled:false KexAlgorithms:[] Ciphers:[] MACs:[] LoginBannerFile: SetstatMode:0 EnabledSSHCommands:[md5sum sha1sum cd pwd]}"}
{"level":"debug","time":"2020-03-01T22:18:11.306","sender":"utils","connection_id":"","message":"umask not available on windows, configured value 0022 (18)"}
{"level":"info","time":"2020-03-01T22:18:11.306","sender":"sftpd","connection_id":"","message":"No host keys configured and \"C:\\\\ProgramData\\\\SFTPGO\\\\id_rsa\" does not exist; creating new private key for server"}

It just keeps repeating those lines over and over when the service restarts itself. It would help if it wrote better error messages in the sftpgo log file.

This unhelpful error message appears in the Windows Event Log called "System".

The SFTPGo service terminated with the following service-specific error: 
Incorrect function.

It would be nice if that Event Log error was also more helpful. Not sure if you have any control over that though, unless you can catch the exception and rethrow it with a more helpful "message" property.

Using goreleaser for binary releases

Hey there,

great project. What do you think about using goreleaser (https://goreleaser.com/) for binary releases?
I used it in some projects and it was a great tool for binary releases.

See https://github.com/sandreas/graft/blob/master/.goreleaser.yml for an example.

sandreas

Create a CLI

The REST API is great, of course, but a more intuitive CLI would also be awesome.

Two options:

• build a separate tool (in any language, but I personally would prefer golang)
• make the daemon also the CLI (see eg. docker, vault and nomad)

In the first scenario, you will need to keep both projects in sync where relevant.

In the second case, you should use a specific "command" argument to run the server, so you don't accidentally run it and create all kinds of default files (log, id_rsa)

eg.:

$ sftpgo serve --config-dir=/etc/sftpgo/
$ sftpgo user create foo --uid=1000 --homedir=/var/lib/sftpgo/foo --keys=@/path/to/authorized_keys --no-password

Environmental variable configuration for data_provider not working (user error?)

Is this the correct syntax? When I do not provide a configuration file and instead set these env variables it doesn't seem to be applying those values.

SFTPGO_DATA_PROVIDER__DRIVER=mysql
SFTPGO_DATA_PROVIDER__NAME=mydb
SFTPGO_DATA_PROVIDER__HOST=127.0.0.1
SFTPGO_DATA_PROVIDER__PORT=330

sftp    | {"level":"info","sender":"cmd","time":"2019-09-03T20:47.33.437","message":"starting SFTPGo, config dir: ., config file: sftpgo, log max size: 10 log max backups: 0 log max age: 0 log verbose: true, log compress: false"}
sftp    | {"level":"warn","sender":"config","time":"2019-09-03T20:47.33.438","message":"error loading configuration file: Config File \"sftpgo\" Not Found in \"[/app/.config/sftpgo /etc/sftpgo /app]\". Default configuration will be used: {SFTPD:{Banner:SFTPGo BindPort:2022 BindAddress: IdleTimeout:15 MaxAuthTries:0 Umask:0022 UploadMode:0 Actions:{ExecuteOn:[] Command: HTTPNotificationURL:} Keys:[] IsSCPEnabled:false} ProviderConf:{Driver:sqlite Name:sftpgo.db Host: Port:5432 Username: Password: SSLMode:0 ConnectionString: UsersTable:users ManageUsers:1 TrackQuota:1} HTTPDConfig:{BindPort:8080 BindAddress:127.0.0.1}}"}
sftp    | 2019-09-03T20:47.33.438 WRN error loading configuration file: Config File "sftpgo" Not Found in "[/app/.config/sftpgo /etc/sftpgo /app]". Default configuration will be used.
sftp    | {"level":"warn","sender":"dataProvider","time":"2019-09-03T20:47.33.438","message":"sqlite database file does not exists, please be sure to create and initialize a database before starting sftpgo"}
sftp    | {"level":"error","sender":"cmd","time":"2019-09-03T20:47.33.438","message":"error initializing data provider: stat sftpgo.db: no such file or directory"}
sftp    | 2019-09-03T20:47.33.438 ERR error initializing data provider: stat sftpgo.db: no such file or directory
sftp exited with code 1

Feature request: Keyboard interactive auth

Hi, first of all I'd like to thank you for this amazing project. I was looking for such a simple and hackable sftp server for a very long time. Great work!

Are you interested in implementing Keyboard Interactive Authentication? It could be used to perform multi factor authentication. I think it's necessary to allow to specify custom executables in this case in order to do dynamic questions / answers handling (similar to how external auth program works now).

unmarshal error for field Language of type disconnectMsg

After updating to the latest version I get this error message when attempting to connect:

unmarshal error for field Language of type disconnectMsg

And then it times out. Any ideas?

Support for rsync

Since you now support scp, I think rsync would be a logical next step :-)

can't login with public key after updating to the latest code and updating database

I just updated my test setup to the latest master, and ran the sql update line:

ALTER TABLE "users" RENAME COLUMN "public_key" TO "public_keys";

Now I can no longer login with the existing keys.

I reverted to the commit 5ad222fc53c01339b2acecd6849555b7006825b1: it works.
I cherry-pick 5ad222fc53c01339b2acecd6849555b7006825b1 and run the SQL line: it no longer works.

Should I change something else?

Better user tracking in the logs

I have a few thoughts/requests, I can make separate issues if you think that would be better.

a) It would be great to be able to identify connecting users from the logs by the fingerprint and the comment (the last part) of the public key. Now we only see the user name.

b) I only see the user name when the user does actions. Is the user not authenticated earlier?

c) I think the logs should use the connection_id to be able to trace connections/actions.

d) Somtimes paths are quoted in the logs, sometimes not; this should obviously be consistent :-)
"requested list/stat" entries are not quoted
"fileread requested" is quoted

What I see now:

{"level":"debug","sender":"sftpd","time":"2019-09-05T10:03.56.827","message":"accepted inbound connection, ip: 10.10.10.1:4727"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:03.56.829","message":"connection added, num open connections: 1"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:03.58.496","message":"requested list file for dir: /var/ftp/myuser user: myuser"} 
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.01.497","message":"requested stat for file: /var/ftp/myuser/in user: myuser"} 
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.02.910","message":"requested list file for dir: /var/ftp/myuser/in user: myuser"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.05.315","message":"requested stat for file: /var/ftp/myuser/in/somefile.xlsx user: myuser"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.11.706","message":"fileread requested for path: \"/var/ftp/myuser/in/somefile.xlsx\", user: myuser"}
{"level":"info","sender":"Download","elapsed_ms":57,"size_bytes":704072,"username":"myuser","file_path":"/var/ftp/myuser/in/somefile.xlsx","connection_id":            
"77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","protocol":"SFTP","time":"2019-09-05T10:04.11.763"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.13.278","message":"connection closed, id: 8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.13.279","message":"connection removed, num open connections: 0"}

What I expect to see (more or less):

{"level":"debug","sender":"sftpd","time":"2019-09-05T10:03.56.827","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"accepted inbound connection, ip: 10.10.10.1:4727"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:03.56.829","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"connection added, num open connections: 1"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:03.56.829","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"user authenticated with public key \"jo@desktop\" fingerprint:SHA256:FV3+wlAKGzYy7+J02786fh8N8c06+jga/mdiSOSPT7g"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:03.58.496","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"requested list file for dir: \"/var/ftp/myuser\" user: myuser"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.01.497","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"requested stat for file: \"/var/ftp/myuser/in\" user: myuser"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.02.910","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"requested list file for dir: \"/var/ftp/myuser/in\" user: myuser"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.05.315","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"requested stat for file: \"/var/ftp/myuser/in/somefile.xlsx\" user: myuser"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.11.706","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"fileread requested for path: \"/var/ftp/myuser/in/somefile.xlsx\", user: myuser"}
{"level":"info","sender":"Download","elapsed_ms":57,"size_bytes":704072,"username":"myuser","file_path":"/var/ftp/myuser/in/somefile.xlsx","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","protocol":"SFTP","time":"2019-09-05T10:04.11.763"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.13.278","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"connection closed"}
{"level":"debug","sender":"sftpd","time":"2019-09-05T10:04.13.279","connection_id":"8a329a9d77c9b078117ddc27b61654d50ed54b372f41b14ae6155c42bfe30985","message":"connection removed, num open connections: 0"}

Feature suggest: Let users be disabled and expire

Hi

It would be great to have users to be able to be disabled and expire.

Disabled:
Users in UNIX can be locked. This option doesn't seems to be implemented here.

Expire:
If we can ulitize the actions and http notification webhook to tell the datastore when a user was last used, we can utilise the API to disable/lock user after a certain time, if no last used variable was to be implemeted.

In UNIX we can use the lastlogin command to see when a user has last had it's successful connection.

Include SystemD init file in release

I'm building a pipeline to release an RPM based on your versioned releases (thanks for that!); I'm missing the SystemD init file. I believe this should be included as well?

Port 22 already in use

When attempting to start the server I get this error:

2019-07-31T15:34.16.762 ERR could not start SFTP server: listen tcp 0.0.0.0:22: bind: address already in use

How can I expose the SFTP server on port 22 when ssh is already running on that port?

Some issues with rclone

Hi,

I'm trying to use rclone but ran into some issues.

Path with space and symbol not correctly handled by checksum commands

For example:
/music/DG/C.kleiber Originals Box-dg Recordings(SHM-CD)/Brahms Symphony No.4 Carlos Kreiber Vienna Philharmonic (SHM-CD)/albumart.pamp

rclone generate command:
sftp cmd = /music/DG/C.kleiber\ Originals\ Box-dg\ Recordings\(SHM-CD\)/Brahms\ Symphony\ No.4\ \ Carlos\ Kreiber\ \ Vienna\ Philharmonic\ \(SHM-CD\)/albumart.pamp

In sftpgo's log:

{"level":"debug","time":"2020-02-14T09:47:08.656","sender":"ssh","connection_id":"131d0931bfd5a9069d26124580e4442fb4bdc4fe9d7a044370f414babe2ed0f0","message":"new ssh command: \"md5sum\" args: [/music/DG/C.kleiber Originals Box-dg Recordings (SHM-CD )/Brahms Symphony No.4  Carlos Kreiber  Vienna Philharmonic  (SHM-CD )/albumart.pamp] user: ftp_public, error: <nil>"}
{"level":"warn","time":"2020-02-14T09:47:08.656","sender":"ssh","connection_id":"131d0931bfd5a9069d26124580e4442fb4bdc4fe9d7a044370f414babe2ed0f0","message":"command failed: \"md5sum\" args: [/music/DG/C.kleiber Originals Box-dg Recordings (SHM-CD )/Brahms Symphony No.4  Carlos Kreiber  Vienna Philharmonic  (SHM-CD )/albumart.pamp] user: ftp_public err: open /media/public/music/DG/C.kleiber Originals Box-dg Recordings (SHM-CD )/Brahms Symphony No.4  Carlos Kreiber  Vienna Philharmonic  (SHM-CD )/albumart.pamp: no such file or directory"}

It seems like escape of symbols, eg.\( is also translated into space. It's also reproducible by manually enter md5sum command.

Transfer error (SSH_FX_FAILURE)

Not much information provided by the log.
rclone:

2020/02/14 09:46:53 DEBUG : C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac: multi-thread copy: stream 1/2 (0-171573248) size 163.625M finished
2020/02/14 09:46:53 DEBUG : C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac: multi-thread copy: stream 1/2 failed: sftp: "incomplete download: 173670400/343043570 bytes transferred" (SSH_FX_FAILURE)
2020/02/14 09:46:53 DEBUG : C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac: multi-thread copy: stream 2/2 failed: context canceled
2020/02/14 09:46:53 ERROR : C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac: Failed to copy: sftp: "incomplete download: 173670400/343043570 bytes transferred" (SSH_FX_FAILURE)

sftpgo:

{"level":"debug","time":"2020-02-14T10:45:14.386","sender":"sftpd","connection_id":"b1adc368547b89c4a788affb0d92d392bd090d9341b1a39930f511851e597989","message":"fileread requested for path: \"/media/public/music/DG/C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac\""}
{"level":"debug","time":"2020-02-14T10:45:14.386","sender":"sftpd","connection_id":"73148db23ebefb1ad78cb41e43b302281a2d7813f5909e5d8f18bb1befae4901","message":"fileread requested for path: \"/media/public/music/DG/C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac\""}
{"level":"debug","time":"2020-02-14T10:45:14.387","sender":"sftpd","connection_id":"b1adc368547b89c4a788affb0d92d392bd090d9341b1a39930f511851e597989","message":"requested stat for path: \"/media/public/music/DG/C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac\""}
{"level":"debug","time":"2020-02-14T10:45:14.388","sender":"sftpd","connection_id":"73148db23ebefb1ad78cb41e43b302281a2d7813f5909e5d8f18bb1befae4901","message":"requested stat for path: \"/media/public/music/DG/C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac\""}
{"level":"warn","time":"2020-02-14T10:45:35.825","sender":"sftpd","connection_id":"73148db23ebefb1ad78cb41e43b302281a2d7813f5909e5d8f18bb1befae4901","message":"transfer error: incomplete download: 173670400/343043570 bytes transferred, path: \"/media/public/music/DG/C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac\""}
{"level":"warn","time":"2020-02-14T10:45:36.190","sender":"sftpd","connection_id":"b1adc368547b89c4a788affb0d92d392bd090d9341b1a39930f511851e597989","message":"transfer error: incomplete download: 161316864/343043570 bytes transferred, path: \"/media/public/music/DG/C.kleiber Originals Box-dg Recordings(SHM-CD)/Beethoven Symphonies Nos.5, 7  Carlos Kleiber  Wiener Philharmoniker (SHM-CD)/Carlos Kleiber  Wiener Philharmoniker - Beethoven - Symphonies No. 5 & 7.flac\""}

Support for FTPs

Many SFTP servers support the FTPS protocol too. To ensure compatibility if someone wants to use SFTPgo instead of their FTP server, it would be wise to support FTPs as well.

error building with latest go 1.14

Hello, there are errors building with latest go 1.14:

git pull

remote: Enumerating objects: 62, done.
remote: Counting objects: 100% (62/62), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 35 (delta 29), reused 35 (delta 29), pack-reused 0
Unpacking objects: 100% (35/35), done.
From https://github.com/drakkan/sftpgo
7163fde..833b702 master -> origin/master
Updating 7163fde..833b702

go build

github.com/drakkan/sftpgo/sftpd

sftpd/ssh_cmd.go:279:20: undefined: errors.Is
note: module requires Go 1.13

Rest API via Postman or Curl

Hi,

I used the python rest client and was successful in creating users.

Is it possible to use the rest api via curl or postman?

I tried calling curl http://localhost:8080/add_user?username=test_username&password=test_pwd&home_dir=/tmp/test_home_dir&uid=33&gid=1000&max_sessions=2&quota_size=0&quota_files=3&permissions='*'&upload_bandwidth=100&download_bandwidth=60

but got an error {"error":"","message":"Not Found","status":404}

Thanks,

Ajay

Logging with public keys

Hi drakkan,

Would you have an example of how to create a user with his public key?

Maybe using the python client.

Thanks,

Ajay

Ability to choose the log processing

Hi,

I have dockerized SFTPgo service with a custom systemD script management.
It's working but i must choose a location path for the logfile with --log-file-path argument and i must have to mount a volume only for this. too bad...

Is it possible to have a parameter to change logger process and publish all logs to stdout/stderr ? and avoid to specify a log path and log file ?
With systemD and LogIdentifier parameters, it will be easy to processing logs throuth journalctl and syslog.
I can submit Dockerfile and custom SystemD script with a pull request if you want.

Thanks in advance

Log version when starting

After I start the sftpgo daemon, I expect it to log it's version, similar to the output from sftpgo --version.

suggestion: Windows Service restart behavior

SFTPGo 0.9.5-2be3721-2020-01-12T13:56:20Z
Windows

When you run sftpgo.exe service install, it creates a Windows service with a restart behavior of "always restart". I suggest you change that default behavior to only restart twice, and then if it fails a third time, let it stay stopped. The other Issue I created demonstrates a reason for this: it keeps restarting over and over, and crashes every time. It would be better if it gave up and quit wasting resources. It's doubtful that a service that crashes twice in a row will be successful the third time, so please don't tell it to restart infinitely, just twice.

Feature request: S3 Backend

It would be very convenient to have the ability to use the credentials: accesskey & secretkey of an s3 storage and then connect to the storage itself as backend.

Detect errored upload ?

I have a use case where I want detect broken upload.
I mean i send a file via scp I get the action. this is fine.
I send a file via scp I kill scp. Today the server will fire an action the same way.

Is there any way to detect those ?

Support with S3 integration

I have some questions about the s3 configuration parameters listed in the doc.

• s3 endpoint is always required? if it is, you are referring to an s3 access point?

Any clients ?

A golang cli client ?
Can build a GUI client on top of the cli client too I think.

Let me know what's lacking .. :)

Connecting with WinScp

Hi,

Sftpgo is running fine on my ubuntu server. (the log is fine)

I have manually inserted a user with a password in the sqlite database.
While connecting via WinScp it throws a authentication error.
Hope I am doing the right steps?

Thanks,

Ajay

Restricting SFTP/SCP algorithms for security reasons

Hello,

Very great project.
Is it possible to add feature of restricting the use of SFTP/SCP Algorithms such as Compression algorithms, Key exchanges, ciphers, or MACs for security reasons ?
A white list to choose algorithms

Thanks

Relatively lower performance than OpenSSH

Hi,
Thanks for this great project !

I did some test in my environment and the transfer speed is much lower than OpenSSH.

Server
OS	Debian 10.2 x64
CPU	Ryzen5 3600
RAM	64GB ECC
Disk	3* Intel P4510 4TB RAID0
Ethernet	Mellanox ConnectX-3 40GbE

Client
OS	Windows 10 1909 x64
CPU	Threadripper 1920X
RAM	64GB ECC
Disk	Samsung 960EVO 1TB
Ethernet	Mellanox ConnectX-3 40GbE

Under Filezilla I can get 500MB/s with OpenSSH, but only about 200MB/s with sftpgo.

In both case I'm using AES256-CTR as cipher and SHA-256 as MAC, I've also tried AES128-CTR but nothing changes.

CPU usage of sftpgo is higher than OpenSSH:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND 
 4527 sftp      20   0 1795576  52044   8628 R 133.5   0.6   2:12.13 sftpgo 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND 
27934 xxxxxx    20   0   17112   5360   4188 R  67.8   0.1   0:10.01 sshd                                                  
27942 xxxxxx    20   0   17112   5344   4176 R  27.4   0.1   0:12.52 sshd 

In both case I've got a maximum TCP window size of 4MB.

Support for Logging to stdout/stderr

Hello!

We're using SFTPGo in a Docker environment, and while the flatfile logging option isn't particularly difficult for us to work around (using other tools like Dockerize), it's always much easier if we can just pass output directly to stdout/stderr, where the Docker stack can manage it according to its own log rotation rules.

Would this be rather straightforward to support in the app?

Support multiple public keys per account

It would be useful to support multiple public keys per account. Similar to authorized_keys.

Dockerfile

I am trying to Dockerize the app and trying this:

FROM golang:1.12.7-stretch
RUN mkdir /app 
ADD . /app/ 
WORKDIR /app

RUN apt-get update
RUN apt-get install -yq git-all
RUN go get -u github.com/drakkan/sftpgo
CMD ["sftpgo -config-dir /app"]

However, when I run it with:

docker run -p 8080:8080 sftpgo

I get:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"sftpgo -config-dir /app\": stat sftpgo -config-dir /app: no such file or directory": unknown.

Any ideas?

Feature suggest: Let the API create a UNIX user

Hi

Can it be possible (and it's a huge security risk) to have the API to create a new UNIX user?

It's useful for when you want to use the uid/guid parameters to have more granular control of the folders in the users home folder.