handl3r / proxy-fileserver

Proxy file server integrates with google drive and local file system

Proxy file server

Proxy file server integrates with Google Drive for caching file with high frequency download

Instruction Build and Run

Requirement: Git, golang(version > 1.12), docker, openssl, windows or linux os

1. Generate key pair rsa

• Command

openssl genrsa -out private.pem 512
openssl rsa -in private.pem -outform PEM -pubout -out public.pem

You can use 1024,... for length of key instead of 512, but can not small then 512 Long key make long token

• Output:
  • private.pem is secret key use for your website to generate token or use for the proxy server generate token by api
  • public.pem is key use for proxy for validating token purpose

2. Build

• In your project folder :

git clone https://github.com/handl3r/proxy-fileserver
cd proxy-fileserver

# in windows
go build -o proxy-fileserver.exe cmd/main.go 
# or in linux
go build -o proxy-fileserver cmd/main.go

• Fix .env file with your config:
  • PROXY_SERVER_ENV= dev or prod, prod will come with log in json format
  • SHARED_ROOT_FOLDER_LOCAL= parent folder of shared-folder on server
  • SHARED_ROOT_FOLDER= shared-folder name your shared folder on Google Drive, same name with mirror folder on server
  • SHARED_ROOT_FOLDER_ID= your shared folder id on Google Drive, example: 1eQ-6Ftg1G1ewA_-7puGpV_YCjHkpmkVe
  • CACHE_TIME_LOCAL_FILE_SYSTEM= unit minute, time to delete a cached file on server from last download time
  • AUTH_PUBLIC_KEY= path to your public key that generated by openssl, example certificates/public512.pem
  • PRIVATE_KEY_LOCATION=certificates/private512.pem on proxy server
  • EXPIRED_TIME_TOKEN=10M expired for token
  • CYCLE_TIME_CLEANER= unit minute, cycle time cleaner wake up to clean expired files
  • MYSQL_USER= username database
  • MYSQL_PASSWORD= password database
  • MYSQL_PORT=3306 port database mysql
  • MYSQL_HOST= host database
  • MYSQL_DATABASE= name database
  • HTTP_PORT=8080 port http
  • TOKEN_MODE token mode in 1, 2, 3
  • GOOGLE_APPLICATION_CREDENTIALS= path to credential file (json) of service account cloud google
  • CREDENTIAL_GOOGLE_OAUTH2_FILE=certificates/credentials.json google oauth drive credential
  • TOKEN_GOOGLE_OAUTH2_FILE=certificates/token.json path to save token
  • GOOGLE_OAUTH2_ENABLE=ON default is ON // if set to OFF, use must config service account
  • INTERACTIVE_MODE=OFF default is off. Set to ON when use want to interact with terminal to exchange google access token
  • TELEGRAM_BASE_URL= https://api.telegram.org base url for telegram api
  • TELEGRAM_BOT_TOKEN= your token bot
  • TELEGRAM_CHANNEL_ID= id of channel that bot will send message to

• .env and binary file must be in the same folder
• TOKEN_MODE:
  • 1: no require token
  • 2: token with no strict path: use medium_level_token
  • 3: token strict path: use high_level_token

3. Setup

• On Cloud Google:

  • Create service account or create OAuth client ID if you want to use GOOGLE_OAUTH2_ENABLE and get certificate in json file (1)
  • Enable Drive api

• On Google Drive:

  • Files storage in 'shared-folder'
  • Share your 'shared-folder' with service account of google cloud if you use service account
  • Get ID of 'shared-folder' on url

• On server:

  • Create parent folder: example 'temp'
  • Create 'shared-folder' inside 'temp'
  • Put your public key and certificate from (1) on somewhere: example 'certificates/cer.json', ' certificates/public512.pem'
  • Define your .env file (see .env.example)

• Use exg tool if you set INTERACTIVE_MODE=OFF to pre-generate token. exg-tool

Example structure of tree folder tree: example-tree-folder

4. Run

On workspace:

Run service

./proxy-fileserver

Access api on port 8080:

• Request example:

  http://localhost:8080/avt.jpg?token=dsahdha.dsad.ewuegud
  

• Response example:
  • 204 for no file exist
  • 500 for system error
  • 200 for success response

Note

For test purpose: use this script for generate token:

package main

import (
	"crypto/rsa"
	"fmt"
	"github.com/dgrijalva/jwt-go"
	"io/ioutil"
	"log"
	"time"
)

func GetCertificate(location string) (*rsa.PrivateKey, error) {
	privateKeyBytes, err := ioutil.ReadFile(location)
	if err != nil {
		return nil, err
	}
	privateKey, err := jwt.ParseRSAPrivateKeyFromPEM(privateKeyBytes)
	if err != nil {
		return nil, err
	}
	return privateKey, nil
}
func main() {
	privateKey, err := GetCertificate("private512.pem")
	if err != nil {
		panic(err)
	}
	t := jwt.New(jwt.GetSigningMethod("RS256"))
	claims := jwt.MapClaims{}
	claims["iat"] = time.Now().Unix()
	claims["exp"] = time.Now().Add(1 * time.Minute).Unix()
	t.Claims = claims
	token, err := t.SignedString(privateKey)
	if err != nil {
		log.Printf("Error when generate token: %s", err)
	}
	fmt.Println(token)
}

API:

API Download file:

• Request:

curl 'http://localhost:8080/shared-folder/avt.jpg?token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MTU0NzE2MTYsImlhdCI6MTYxNTQ3MTAxn0.jyX3RIaENdI6JTZdziN3c86cvpqj2M7hpFZTuCATMqtU8uzbs9tLjev21Gng9xwSikb5nY4BcCQRtx9ie29SwQ' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36' \
  --compressed

• Response:
  • Get file successfully:
    • HTTP Code: 200
  • Invalid token:
    • HTTP Code: 401
  • File not found:
    • HTTP Code: 404
  • System Error:
    • HTTP Code: 500
    • Body: "System error. Please contact admin!"

API get token:

• Medium level token:
  • Request:

    curl --location --request POST 'localhost:8080/auth' \
    --header 'Content-Type: application/json' \
    --data-raw '{
    "type": "medium_level_token"
    }'

  • Response:

    {
        "token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MTgxMzA4ODMsImlhdCI6MTYxODEyOTY4M30.Lyjr8-fIZdeUTueZHx47LwH0ynIEbgPp_gVYRZtdnikYt_zReFTIxLsOCMx1FZz9HSraE57TJc3D4avDZvbXuw"
    }

• High level token:
  • Request:

    curl --location --request POST 'localhost:8080/auth' \
    --header 'Content-Type: application/json' \
    --data-raw '{
        "type": "high_level_token",
        "path": "/avt.jpg"
    }'

  • Response:

    {
        "token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MTgxMzEyMjQsImlhdCI6MTYxODEzMDAyNCwicGF0aCI6Ii9hdnQuanBnIn0.fAbwt13Hw8qXfYzAXU9ap_eAGngilQRgVwq63yN_PSRCYF3ye1Hv2YGirwhukc8MwPb6c3m0gxTkdvooZxWdxg"
    }

API verify token:

• Request:

  curl --location --request POST 'localhost:8080/verify' \
  --header 'Content-Type: application/json' \
  --data-raw '{
      "token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MTU0NzE2MTYsImlhdCI6MTYxNTQ3MTAxNn0.jyX3RIaENdI6JTZdziN3c86cvpqj2M7hpFZTuCATMqtU8uzbs9tLjev21Gng9xwSikb5nY4BcCQRtx9ie29SwQ"
  }'

• Response:

  • Valid:
    • HTTP Code: 200
    • Body: null
  • Invalid:
    • HTTP Code: 401
    • Body: null

FAQ

• If any error log with Google Drive, re-share drive folder with service account and wait some minutes or maybe some hours. Google Drive need time to re-index your share-across-domain and longer with old files but immediately on new files.

• Delete all records in database and all file in shared-folder before re-run

• More information to config cloud google:

  • https://developers.google.com/drive/api/v3/quickstart/go

• Confuse about INTERACTIVE_MODE:

  • When interactive mode set to OFF, you must had access token and refresh token storage in TOKEN_GOOGLE_OAUTH2_FILE
  • Encourage to use google_token_exchange tool to pre-generate token, then use INTERACTIVE_MODE=OFF in proxy server