ac;pic is an application that allows you to store and manage your pictures. ac;pic is built by Altocode. While the service itself is paid, Altocode freely shares the code for all purposes, including commercial ones.
To understand why we're sharing the source code of a commercial product, please read our manifesto. If that's too long to read, in a nutshell: we want to share our code so that others can learn from it and contribute to us. Sharing is the way to progress.
All non-code documents related to the project are published in this open folder.
The application is currently under development and has not been launched yet. We estimate to have an alpha by 2020.
The author wishes to thank Browserstack for providing tools to test cross-browser compatibility.
- Client
- New interface.
-
Migrate to gotoB v2
-
Add logic & endpoint to send to server latency of
query
requests. -
Server
- httponly cookie & csrf token
- Security: figure out if workaround for package-lock is necessary with nested dependencies that are not pegged.
- ac;tools
-
Admin & deploy
- Manage payments.
- Favicon & icons.
- Report pictures.
- Security writeup: xss routes with user input (tags, eventually picture titles; image contents are left to browsers but images have to be valid as per imagemagick), cookie as signed token, CSRF prevented by sending cookie (not by headers, https://blog.appsecco.com/exploiting-csrf-on-json-endpoints-with-flash-and-redirects-681d4ad6b31b), s3 encryption, we encrypt the cookie only for distinguishing expired cookie from attack (https://hueniverse.com/on-securing-web-session-ids-b90f566b3786)
-
Account
- Account view
- Delete account.
- Change email & password.
- Export all data.
- Re-import your data (won't reset what you have. do it through the proper endpoints, change ids).
- Payment.
- Payment late: 2 week notice with download.
- Freeze me out.
- API tokens.
- Status & stats page.
- Languages.
-
Share
- Share/unshare with authorization & automatic email.
- Share/unshare tag with a link (takes you to special view even if you're logged in, with go back to my pictures). Query against it as well with tags that are in those too?
- Upload to shared tag.
- Tags with same name (local vs shared, put a @).
- QR code to share.
-
Upload
- Client-side hashes for fast duplicate elimination.
- Client-side hashes to avoid deleted pictures on folder upload mode (with override).
- Folder upload on Android & mobile.
-
Organize
- Hidden tags.
- Mobile/tablet view.
- Add colors to tags.
- Show dates in upload mode.
- Smaller level of scale to go faster
- Upload video.
- Add title (based on title of pic but optional)
- Enable GPS detection.
- Create tag that groups tags (can also have pictures directly assigned).
- Create group that groups people.
- Filters.
- Themes for the interface.
- Set date manually?
- Order pictures within tag? Set priorities! Manual order mode.
-
Upload
- Allow only jpeg & png.
- Auto thumbnail generation.
- Server-side encryption.
- See progress when uploading files, using a progress bar.
- Ignore images that already were uploaded (by hash check).
- Upload more files while uploading files.
- Allow to go back to browse while files are being uploaded in the background.
- Retry on error.
- Add one or more tags to a certain upload batch.
-
Carrousel
- Full screen viewing.
- Use etags to save bandwidth.
- Carrousel with wrap-around and preloading of the next picture.
- Show date & tags.
-
Organize
- Multiple selection with shift & ctrl.
- Tag/untag.
- Delete pictures.
- Sort by newest, oldest & upload date.
- Autocomplete tags when searching.
- Allow for more than one tag to be searched at the same time.
- See number of tags & date below each picture.
- Rotate pictures.
- Select all pictures in query even if they were not loaded.
- Consider 1900-2100 as automatic tags for search.
- Refresh list of pictures if there's an upload in the background.
-
Account
- Signup with invite.
- Login/logout.
- Recover/reset/change password.
-
Admin
- Metering requests, downloads & space stored.
- Block further uploads if storage limits are exceeded.
- Invites.
- Stats endpoint.
- Share
- Comments.
- Share to social platforms.
- Serve images as hosting.
- Share certain tags only on shared pictures.
- Home pages.
If any route fails with an internal error, a 500 code will be returned with body {error: ...}
.
All POST
requests must have a content-type
header of application/json
and their bodies must be valid JSON objects. The only exception is POST /pic
, which must be of type multipart/form-data
.
All non-auth routes (unless marked otherwise) will respond with a 403 error with body {error: 'nocookie'}
if the user is not logged in.
If a cookie with invalid signature is sent along, the application will respond with a 403 error with body {error: 'tampered'}
.
If a cookie with valid signature but that has already expired is sent along, the application will respond with a 403 error with body {error: 'session'}
.
All POST requests (unless marked otherwise) must contain a cookie
field equivalent to the cookie
provided by a successfull call to POST /auth/login
. This requirement is for CSRF prevention. In the case of POST /upload
, the cookie must be present as a field within the multipart/form-data
form. If this condition is not met, a 403 error will be sent.
POST /requestInvite
.- Body must be
{email: STRING}
, otherwise a 400 will be sent.
- Body must be
-
POST /auth/login
.- Does not require the user to be logged in.
- Body must be
{username: STRING, password: STRING, tz: INTEGER}
. If not, a 400 code will be returned with body{error: ...}
. username
is lowercased and any leading & trailing space is removed from it (and intermediate spaces or space-like characters are reduced to a single space).username
can be either theusername
or theemail
associated to ausername
.- If the username/password combination is not valid, a 403 code will be returned with body:
{error: 'auth'}
. - If the username/password combination is valid but the email hasn't been verified yet, a 403 code will be returned with body
{error: 'verify'}
. tz
must be the output ofDate.getTimezoneOffset ()
, an integer expressing the number of minutes behind (or ahead) of UTC in the local time.
-
POST /auth/signup
.- Does not require the user to be logged in.
- Body must be
{username: STRING, password: STRING, email: STRING, token: STRING}
. The email must be a valid email. If not, a 400 code will be returned with body{error: ...}
. - Both
username
andemail
are lowercased and leading & trailing space is removed from them (and intermediate spaces or space-like characters are reduced to a single space).username
cannot contain any@
or:
characters. - If there's no invite associated with the token, a 403 is returned with body
{error: 'token'}
. - If there's already an account with that email, a 403 is returned with body
{error: 'email'}
. - If there's already an account with that username, a 403 is returned with body
{error: 'username'}
.
-
GET /auth/verify/TOKEN
(for verifying email ownership).- Does not require the user to be logged in.
- If successful, a 302 is returned redirecting to
/
. - If the token provided doesn't match what's on the database, a 403 is returned.
-
POST /auth/recover
.- Does not require the user to be logged in.
- Body must be
{username: STRING}
. The username can be either the username or the email. - If username is invalid, a 403 is returned.
- If successful, it sends an email to the associated email address with a link for password recovery.
-
POST /auth/reset
.- Does not require the user to be logged in.
- Body must be
{username: STRING, password: STRING, token: STRING}
. The username can be either the username or the email. - If username is invalid, a 403 is returned.
- If successful, it sends an email to the associated email address with a link for password recovery.
-
GET /auth/logout
.- Unless there's an error, the route will return a 302 code with the
location
header set to/
.
- Unless there's an error, the route will return a 302 code with the
-
POST /auth/delete
.- Temporarily disabled (route always returns 501).
- User must be logged in, otherwise a 403 is returned.
- The body is ignored.
- If successful, a 302 is returned redirecting to
/
.
-
POST /auth/changePassword
.- Body must be
{old: STRING, new: STRING}
.
- Body must be
POST /feedback
-
Body must be an object of the form
{message: STRING}
(otherwise 400). -
If successful, returns a 200.
-
GET /pic/ID
- Pic must exist and the user must have permissions to see it (otherwise, 404).
- Depending on ETag, a 200 or 304 is returned.
- If the file is not found, a 404 is returned.
-
GET /thumb/ID
- Thumb must exist and the user must have permissions to see it (otherwise, 404).
- Depending on ETag, a 200 or 304 is returned.
- If the file is not found, a 404 is returned.
-
POST /pic
- Must be a multipart request (and it should include a
content-type
header with valuemultipart/form-data
). - Must contain fields (otherwise, 400 with body
{error: 'field'}
). - Must contain one file (otherwise, 400 with body
{error: 'file'}
). - Must contain a field
uid
with an upload id (otherwise, 400 with body{error: 'uid'}
. - Must contain no extraneous fields (otherwise, 400 with body
{error: 'invalidField'}
). The only allowed fields areuid
,lastModified
andtags
; the last one is optional. - Must contain no extraneous files (otherwise, 400 with body
{error: 'invalidFile'}
). The only allowed file ispic
. - Must include a
lastModified
field that's parseable to an integer (otherwise, 400 with body{error: 'lastModified'}
). - If it includes a
tag
field, it must be an array (otherwise, 400 with body{error: 'tags'}
). None of them should be'all
','untagged'
or a four digit string that when parsed to an integer is between 1900 to 2100 (otherwise, 400 with body{error: 'tag: TAGNAME'}
). - The file uploaded must be
.png
or.jpg
(otherwise, 400 with body{error: 'format'}
). - If the same file exists for that user, a 409 is returned with body
{error: 'repeated'}
. - If the storage capacity for that user is exceeded, a 409 is returned with body
{error: 'capacity'}
. - If the upload is successful, a 200 is returned.
- Must be a multipart request (and it should include a
-
DELETE /pic/ID
- If the picture is not found, or if it does not belong to the user attempting the deletion, a 404 is returned.
- If the deletion is successful, a 200 is returned.
-
POST /rotate
- Body must be of the form
{ids: [STRING, ...], deg: 90|180|-90}
(otherwise, 400 with body{error: ...}
). - All pictures must exist and user must be owner of the pictures, otherwise a 404 is returned.
- There should be no repeated ids on the query, otherwise a 400 is returned.
- If the rotation is successful, a 200 is returned.
- Body must be of the form
-
POST /tag
- Body must be of the form
{tag: STRING, ids: [STRING, ...], del: true|false|undefined}
tag
will be trimmed (any whitespace at the beginning or end of the string will be eliminated; space-like characters in the middle will be replaced with a single space).tag
cannot be a stringified integer between 1900 and 2100 inclusive. It also cannot beall
oruntagged
.- If
del
istrue
, the tag will be removed, otherwise it will be added. - All pictures must exist and user must be owner of the pictures, otherwise a 404 is returned.
- There should be no repeated ids on the query, otherwise a 400 is returned.
- If successful, returns a 200.
- Body must be of the form
-
GET /tags
- Returns an object of the form
{tag1: INT, tag2: INT, ...}
. Includes a field foruntagged
and one forall
.
- Returns an object of the form
-
POST /query
- Body must be of the form
{tags: [STRING, ...], mindate: INT|UNDEFINED, maxdate: INT|UNDEFINED, sort: newest|oldest|upload, from: INT, to: INT}
. Otherwise, a 400 is returned with body{error: ...}
. body.from
andbody.to
must be positive integers, andbody.to
must be equal or larger tobody.from
. For a given query, they provide pagination capability. Both are indexes (starting at 1) that specify the first and the last picture to be returned from a certain query. If both are equal to 1, the first picture for the query will be returned. If they are 1 & 10 respectively, the first ten pictures for the query will be returned.all
cannot be included onbody.tags
. If you want to search for all available pictures, setbody.tags
to an empty array. If you send this tag, you'll get a 400 with body{error: 'all'}
.- If defined,
body.mindate
&body.maxdate
must be UTC dates in milliseconds. body.sort
determines whether sorting is done bynewest
,oldest
, orupload
. The first two criteria use the earliest date that can be retrieved from the metadata of the picture, or thelastModified
field. In the case of theupload
, the sorting is by newest upload date; there's no option to sort by oldest upload.- If the query is successful, a 200 is returned with body
pics: [{...}], total: INT}
.- Each element within
body.pics
is an object corresponding to a picture and contains these fields:{date: INT, dateup: INT, id: STRING, t200: STRING|UNDEFINED, t900: STRING|UNDEFINED, owner: STRING, name: STRING, dimh: INT, dimw: INT, tags: [STRING, ...], deg: INT|UNDEFINED}
. body.total
contains the number of total pictures matched by the query (notice it can be larger than the amount of pictures inbody.pics
).
- Each element within
- Body must be of the form
POST /share
- Body must be of the form
{tag: STRING, who: ID, del: BOOLEAN|UNDEFINED}
. - The target user (
body.who
) must exist, otherwise a 404 is returned. - If the tag being shared is
all
oruntagged
, a 400 is returned with body{error: 'tag'}
. - If try to share with yourself, a 400 is returned with body
{error: 'self'}
. - If successful, returns a 200.
GET /share
- If successful, returns a 200 with body
{sho: [[USERNAME1, TAG1], ...], shm: [[USERNAME2, TAG2], ...]}
. body.sho
lists the tags shared with others by the user.body.shm
lists the tags shared with the user.
GET /account
- If successful, returns a 200 with body
{username: STRING, email: STRING, type: STRING, created: INTEGER, used: [INTEGER_USED, INTEGER_MAXIMUM], logs: [...]}
.
POST /error
- This route does not require the user to be logged in.
- Body must be JSON, otherwise a 400 is returned.
All the routes below require an admin user to be logged in.
POST /admin/stats
- Publicly accessible.
- Returns all stats information.
POST /admin/invites
- Body must be
{email: STRING}
andbody.email
must be an email, otherwise a 400 is returned with body{error: ...}
.
State.view
: can be 'auth'
or 'main'
.
State.subview
: for view 'auth'
: 'login'
/'signup'
. For view 'main'
, 'browse'
/'upload'
.
State.notify
: for printing messages, {color: STRING, message: STRING, timeout: TIMEOUT FOR CLEARING State.notify}
.
State.query
: {tags: [...], sort: 'newest'/'oldest'/'upload'}
.
State.shift
: true|false|undefined
, truthy when the shift
key is depressed.
State.ctrl
: true|false|undefined
, truthy when the ctrl
key is depressed.
State.autotag
: STRING denoting the tag being entered by the user for tagging pictures or searching for an existing tag with which to tag pictures.
State.refreshQuery
: undefined|timeout
. If there are pending uploads in State.upload.queue
, this timeout retrieves pics. It's executed once per second.
State.autoquery
: undefined|string
, used to search for tags in the query box.
State.upload
: used for queuing uploads {queue: [FILE1, FILE2, ...], error: [[error, file], ...], invalid: [filename, ...], done: INT, repeated: INT, tags: [...]}
.
State.uploadFolder
: undefined|boolean
. If true
, the input for uploading files will upload entire directories instead.
State.uploadModal
: undefined|true
, if true, show the upload modal.
State.lastClick
: undefined|{id: PICID, time: INT}
, marks the last picture clicked and the time when it happened, to implement the folllowing: picture selection, picture selection by range, opening canvas view.
State.lastScroll
: undefined|{y: INT, time: INT}
, marks the time of the last scroll, and the last Y position of the window (window.scrollY
).
State.canvas
: undefined|PIC
, if not undefined contains the picture object that's being shown on the canvas
view.
State.nextCanvas
: undefined|PIC
, used to preload the next picture in the canvas
view.
State.showPictureInfo
: undefined|boolean
, if truthy, picture information is shown on the canvas
view.
State.screen
: {w: window.innerWidth, h: window.innerHeight}
. Used by the canvas
view.
State.selected
: {id1: true, id2: true, ...}
. Lists the ids all of selected pictures.
State.slider
: true|false|undeined. If no pictures selected, falsy is all tags, truthy is selected tags. If pictures are selected, falsy is add tags, truthy is remove tags.
State.loading
: true|undefined, to see whether pics are being loaded.
Data.pics
: [...]
; comes from body.pics
from POST /query
. A selected
boolean can be added by the client to denote selection of the picture, but that never reaches the server.
Data.total
: number of pics matched by query. Also comes from POST /query
.
Data.tags
: {all: INT, untagged: INT, ...}
; the body returned by GET /tags
.
Data.account
: {username: STRING, email: STRING, type: STRING, created: INT, logs: [{...}, ...], used: [INT, INT]
; the body returned by GET /account
.
- users:USERNAME (hash):
pass: STRING,
username: STRING,
email: STRING,
type: STRING (one of tier1|tier2),
created: INT
s3:bget: INT (bytes GET from s3),
s3:buse: INT (space used in S3),
- emails (hash): key is email, value is username
- emailtoken:TOKEN (hash): key is token, value is email. Used to verify email addresses.
- invites (hash): key is email, value is {firstName: STRING, token: ..., sent: INT (date), accepted: UNDEFINED|INT (date)}
- verify (hash): key is token, value is email. Deleted after usage.
- upic:USERID (set): contains hashes of the pictures uploaded by an user, to check for repetition.
- upicd:USERID (set): contains hashes of the pictures deleted by an user, to check for repetition when re-uploading files.
- thu:ID (string): id of the corresponding pic.
- pic:ID (hash)
id: STRING (uuid),
owner: STRING (user id),
name: STRING,
dateup: INT (millis),
dimw: INT (width in pixels),
dimh: INT (height in pixels),
by: INT (size in bytes)
hash: STRING,
dates: STRING (stringified array of dates belonging to the picture, normalized and sorted by earliest first),
orientation: STRING (stringified array of orientation data) or absent
deg: 90|-90|180 or absent,
date: INT (latest date within dates),
t200: STRING or absent,
by200: INT or absent,
t900: STRING or absent,
by900: INT or absent,
xt2: INT or absent, number of thumb200 downloaded (also includes cached hits)
xt9: INT or absent, number of thumb900 downloaded (also includes cached hits)
xp: INT or absent, number of pics downloaded (also includes cache)
- pict:ID (set): list of all the tags belonging to a picture.
- tag:USERID:TAG (set): pic ids.
- shm:USERID (set): USERA:TAG, USERB:TAG (shared with me)
- sho:USERID (set): USERA:TAG, USERB:TAG (shared with others)
- tags:USERID (set): list of all tags created by the user. Does not count tags shared with the user.
- ulog:USER (list): stringified log objects with user activity. Leftmost is most recent.
- For login: {t: INT, a: 'log', ip: STRING, ua: STRING, tz: INTEGER}
- For signup: {t: INT, a: 'sig', ip: STRING, ua: STRING}
- For recover: {t: INT, a: 'rec', ip: STRING, ua: STRING, token: STRING}
- For reset: {t: INT, a: 'res', ip: STRING, ua: STRING, token: STRING}
- For password change: {t: INT, a: 'chp', ip: STRING, ua: STRING, token: STRING}
- For destroy: {t: INT, a: 'des', ip: STRING, ua: STRING}
- For uploads: {t: INT, a: 'upl', id: STRING, uid: STRING (id of upload), tags: ARRAY|UNDEFINED}
- For deletes: {t: INT, a: 'del', id: STRING}
- For rotates: {t: INT, a: 'rot', id: STRING, deg: 90|180|-90}
- For (un)tags: {t: INT, a: 'tag', tag: STRING, d: true|undefined (if true it means untag), ids: [...]}
- For (un)shares: {t: INT, a: 'sha', tag: STRING, d: true|undefined (if true it means unshare), u: STRING}
- sti:d:DATE (string): picture/thumb downloads in the last 10 minutes. Time is Date.now () divided by 100000.
- sti:u:DATE (string): uploads in the last 10 minutes. Time is Date.now () divided by 100000.
- sti:t:DATE (string): tag operations in the last 10 minutes. Time is Date.now () divided by 100000.
- sti:l:DATE (string): total milliseconds for all responses, to calculate average, in the last 10 minutes. Time is Date.now () divided by 100000.
- sti:hxxx:DATE (string): responses with HTTP status code XXX in the last 10 minutes. Time is Date.now () divided by 100000.
- stp:a:DATE (hyperloglog or string): unique active users in the last 10 minutes. Time is Date.now () divided by 100000. Entries older than 10 minutes will be converted from hyperloglog to a string with a counter.
- stp:A:DATE (hyperloglog or string): unique active users in the last 24 hours. Time is Date.now () divided by 100000. Entries older than a day will be converted from hyperloglog to a string with a counter.
- stp (set): list of all hyperloglog entries.
- cachestats (string): stringified object with cache of /admin/stats. Lasts 60 seconds only.
Used by giz:
- session:ID (string): value is username. Used for sessions. Expires automatically.
- token:ID (string): value is username. Used for password recovery. Expires automatically.
- users:USERNAME (hash): covered above, giz only cares about `pass`.
On /etc/security/limits.conf
, the following two lines to increase file handles per process:
* soft nofile 99999
* hard nofile 99999
On /etc/sysctl.conf
:
# increases TCP maximum queue length to 4096
net.core.somaxconn=4096
# for giving more memory to redis to create background processes for saving the DB to disk
vm.overcommit_memory=1
start.sh
(must be executable, set with chmod 777 start.sh
):
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"
echo never > /sys/kernel/mm/transparent_hugepage/enabled
service redis-server restart
cd /root/acpic && mg restart
crontab with @reboot /root/start.sh
Installation of certbot:
sudo add-apt-repository ppa:certbot/certbot -y
sudo apt-get update
sudo apt-get install python-certbot-nginx -y
On your relevant nginx configuration (either at /etc/nginx/sites-available
or /etc/nginx/sites-enabled
), add this line:
server_name YOURDOMAIN
Make sure your domain points to the server where you're adding these changes. Then run these commands:
service nginx reload
sudo certbot --nginx -d YOURDOMAIN
Add the following to your crontab, to renew the certificates at the hour HH:YY every day.
crontab: YY HH * * * sudo certbot renew
If you find a security vulnerability, please disclose it to us as soon as possible (info AT altocode.nl
). We'll work on it with the utmost priority.
ac;pic is written by Altocode and released into the public domain.