acheong08 / obi-sync

Reverse engineering of the native Obsidian sync and publish server

Rev Obsidian Sync

Reverse engineered obsidian sync server (NOT OFFICIAL).

Warning

The main branch is the development branch. For stable usage, use the latest release.

Note

The plugin is broken on obsidian >= 1.4.11. This is intentional by the official ObsidianMD team. They have made clear their dissatisfaction with this project. The Path Forward - We are in the early stages of designing an alternative plugin that does not make use of existing code by ObsidianMD team. It is still in the design phase and help is needed. For now, we can manually patch the files from https://github.com/obsidianmd/obsidian-releases/releases/. I will write up a wiki page or automate that soon.

Features

• End to end encryption
• Live sync (across devices)
• File history/recovery/snapshots
• Works natively on IOS/Android/Linux/MacOS/Windows... (via the plugin)
• Vault sharing
• Publish (markdown only. no rendering yet)

Experimental

These features are not in the latest release but in the main branch. They might not be fully tested and are probably unstable.

• N/A

To do

• Fix bugs
• Improve publish

Quickstart

Note

The comprehensive documentation by @Aetherinox can be found in the wiki.

Quickstart with Docker

Environment variables

Required:

• DOMAIN_NAME - The domain name or IP address of your server. Include port if not on 80 or 433. The default is localhost:3000

Optional

• ADDR_HTTP - Server listener address. The default is 127.0.0.1:3000
• SIGNUP_KEY - Signup API is at /user/signup. This optionally restricts users who can sign up.
• DATA_DIR - Where data is saved. Default .
• MAX_STORAGE_GB - The maximum storage per user in GB. Default 10
• MAX_SITES_PER_USER - The maximum number of sites per user. Default 5

Building & Running

• git clone https://github.com/acheong08/obsidian-sync
• cd obsidian-sync
• go run cmd/obsidian-sync/main.go

Optional:

• Configure nginx
• HTTPS is recommended.

When you're done, install and configure the plugin

Adding a new user

go run cmd/signup/main.go

Alternatively:

curl --request POST \
  --url https://yourdomain.com/user/signup \
  --header 'Content-Type: application/json' \
  --data '{
	"email": "[email protected]",
	"password": "example_password",
	"name": "Example User",
	"signup_key": "<SIGNUP_KEY>"
}'

You can set the signup key via the SIGNUP_KEY environment variable. If it has not been set, you can exclude it from the request.

obi-sync's Issues

How to use publish

I wonder how to use publish by this repo

docker version sync error

I am using docker version now
it works perfectly between my two pcs
but it does not work on my Android phone
it keeps reporting error code 1006

could please help look into it?

HTTPS required when server not on localhost

I set it up by referring to your readme. However it does not open the synchronization process properly. I don't know if I need to apply https certificate? The specific error message is as follows:

How to sync .obsidian folder/custom files, etc...

I have installed the server, and it's working perfectly, but I see that it only syncs specific files, like .md, .png,... How to make it sync all the files/folder (like .obsidian folder)? By the way, thank you for your awesome work.

The path forward

As we've learned, Obsidian will keep trying to prevent the sync plugin from working on newer versions. On desktop, just replacing the asar is possible, but on mobile it is not.

There has to be a better solution for the problem with the plugin than to keep on playing cat and mouse with the obsidian devs.

We could:

• Reverse engineer the actual sync plugin so we don't have to ride off of code they might patch
• Write a sync server and client from scratch

There's this somewhat abandoned project called obsidian-diy-sync which could serve as a starting point for this

Completed: Use a plugin to override API endpoint

I wonder if we could use a regular plugin to override the api.obsidian.md endpoint, instead of patching the Obsidian package. A library like https://www.npmjs.com/package/@mswjs/interceptors could handle the plumbing.

If PRs are welcome I might give it a try in the next few weeks

Permission Denied

Getting fork/exec /tmp/go-build988693008/b001/exe/main: permission denied Wwhen running main.go

failed to look up field size from DDL CREATE TABLE sites

version: 0.1.0

Docker Install

Maybe I'm losing my mind. Running on Ubuntu with docker.
Pulled using

docker pull ghcr.io/acheong08/obi-sync:latest

And received:

no matching manifest for linux/amd64 in the manifest list entries

However if I use the following since I'm on Linux:

docker pull ghcr.io/acheong08/obi-sync:latest@sha256:97f08009a21f0c78f07c115df42df79e5acef3bf8beab40a70f4a3906fc10283

It will download properly, but when I issue:

docker compose up -d

I get:

The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested

Granted, I've only been using docker for a few months now, but thus far I've installed 7 projects without issue, and this is the first one to throw headaches. Not sure what I'm missing.

I think it's the same issue referenced in #18 (comment)

Thanks

docker support？

thanks for your work，are there any possibilities that publish a docker version？

obsidian 1.4.5 supported? - yes

obsidian 1.4.5 supported？

Confirmation of Connection

I have nginx and docker setup, with the container running fine. Went into Obsidian, enabled publish / sync, and signed in. Created a new vault, and the progress says that it's uploading files successfully.

However, the volume folders are empty, and I can't access anything via the website using

https://vault.domain.com/published/Vaultname/Path/To/File.md

There a step I'm missing? I even executed the CURL command to create a new user, and I can't even find out where that data is stored, no config files found, no db files, etc.

The data / volume folder is also empty.

var/lib/docker/volumes/obsidian-sync_sync_data/_data

Docker compose issue

Hi there,

First congrats on the progress so far, this project looks promising! I tried to run the sync server with docker-compose file provided in the Wiki but to no avail.

Environment info:
Two VM's running Debian 11 x64, Docker 24.0.5, using the docker compose config as-is.

Issue encountered:
The sync server was started with docker compose up, and was listing on 127.0.0.1:3000, which was exposed as port 3000 on the host. I tried access the server using both IP:3000 and nginx reverse proxy: the IP:3000 returns "curl: (56) Recv failure: Connection reset by peer" no matter accessing from localhost or Internet, and the nginx proxy shows 502 bad gateway, which means it's not talking to the upstream.

I've checked the provided docker-compose file, it's pretty straight-forward, and it looks just fine, no idea why it's not working as intended.

multi-arch support for docker image

i was interested in this project and wanted to host it on my raspberry pi but got an error when using docker-compose.

ERROR: no matching manifest for linux/arm64/v8 in the manifest list entries

checked it out and the docker images are only built for amd64 at the moment meaning nobody with arm CPUs will be able to use the server.

After upgrading to version 1.5.3, it's not possible to create new vault in the vault list interface. How can this issue be resolved?

          After upgrading to version 1.5.3, it's not possible to create new vault in the vault list interface. How can this issue be resolved?

Originally posted by @xcnick in acheong08/obi-sync-lib#1 (comment)

The Server is not working in windows

So i am trying to run the server in windows i am able to create a user but not able to login. I am getting the error sql: no rows in result set

Readme / Wiki

I don't recall if closed issues give notifications. Been awhile since I've messed with issues/tickets.

Here's the doc: https://github.com/Aetherinox/obi-sync-docs/blob/main/README.md

I tried to keep it as simple as possible, but also cover all the bases. You can pick out parts if you want to shorten the readme, or if you want to create a few Wiki pages and copy/paste the contents. The images are uploaded to github, so you don't need to go shuffling them around. I also confirmed with Github to make sure the images won't expire or poof / disappear, and they won't. As long as there's no Terms of Service violation; which there isn't.

It pretty much covers Sync and Publish, along with configuration. It went through it at a few different occasions today to re-read and check for grammar, shorten the words, etc. For the most part everything looks OK.

Also provided a revised template for people who want to use an .env file for variables instead of editing the docker-compose.yml which is located here.

Hope it helps.

Publish Issues

Hey, had to step away, this is part 2 of #24

In short, the issue still exists that I cannot view the online published MD files using publish.domain.com (with your nginx template). The site itself works and I get json formatted error messages, but it can never find the pages / slug.

So I tried out a series of tests.
When originally executing:

curl --request POST --url https://api.domain.com/publish/list --header 'Content-Type: application/json' --data '{ "token": "xxx"}'

It would return:

{"limit":1,"shared":[],"sites":[]}

I tried numerous re-creations of the vault, wiped info, etc. Nothing ever worked.

I then manually entered:

curl --request POST --url https://api.domain.com/publish/create --header 'Content-Type: application/json' --data '{ "token": "xxx"}'

And it created a slug for me as 755625b4-86c5-43cd-84fa-89f2b415596c

I used that slug as part of the domain url https://publish.domain.com/SLUG_HERE and I actually get information returned in json now

{
  "error": "record not found",
  "path": "",
  "site": {
    "id": "3540989e-c7c6-5aac-ba5f-7e7127baac43",
    "host": "domain.com",
    "slug": "755625b4-86c5-43cd-84fa-89f2b415596c"
  }
}

Before I was just getting Record Not Found and nothing else.

I'm not sure if I'm onto something here, I'm just tracing all the code backward to figure out where the issue could be, in case any of this helps out. It seems like my vaults aren't getting a slug, so the webserver can't find the pages. I've tried numerous vaults now, and can access no .md pages.

Edit: Did a wipe and started fresh.
Linked obsidian account, enabled sync / public and then synced my starter vault. Which gives me the sites list of:

{"limit":1,"shared":[],"sites":[]}

Decided to use this command to see if I could assign a slug to the newly created vault uid with:

curl --request POST --url https://obsidian.domain.com/api/slug --header 'Content-Type: application/json' --data '{ "id": "06d68834-b7e4-42e9-b72a-592d456ff3a3", "slug": "Test", "token": "xxxx"}'

Which returns

{"error":"You do not have permission to change this site's slug"}

Attempted to access the online published site using the URLs:

https://publish.domain.com/Test/Welcome.md
https://publish/domain.com/06d68834-b7e4-42e9-b72a-592d456ff3a3/Welcome.md

Test being the name I made for the vault, and the 2nd one being the UID auto assigned to the vault to see if either one of the two would work.

Both URL attempts return

{"error":"Site not found"}

On a positive note, after the endless hours of screwing around with this, I was able to edit the docker-compose file and get the vault database to save in a specified location. For some reason, the default docker-compose template doesn't have a period at the front of the path, but as soon as I added a period, it started saving in the folder where I originally specified it to go.

Maintain compatibility with newer versions of Obsidian sync? - yes

Hi, I was going to start using this server but just saw that you stopped using Obsidian.

Does this mean that this server will no longer support newer versions of Obsidian?
Or will you still maintain compatibility even though you aren't using it anymore?

Thanks for your work

1.4.11 patched the plugin again

line 1056 of main.js when ran through js-beautify

e.onBeforeRequest = r, e.onBeforeSendHeaders = r, e.onHeadersReceived = r;

r being a function which returns !1