GithubHelp home page GithubHelp logo

juxt / site Goto Github PK

View Code? Open in Web Editor NEW
129.0 31.0 20.0 103.87 MB

A web and API server, powered by xtdb.com

Clojure 80.29% Emacs Lisp 0.05% HTML 2.04% Shell 0.47% Makefile 0.16% TypeScript 12.75% CSS 0.51% JavaScript 3.44% Dockerfile 0.30%

site's Introduction

Important
This is the repository for Site 1.0. If you are looking for the latest version (Site 2.0), it is at https://github.com/juxt-site/site.

Site

CircleCI

Site is a project from JUXT to build a Resource Server out of our open-source XTDB database.

Site supports web content, and OpenAPI. You can use Site as a versioned Content Management System.

An official (work in progress) documentation website can be found here.

Note
A new version of Site is being developed here: https://github.com/juxt-site/site

What is Site?

Site is a Resource Server, built on the XTDB database.

You can put things into Site with (HTTP) PUT requests. When you do this, Site will put (the representation of) your thing (document, image, video, data…) into the database. You can get these later with a (HTTP) GET request with the same URI. In this way, Site behaves like a web server, with an immutable bitemporal content store.

APIs

If you PUT a JSON document with a Content-Type of application/vnd.oai.openapi+json;version=3.0.2, Site will treat this as an OpenAPI API definition, and serve that API for you. This OpenAPI API definition will contain the API endpoints, and provide schemas for the data transferred by the API. This tells the server how to validate data coming in to the API, and how to construct data on the way out.

APIs served from Site are good web citizens. They implement HTTP method semantics properly, with support for content negotiation, conditional requests, range requests and authentication.

APIs are also able to benefit from Site’s authorization module, Pass, providing Policy-Based Access Control, loosely based on XACML.

Test drive

Requirements

Before you start, you’ll need to have the following installed:

  • Java 9+ (Site is not compatible with Java 8 and below!)

  • Clojure and

  • Babashka installed on your system.

Clone this repo

$ git clone https://github.com/juxt/site

macOS (>= 10.15)

It is recommended to install openssl and openjdk and make sure JAVA_HOME is set correctly. At the time of writing brew installs openjdk 17.02 and openssl 3.0.1:

$ brew install openssl
$ echo export JAVA_HOME=/usr/local/Cellar/openjdk/17.0.2/libexec/openjdk.jdk/Contents/Home >> ~/.zshrc
$ exec zsh -l

Check that Java can find libcrypto:

$ cd site # (if needed)
$ clojure -A:dev -M:test -m kaocha.runner --focus juxt.site.authz-test

The test should run just fine. If instead $JAVA_HOME/bin/java aborts (signal 6) with the message:

$ ...
$ ... WARNING: ${JAVA_HOME}/bin/java is loading libcrypto in an unsafe way

you need to make libcrypto visible to $JAVA_HOME/bin/java by running e.g.:

$ ln -s  /usr/local/Cellar/openssl@3/3.0.1/lib/libcrypto.dylib $JAVA_HOME/lib

Configure

There’s a sample configuration in etc you should copy to $HOME/.config/site/config.edn.

$ mkdir -p $HOME/.config/site
$ cp site/etc/config.edn $HOME/.config/site/config.edn

Configure a password

Tip

If you have pass installed, this is a good time to configure a password for your superuser. We’ll assume the superuser will be named admin, but feel free to choose your own username here.

$ pass generate -n site/local/admin

Start the server

Start the Site server:

$ site/bin/site-server
Note
Alternatively, if you’re familiar with Clojure development, you can start the server via the deps.edn file and simply 'jack-in' with your editor or IDE as normal.

Start multiple instances of the server

If you require multiple Site servers to coexist on the same machine, you can start site passing a different configuration file as follows:

$ SITE_CONFIG=/absolute/path/custom-site-config.edn site/bin/site-server

In this case please be sure to change the configuration so ports are different and XTDB files are stored in a separate folder than the ones specified in the example configuration file. You’ll also need to specify Site host:port when using site commands, for example:

$ SITE_BASE_URI=http://localhost:5509 site/bin/site get-token -u admin

Start the server from the provided Docker image

Optionally, you can also get Site up and running using the provided Docker image. You need Docker installed in your system, then execute the following from the command line (sudo might not be necessary depending on your installation):

sudo docker build -t juxt/site:latest .
sudo docker run -p 2021:2021 -p 50505:50505 -d juxt/site:latest

The REPL

If you’ve run Site via your development environment and 'jacked-in' you’ll already have a REPL. Proceed to the next step.

If you’re running Site with site/bin/site-server, you’ll need to connect a terminal to Site to access the REPL. You can do this via port 50505, which is a socket REPL that Site starts by default.

How you connect to this port is up to you. One way is via ncat, but you can replace ncat with telnet, or netcat, depending on what’s available for your system.

Note

Arch users can install ncat by installing the nmap package:

$ sudo pacman -Sy nmap
$ ncat localhost 50505
Tip

Prefix the command with rlwrap if you have it installed.

$ rlwrap ncat localhost 50505

Bootstrap

Bootstrap the new system by adding the minimum resources that are required to allow remote access.

Site by JUXT. Copyright (c) 2021, JUXT LTD.
Type :repl/quit to exit

[ ]  Site API not installed.  Enter (put-site-api!) to fix this.
[ ]  Authentication resources not installed.  Enter (put-auth-resources!) to fix this.
[ ]  Role of superuser not yet created. Enter (put-superuser-role!) to fix this.
[ ]  No superusers exist. Enter (put-superuser! <username> <fullname>)
     or (put-superuser! <username> <fullname> <password>) to fix this.
site>

Install the Site API:

site> (put-site-api!)

Install the authentication rules:

site> (put-auth-resources!)

Install the superuser role:

site> (put-superuser-role!)

Finally, create a superuser. If you have pass installed, this will fetch the password directly:

site> (put-superuser! "admin" "Administrator")
Note
We recommend that you generate a password with pass.

If you don’t have pass installed, you can add a password as a final argument to put-superuser!.

site> (put-superuser! "admin" "Administrator" "admin")

Replace "admin", "Administrator" and "admin" with your own username, full name and password respectively.

Quit the REPL, for example, with Ctrl-C or by typing :repl/quit.

Run the site tool

The site tool is a command-line utility that allows you to remotely administer site.

If you’re on MacOS, you will need to install the gnu version of readlink. You can do so with brew:

brew install coreutils
ln -s /usr/local/bin/greadlink /usr/local/bin/readlink

We must first get a token that we can use for API access. This process authenticates to the site server using your password.

Here, replace admin with your username (or let it default to your OS username)
$ site/bin/site get-token -u admin

Now we can use the site tool for remote administration. Try the following:

$ site/bin/site list-users

Configure the expiry time for tokens

By default, tokens last for an hour. That can sometimes mean they expire during work sessions. You can set the expiry time of new tokens via the REPL.

(put! (assoc (e "http://localhost:2021/_site/token")  ::pass/expires-in (* 24 3600)))

License

The MIT License (MIT)

Copyright © 2020-2021 JUXT LTD.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

site's People

Contributors

armincerf avatar fiv0 avatar govindkrjoshi avatar herichovadajana avatar joelittlejohn avatar johantonelli avatar jsulmont avatar malcolmsparks avatar msladecek avatar reborg avatar refset avatar rhcarvalho avatar rhizomaticflow avatar tggreene avatar vollcheck avatar wkok avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

site's Issues

Cannot run on M1 chips on macOS

Run ./bin/site-server gives the following error:

Site by JUXT. Copyright (c) 2021, JUXT LTD.
00:00:06.764 [main] INFO juxt.site.alpha.main - Starting system
00:00:06.765 [main] INFO juxt.site.alpha.main - Configuration profile: prod
00:00:06.766 [main] DEBUG juxt.site.alpha.main - Loading configuration from /Users/matthew/Drive/dev/samples/site/dev-local/config.edn
2022-01-01 00:00:08.339:INFO::main: Logging initialized @2575ms to org.eclipse.jetty.util.log.StdErrLog
00:00:11.133 [main] INFO juxt.site.alpha.db - Starting XT node
Execution error (UnsatisfiedLinkError) at jdk.internal.loader.NativeLibraries/load (NativeLibraries.java:-2).
/private/var/folders/2k/05rf5_b108d1xl0mcksf5z980000gn/T/xtdb_rocksdb-6.12.7/librocksdbjni-osx.jnilib: dlopen(/private/var/folders/2k/05rf5_b108d1xl0mcksf5z980000gn/T/xtdb_rocksdb-6.12.7/librocksdbjni-osx.jnilib, 0x0001): tried: '/private/var/folders/2k/05rf5_b108d1xl0mcksf5z980000gn/T/xtdb_rocksdb-6.12.7/librocksdbjni-osx.jnilib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/librocksdbjni-osx.jnilib' (no such file), '/usr/lib/librocksdbjni-osx.jnilib' (no such file)

Full report at:
/var/folders/2k/05rf5_b108d1xl0mcksf5z980000gn/T/clojure-14123307117178288585.edn

401 Unauthorized when trying to get admin token

Following the Test drive I get the following when trying to get-token for the admin user

site > bin/site get-token -u admin
site: Unix pass prefix: site/local/
site: Get token from http://localhost:2021/_site/token
site: Server response: 401 Unauthorized

site: Are your credentials valid? request-id is http://localhost:2021/_site/requests/50ba7d0fad679394de1aff2c

Looking at the log output it looks like the username being passed to juxt.pass.alpha.authentication/lookup-user is nil

17:16:43.457 [qtp1821409160-23] DEBUG juxt.site.alpha.handler - Resource provider: :juxt.site.alpha.handler/db
17:16:43.461 [qtp1821409160-23] ERROR juxt.pass.alpha.authentication - #error {
 :cause nil
 :via
 [{:type java.lang.NullPointerException
   :message nil
   :at [java.util.regex.Matcher getTextLength Matcher.java 1770]}]
 :trace
 [[java.util.regex.Matcher getTextLength Matcher.java 1770]
  [java.util.regex.Matcher reset Matcher.java 416]
  [java.util.regex.Matcher <init> Matcher.java 253]
  [java.util.regex.Pattern matcher Pattern.java 1133]
  [clojure.core$re_matcher invokeStatic core.clj 4856]
  [clojure.core$re_matches invokeStatic core.clj 4886]
  [clojure.core$re_matches invoke core.clj 4886]
  [juxt.pass.alpha.authentication$lookup_user invokeStatic authentication.clj 105]
  [juxt.pass.alpha.authentication$lookup_user invoke authentication.clj 101]
  [juxt.pass.alpha.authentication$authenticate invokeStatic authentication.clj 263]
  [juxt.pass.alpha.authentication$authenticate invoke authentication.clj 227]
  [juxt.site.alpha.handler$wrap_authenticate$fn__16783 invoke handler.clj 800]
  [juxt.site.alpha.handler$wrap_negotiate_representation$fn__16778 invoke handler.clj 792]
  [juxt.site.alpha.handler$wrap_find_current_representations$fn__16773 invoke handler.clj 787]
  [juxt.site.alpha.handler$wrap_redirect$fn__16767 invoke handler.clj 773]
  [juxt.site.alpha.handler$wrap_locate_resource$fn__16762 invoke handler.clj 760]
  [juxt.site.alpha.handler$wrap_method_not_implemented_QMARK_$fn__16758 invoke handler.clj 754]
  [juxt.site.alpha.handler$wrap_error_handling$fn__16923 invoke handler.clj 1240]
  [juxt.site.alpha.handler$wrap_security_headers$fn__16828 invoke handler.clj 920]
  [juxt.site.alpha.handler$wrap_cors_headers$fn__16874 invoke handler.clj 1029]
  [juxt.site.alpha.handler$wrap_store_request_in_request_cache$fn__16965 invoke handler.clj 1380]
  [juxt.site.alpha.handler$wrap_store_request$fn__16969 invoke handler.clj 1387]
  [juxt.site.alpha.handler$wrap_log_request$fn__16975 invoke handler.clj 1405]
  [juxt.site.alpha.handler$wrap_service_unavailable_QMARK_$fn__16983 invoke handler.clj 1428]
  [juxt.site.alpha.handler$wrap_initialize_request$fn__16951 invoke handler.clj 1342]
  [juxt.site.alpha.handler$wrap_healthcheck$fn__16979 invoke handler.clj 1412]
  [juxt.site.alpha.handler$wrap_ring_1_adapter$fn__16959 invoke handler.clj 1369]
  [ring.adapter.jetty$proxy_handler$fn__8527 invoke jetty.clj 27]
  [ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a handle nil -1]
  [org.eclipse.jetty.server.handler.HandlerWrapper handle HandlerWrapper.java 127]
  [org.eclipse.jetty.server.Server handle Server.java 516]
  [org.eclipse.jetty.server.HttpChannel lambda$handle$1 HttpChannel.java 388]
  [org.eclipse.jetty.server.HttpChannel dispatch HttpChannel.java 633]
  [org.eclipse.jetty.server.HttpChannel handle HttpChannel.java 380]
  [org.eclipse.jetty.server.HttpConnection onFillable HttpConnection.java 273]
  [org.eclipse.jetty.io.AbstractConnection$ReadCallback succeeded AbstractConnection.java 311]
  [org.eclipse.jetty.io.FillInterest fillable FillInterest.java 105]
  [org.eclipse.jetty.io.ChannelEndPoint$1 run ChannelEndPoint.java 104]
  [org.eclipse.jetty.util.thread.QueuedThreadPool runJob QueuedThreadPool.java 773]
  [org.eclipse.jetty.util.thread.QueuedThreadPool$Runner run QueuedThreadPool.java 905]
  [java.lang.Thread run Thread.java 834]]}
17:16:43.461 [qtp1821409160-23] DEBUG crux.query - :query {:find [rule], :where [[rule :juxt.site.alpha/type "Rule"]], :in []}

[Feature] Add a coerce option for endpoint parameters in openapi.edn

Add a coerce option (or even more generalized transform option) in parameters passed to site endpoints so that they can be used as their proper type/shape from within the Datalog query.

For example:

{:name "sort-order"
 :in "query"
 :description "The order to sort by"
 :required false
 :default "asc"
 :schema {:type "string"}
 :coerce keyword}

so that i can then use it in my datalog query as:

{:find [(pull e [*]) name]
           :where [[e :juxt.site.alpha/type "Entity"]
                        [e :name name]]
           :order-by  [[name {:in "query" :name "sort-order"}]]}

Handling of unaccepted/wrong content-type

Currently specifying a bad content-type results in a server error. We could do two things to improve the dev experience:

  • Return a 406 Not Acceptable instead of a server error.
  • Include the error and irritating content-type in the response.

The same is probably true for content-language and content-encoding.

graphiql build breakage

npx gulp is failing

npx gulp
[04:51:06] Using gulpfile ~/src/github.com/juxt/site/opt/graphiql/gulpfile.js
[04:51:06] Starting 'default'...
[04:51:06] 'default' errored after 20 ms
[04:51:06] Error in plugin "gulp-inline-source"
Message:
    ENOENT: no such file or directory, open '/home/mal/src/github.com/juxt/site/opt/graphiql/build/_site/graphiql/static/css/2.dff3c244.chunk.css'
Details:
    errno: -2
    syscall: open
    code: ENOENT
    path: /home/mal/src/github.com/juxt/site/opt/graphiql/build/_site/graphiql/static/css/2.dff3c244.chunk.css
    domainEmitter: [object Object]
    domainThrown: false

make: *** [Makefile:6: build] Error 1

The problem seems to have arisen since the inclusion of PUBLIC_URL in .env

mal@malxps graphiql]$ git show 5ec730494d42463538e6b6b2d8202e4b3d515d23 -- .env
commit 5ec730494d42463538e6b6b2d8202e4b3d515d23
Author: Alex Davis <[email protected]>
Date:   Thu Oct 14 14:30:17 2021 +0100

    Add 'put-static-site' cli tool

diff --git a/opt/graphiql/.env b/opt/graphiql/.env
index 7b6d6ef..9eb4531 100644
--- a/opt/graphiql/.env
+++ b/opt/graphiql/.env
@@ -1,3 +1,4 @@
 INLINE_RUNTIME_CHUNK=false
-GENERATE_SOURCEMAP=false
+GENERATE_SOURCEMAP=true
 SKIP_PREFLIGHT_CHECK=true
+PUBLIC_URL=/alextest

This fails regardless of the value of PUBLIC_URL. Workaround: Removing PUBLIC_URL fixes this.

Extraneous Input regex.Matcher

I'm getting the following error on some requests, it looks like pattern matching on the accept-content header throwing an exception:

Jun  1 01:41:49 ip-172-31-26-211 web: Extraneous input: {:matcher #object[java.util.regex.Matcher 0x58f9dc87 "java.util.regex.Matcher[pattern=.* region=34,52 lastmatch=*; q=.2, */*; q=.2]"], :result ({:juxt.reap.alpha.rfc7231/media-range "text/html", :juxt.reap.alpha.rfc7231/type "text", :juxt.reap.alpha.rfc7231/subtype "html", :juxt.reap.alpha.rfc7231/parameters {}} {:juxt.reap.alpha.rfc7231/media-range "image/gif", :juxt.reap.alpha.rfc7231/type "image", :juxt.reap.alpha.rfc7231/subtype "gif", :juxt.reap.alpha.rfc7231/parameters {}} {:juxt.reap.alpha.rfc7231/media-range "image/jpeg", :juxt.reap.alpha.rfc7231/type "image", :juxt.reap.alpha.rfc7231/subtype "jpeg", :juxt.reap.alpha.rfc7231/parameters {}}), :remainder "*; q=.2, *…", :crux.db/id nil, :juxt.site.alpha/type "Request", :ring.request/headers nil}
Jun  1 01:41:49 ip-172-31-26-211 web: clojure.lang.ExceptionInfo: Extraneous input
Jun  1 01:41:49 ip-172-31-26-211 web: at juxt.reap.alpha.combinators$complete$fn__6026.invoke(combinators.clj:244)
Jun  1 01:41:49 ip-172-31-26-211 web: at juxt.reap.alpha.ring$headers__GT_decoded_preferences$iter__12626__12630$fn__12631$fn__12632.invoke(ring.clj:36)
Jun  1 01:41:49 ip-172-31-26-211 web: at juxt.reap.alpha.ring$headers__GT_decoded_preferences$iter__12626__12630$fn__12631.invoke(ring.clj:35)
Jun  1 01:41:49 ip-172-31-26-211 web: at clojure.lang.LazySeq.sval(LazySeq.java:42)

nmap scan causes null pointer exception

Site Commit Hash: a696e48

Executing the following:

nmap -A -v -v localhost

on a Linux machine running site-server causes a null pointer exception.

See the attached log file site-server.log.

Steps to reproduce:

  1. Start a clean site instance by following the steps in README.adoc (i.e., bin/site-server).
  2. Execute nmap command, and wait for a short period of time.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.