GithubHelp home page GithubHelp logo

babashka / neil Goto Github PK

View Code? Open in Web Editor NEW
352.0 8.0 26.0 319 KB

A CLI to add common aliases and features to deps.edn-based projects

License: MIT License

Clojure 93.24% Batchfile 0.04% Nix 0.57% Emacs Lisp 6.15%
babashka deps-edn clojure

neil's Introduction

neil

A CLI to add common aliases and features to deps.edn-based projects.

Introduction

See the New Clojure project quickstart blog post for a gentle introduction into neil.

Installation

Homebrew (Linux and macOS)

$ brew install babashka/brew/neil

Scoop (Windows)

$ scoop bucket add scoop-clojure https://github.com/littleli/scoop-clojure
$ scoop install neil

For detailed information about scoop installer check scoop-clojure.

Nix

$ nix-shell -p neil

# Alternatively, if your nix channel doesn't have neil yet:
$ nix-shell -I nixpkgs=channel:nixos-unstable -p neil

Clojure

Add the following alias to your global or project-local deps.edn:

:neil {:deps {io.github.babashka/neil {:git/tag "v0.1.41"
                                       :git/sha "77288d4"}}
       :main-opts ["-m" "babashka.neil"]}

Then invoke clj -M:neil.

Manual

  • Install babashka
  • Download the neil script to somewhere on your PATH. In Windows, also download the neil.bat script and place it in the PATH.

Status

As we're still finding out the best UX, neil may undergo breaking changes from version to version.

Usage

Type neil to see the help:

Usage: neil <subcommand> <options>

Most subcommands support the options:
  --alias      Override alias name.
  --deps-file  Override deps.edn file name.

Subcommands:

add
  dep    Alias for `neil dep add`.
  test   adds cognitect test runner to :test alias.
  build  adds tools.build build.clj file and :build alias.
  kaocha adds kaocha test runner to :kaocha alias.
  nrepl  adds nrepl server to :nrepl alias.

dep
  add: Adds --lib, a fully qualified symbol, to deps.edn :deps.
    Run `neil dep add --help` to see all options.

  search: Search Clojars for a string in any attribute of an artifact
    Run `neil dep search --help` to see all options.

  upgrade: Upgrade libs in the deps.edn file.
    Run `neil dep upgrade --help` to see all options.

  versions: List available versions of a library (Clojars libraries only)
    Run `neil dep versions -h` to see all options.

  update: Alias for `upgrade`.

license
  list   Lists commonly-used licenses available to be added to project. Takes an optional search string to filter results.
  search Alias for `list`
  add    Writes license text to a file
    Options:
    --license The key of the license to use (e.g. epl-1.0, mit, unlicense). --license option name may be elided when license key is provided as first argument.
    --file    The file to write. Defaults to 'LICENSE'.

new
  Create a project using deps-new
    Run `neil new --help` to see all options.

version
  Commands for managing the :version key in the deps.edn project config.
    Run `neil version --help` to see all options.

test
  Run tests. Assumes `neil add test`. Run `neil test --help` to see all options.

add dep

This will add the newest version of clj-kondo to the :deps map in deps.edn:

$ neil add dep :lib clj-kondo/clj-kondo

The :lib keyword may be elided if the libname is the first argument after dep:

$ neil add dep clj-kondo/clj-kondo

The add dep command will always overwrite an existing dependency.

To add a git library from Github you can use :sha to provide a SHA or :latest-sha to pick the latest sha from the default branch:

$ neil add dep org.babashka/sci :latest-sha true

add build

To add tools.build related features, use:

$ neil add build :deps-deploy true

After that you can run tasks like:

$ clojure -T:build uber
$ clojure -T:build deploy

If you didn't use the full app template when starting your project, then you'll need to configure a couple things in order to get a working uberjar. See the uberjar section here.

add test

$ neil add test

This will add the Cognitect test-runner to your deps.edn so you can execute:

$ clojure -X:test

A similar option is supported for kaocha:

$ neil add kaocha

To change the alias you can provide an option like:

$ neil add kaocha :alias kaocha2

dep search

Search Clojars for a string in any attribute of an artifact:

$ neil dep search "babashka.nrepl"
:lib babashka/babashka.nrepl :version 0.0.6

Note that Clojars stores the namespace and name of a library as separate attributes, so searching for a ns-qualified library will not necessarily return any matches:

$ neil dep search "babashka/babashka.nrepl"
Unable to find babashka/babashka.nrepl on Clojars.

But a search string can be matched in a library's description:

$ neil dep search "test framework"

will return libraries with 'test framework' in their description.

license list

List/search for licenses that can be added to a project with neil. This functionality uses Github's license API, which is also used by choosealicense.com. With no search term, a list of commonly-used licenses is returned:

$ neil license list
:license agpl-3.0 :name GNU Affero General Public License v3.0
:license apache-2.0 :name Apache License 2.0
...

A search term can be added to filter the commonly-used list with a case-insensitive search against the license name:

$ neil license list "lesser general"
:license lgpl-2.1 :name GNU Lesser General Public License v2.1

The full collection of available licenses can be found in the license API repo.

license search is an alias for license list.

license add

Retrieve license text from Github's license API and write it to a file. See the license list help for details on available licenses.

$ neil license add :license mit :file myproj/license.txt

Will write the MIT license to the file myproject/license.txt. The :license keyword can be left out if the license key is the first argument, and :file defaults to LICENSE, so a minimal usage:

$ neil license add epl-1.0

Will create a LICENSE file in the current directory with the EPL 1.0 text.

Emacs Integration

neil.el is a companion Emacs package.

Load it using your preferred Emacs package manager, e.g., for Doom Emacs:

;; packages.el

(package! neil :recipe (:host github :repo "babashka/neil" :files ("*.el")))

;; config.el

(use-package! neil
  :config 
  (setq neil-prompt-for-version-p nil
        neil-inject-dep-to-project-p t))

Github's Rate Limit

Github's API has a 60 hit/hour rate-limit. The workaround for this is creating a personal access token and setting two env vars:

  • NEIL_GITHUB_USER
  • NEIL_GITHUB_TOKEN

Roadmap

  • Add bb.edn-related features for invoking test and build tasks
  • Option to add cljs-test-runner

Contributing

If this project shows potential to you, I'd be happy to discuss and receive contributions.

Dev

See neil.rb for the brew Formula. You can install this formula locally with:

$ brew reinstall --build-from-source ./neil.rb

Or install a development version with bbin:

$ bbin install . --as neil-dev --main-opts '["-m" babashka.neil/-main]'

You can choose your own binary name with the --as YOUR_BINARY option. Rebuilding is not required when installing with bbin - any changes to src/babashka/neil.clj will count the next time you run neil-dev (YOUR_BINARY).

License

Copyright © 2022 Michiel Borkent

Distributed under the MIT License. See LICENSE.

neil's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

neil's Issues

Some kind of messed up dependency resolution

I removed ~/.m2/repository (for unrelated reasons), and now I cannot run neil anymore. I got the latest main; ran bb run gen-script, and then tried doing: ./neil dep search foo. It's throwing this:

----- Error --------------------------------------------------------------------
Type:     java.nio.file.NoSuchFileException
Message:  /Users/ag/.m2/repository/borkdude/rewrite-edn/0.0.2/rewrite-edn-0.0.2.jar
Location: /Users/ag/sandbox/babashka_neil/./neil:13:1

----- Context ------------------------------------------------------------------
9:
10: (ns babashka.neil
11:   {:no-doc true})
12:
13: (require '[babashka.curl :as curl]
    ^--- /Users/ag/.m2/repository/borkdude/rewrite-edn/0.0.2/rewrite-edn-0.0.2.jar
14:          '[babashka.fs :as fs]
15:          '[borkdude.rewrite-edn :as r]
16:          '[cheshire.core :as cheshire]
17:          '[clojure.edn :as edn]
18:          '[clojure.string :as str])

----- Stack trace --------------------------------------------------------------
sun.nio.fs.UnixFileAttributeViews/Basic      - <built-in>
java.util.zip.ZipFile/Source                 - <built-in>
java.util.zip.ZipFile/CleanableResource      - <built-in>
babashka.impl.classpath/path-from-jar        - <built-in>
babashka.impl.classpath.Loader/fn--17130     - <built-in>
clojure.core/some                            - <built-in>
babashka.impl.classpath/source-for-namespace - <built-in>
babashka.main/exec/load-fn--34829            - <built-in>
babashka.neil                                - /Users/ag/sandbox/babashka_neil/./neil:13:1

I'm not sure what's going on. Running clj -Stree shows this:

Picked up JAVA_TOOL_OPTIONS: -Dapple.awt.UIElement=true
org.clojure/clojure 1.11.1
. org.clojure/spec.alpha 0.3.218
. org.clojure/core.specs.alpha 0.2.62
babashka/babashka.curl 0.1.0
babashka/fs 0.1.1
rewrite-clj/rewrite-clj 1.0.699-alpha
. org.clojure/tools.reader 1.3.6
borkdude/rewrite-edn 0.1.0
X rewrite-clj/rewrite-clj 1.0.572-alpha :use-top                <---- see this? 
cheshire/cheshire 5.10.1
. com.fasterxml.jackson.core/jackson-core 2.12.4
. com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.12.4
    . com.fasterxml.jackson.core/jackson-databind 2.12.4
    . com.fasterxml.jackson.core/jackson-annotations 2.12.4
    . com.fasterxml.jackson.core/jackson-core 2.12.4
    . com.fasterxml.jackson.core/jackson-core 2.12.4
. com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.12.4
    . com.fasterxml.jackson.core/jackson-databind 2.12.4
    . com.fasterxml.jackson.core/jackson-core 2.12.4
. tigris/tigris 0.1.2

Can someone tell me what's happening? What have I screwed up on my machine?

Proposal: Make neil command automatically run clojure -X:alias or clojure -M:alias

Proposal

I was thinking it could be nice if:

neil command

Would be smart enough to look at the deps.edn, lookup an alias for command, and figure out if it needs to be called with -X or -M or -T, and than basically run it with either:

clojure -X:command
or
clojure -M:command
or
clojure -T:command

If command clashed with neil's own commands, I would put a message like:

conflicts with neil command and your deps.edn command, defaulting to running neil's command, if you want to run deps's command explicitly call: neil clojure command

So for example, one could do:

deps.edn

{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}}
 :aliases
 {:run {:main-opts ["-m" "myapp.myapp"]}
  :run-x {:ns-default myapp.myapp
          :exec-fn greet
          :exec-args {:name "Clojure"}}
  :build {:deps {io.github.clojure/tools.build {:git/tag "v0.8.3" :git/sha "0d20256"}}
          :ns-default build
          :neil/run-as :tool}
  :test {:extra-paths ["test"]
         :extra-deps {org.clojure/test.check {:mvn/version "1.1.1"}
                      io.github.cognitect-labs/test-runner
                      {:git/tag "v0.5.1" :git/sha "dfb30dd"}}
         :main-opts ["-m" "cognitect.test-runner"]
         :exec-fn cognitect.test-runner.api/test
         :neil/run-as :main}}}

And now from command line:
$ neil run - runs clojure -M:run
$ neil run-x - runs clojure -X:run-x
$ neil build uber - runs clojure -T:build uber
$ neil test - runs clojure -M:test

When it is ambiguous what the run-as should be, such as here:

:build {:deps {io.github.clojure/tools.build {:git/tag "v0.8.3" :git/sha "0d20256"}}
          :ns-default build
          :neil/run-as :tool}

I think neil could prompt:

  1. Run as Tool
  2. Run as Main
  3. Run as Exec

Or if the alias contains a :neil/run-as than that is used instead.

Maybe we can also have a neil command to set the :neil/run-as or after it prompts the first time we can have it set it automatically for future invocation.

This would basically allow any alias to become a neil command.

Alternative

Alternatively, we can put this behavior behind a command, so maybe:

neil run alias

It would behave the same, but you'd have to do:
$ neil run run - runs clojure -M:run
$ neil run run-x - runs clojure -X:run-x
$ neil run build uber - runs clojure -T:build uber
$ neil run test - runs clojure -M:test

And maybe in this case we could also have an explicit run-as command:
$ neil run-as main run - runs clojure -M:run
$ neil run-as exec run-x - runs clojure -X:run-x
$ neil run-as tool build uber - runs clojure -T:build uber
$ neil run-as main test - runs clojure -M:test

Add `bb dev` task to auto-build `neil` script

I've been using the following command for neil development to re-run the bb gen-script task when a file changes:

find . -iname '*.clj' -o -iname 'prelude' | entr -s 'bb gen-script && date'

Would it be helpful to add a bb watch-script task so this is built-in to the neil repo? It would be nice to not have to use entr for this.

If so, I'm assuming pod-babashka-fswatcher would be preferred over pod-babashka-filewatcher for implementation since it has more recent updates. Edit: Actually I see now that fswatcher does not have a brew formula yet, so I'm not so sure.

Proposal: Formatted and Paginated Help

Features

  • When using a terminal, add ANSI escape codes for bold and underlined text (particularly useful for Usage text)
  • When using a terminal, pipe help output to --pager/$PAGER//usr/bin/less -is (disabled in Windows?)
  • When --pager is set and falsy, disable pager

Docs from man man

-P  pager
              Specify which pager to use.  This option overrides the MANPAGER environment variable, which in turn overrides the
              PAGER variable.  By default, man uses /usr/bin/less -is.

Example (final formatting TBD)

help-color

neil new

Create new project

Supported options:

:name foo/bar (the main namespace)
:build true/false (defaults to true)
:deps-deploy true/false (defaults to true)
:license (mit / epl1 / etc, defaults to ???)

Proposal: `neil add nrepl`

This is an alias I find myself using often, so I thought it might be useful to add it to neil. I'm happy to help with implementation if the feature is approved.

Example:

$ neil new scratch nrepl-example
Creating project from org.corfield.new/scratch in nrepl-example

$ cd nrepl-example
$ cat deps.edn
{:paths ["src"]
 :deps  {}}

$ neil add nrepl
$ cat deps.edn
{:paths ["src"]
 :deps  {}
 :aliases
 {:nrepl
  {:extra-deps {nrepl/nrepl {:mvn/version "0.9.0"}}
   :main-opts ["-m" "nrepl.cmdline" "--interactive" "--color"]}}}

$ clj -M:nrepl
nREPL server started on port 63051 on host localhost - nrepl://localhost:63051
nREPL 0.9.0
Clojure 1.11.1
OpenJDK 64-Bit Server VM 11.0.15+10
Interrupt: Control+C
Exit:      Control+D or (exit) or (quit)
user=>

Support all possible "lib to url" cases

Source: https://clojure.org/reference/deps_and_cli#_coord_attributes
Related: https://clojure.atlassian.net/browse/TDEPS-228

Lib format Inferred :git/url
io.github.ORG/PROJECT "https://github.com/ORG/PROJECT.git"
com.github.ORG/PROJECT "https://github.com/ORG/PROJECT.git"
io.gitlab.ORG/PROJECT "https://gitlab.com/ORG/PROJECT.git"
com.gitlab.ORG/PROJECT "https://gitlab.com/ORG/PROJECT.git"
io.bitbucket.ORG/PROJECT "https://bitbucket.org/ORG/PROJECT.git"
org.bitbucket.ORG/PROJECT "https://bitbucket.org/ORG/PROJECT.git"
io.beanstalkapp.ORG/PROJECT "https://ORG.git.beanstalkapp.com/PROJECT.git"
com.beanstalkapp.ORG/PROJECT "https://ORG.git.beanstalkapp.com/PROJECT.git"
ht.sr.ORG/PROJECT "https://git.sr.ht/~ORG/PROJECT"

I think this should apply to any case where we currently infer the GitHub URL. That said, it may be easiest to start with neil new and work from there since the deps management for that command is standalone.

`neil add`: Incorrect formatting when `:aliases` key is missing

Steps to reproduce:

$ neil new scratch foobar
$ cd foobar
$ neil add nrepl
$ cat deps.edn
{:paths ["src"]
 :deps  {} :aliases {
 :nrepl ;; added by neil
 {:extra-deps {nrepl/nrepl {:mvn/version "0.9.0"}}
  :main-opts ["-m" "nrepl.cmdline" "--interactive" "--color"]}}}

Expected output:

{:paths ["src"]
 :deps  {}
 :aliases
 {:nrepl ;; added by neil
  {:extra-deps {nrepl/nrepl {:mvn/version "0.9.0"}}
   :main-opts ["-m" "nrepl.cmdline" "--interactive" "--color"]}}}

Extra space in `deps.edn` when creating scratch project

There's an extra space in the last line. Small issue but it's bugging me so I'm just going to fix it.

$ neil new scratch rads/foo
Creating project from org.corfield.new/scratch in foo

$ cat foo/deps.edn
{:paths ["src"]
 :deps  {}
 :aliases
  {:neil {:project {:name rads/foo}}}}

Dev: Run `clj-kondo --lint` in CI

I noticed CI was passing in my PR despite having an unused require. Usually IntelliJ catches this for me (there's a plugin for clj-kondo integration) but for some reason Cursive is getting errors in this project and my usual editor linting is not working.

Do we want to block on clj-kondo --lint in GitHub Actions so we can catch these issues before merging to main?

Dev: Organize tests by namespace

Right now the bb tests task is hard-coded to run tests.clj.

To keep the test suite manageable going forward it would be helpful if we could support the standard clojure.test conventions with a test directory containing a _test.clj file for each namespace.

I think we could work around this temporarily by manually requiring more test namespaces inside this file, but it's not ideal.

Add "dep search" command to neil help

I noticed that "dep search" command is missing which makes it harder for the users to discover.
It's well described in the README and I think it's the only one that's missing:

neil
Usage: neil <subcommand> <options>

Most subcommands support the options:
  --alias      Override alias name.
  --deps-file  Override deps.edn file name.

Subcommands:

add
  dep    Alias for `neil dep add`.
  test   adds cognitect test runner to :test alias.
  build  adds tools.build build.clj file and :build alias.
    Options:
    --deps-deploy Adds deps-deploy as dependency and deploy task in build.clj
  kaocha adds kaocha test runner to :koacha alias.
  nrepl  adds nrepl server to :nrepl alias.

dep
  add: Adds --lib, a fully qualified symbol, to deps.edn :deps.
    Run neil add dep --help to see all options.

new:
  Create a project using deps-new
    Run neil new --help to see all options.

  Examples:
    neil new scratch foo --overwrite
    neil new io.github.rads/neil-new-test-template foo2 --latest-sha

license
  list   Lists commonly-used licenses available to be added to project. Takes an optional search string to filter results.
  search Alias for `list`
  add    Writes license text to a file
    Options:
    --license The key of the license to use (e.g. epl-1.0, mit, unlicense). --license option name may be elided when license key is provided as first argument.
    --file    The file to write. Defaults to 'LICENSE'.

Error when invoking `clojure -M:neil`

deps.edn:

{:aliases
 {:neil {:deps {io.github.babashka/neil {:git/tag "v0.1.39" :git/sha "1f747a5"}}
         :main-opts ["-m" "babashka.neil"]}}}
$ clojure -M:neil
Syntax error (IllegalStateException) compiling fs/windows? at (babashka/neil/curl.clj:14:21).
var: #'babashka.fs/windows? is not public

Full report at:
/var/folders/tt/73f6l1j971n03fsxstmz_g280000gn/T/clojure-13803427228737455161.edn

Idea: Extensible `neil add`

Right now the only way to extend the neil add subcommand is by making a pull request. In contrast, neil new can be extended with any deps.edn lib.

Would it make sense for neil add to work in a similar way where the user can provide their own “template” if needed?

My main motivation for this is allowing the community to extend neil without having to mess around with its internal code. If there is a particularly popular or useful neil add extension, that could be a candidate for being included as a built-in.

# Built-in command (unqualified name)
$ neil add shadow-cljs

# External command (qualified name)
$ neil add io.github.rads/shadow-cljs

Commands to bump or set version patch | minor | major

Npm has npm version:

$ npm version --help
npm version

Bump a package version

Usage:
npm version [<newversion> | major | minor | patch | premajor | preminor | prepatch | prerelease | from-git]

I think neil could have the same (but for now only these three):

neil version patch
neil version minor
neil version major

similar to what npm does: it increments the current number.

And then use this {:neil {:project {:version ...}}} in build.clj

Npm also creates a git tag but I think we could have this as a separate command:

neil tag

which takes the current version and creates a git tag.

Or does it make sense to do it as npm does it and combine these two?

I think it makes sense to store the version numbers separately: {:major ".." :minor ".." :patch ".." :qualifier ".."} similar to how *clojure-version* is stored.

Maintain data in deps.edn

We could consider maintaining some data in e.g. the deps.edn:
{:org.babashka/neil {:lib foo.bar}} or {:aliases {:org.babashka/neil {:lib foo.bar}}}.

And you could add this with neil set :lib foo.bar or so. And then neil add test would generate the first test for you based on the :lib name.

We could do this (add the lib name to deps.edn) while you're doing neil new even.

Doc missing for `neil dep versions`

Entrypoint for neil dep versions appears to be missing from the global helptext. But the command works fine :)

# On master, as of 2022-08-16
$ (cd ~/dev/teodorlu/neil && git rev-parse HEAD)
e21bc88cdb0cfcb9e9d3bb4f5d8640885735ca9f
$ neil
Usage: neil <subcommand> <options>

Most subcommands support the options:
  --alias      Override alias name.
  --deps-file  Override deps.edn file name.

Subcommands:

add
  dep    Alias for `neil dep add`.
  test   adds cognitect test runner to :test alias.
  build  adds tools.build build.clj file and :build alias.
  kaocha adds kaocha test runner to :koacha alias.
  nrepl  adds nrepl server to :nrepl alias.

dep
  add: Adds --lib, a fully qualified symbol, to deps.edn :deps.
    Run `neil dep add --help` to see all options.

  search: Search Clojars for a string in any attribute of an artifact
    Run `neil dep search --help` to see all options.

new:
  Create a project using deps-new
    Run neil new --help to see all options.

test:
  Run tests. Assumes `neil add test`. Run `neil test --help` to see all options.

license
  list   Lists commonly-used licenses available to be added to project. Takes an optional search string to filter results.
  search Alias for `list`
  add    Writes license text to a file
    Options:
    --license The key of the license to use (e.g. epl-1.0, mit, unlicense). --license option name may be elided when license key is provided as first argument.
    --file    The file to write. Defaults to 'LICENSE'.
# But `dep versions` still works fine
$ neil dep versions hiccup/hiccup
:lib hiccup/hiccup :version 2.0.0-alpha2
:lib hiccup/hiccup :version 2.0.0-alpha1
:lib hiccup/hiccup :version 1.0.5
:lib hiccup/hiccup :version 1.0.4
:lib hiccup/hiccup :version 1.0.3
:lib hiccup/hiccup :version 1.0.2
:lib hiccup/hiccup :version 1.0.1
:lib hiccup/hiccup :version 1.0.0
:lib hiccup/hiccup :version 1.0.0-RC3
:lib hiccup/hiccup :version 1.0.0-RC2

Search fails for some deps

Version 0.0.23

neil dep search "org.clojure/tools.cli"
----- Error --------------------------------------------------------------------
Type:     java.lang.NullPointerException
Location: /usr/local/bin/neil:337:11

----- Context ------------------------------------------------------------------
333:   (let [search-term (first (:cmds opts))
334:         url (str "https://clojars.org/search?format=json&q=" search-term)
335:         {search-results :results
336:          results-count :count} (curl-get-json url)]
337:     (when (zero? results-count)
               ^---
338:       (binding [*out* *err*]
339:         (println "Unable to find" search-term  "on Clojars.")
340:         (System/exit 1)))
341:     (doseq [search-result search-results]
342:       (println :lib (format  "%s/%s"

----- Stack trace --------------------------------------------------------------
clojure.core/zero?       - <built-in>
babashka.neil/dep-search - /usr/local/bin/neil:337:11
babashka.neil/dep-search - /usr/local/bin/neil:332:1
babashka.neil/dep        - /usr/local/bin/neil:414:16
babashka.neil/dep        - /usr/local/bin/neil:408:1
babashka.neil/-main      - /usr/local/bin/neil:420:13
babashka.neil/-main      - /usr/local/bin/neil:416:1
babashka.neil            - /usr/local/bin/neil:425:3

Consider using `deps-info` for `neil new` implementation

I needed to use the same tools.deps inference for bbin so I moved the code to a shared library:

https://github.com/rads/deps-infer

I suggest we add a dependency for this lib in neil and update the neil new implementation to use this.

I plan to maintain deps-infer going forward since it's a useful library on its own without bbin or neil.

bb --config <(echo "{:deps $(deps-infer --lib io.github.rads/watch)}") \
  -e "(require 'rads.watch) (rads.watch/print-help nil)"

Alias option ignored in neil dep add library --alias alias

Am I just missing something? I'm unable to target custom aliases when adding dependencies to a deps.edn.

Expected behavior-

neil dep add vlaaad/reveal --alias :reveal would add the latest version of reveal to an alias :reveal under its :extra-deps

Apparent behavior-

The dependency is added to :deps.


These commands are working as expected:
  • neil add nrepl --alias :nrepl-custom-alias
  • neil add kaocha --alias :kaocha-custom
  • etc.
  • neil add dep vlaaad/reveal --version (custom-version)
  •              "             :version (custom-version)
  •              "             --as different-name
In these commands, `--alias` seems to be ignored:
  • neil add dep vlaaad/reveal --alias :reveal
  •              "             --alias reveal
  •              "             --alias "reveal"

Same goes with all versions of the command with neil dep add.

So, the alias flag is working both as a CLI flag and a keyword for all commands except add dep. All other flags for add dep besides alias also work.


Taking a look at neil.clj, dep-add is the only fn that adds dependencies and doesn't call add-alias or check for the keyword.

Cannot install neil with homebrew

brew install babashka/brew/neil
=>

==> Tapping babashka/brew
Cloning into '/usr/local/Homebrew/Library/Taps/babashka/homebrew-brew'...
remote: Enumerating objects: 44, done.
remote: Counting objects: 100% (44/44), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 44 (delta 7), reused 43 (delta 6), pack-reused 0
Receiving objects: 100% (44/44), 5.42 KiB | 1.81 MiB/s, done.
Resolving deltas: 100% (7/7), done.
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/babashka/homebrew-brew/Formula/[email protected]
No available formula with the name "[email protected]". Did you mean [email protected] or [email protected]?
In formula file: /usr/local/Homebrew/Library/Taps/babashka/homebrew-brew/Formula/[email protected]
Expected to find class NeilAT008, but only found: Neil.
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/babashka/homebrew-brew/Formula/[email protected]
No available formula with the name "[email protected]". Did you mean [email protected] or [email protected]?
In formula file: /usr/local/Homebrew/Library/Taps/babashka/homebrew-brew/Formula/[email protected]
Expected to find class NeilAT009, but only found: Neil.
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/babashka/homebrew-brew/Formula/neil.template.rb
No available formula with the name "neil.template".
In formula file: /usr/local/Homebrew/Library/Taps/babashka/homebrew-brew/Formula/neil.template.rb
Expected to find class NeilTemplate, but only found: Neil.
Error: Cannot tap babashka/brew: invalid syntax in tap!

Error when running `neil dep upgrade`

upgrade still hasn't been pushed to nixpkgs, so I overrode it with the latest sha from github:

  neil = super.neil.overrideAttrs (oldAttrs: rec {
    src = super.fetchFromGitHub {
      owner = "babashka";
      repo = "neil";
      rev = "3b61436e3";
      hash = "sha256-mrvxfSHI1nIQyhiMLgFUh8dnzgOv16DYEKwjIoQsQo8=";
    };
  });

When I run neil dep upgrade from a nix shell with the above overlay, I receive the following error:

----- Error --------------------------------------------------------------------
Type:     clojure.lang.ExceptionInfo
Message:  Could not resolve symbol: fs/windows?
Location: /nix/store/x4k2binyymh02qwgisgmhziypiiknl2b-neil-0.0.13/bin/.neil-wrapped:34:29
Phase:    analysis

----- Context ------------------------------------------------------------------
30: (def dev-github-token (System/getenv "BABASHKA_NEIL_DEV_GITHUB_TOKEN"))
31: 
32: (def curl-opts
33:   (merge {:throw false
34:           :compressed (not (fs/windows?))}
                                ^--- Could not resolve symbol: fs/windows?
35:          (when (and dev-github-user dev-github-token)
36:            {:basic-auth [dev-github-user dev-github-token]})))
37: 
38: (defn curl-get-json [url]
39:   (-> (curl/get url curl-opts)

Operating System: NixOS 22.05 (Quokka)
Kernel: Linux 5.15.72
Architecture: x86-64

Possibly more intuitive CLI command / subcommand naming?

Working on #16, I didn't find any place to fit in "list available versions for a library" into the current subcommand stucture. Toplevel add doesn't fit - we're not adding versions. Yet, listing available versions is very much related to adding a version.

Grouping dependency stuff together makes sense to me. For example:

# :lib as option - or as the first command
neil dep add :lib org.clojure/clojure
neil dep add org.clojure/clojure

# dep versions subcommand can have same structure.
neil dep versions :lib org.clojure/clojure
neil dep versions org.clojure/clojure

Not sure about what to do about the rest of the add subcommand. Perhaps just keep it. We woiuld have two ways of adding a dependency (neil dep add and neil add dep).

CLI for listing available versions of a library

WIP PR: #17

For example:

$ neil dep versions hiccup/hiccup
:lib hiccup/hiccup :version 2.0.0-alpha2
:lib hiccup/hiccup :version 2.0.0-alpha1
:lib hiccup/hiccup :version 1.0.5
:lib hiccup/hiccup :version 1.0.4
:lib hiccup/hiccup :version 1.0.3
:lib hiccup/hiccup :version 1.0.2
...

If we add this, we could create a one liner to add a specific version of a dependency to deps.edn. For example using fzf.

neil dep versions hiccup/hiccup | fzf | xargs neil add dep

Example grepping for version 1.0:

neil dep versions hiccup/hiccup | grep ":version 1.0" | fzf

dep info flag

How difficult would be to add dep info option?
It would be nice to be able to jump to the homepage of a library.

also, somewhat related: dep search, in addition to lib-name and version, could also return the description line for each package.

Proposal: `neil add shadow-cljs`

Rationale

I recently converted a project from figwheel-main to shadow-cljs and now have a pretty good understanding of the challenges when using these libraries in 2022. Though figwheel-main was working fine once I set it up, I ended up switching to shadow-cljs for more community support. That said, when setting up both build tools, I had to parse a lot of information to build a minimal but complete dev workflow that included a ClojureScript REPL, tests, CI, etc...

One option to make this easier now that neil new supports external templates would be to create a configurable shadow-cljs template, which is similar to the approach of frameworks like kit-clj. That's a totally valid approach and it's why I'm also working on a PR for kit-clj to add deps-new compatibility which will enable this top-down workflow with neil.

That said, the template approach doesn't work as well for adding shadow-cljs to an existing project. This is where neil has an opportunity to shine, providing a bottom-up workflow in addition to the top-down one. Beyond that, if we could have a tool that knows how to add shadow-cljs features incrementally with sane, modern defaults, this would be a boon to ClojureScript developers everywhere.

Example

# Create empty project
$ neil scratch my-project
$ cd my-project
$ neil add shadow-cljs

# Start a watcher for app and tests
$ clj -M:shadow-cljs watch app test
shadow-cljs - HTTP server available at http://localhost:9500
shadow-cljs - server version: 2.19.8 running at http://localhost:9630
shadow-cljs - nREPL server started on port 51484
shadow-cljs - watching build :app
shadow-cljs - watching build :test
[:app] Configuring build.
[:test] Configuring build.
[:app] Compiling ...
[:test] Compiling ...
[:test] Build completed. (147 files, 1 compiled, 0 warnings, 5.38s)
[:app] Build completed. (417 files, 0 compiled, 0 warnings, 5.55s)

# Auto-run and hot-reload CLJS tests
$ open http://localhost:9500

# Generate an optimized bundle for production
$ clj -T:build bundle
added 652 packages, and audited 653 packages in 6s
[:app] Compiling ...
[:app] Build completed. (384 files, 0 compiled, 0 warnings, 22.52s)

# Generate an uberjar with the bundle step included
$ clj -T:build uber
$ cd target && jar tf *.jar | grep main_bundle.js
public/js/main_bundle.js

Requirements

Note: Due to the number of features in shadow-cljs, we may consider options within neil add shadow-cljs to generate only a subset of the features below (e.g. --karma-tests false if you don't want karma.conf.js).

  • neil add shadow-cljs should fill in the following files as needed:
    • deps.edn
    • shadow-cljs.edn
      • Add :deps true to enable tools.deps integration
      • Add :builds :app with :target :browser
      • Add :builds :test with :target :browser-test
      • Add :dev-http to run tests from :builds :test
      • Add :builds :ci with :target :karma
    • build.clj
      • Add bundle function
      • Update uber function to include bundle
    • package.json and package-lock.json
      • Includes dependencies for :target :karma
      • Used for adding NPM deps with shadow-cljs
      • Can also be used to enable npx shadow-cljs command, but this is optional with tools.deps integration
    • karma.conf.js
      • Used for :test build in shadow-cljs.edn
    • .gitignore
      • Ignore .shadow-cljs directory
    • .github/workflows/cljs-tests.yml
      • Auto-run :ci build on push to main

Dev: Avoid GitHub API rate limits with basic auth

Quick example for dev:

;; babashka.neil.curl

(def dev-github-user (System/getenv "BABASHKA_NEIL_DEV_GITHUB_USER"))
(def dev-github-token (System/getenv "BABASHKA_NEIL_DEV_GITHUB_TOKEN"))

(def curl-opts
  (merge {:throw false
          :compressed (not (fs/windows?))}
         (when (and dev-github-user dev-github-token)
           {:basic-auth [dev-github-user dev-github-token]})))
$ bb -e "(-> ((requiring-resolve 'babashka.neil.curl/curl-get-json) \"https://api.github.com/rate_limit\") :rate)"
{:limit 60, :used 8, :remaining 52, :reset 1660690276}

$ export BABASHKA_NEIL_DEV_GITHUB_USER=rads
$ export BABASHKA_NEIL_DEV_GITHUB_TOKEN=op://Private/Github/access-token
$ op run -- bb -e "(-> ((requiring-resolve 'babashka.neil.curl/curl-get-json) \"https://api.github.com/rate_limit\") :rate)"
{:limit 5000, :used 24, :remaining 4976, :reset 1660688942}

This would be the simplest solution to avoid the GitHub rate limits when running bb tests locally. Using basic auth raises the limit from 60 requests per hour to 5000.

The rate limit seems to be a common pitfall for new contributors to this project so we may want to add something to the README as well. Maybe some better error handling for the rate-limiting HTTP response too.

If this is useful outside of dev we could add it as an command line argument, but I think the env variable would solve the immediate problem since hitting the rate limit is usually due to running bb tests. I don't think we should support passwords, only Personal Access Tokens.

Add license

It'd be nice if you can add a license, it'd make easier to create a Neil package for Linux distros.
And thanks for Neil, it's small but quite helpful :)

Proposal: Local curl request caching

Background

Neil makes use of the github api when checking for the latest version, in both the neil dep add and neil dep upgrade flows (and likely others). Github has a default rate-limit of 60 requests per hour - after that, requests for latest commits will return a 403. Github rate limit docs here.

This ends up nil-punning in the current code, eventually leading to a misleading "no remote found for lib" error here - improving this would be some low-hanging fruit, and probably the least amount of work and complexity.

This has been discussed in a few other places:

  • #111 in particular proposed a solution that local devs and end users can take advantage of, by creating and setting a personal access token as an env var.
  • #127 some further discussion happened here (I hadn't seen the previous issue yet)

the (potential) problem

This rate limit could be a problem to local dev and testing, especially as the tests grow. The upgrade tests that merged this week now make quite a few network calls (6 to github, from my read of them) - this makes it much easier to hit the rate limit even when working on just those tests.

It's possible for neil's end users to hit this limit as well. A project with 31+ git dependencies (which is extreme) would only be able to update their deps once per hour, assuming neil is the only tool they are hitting the github api with - the rate-limit is also ip-based, so the 60 hits per hour could be shared. This means if you undo your upgrade, then try to redo it, it will fail. Definitely an extreme case!

a potential solution: a local cache

One way to deal with this could be with a local cache that cuts off repeated web requests. Repeated calls to neil dep upgrade could return without making any web requests (after the first call). Users should be able to clear the cache themselves, but otherwise updates to libraries are rare enough that caching for some amount of time (an hour) seems fine.

This would speed up the tests, and mostly dodge this whole rate limit thing for end users, unless they forced it.

Caching introduces complexity that neil doesn't have to deal with at all right now - it could create some unexpected behavior for users or devs, if it's not clear there's a caching layer. It also adds another artifact/bit of state to manage, because it's another whole file to read/write from. So, definitely some tradeoffs/drawbacks.

I implemented something like this on a personal project, and had good success with an in-memory cache that was just a map with request parameters as keys and the response object as a value, so that's the direction I'd recommend going:

;; something like ~/.config/neil/request-cache.edn, a map
{;; the keys are the request params we would pass to curl
 "<request-url>"
 ;; the values are a tuple of request-time and response-object
 [request-timestamp response-parsed-json-blob]}

Actual Impact?

Really, I don't think this is too critical to neil end-users at the moment. I'm motivated to address it because I introduced a number of github requests in the PRs that merged this week, and developers on this project may now be forced to use the new env vars to work with the repos tests.

Working on neil offline is not really possible - this solution would make it reasonable for the tests to completely mock requests with some fixtures... but that's probably not too relevant, as neil is definitely not an offline tool.

TLDR

Sorry for the short novel!

This is all a long way of saying we could cache/memoize the curl-get-json function in neil/curl.clj. That feels like a good spot to cut off requests that we don't need to make repeatedly, when the result is the often the same (unless you just released a new version, in which case we should provide a cache-clearing command).

It'd require reading/writing from a cache file somewhere, so we'd need to decide where we're ok with that file living. This would also require some new commands and help text to make it clear to users that a cache is in play.

In the meantime, devs and users can take advantage of the new env vars, once they setup a personal access token.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.