nixos / nixos-search

Search NixOS packages and options

search.nixos.org

This repository contains the scripts and the web application for search.nixos.org.

How this project came to be

Initial idea was to replace NixOS packages and options search which was fetching one JSON file which contained all packages (or options). This approach is good for its simple setup, but started to show its problems when packages number was getting bigger and bigger. I'm sure we could optimize it further, but ideas what all could we do if there would be some database in the back were to tempting not to try.

For backend we are using Elasticsearch instance which is kindly sponsored by bonsai.io. On the frontend we are using Elm.

How search works?

The use case we want to solve is that a visitor want to see if a package exists or to look up certain package's details.

A user wants to converge to a single result if possible. The more characters are added to a search query the more narrow is search is and we should show less results.

Very important is also ranking of search results. This will bring more relevant search results to the top, since a lot of times it is hard to produce search query that will output only one result item.

A less important, but providing better user experience. are suggestions for writing better search query. Suggesting feature should guide user to write better queries which in turn will produce better results.

Development

To start developing open a terminal and run:

env --chdir=frontend nix develop -c yarn dev

You can point your browser to http://localhost:3000 and start developing. Any changes to source files (./frontend/src) will trigger a hot reload of an application.

Deploying

• On each commit to main branch a GitHub Action is triggered.
• GitHub Action then builds production version of the web application using yarn prod command.
• The built web application (in ./dist) is then deployed to Netlify.
• GitHub Action can also be triggered via Pull Request, which if Pull Request was created from a non-forked repo's branch, will provide a preview url in a comment.

Adding flakes

To add your own flakes to the search index edit ./flakes/manual.toml.

Possible types are github, gitlab, sourcehut, and git (which is the fallback for any kind of git repository but requires to set a revision key manually as of now).

To test whether your flake is compatible with nix flake-info you can try running flake-info against it

$ nix run github:nixos/nixos-search#flake-info -- flake <your flake handle>

nixos-search's Issues

[review] Navbar "traps" users on the search sub-site

As part of the review for NixOS/nixos-homepage#454

---

The navbar's logo should probable link back to the nixos.org website.

I would additionally add a first option between the logo and "Packages" that says something like "Back to nixos.org" or similar, just to ensure people "are not trapped" on the packages website.

(Yes, I know our target audience is tech-savvy and knows about the back button, but this is the kind of detailing work that can make or break changes in sites).

---

Additionally (minor nitpick), it is pretty weird to see the navbar being a bit "wider" there.

Mark or hide unsupported/depricated packages from pkg search

There are a couple times where I've been looking at packages only to find out they're depricated, have outstanding security warnings, etc. only to find the notice after trying to install the package or view it on the nixpkgs repo

It'd be nice if the search result marked the item with these warnings or hid them

cleanup how we handle licenses

How licenses are currently handled in nixpkgs makes it impossible to show them in a reliable way. We can of course guess but it would be much nicer if we could fix the issue in nixpkgs and require certain structure for meta.licence.

This would require to write some assert statements in check-meta.nix

Package search: Add link to issues

My idea was to add a "Related issues" row to the package datails with the number of related GitHub issues. This could be hidden with JS when there are no open issues.

Mockup

We can probably get the number of open issues from the GitHub API when expanding the details and the link is just a search like https://github.com/NixOS/nixpkgs/search?q=pipenv&state=open&type=Issues

This would add a soft-dependency on GitHub API since the number will not appear when it can't be reached. But when GitHub is down, we have bigger problems...

What do you think?

integrate into nixos.org website and remove existing packages/options search

[review] Options description are not parsing docbook contents

As part of the review for NixOS/nixos-homepage#454

---

This is a major downgrade, and a pretty major issue.

Compare services.firefox.syncserver.enable.

This also currently directly uses the bytes without consideration for XSS!!!!

See how it's been done in the previous implementation for known cases.

Require certain fields for each maintainer

Currently we need to guess which fields are there. It would be nice to have some certainty about this.

html double escape in options.html

the javascript in https://github.com/NixOS/nixos-homepage/blob/master/nixos/options.tt#L233 appears to escape < and > twice

an example can be seen on https://nixos.org/nixos/options.html by searching for services.znc.confOptions.passBlock

Rewrite how we look up for packages

Currently we use the following command:

nix-env \
    -f '<nixpkgs>' \
    -I nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/{evaluation['git_revision']}.tar.gz \
    --arg config 'import {CURRENT_DIR}/packages-config.nix' \
    -qa \
    --json

Which works ok, but we might consider extracting more information:

• dependencies, which would then allow us to create some sort of navigation between packages
• package args (withPython, etc...)

Implement Error view for search results

Implement NotAsked view for seach results

second click on Submit button clears up the search

it might be related but also first visit with "query" querystring does not trigger search.

[review] Off-by-one on "last" page

As part of the review for NixOS/nixos-homepage#454

---

At the very least, that's what it feels like.

Reproduction:

• Search for results that give a multiple of the results count (e.g. 30 results for 15 per page)
• https://search.nixos.org/packages?query=firefox&channel=unstable
• Use the Last button
• Observe how it's giving you an empty page
• Observe URL: from=30&size=15

[review] Lack of permalinks

As part of the review for NixOS/nixos-homepage#454

---

This new search lacks the attrname-based permalink of the previous implementation:

• https://nixos.org/nixos/packages.html?attr=linuxPackages_4_14.kernel

Note that this shows only the relevant result. This is especially good for third parties that want to link to the precise page of a package. I'm not sure, but I think the wiki has a template that uses this feature so an attribute name can be linked directly to.

I see this could be "replaced" by an opaque ID like fpxcfHIBjCujEan6IXYu, but I believe it would be a mistake, as using those opaque IDs in user-facing ways will only serve to create URLs that rot. An "evergreen" attribute name based way to get a package is required.

In my opinion this cannot be shipped without readable attrname-based URLs.

Channel change in links is immediate (while search is on activating the search action)

(Observed when verifying #72 and #70)

---

This could be seen more as a nitpick, but this is more about keeping things legit and well-organized all the way.

It seems that the links are not built using the state from the current search, but from an amalgamation of the current state of the form + the state of the search. I would expect that, for all results, things always end up changing to either pressing the search button or all form changes. This is currently a mish-mash.

• Be on unstable channel.
• Search for hello.
• Open the hello result
• Observe where links point to.
• Change the channel to 20.03 DO NOT update using Search.
• Observe the links changing.

For hydra links, this is not an issue, but for GitHub location, it is. The location for a result across channels can and does change!

Improve search quality / UX

• no expensive wildcard, #35
• ranking, #36
• suggestions / completion suggesters, #74
• facet search, aggregation (grouping of package sets)

[review] Platforms list in search result should be limited

As part of the review for NixOS/nixos-homepage#454

---

It should be limited to the main supported platforms, here's the previous implementation:

https://github.com/NixOS/nixos-homepage/blob/c5780d9cff360d5a0af9a7d0dac2e12b991386ef/packages-explorer/src/gui/result.js#L11-L16

This is especially needed since most of the platforms shown will actually be broken since they are not tested. Take for example firefox, listing:

Most of those are not actually working.

Note that there is an additional filtering against darwin when the channel is nixos-*

https://github.com/NixOS/nixos-homepage/blob/c5780d9cff360d5a0af9a7d0dac2e12b991386ef/packages-explorer/src/gui/result.js#L105-L106

This is for #72.

[review] Changing channel does not reload the contents without an action

As part of the review for NixOS/nixos-homepage#454

---

The dropdown is after the search "action", but requires backtracking to be taken into consideration.

• Enter search terms
• Activate Search button
• Change channel
• Observe the contents have not refreshed

I would expect the channel results to change since the search button is intimately linked to the input box.

A solution would involve placing the channel selection before the action button. If we think hierarchically, the channel is at the "root" of the tree for our taxonomy, in a way. I think this means that the channel selection should be before the search box. Though it would be fine before the button, and after the search query, but harder to integrate in the flow visually.

I know that it is by design that results don't change when inputs change, but I believe the drop down could be an exception to this rule, considering its position.

implement details view for Packages

Option default and example value is somehow munged when indexed

From the API:

The value is actually false. This makes me concerned that the indexing could be munging other kind of values into different representations of their actual definition.

For some reason search is always performed on "nixos-unstable-packages" index even if on Options list

Implement different view for empy list of successfully fetched items

show under as search.nixos.org

paginate results

Add support - by default - to search with * on both sides.

Reproduce: searching for "service" works and shows results, while searching for "servic" doesn't return any results

[review] Loss of search-as-you-type is a major loss of functionality

As part of the review for NixOS/nixos-homepage#454

---

This is especially for the options, as it affects packages a bit less.

I'm bad at remembering things. All things. I generally don't remember the option names right, but only parts of them. I actually use the search-as-you-type feature of the site to work around my failings.

Losing search as you type in options would actively harm, at the very least, my ability to give help regarding options, in addition to slow down considerably my own personal use of options.

If this is not possible, I will likely have to implement an additional, web-based or not, fuzzy-finder that searches as you type for option names.

[review] Default or example values can accidentally disappear

As part of the review for NixOS/nixos-homepage#454

---

The option example value "None", and empty string "" are deemed magical.

nixos-search/src/Page/Options.elm

Lines 185 to 194 in 0a6369e

	withDefault wrapWith value =
	case value of
	"" ->
	text default
	"None" ->
	text default
	_ ->
	wrapWith value

This actually clashes with existing values.

See:

For "":

• https://search.nixos.org/options?showDetailsFor=-NNdfHIBaoxxchBBuEZ8&query=printing&from=0&size=15&channel=unstable (services.printing.extraFilesConf).
• https://nixos.org/nixos/options.html#extrafilesconf

For "None":

• https://search.nixos.org/options?showDetailsFor=RNNdfHIBaoxxchBBuEd8&query=collectd&from=15&size=15&channel=unstable (services.prometheus.exporters.collectd.collectdBinary.securityLevel)
• https://nixos.org/nixos/options.html#services.prometheus.exporters.collectd.collectdbinary.securitylevel

Implement details view for Options

run import script daily (via github actions)

Add the possibility to search for files in the packages, like a command name

It would interesting to have a page that permits the users to search for a file in the packages. A good example is a program that you need but you only now the executable name that differ from the package name. For example, I was looking for the readelf command and I was not able to find it from the Packages search page or the nix search tool.

Debian has a good tool for that:
https://www.debian.org/distrib/packages#search_contents
The readelf example:
https://packages.debian.org/search?searchon=contents&keywords=%2Fbin%2Freadelf&mode=path&suite=stable&arch=any

It would be very nice to have a similar page in the Nixos site.

Project should be built by nix

Currently we run nix-shell --command "yarn build" and then copy ./dist folder. It would be very nice to be just able to do nix-build.

When trying this initially something was happening with Elm that things didn't work. Need to give it another try.

We should also make sure to create an environment to develop ./scripts and that we are suing mypy/black there.

Also search in other fields

This what the query could look like:

{
        "query": {
            "bool": {
                "should": [
                    {"wildcard": {"attr_name": "*nixpkgs-review*"}},
                    {"wildcard": {"name": "*nixpkgs-review*"}},
                    {"wildcard": {"homepage": "*nixpkgs-review*"}}
                ],
                "minimum_should_match" : 1,
                "boost" : 1.0
            }
        }
    }

It would be also useful to look in description and longDescription.

[review] Search bar uses visually mismatched items

As part of the review for NixOS/nixos-homepage#454

---

The search input row is pretty visually mismatched.

First of all, it uses different sizes for the inputs. Already it hits quite hard. Though I think the real crime is the mostly unstyled <select /> next to a completely styled text box.

I know, styling inputs is fraught with perils, as if it was an afterthought with web browsers. Though it really clashes.

Index NUR packages

Would it be possible to also import NUR? We don't need to show search results for that on the NixOS homepage however, this would allow it to re-use nixos-search for NUR as well.
We already generate a simple packages.json over all repositories that can be imported. It would be great if there was a unified search that would first search nixpkgs and than fallsback to NUR in case nixpkgs does not have the package.

show details should be included in url

[review] Searching for a partially or fully qualified option name fails

As part of the review for NixOS/nixos-homepage#454

---

Search for boot.initrd, no result. Even boot initrd fails.

This is a critical issue.

Add footer: Supported by bonsai.io :heart: Thank you!

Search result issues

The first thing i tested already fails:

https://search.nixos.org/options?query=bees&from=0&size=15&channel=unstable

this is what should be found:

https://nixos.org/nixos/options.html#bees

and then try to search for "beesd". Nothing is found anymore!

When you found the option you need, you want to copy them next. That also fails! https://peertube.social/videos/watch/9d1ccb0c-9811-4107-aece-fa2648c0b213

Haven't tested more yet, but there might be more issues.

IPv6 support of search backend

Hate to bring it up again but the backend we are using for the search doesn't seem to support IPv6. While Elasticsearch is perfectly capable of IPv6 the host (despite being on AWS) failed to enable it. This is a discrimination against people with poor (or none) IPv4 connectivity on this planet.

Like explained in my NixOS.org Netlify post we should try to have this as a baseline requirement for new services that will - to some degree - be mandatory/required to interacting with our services/websites.

[review] Link to hydra job missing

As part of the review for NixOS/nixos-homepage#454

---

https://nixos.org/nixos/packages.html?attr=firefox&channel=nixpkgs-unstable&query=firefox

The Platforms link link to the job for the given platform for the given attribute. It is currently missing from the new implementation.

[review] Navigation between packages↔options rudely loses input

As part of the review for NixOS/nixos-homepage#454

---

Imagine I input my search term in the packages search, and oops, I wanted to search into options instead. Changing section loses the input.

This is a mostly minor fault, but since this is a richer app, we have the ability to change sections in a flexible manner, navigating while keeping the values entered by the user. It is minor, but helps smooth out the experience.

This can be implemented after the release and is of lower priority

option_name and attr_name should be indexed as keywords not as text

[review] More user-friendly "network error" message

As part of the review for NixOS/nixos-homepage#454

---

The following message is vague, and considering our audience, most likely wrong.

Network Error!
Please check your network connection.

We may want to introduce a note about content blockers, something like the following (if we can distinguish third party from first party)

Network Error!
A network request [to a third party domain] failed. This is either due to a content blocker or a networking issue.

[review] Navigation to a selected row can be wonky (not scrolled)

As part of the review for NixOS/nixos-homepage#454

---

Navigating to an open row is not good.

• Resize your viewport or zoom so that a search's results last entries fall under the fold.
• Scroll down
• Click the last option
• Observe (minor) the whole contents vanish for a brief moment (mostly cosmetic?)
• Observe (major) you are scrolled up to the top, with no clear indication that something really happened "down there".

Coincidentally, or maybe not, it looks like whatever handles the history API navigation does not handle resetting the scroll to where the user was. (TBF, the previous search didn't either, it is more of a clue.)

I think though, that the problem actually comes from the mostly cosmetic vanishing of contents. As the DOM is emptied, scroll goes naturally to the top.

Furthermore, if the solution to permalinks are navigation to an opened row, not scrolling the open result in view will look broken.

Add support to search different channels

Implement Loading view for search results

second click on open item in search should close the details

Option github links to the right channel TODO to be done

• nixos-search/src/Page/Options.elm

  Lines 176 to 178 in 0a6369e

  	-- TODO: this should take channel into account as well
  	githubUrlPrefix =
  	"https://github.com/NixOS/nixpkgs-channels/blob/nixos-unstable/"

Right now, a link to the unstable channel branch name is hardcoded. This should use the revision the options listing was indexed with.

Showing related packages

It might be interesting to add a meta field relatedPackages which would describe some relations between certain packages. This is similar how Debian does it, where they say:

Suggests

This is used to declare that one package may be more useful with one or more others. Using this field tells the packaging system and the user that the listed packages are related to this one and can perhaps enhance its usefulness, but that installing this one without them is perfectly reasonable.

This could be

Example: docker would define docker-compose as related package

meta.relatedPackages =
  { docker-compose = {
      description = "  docker-compose can be used...";
    };
  };

This might also be used in future nix cli

reduce shards in elastic search

right now we create 2 indexes per channel. one for packages and one for options. the data from one evaluation should be in one index.