haskell / hackage-server Goto Github PK

For packages that define only executables, and no libraries, we record the corresponding docs as having "failed to build" rather than "not applicable".

Some docs contain references to localhost as well as wrong URLs (some sort of cross between new and old Hackage styles), due to a (now fixed) bug in the build client. All we need to do is to delete all the docs and re-import / rebuild them.

At the moment the doc builder submits build reports for all packages it builds. Since it rebuilds all dependencies for all cases, that means that for some packages we submit the same build report over and over again (the beta hackage 2 server already has 112,000 build reports, after a few days of doc building, with some packages nearly 2000 build reports).

Renaming or coalescing tags (applying to all tags on all packages, not just individual package)

At the moment, all we have is the utterly trivial rundocs.sh script; this should be more in line with the continuous mode of the mirror client (with support for --keep-going too, probably).

So I tried to register as a new user at http://beta.hackage.haskell.org/users/register-request but when I clicked on "request account" I got the error:

Server error: sendmail exited with error code ExitFailure 1

My email address was correct and as far as I can tell this should be working (ie there's no messages saying it isn't enable yet).

Without threaded CTRL-C handling does not work correctly:

<starting to build package X>
<notice that package X failed>
<continue with package Y>
<cabal starts to download a dependency of Y>
<CTRL-C during the download>

At this point both cabal process is interrupted, but the doc builder process is not; a subsequent CTRL-C interrupts the doc builder but now the exception handler (installed by "finally") which writes out the "failed" state does not get run. With the -threaded option this does not happen: the CTRL-C interrupts both cabal and the doc builder, and the finally handler does run. Not necessarily a problem to need -threaded, but it would be nice to understand why.

I did modify the code to do that, and I think it worked at some point, so is probably a simple mistake somewhere.

Enable garbage collection of more things

Apparently we have lots (#31), but where are they?

For example:

Hackage has been upgraded to use a more secure login system. Please go to http://hackage:8081/users/htpasswd-upgrade to re-enable your account and for more details about this change.

I.e. keywords appearing close together and in order score higher.

In particular, the number of subdirectories in core/packages can exceed (Linux) OS limits, making it impossible to unarchive the tarfile; that will fail with

tar: backup-2013-08-30/core/package/yesod-core-1.1.8.3: Cannot mkdir: Too many links

We need a nice monadic style where we can fail with e.g. internal server errors without propagating Either blah blah all over the place. See for example how we cannot currently abstract over takeRequestBody.

For example, during backup, we issue a warning when we attempt to export a blob that doesn't exist on disk (39e050b); however, these warnings can easily get lost in the rest of the noise.

We should probably be using some sort of logging library, and be able to issue warnings and errors in such a way that log monitoring tools can pick them up and notify admins.

Hackage 2 allows some updates for the package descriptions for already uploaded package, and stores the various revisions. Could Hackage use Git as backend and push changes to a public readonly mirror somewhere? I imagine that people can easily take a look at it; and probably it would be somewhat easy, for instance, to use Git commit ids for people interested in repeatable builds.

I mentioned this today at Zurihac, but I fear I didn't explain myself clearly.

Right now the query "json" doesn't find the most popular JSON package aeson due to aeson using a capital "JSON" in its description.

While we have at least switched it all to use the haddock "Ocean" theme, there's a lot more that could be done by someone good at web design. The pages are currently constructed of a mixture of templates and pages generated in code. We should try to do that consistently with templates and make it easier to adjust the site. It should also be possible to do more JS stuff, rather than classic html4 POST forms.

Package candidates lets you upload a "candidate" giving it a url that others can download from, and gives an opportunity for doc builders and build bots to try the package out. Once the author is satisfied then they can publish to the main package index. This feature is about 95% complete (we think).

In addition to package maintainers, hackage trustees are people who can help curate a collection of packages. While they cannot upload new versions of packages directly, the intention is that they can edit package metadata, including adjusting dependencies to match reality. Currently the only things they can edit are package tags and upload documentation.

(analogous to the mirror client)

This feature was implemented but is currently turned off because it used to much memory. This should be investigated, the data structured adjusted and the feature re-enabled. The number of reverse dependencies should be an important component of a package popularity/quality metric.

There is partial support for uploading build reports and build logs. This needs to be extended and should include anonymous build reports that the cabal-install client can generate (currently we only support the non-anonymous/detailed build reports). This way we can gather huge amounts of data on what packages work in what circumstances. We can use that to help inform maintainers and users and to edit package dependencies to match reality.

When interrupting the doc builder with CTRL-C, the package currently being built will be marked as failed.

This should take advantage of tag sets (#24), so that we can automatically translate some category someone uses into the proper tag.

I'm marking this as high priority because if it's a bug in the blob storage code itself, rather than in the build reports feature, it can affect the whole server.

This means that if we try to mirror a package from the old hackage this will fail if that package was uploaded by a newly created user.

This is particularly helpful when others can set up their own Hackages.

On http://beta.hackage.haskell.org/package/system-filepath-0.4.7

The repository info is rendered as

head: bzr branch https://john-millikin.com/branches/system-filepath/0.4/
this: SourceRepo {repoKind = RepoThis, repoType = Just Bazaar, repoLocation = Just "https://john-millikin.com/branches/system-filepath/0.4/", repoModule = Nothing, repoBranch = Nothing, repoTag = Just "system-filepath_0.4.7", repoSubdir = Nothing}

of which the 2nd line is not what a user expects to see there.

Earlier IRC discussion with @dcoutts providing directions how to fix this:

< dcoutts_> hvr: dunno why it renders like that, probably a missing case in the code
< dcoutts_> hvr: see sourceRepositoryToHtml :: SourceRepo -> Html in Distribution.Server.Pages.Package
< dcoutts_> hvr: the current code expects the module, branch and tag to be empty
< dcoutts_> but that's clearly not the case here
< dcoutts_> yours has a tag
< dcoutts_> which isn't wrong, but we don't know how to render that
< dcoutts_> should be easy to fix

(For instance, try http://new-new-hackage.haskell.org/package/hpuz).

(in the blob storage API)

For instance, if we have a 500 error because the disk is full, the log records that.

(particularly with authentication)

And we should use these sets when coalescing (see #23).

With these sets we can actually have a set of tag names that all refer to the same canonical tag name; we would use this in the search feature, when we search on tags. This means we can tidy up the tags and make them useful for organising and searching since at the moment people pick different names for the same thing

Many resources have json or other machine readable formats, but not all yet. See the api page (which itself has a json version). This is an easy way to contribute to the development.

DELETE, PUT, and (in some cases) POST should not redirect, but rather return a document indicating the location of the new resource, if necessary, or other cues to continue browsing.