GithubHelp home page GithubHelp logo

hound-search / hound Goto Github PK

View Code? Open in Web Editor NEW
5.6K 135.0 568.0 3.21 MB

Lightning fast code searching made easy

License: MIT License

Go 26.62% Shell 0.19% JavaScript 70.32% CSS 0.93% HTML 0.15% Makefile 0.14% Dockerfile 0.05% Less 1.60%
hacktoberfest

hound's Introduction

Hound

Build Status .github/workflows/go.yaml

Hound is an extremely fast source code search engine. The core is based on this article (and code) from Russ Cox: Regular Expression Matching with a Trigram Index. Hound itself is a static React frontend that talks to a Go backend. The backend keeps an up-to-date index for each repository and answers searches through a minimal API. Here it is in action:

Hound Screen Capture

Quick Start Guide

Building hound

  1. Install go (minimum version required: 1.16) and npm

  2. Clone the repository and run make.

git clone https://github.com/hound-search/hound.git
cd hound
make

The resulting binaries (hound, houndd) can be found in the .build/bin/ directory.

  1. Create a config.json file and use it to list your repositories. Check out our example-config.json to see how to set up various types of repositories. For example, we can configure Hound to search its own source code using the config found in default-config.json:
{
  "dbpath" : "db",
  "repos" : {
    "Hound" : {
      "url" : "https://github.com/hound-search/hound.git",
      "vcs-config" : {
        "ref" : "main"
      }
    }
  }
}

A complete list of available config options can be found here.

  1. Run the Hound server with houndd in the same directory as your config.json. You should see output similar to:
2015/03/13 09:07:42 Searcher started for statsd
2015/03/13 09:07:42 Searcher started for Hound
2015/03/13 09:07:42 All indexes built!
2015/03/13 09:07:42 running server at http://localhost:6080
  1. By default, hound hosts a web ui at http://localhost:6080 . Open it in your browser, and start searching.

Using Docker (1.14+)

  1. Install docker if you don't have it. We need at least Docker >= 1.14.

  2. Create a config.json file and use it to list your repositories. Check out our example-config.json to see how to set up various types of repositories. For example, we can configure Hound to search its own source code using the config found in default-config.json.

Run with image from github

docker run -d -p 6080:6080 --name hound -v $(pwd):/data ghcr.io/hound-search/hound:latest

You should be able to navigate to http://localhost:6080/ as usual.

Build image and container yourself

  1. Clone repository
git clone https://github.com/hound-search/hound.git
cd hound
  1. Build the image
docker build . --tag=hound
  1. Create the container
docker create -p 6080:6080 --name hound -v $(pwd):/data hound
  1. Starting and stopping the container
docker start hound
docker stop hound

Running in Production

There are no special flags to run Hound in production. You can use the --addr=:6880 flag to control the port to which the server binds. Currently, Hound does not support TLS as most users simply run Hound behind either Apache or nginx. However, we are open to contributions to add TLS support.

Why Another Code Search Tool?

We've used many similar tools in the past, and most of them are either too slow, too hard to configure, or require too much software to be installed. Which brings us to...

Requirements

  • Go 1.16+

Yup, that's it. You can proxy requests to the Go service through Apache/nginx/etc., but that's not required.

Support

Currently Hound is only tested on MacOS and CentOS, but it should work on any *nix system. Hound on Windows is not supported but we've heard it compiles and runs just fine (although it helps to exclude your data folder from Windows Search Indexer).

Hound supports the following version control systems:

  • Git - This is the default
  • Mercurial - use "vcs" : "hg" in the config
  • SVN - use "vcs" : "svn" in the config
  • Bazaar - use "vcs" : "bzr" in the config
  • Local - use "vcs" : "local" in the config

See config-example.json for examples of how to use each VCS.

Private Repositories

There are a couple of ways to get Hound to index private repositories:

  • Use the local pseudo-vcs driver. This allows you to index a local directory. You can set "watch-changes" : true to calculate a recursive hash of all the files in the directory and automatically re-index.
  • Use the file:// protocol. This allows you to index a local clone of a repository. The downside here is that the polling to keep the repo up to date will not work. (This also doesn't work on local folders that are not of a supported repository type.) If you're using Docker, you must mount a volume to your repository (e.g., -v $(pwd)/src:/src) and use the relative path to the repo in your configuration.
  • Use SSH style URLs in the config: "url" : "[email protected]:foo/bar.git". As long as you have your SSH keys set up on the box where Hound is running this will work.

Keeping Repos Updated

By default Hound polls the URL in the config for updates every 30 seconds. You can override this value by setting the ms-between-poll key on a per repo basis in the config. If you are indexing a large number of repositories, you may also be interested in tweaking the max-concurrent-indexers property. You can see how these work in the example config.

Editor Integration

Currently the following editors have plugins that support Hound:

Hacking on Hound

Editing & Building

Requirements:

  • make
  • npm (Usuall npm comes bundled with Node.js. If that's not the case on the system you're using, you can get it here)
git clone https://github.com/hound-search/hound.git
cd hound
make

The hound executables will be available in .build/bin.

Testing

There are an increasing number of tests in each of the packages in Hound. Please make sure these pass before uploading your Pull Request. You can run the tests with the following command. To run the entire test suite, use:

make test

If you want to just run the JavaScript test suite, use:

npm test

Any Go files that end in _test.go are assumed to be test files. Similarly, any JavaScript files that ends in .test.js are automatically run by Jest, our test runner. Tests should live next to the files that they cover. Check out Jest's docs for more details on writing Jest tests, and check out Go's testing docs for more details on testing Go code.

You need to install Node.js >= 12 and install jest by npm install jest to run the JS tests.

Working on the web UI

Hound includes a web UI that is composed of several files (html, css, javascript, etc.). To compile UI changes use:

make ui

To make development easier, there is a flag that will read the files from the file system (allowing the much-loved edit/refresh cycle).

First you should ensure you have all the dependencies installed that you need by running:

make dev

Then run the hound server with the --dev option:

.build/bin/houndd --dev

Get in Touch

Created at Etsy by:

Hound is maintained by:

hound's People

Contributors

adrienp avatar avleen avatar belak avatar blobbered avatar davegallant avatar dharmabruce avatar dschott68 avatar gmcaguilar avatar hanguofeng avatar hoorayimhelping avatar igorwwwwwwwwwwwwwwwwwwww avatar ivantsepp avatar jacobrose avatar jeffswensen avatar jklein avatar jnackman avatar jrarmstro avatar jsok avatar kellegous avatar kinghrothgar avatar mrexodia avatar reinhardt1053 avatar rfan-debug avatar sahildua2305 avatar salemhilal avatar sstarcher avatar stanistan avatar tgulacsi avatar twizzyyanki avatar yayachiken avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hound's Issues

Indexed files removed by other repos

When searching I received errors of the form:

ERROR:open <path to missing file>: no such file or directory

I think this is due to the expungeOldIndexes function in searcher.go
If, for example, you have two projects: 'my-project' and 'my-project-tests', indexing of 'my-project' removes directories matching 'my-project-', which includes the directories for 'my-project-tests'.

Tag Support

We're searching 100+ repositories and would like to be able to tag repos. Then we can restrict searches to repos with a given tag. Here's what I imagine the config could look like. Happy to give this a go, but would appreciate some input on where to get started.

 "SomeGitRepo" : {
            "url" : "https://www.github.com/YourOrganization/RepoOne.git",
           "tags" : ["client-1", "ruby"]
        }.
 "SomeRubyRepo" : {
            "url" : "https://www.github.com/YourOrganization/RepoOne.git",
           "tags" : ["client-1", "objective-c"]
        }

Issue with UTF16 files

We have a mixture of UTF8 and UTF16 files. Hound seems to search them correctly, as in it will say 16 files searched, but only show 4. When I go and manually switch save as UTF8 and check in then the files will show in the UI.

Support for Stash (and other) URLs

Is there a way to configure the format/pattern of the URL where the results are linked to? It works fine with gitlab and github, but with stash, another format is expected:

https://mystash.tld/projects/PROJECT/repos/REPO/browse/common/src/main/...

Install with go

I tried following the instruction by doing the following:

git clone [email protected]:etsy/Hound.git
cd Hound
go get github.com/etsy/hound/cmds/houndd
houndd

However, I got this error bash: houndd: command not found. I am not familiar with go at all, so I am not sure what I am doing wrong. Could someone point me in the right direction?

not a Rake fan

I'm not a fan of using Rake as a replacement for a simple make or simple bash. But to illustrate a point similar to the go authors... using go to compile go. Here is a Gist that you are welcome to incorporate into your project. I will adhere to whatever license you prefer:

https://gist.github.com/rbucker/2f362aeb0ddec24e8b8f

List all repos

A small info page that could list all the repos configured to be searchable. So that others can have a look and recommend additions.

Provide prebuilt binaries

Since this project can be built into a single binary, it would be great it could be downloaded without having to have any build environemnt

Backend pointing to the existing local repositories instead of a git clone

Two questions:

  1. Can the config.json be pointed to local path instead of a git location? For example if I deploy Hound on a stash repo box (which has the all source code) and point the config so that Hound's data/ directory symlinks to the stash repo. The config would looks something like below:

config.json

{
"dbpath" : "data",
"repo-type": "local",
"repos" : {
"SomeRepo" : {
"url" : "/MyRepositories/SomeRepo"
},
"AnotherRepo" : {
"url" : "
/MyRepositories/AnotherRepo"
}
}
}

Add symlink
~/Hound/data symlinks to ~/MyRepositories/

  1. Does it auto index when new code is pushed to the repository?

Local repo

Does this support indexing local repositories?

The example config has a url key, but can I use, say path and set the value to a folder on my machine?

Document rough memory/disk usage requirements

Hound is super-simple to get running, which is awesome! I'm trying to index a number of repos on a host that's got lots of disk, but little memory. (1GB... I know. I want to increase that but can't at the moment for Reasons.) The indexer runs for a brief period and then hits a panic:

panic: fork/exec /usr/bin/git: cannot allocate memory

My currently configured repos (100 of em) are just a small fraction of the total I'd like to index. I'll be moving to a different host, but I'm wondering if you have any rough rules of thumb for resource requirements. Obviously this is very repo dependent, but perhaps something normalized like a ratio of source repo size on disk to index size and indexer memory requirements?

All links are 404s

Here's an example:

http://localhost:6080/[email protected]:Internetworkexpert/Members-Dash/blob/master/assets/bower_components/bootstrap/less/type.less#L114

Naturally you won't be able to see the original on github, because it's private.

Don't exit if a corrupt index is found

I'm setting up Hound to run against about 3000 Stash repos (PR coming to handle the different file link format) and noticed that a small number of repos weren't indexed properly the first time. As soon as Hound finds a bad index, it immediately shuts down.

Deleting the source code and index for each repo and letting Hound re-clone and reindex seemed to work every time.

I'd prefer not to have Hound exit as soon as it finds a bad index, but log a severe error instead. Rather than submitting a PR for this behaviour, I wanted to check what other people think first - maybe the option should be configurable for example ?

Set indexer frequency

Setting up Hound locally now, but its unclear from the README when does the indexer actually run after the first indexing. It should be configurable, I think.

Hover hint on button next to search hits.

Hey,

What does the button next to each hit do? Can you add some mouseover text so we know.

Looks like a clipboard. Is it "copy to clipboard"? Doesn't seem to work if so.

Thanks

Google App Engine is unsupported

I'd like to use this on Google App Engine, but GAE doesn't support the unsafe or syscall packages (source). Otherwise, it appears this will work and would open Hound up for developers on Google App Engine.

This could also be an alternative for issue #41.

Is it possible to refactor?

Mysterious crashes when starting up

I have no idea what this means:

2015/01/27 17:13:05 Searcher started for inventory
panic: exit status 128

goroutine 1 [running]:
main.main()
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/cmds/houndd/main.go:304 +0x5a6

goroutine 6 [sleep]:
hound/searcher.func·001()
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:183 +0x54b
exit status 2

I found that deleting the relevant entries from my config file allowed it to continue. (There were maybe half a dozen URLs that causes crashes like this).

Allow recursive directories to be indexed with file://

If I have a folder containing all my repos (local or on shared network space using gitlab or similar) the entire directory contains sub directories of different git repos. If I want to index all of the repos, including newly added repos, I would hope to be able to use the following config to index all of the repos at once.

{
  "repos" : {
    "gitlab" : {
      "url" : "file://path/to/repos/"
      "recurse" : "true"
    }
  }
}

This would hopefully help with managing dozens or hundreds of repos needing to be put into a config.json file.

failed to build

src/hound/index/grep.go:64: function ends without a return statement
src/hound/index/grep.go:137: function ends without a return statement

src: no such file or directory

I am completely new to go so I am not sure what I'm doing wrong, but I got this error trying to build the docker image. Could anyone help?

Step 1 : COPY src /go/src
INFO[0000] src: no such file or directory

Allow putting a GitHub Organization in the config and index all repos under it

It would be really helpful if you could just tell it to search/index all the repositories under a namespace. With some organisations having hundreds of repo's needing to explicitly define every single one of them is needless work.

If it's only for repo's on Github it could be automated through the Github API.

Change name?

Howdy folks,

This project looks cool. Nice idea and execution.

I work at thoughtbot. We've run a product, also called Hound, for a year or so. It is a hosted Continuous Integration service to check code style on GitHub pull requests. It is also open source.

We've noticed a number of people expressing confusion over the conflicting "Hound" name for code-related tools:

Would you mind changing the name of this project to something else?

Happy to chat over email if that's easier, too. [email protected].

Dan

make failed on src/hound/index/grep.go

After clone your repo and configure the config file I run make and it stops with that error on Ubuntu 12.04

make
GOPATH=pwd go build -o bin/houndd src/hound/cmds/houndd/main.go
hound/index
src/hound/index/grep.go:64: function ends without a return statement
src/hound/index/grep.go:137: function ends without a return statement
make: *** [bin/houndd] Fehler 2

ability to specify branches

It'd be great if you could specify a branch to use for searching in the config. It'd be even better if you could do it on the fly in the UI

Bitbucket file urls

When click on a file that has been pulled from bitbucket it uses the URL that works for github which is slightly different than bitbucket's.

"blob/master" for github
"src/master" is needed for bitbucket

Syntax highlighting

Have you considered adding support for syntax highlighting? This is complicated by the fact that each line is displayed separately. It seems like it would be much simpler to just show all the code inside a <pre> tag and then use something like google-code-prettify to add line numbers and syntax highlighting. Then just apply the highlighting on the generated HTML using something like the jQuery highlight plugin.

Work with local repos (no clone)

Enhancement suggestion:
I would like Hound to work with repos that are already on my server, without cloning them first. This is for an in-house repo hosting server, for example.

Support simple query language

It would be nice if Hound supports a simple query language like "firstTerm !secondTerm" (search files including "firstTerm", but not "secondTerm"). Narrowing down a search with the regex file path works fine, but sometimes I want to search for specific filenames and match files that I have not yet modified (for example, adding a plugin to several POMs).

Having a simple query like the one above combined with the regex file path would allow me to search for those files. Using AND, OR, NOR, XOR would also be possible.

Do you think this is feasible and in line with the direction you want Hound to move?

Houndd errors after rebuilding index

I ran hound using
go run src/hound/cmds/houndd/main.go

Works fine for a while. Then, I see this in my console—

2015/01/29 14:50:23 Rebuilding fs-scoring for efa7db02762ad04355539f65b1b7a8e7e33b9bbe
2015/01/29 14:50:26 merge 0 files + mem
2015/01/29 14:50:26 2470307 data bytes, 478302 index bytes
HeapInUse = 4.84
HeapIdle  = 473.31

And after that, searching for anything gives messages like these—

ERROR:open /home/aphatarpekar/setups/Hound/data/questionnaire-service-50fab2c00d1251d2b020a4776de59359405285aa/raw/pom.xml: no such file or directory

Private repo support?

Is there private repo support? How does one configure it?

If not, it'd be a great feature to have!

@jdan suggested it on HN a few minutes ago.

cannot clone local mercurial (hg) repository

2015/02/03 14:30:04 Searcher started for ABC-Project
2015/02/03 14:30:04 Failed to clone file://abc/project, see output below
Cloning into 'ABC-Project'...
fatal: '/abc/project' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Continuing...
2015/02/03 14:30:04 Some repos failed to index, see output above
2015/02/03 14:30:04 running server at http://localhost:6080...

The folder exists and I have the access right since I'm running it as root.. It's just that it is local mercurial repo instead of git.. I am using commit d2c1d98

REST API

Would be great to support a RESTful API to support native front ends.

Houndd crashes on empty repositories

One of my github projects is an empty repository at the moment. The houndd/main.go crashes every time it tries to scan it:

go run src/hound/cmds/houndd/main.go                                                                                                                                (no jobs) [master]  Week 05 29.01.15 13:53
2015/01/29 13:54:00 Searcher started for gol
panic: exit status 1

goroutine 1 [running]:
main.main()
    /tmp/Hound/src/hound/cmds/houndd/main.go:304 +0x5a6
exit status 2

Please integrate a check for empty repositories.

Editor integration?

Hi again,

Do you plan to integrate hound into editors? Currently you have to leave my editor, visit a web page, and when you click a hit, you are taken to a github page. It would be nice if users could harness hound in their editor of choice: vim/emacs/eclipse/whatever.

I am a vim user, so I imagine a use case where I can type :hound <search_term>, then a pane would show hits, which I could select from a list to open them as buffers. The ack vim plugin (which is what I currently use) works like this.

I'm not sure how you would account for local edits though.

Quite an ambitious proposal I know, but have you thought about it? Maybe it is too much effort...

Thanks

Freeze when entering an invalid regex

When entering an invalid regex ( it will freeze the tab.

Edit: Including an empty search term.

Google Chrome
Version 40.0.2214.93 (64-bit)
OSX 10.9.5

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.