sproutsocial / botanist

Fast web and command-line code search tool for companies and teams who use github and/or bitbucket.

botanist

a web and command-line code search tool for teams.

Provides the service behind codesearcher

Updates and Deploys

Updates

Updates to this repo will trigger a jenkins job to build, scan, and push the image to ECR.

Deploys

The image launched on the codesearcher instance is managed by infra_chef.

Once a new image is uploaded, to deploy it requires an infra_chef PR to update the botanist_version attribute

(Don't forget to also bump the cookbook version in metadata.rb and update CHANGELOG)

Web search

Command-line search

Packages

some software packages are included to make this work. they are included in this repository for now for compatibility purposes.

Code search

The fast searching is made possible by the excellent codesearch tool from Google:

https://code.google.com/p/codesearch/ (cindex running on a periodic cron to re-index)

Bitbucket repository fetching

https://github.com/samkuehn/bitbucket-backup (running on a periodic cron to update the source code to search) NOTE: Enclosed under ./packages is a custom fork of samkeuhn/bitbucket-backup. I've issued several PR's that he's accepted so he is really quick and amenable to updates, I just haven't quite finished cleaning up the additions I made recently.

GitHub repository fetching

wrote up something to do this in Python based on https://github.com/celeen/gitter

The default GitHub branch name is "main".

Pull requests welcome!

Running locally using docker

1. Create location for repositories on disk

mkdir -p $HOME/botanist/repos

2. Set up credentials

# copy env file to env.local, and set GH_USER, GH_ORGS, GH_PW so you can fetch code to index
# GH_PW should be a personal access token with repo scope
# if the org you want to back up uses SSO, you also need to authorize the token to the org
# under "Configure SSO"
cp env.template env.local

3. Start botanist

docker-compose stop && docker-compose rm -f && docker-compose up --build -d

4. Fetch code repositories using the image you just built

docker run --env-file env.local -v $HOME/botanist/repos:/botanist/repos botanist-webapp /botanist/bin/fetch-code.sh

5. Index the code repositories using the image you just built

docker run --env-file env.local -v $HOME/botanist/repos:/botanist/repos botanist-webapp /botanist/bin/index.sh

botanist's Issues

remove deleted repos

Files with lines longer than 2000 characters do not get indexed

This is due to this hardcoded limit in Google's codeseacher:

https://github.com/google/codesearch/blob/master/index/write.go#L95

google/codesearch#30

feature request:

rename with ARCHIVED- as prefix
and we can blacklist repos that start with that
i could even add an option to botanist to optionally include ARCHIVED- repos

btgrep throws 500

$ btgrep something

Traceback (most recent call last):
  File "/Users/stantonk/Scripts/btgrep", line 30, in <module>
    r = urllib2.urlopen(BOTANIST_DOMAIN + '/search/results.json?' + params).read()
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR

bitbucket deep links to lines of code in files has changed

old format:

https://bitbucket.org/orgname/reponame/src/tip/date_range.py#cl-38

new format:

https://bitbucket.org/orgname/reponame/src/tip/date_range.py#date_range.py-38

Default branch on a repo is not respected in deep links to code

When a repo is configured with a default branch, that branch is correctly used when indexing content. However, deep links in the code ignore this setting and link to the file and line numbers in the master branch, which may or may not exist. To see this problem in action:

1 - Navigate to http://codesearcher.int.sproutsocial.com/search/?q=oauth_is_valid&case=sensitive#web-app-messaging_github
2 - Click the first link

Currently this is a link to https://github.com/sproutsocial/web-app-engagement/blob/master/application/controllers/support/start.php#L2503, which does not exist. The link should go to the repo's default branch, which is "engagement", and the link should be https://github.com/sproutsocial/web-app-engagement/blob/engagement/application/controllers/support/start.php#L2503