GithubHelp home page GithubHelp logo

ag91 / code-compass Goto Github PK

View Code? Open in Web Editor NEW
141.0 12.0 14.0 3.77 MB

A set of code analyses that assist you in tackling software complexity

License: GNU General Public License v3.0

Emacs Lisp 61.74% Python 26.39% JavaScript 10.09% CSS 1.34% HTML 0.44%

code-compass's Introduction

Code compass

This package shall guide you in your software development within Emacs. For example, it will point at the code that requires your changes the most and it will suggest you who to ask for help when you are lost.

I have presented some of the capabilities of code-compass in this talk of EmacsConf2020.

Credits

A significant part of this project relies on code-maat and the bright mind of his author Adam Tornhill. His books are inspiring and a suggested read.

Features

Use code-compass-cheatsheet to have a quick reference of the available commands and their aim. Check the Release Checklist heading for more information about them.

Installation and Dependencies

Please run =code-compass-doctor= if you encounter errors in running this tool

Note: I label dependencies as optional when the commands needing the dependency warn you they cannot execute because the dependency is missing.

This project depends on the following external dependencies:

  • Git
  • Python 3
  • Java 8 or above
  • code-maat
  • cloc
  • graph-cli (optional - only few commands will not succeed)
  • gource (optional)

And the following Emacs packages:

  • async.el
  • dash.el
  • f.el
  • s.el
  • simple-httpd

If you use use-package and you are on a Linux system, this will take care of the Emacs installation:

(use-package async)
(use-package dash)
(use-package f)
(use-package s)
(use-package simple-httpd)

(use-package code-compass
  :load-path "~/.emacs.d/lisp")

For Docker, the Code Maat image, cloc and graph-cli run the following script:

# Docker is only necessary if you want to use that instead of the (automatically downloaded) JAR file
# see here for how to install in systems different from Linux Debian: https://gist.github.com/rstacruz/297fc799f094f55d062b982f7dac9e41
#sudo apt install docker.io;
#sudo systemctl start docker;
# after you manage to run docker successfully
#git clone https://github.com/adamtornhill/code-maat.git;
#cd code-maat;
#docker build -t code-maat-app .

sudo apt install cloc;

cd /tmp;
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py; # from here https://pip.pypa.io/en/stable/installing/
python get-pip.py --user
pip3 install --user graph-cli # https://github.com/mcastorina/graph-cli

For trying things out in a clean Emacs:

(require 'package)
(eval-and-compile
  (setq
   package-archives
   '(("melpa-stable" . "https://stable.melpa.org/packages/")
     ("melpa" . "https://melpa.org/packages/")
     ("marmalade"   . "https://marmalade-repo.org/packages/")
     ("org"         . "https://orgmode.org/elpa/")
     ("gnu"         . "https://elpa.gnu.org/packages/"))))
(package-initialize)

 ;;; Bootstrap use-package
;; Install use-package if it's not already installed.
(unless (package-installed-p 'use-package)
  (package-refresh-contents)
  (package-install 'use-package))
(setq use-package-always-ensure 't)
(require 'use-package)
(require 'diminish)
(require 'bind-key)
(use-package async)
(use-package dash)
(use-package f)
(use-package s)
(use-package simple-httpd)
emacs -Q -l /tmp/code-compass-minimal-setup.el -l ./code-compass.el

Limitations

The limitations I know:

  1. only Git support for now, but I am open to PRs (should be easy because code-maat partially support other VCS already)
  2. Adam said that code-maat may fail for code bases larger than 5 million lines. Please report if you observe that is the case, we will find a solution.
  3. most likely others I will eventually discover from the issues ;)

Release Checklist

Releasing this in the wild is exciting, but it will take some time. Here what you can expect.

install instructions and dependencies

hotspots

software complexity

code churn

change coupling

use case of coupling: find coupled files

code communication

code knowledge

code stability

fragmentation

https://ag91.github.io/blog/2021/02/11/emacs-as-your-code-compass-how-fragmented-is-the-knowledge-of-this-file/

word analysis

use case of coupling: generate todos for current file

integrate gource

https://ag91.github.io/blog/2021/03/19/emacs-as-your-code-compass-watch-history-with-gource/

file-churn icon

main contributors notification

code refactoring

https://ag91.github.io/blog/2022/11/23/emacs-as-your-code-compass-who-is-the-person-who-refactored-most-in-this-project/

functions complexity

oh well I needed tree-sitter for this, see here: https://github.com/ag91/moldable-emacs/blob/master/molds/contrib.el#L242

License

GPLv3

Dependencies

d3:

License: BSD-3

Copyright 2010-2020 Mike Bostock

Contributing

If you have ideas or wishes, just open an issue and I will look into it! Thanks for caring.

Testing

Functions without side effects have tests in their documentation.

To run those install https://github.com/ag91/doctest (at the time of writing my fork as enhancements over the original) and run doctest.

Alternatives

  • CodeScene: this is the code analysis tool of Adam Tornhill which organizations can use to manage their software and organizational complexity. Code-compass learns from CodeScene and adapts to empower you.
  • code-risk: this is a set of scripts Noah Sussman’s uses to find quality issues in repositories. Code-compass includes these and make them easily accessible to you.
  • code-forensics: this makes available code-maat analyses in a node application. Code-compass offers a subset of these for now and focuses more on supporting you while you edit your project. (Thanks @BlankSpruce to share this repository!)
  • git-deps: this shows you dependencies between git commits. Hopefully code-compass will integrate this project to help you when, for example, you are struggling to identify the commit that broke your release.
  • ???

code-compass's People

Contributors

ag91 avatar ambirdsall-gogo avatar blankspruce avatar gopar avatar larebsyed avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

code-compass's Issues

Permission issue while copying scripts

To reproduce:

  1. M-x c/show-coupling-graph-sync
  2. M-x c/show-coupling-graph-sync

The second time, it fails:

c/copy-file: Opening output file: Permission denied, /tmp/code-compass-<REDACTED>/coupling_csv_as_edge_bundling.py

This is because most of the script files are read-only (linux permissions).

Probably these should have read-write permissions in the Git repo?

Slack Features

The experimental slack features error out, eg calling M-x c/slack-main-contributor produce:

Symbol’s value as variable is void: slack-current-team

Use fboundp to see if user has slack installed, if not then show message about it as intended

Improve filtering of reports by directory, particularly for use with monorepos

First, thank you for an interesting and inspiring project, and for presenting such an effective introduction at emacsconf.

My company's product, a web app, is built on a ten year-old codebase that was written by a much larger team than we have now; it is important for us to not trip over the large amount of code we didn't write ourselves and understand where old code stands in the way of new projects. Code compass looks like a very nice way to measure and communicate these things. However, I'm finding I need to make certain workarounds as I experiment with code-compass in my company's monorepo, and I think I could expand some of these into general improvements. It's worth noting that I am at the beginning of an incremental investigation of the different functions code-compass provides; once I am satisfied with my ability to generate a hotspots visualization, I will move onto other types of report, which may uncover new issues to be solved for my use case.

I find that generating a visualization like (c/show-hotspots path-to-monorepo "3m") visualizes so many files that it causes javascript performance issues in the generated page and makes it hard to drill down on any particular sub-project. I have observed two distinct causes:

  1. the sheer amount of potentially-relevant source code in a ten-year-old monorepo containing many sub-projects is just too big. There is a large monolithic rails app, several backend services, and multiple frontend codebases spanning two major frontend frameworks in both javascript and typescript.
  2. the javascript codebases each have their own node_modules directory of 3rd party library code, and these are not filtered out by default

The second is simpler, so I am starting there: since the locally-cached library code is kept out of version control by .gitignore, I believe the only necessary step is providing an --exclude-dir argument to the cloc executable. For my personal workaround, I simply added the argument to the command string directly, for convenience, but for general consumption, extracting a customizable list like the following seems reasonable:

(defcustom c/exclude-directories
  '("node_modules" "vendor" "tmp")
  "A list of directory patterns to exclude from reports. Contents are passed to the cloc executable via its --exclude-dir argument."
  :group 'code-compass)

(defun c/produce-cloc-report (repository)
  "Create cloc report for REPOSITORY."
  (message "Producing cloc report...")
  (shell-command
   (format "(cd %s; cloc ./ --by-file --csv --quiet --skip-uniqueness --exclude-dir=%s) > cloc.csv" repository (string-join c/exclude-directories ",")))
  repository)

If the basic approach for filtering out low-value files I outlined above looks reasonable to you, I can open a pull request in the coming days; I filled in my current best guess for sane defaults, though I'm sure it could be improved.

The first issue, where I would like to filter reports to one or more specific project directories within a monorepo, does not have as obvious a solution, either in terms of implementation or user interface. It might be possible to hack filtered reports together by abusing the --exclude-dir argument—it may even be possible to build a clean API on top of such a hack—but that seems suboptimal for several reasons (for example, the git operations would still operate over the entire repo, and I'm still in the process of figuring out how this may affect the generated reports). I will continue to think on this problem as I continue my experimentation and learn the workings of code-compass better, and would welcome any suggestions or targeted question you may be able to offer.

Error handling should fail earlier and louder

After pressing pause on trying to make code-compass work (thank you by the way on fixing the variables not being passed to the async process!), I tried again today on my repo.
But c/show-hotspots failed (as in, the displayed zoomable.html does not work with a javascript error) while I had no error in Emacs whatsoever 😭

I just spent some time investigating why and while I found the problem (I will open another issue about that), I think that code-compass should make errors more visible to the user, and fail fast to simplify investigation.

Example:

  • when the code-maat command for revisions fails (shell return code 1), the process continues rather than aborting immediately. The generated revisions.csv contains garbage content:
Invalid argument:  git2: Failed to parse the given file - is it a valid logfile?
This is Code Maat, a program used to collect statistics from a VCS.
Version: 1.0-SNAPSHOT

Usage: program-name -l log-file [options]

Options:
  -l, --log LOG                                         Log file with input data
  -c, --version-control VCS                             Input vcs module type: supports svn, git, git2, hg, p4, or tfs
[...]
  • the process continues like everything is ok then spectacularly 😄 crashes:
Produce json...
Traceback (most recent call last):
  File "/tmp/code-compass-<REDACTED>/csv_as_enclosure_json.py", line 232, in <module>
    run(args)
  File "/tmp/code-compass-<REDACTED>/csv_as_enclosure_json.py", line 195, in run
    raw_weights = parse_csv(
  File "/tmp/code-compass-<REDACTED>/csv_as_enclosure_json.py", line 90, in parse_csv
    return [parse_action(row) for row in r]
  File "/tmp/code-compass-<REDACTED>/csv_as_enclosure_json.py", line 90, in <listcomp>
    return [parse_action(row) for row in r]
  File "/tmp/code-compass-<REDACTED>/csv_as_enclosure_json.py", line 114, in parse_element_weight
    weight = float(csv_row[weight_column])  # Assert not zero?
ValueError: could not convert string to float: ' a program used to collect statistics from a VCS.'

Below the full logs:

Producing git report...
(Shell command succeeded with no output)
Producing code-maat revisions report for <REDACTED>...
(Shell command failed with code 1 and no output)
Producing cloc report...
(Shell command succeeded with no output)
Produce json...
Traceback (most recent call last):
  File "/tmp/code-compass-<REDACTED>/csv_as_enclosure_json.py", line 232, in <module>
    run(args)
  File "/tmp/code-compass-<REDACTED>/csv_as_enclosure_json.py", line 195, in run
    raw_weights = parse_csv(
  File "/tmp/code-compass-<REDACTED>/csv_as_enclosure_json.py", line 90, in parse_csv
    return [parse_action(row) for row in r]
  File "/tmp/code-compass-<REDACTED>/csv_as_enclosure_json.py", line 90, in <listcomp>
    return [parse_action(row) for row in r]
  File "/tmp/code-compass-<REDACTED>/csv_as_enclosure_json.py", line 114, in parse_element_weight
    weight = float(csv_row[weight_column])  # Assert not zero?
ValueError: could not convert string to float: ' a program used to collect statistics from a VCS.'

⚠️ notice in particular (Shell command failed with code 1 and no output)

Incorrect assumption about `/tmp` vs `/data`

The code assumes that:

  • /tmp is used when using java -jar xxx.jar value for c/code-maat-command
  • /data is used when using the Docker value for c/code-maat-command

An example of such an assumption:
https://github.com/ag91/code-compass/blob/main/code-compass.el#L336

I use neither a Java command nor Docker (I provide directly a code-maat executable via Nix).
In most places the default used is /tmp when not Docker, but in this line of code (and maybe others?) it uses /data when not Java.

It means that my set-up breaks because it sometimes uses /tmp and sometimes /data.

How about the path where dummy directories are created becomes configurable?

  • it makes less assumptions about the running system (e.g. what if one does not have /tmp? Or not write-permission?)
  • even for those using Docker, it becomes possible to fine-tune the location

In a similar way, I also suggest (but we can move that in another issue if you will) making the global path where code-compass downloads stuff (e.g. dependencies/d3.v3.min2js) as customizable please 🙏
A use case I have is that I provide code-compass Git repository as a Nix package, i.e. it ends up in a read-only directory, which then crashes when code-compass tries to write to it.

Run `checkdoc` on project

checkdoc is still used for projects, and is used by maintainers to ready up project for release

Will help with #20

Extend Hashmap on cache load instead of overwritting

Just realized I didn't think something through.

Right now, caching works by overwritting code-compass-coupling-project-map with whatever is in the cache.

This isn't ideal, it would be better preferred if it extended the current hashmap instead.

Reasoning:

  1. Working on Project X
  2. Run command to get coupled files, so code-compass-coupling-project-map is now populated.
  3. Go to Project Y
  4. Run command to get coupled files, (we see cache exists for this project so we overwrite code-compass-coupling-project-map, hence loosing all previous coupling info for Project X)

Couple Files finds non-existing files

I've noticed that when calling code-compass-coupling-files, the result may sometimes spit out file(s) that no longer exists.

There should be some type of validator against the files it is suggesting.

Publish on MELPA

Hi,

Have you considered publishing code-compass on MELPA or GNU ELPA? That would make it simpler to use 🙏

Change hotspot color interpolation

Hi,
Thank you for this great selection of tools! It really helps me to understand some new repositories.

The one thing I would like to suggest is to change the colors for the hotspot graph to a gradient going from blue to green to red.
For me at least this is visually clearer where the hotspots are.

Naming convention

Hey, thanks for the package. I see great potential for this! \o/

One thing I noticed is that everything is prefixed with c/ which is not common among package development.
prefix would ideally be code-compass- (bit verbose but that is the norm).

Are you open to updating?
I can make PR and still keep backwards compatibility (can be done with defalias)

Error in browser console (n is undefined)

Hi there! I'm in the process of trying out code-compass, and I'm finding that I see the following error in the browser:

image

The page itself is blank. To get to that result I opened one of my repositories, and ran M-x c/show-hotspots. No errors are reported in emacs.

Unable to use graph commands

When I try a command like c/show-complexity-over-commits. I get the following error:

Sorry, didn't recognize that command!
Type graph --help to view common commands.

Any help is appreciated.

Void func c/get-coupling-alist-sync

I have scanned through the code and it looks like this function is in fact missing. Also, there is this file called dot-code-maat.el which is also not available in the package directory

Unclear how to use code-compass

I'm a bit of an emacs newbie, and I'm completely unsure how to use this. I'm using Doom Emacs. Here is the relevant part of my packages.el

;; code-compass https://github.com/ag91/code-compass
(package! async)
(package! dash)
(package! f)
(package! s)
(package! simple-httpd)

(package! code-compass)

I don't see any instructions on how to use this package in the readme. When doing M-x compass nothing shows up, but I'm not sure if that's how to use code-compass

Don't show full path for coupled files results

When calling code-compass-find-coupled-files, the results will show the full path of the suggested files (see below)

Would be nice to trim out project root from path since we know in what repo we are working on.

Screen Shot 2023-02-24 at 10 07 36 AM

Store coupling file info on disk

Would be nice to save a cached version of the coupling results.

Thoughts:
when calculating the coupling results it would be nice to take a snapshot of the current head commit and branch.
Reasoning:

  • if head commit changes, then cached results might not be valid
  • Same reasoning for current branch
  • if we match on both of these, then retrieve results, otherwise re-calculate
  • We could save on directory {$code_compass_home}/.coupling_files/commit_hash ? ¯_(ツ)_/¯

Not sure if this makes sense or is too ambitious.

Rename variable `c/slack-main-contributor`

c/slack-main-contributor gives the impression that its related to slack but when you read the docstring you realize its not related:

"Enable the listing of contributors for a file in the *Messages* buffer."

perhaps c/display-file-contributors or something else?

Some `defcustom`s are not correctly passed to async code

Some defcustoms are not passed to async code. E.g.:

(setq c/code-maat-command "foobar")
(c/show-hotspots)

will incorrectly use the default value of c/code-maat-command rather than "foobar" e.g. in c/run-code-maat.

I managed to make it work with something like that:

@@ -263,6 +263,7 @@
    `(lambda ()
       (setq load-path ',load-path)
       (load-file ,(symbol-file command))
+      (setq c/code-maat-command ,c/code-maat-command)
       (let ((browse-url-browser-function 'browse-url-generic)
             (browse-url-generic-program ,c/preferred-browser))
         (funcall ',command ,repository ,date)))

But I don't know if this is the idiomatic way to go.

Or how is one supposed to customize those defcustoms otherwise?

Scripts are missing in MELPA package ?

After installation, running M-x code-compass-show-hotspots gives me the following

Debugger entered--Lisp error: (file-missing "Opening input file" "No such file or directory" "/Users/msanghvi/.emacs.d/elpa/code-compass-20230411.1032/scripts/csv_as_enclosure_json.py")
  signal(file-missing ("Opening input file" "No such file or directory" "/Users/msanghvi/.emacs.d/elpa/code-compass-20230411.1032/scripts/csv_as_enclosure_json.py"))
  async-handle-result((lambda (result) (when (not nil) (code-compass--run-server-and-navigate "/Users/msanghvi/Repos/sdlc/ice-gradle-pkg/" (or nil code-compass-default-port)))) (async-signal (file-missing "Opening input file" "No such file or directory" "/Users/msanghvi/.emacs.d/elpa/code-compass-20230411.1032/scripts/csv_as_enclosure_json.py")) #<buffer *emacs*>)
  async-when-done(#<process emacs> "finished\n")

I've followed the instructions in the README file:

;;;; init.el
(use-package async)
(use-package dash)
(use-package f)
(use-package s)
(use-package simple-httpd)

;; code-compass
;; for Code-Maat (code as crime scene, etc.)
(use-package code-compass)

Do the script needs to be packaged ? I don't see them in Melpa directory:

(Sun Apr 23 17:37:18)  ~/.emacs.d/elpa/code-compass-20230411.1032
OSX 12.6.5 x86_64 msanghvi % ls
code-compass-autoloads.el	code-compass-pkg.el		code-compass.el			code-compass.elc		dependencies/

(Sun Apr 23 17:37:19)  ~/.emacs.d/elpa/code-compass-20230411.1032
OSX 12.6.5 x86_64 msanghvi %  ls dependencies/
code-maat-1.0.1-standalone.jar

Let me know if there is any more information needed or if you need me to test anything.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.