huridocs / uwazi

Uwazi is a web-based, open-source solution for building and sharing document collections

Uwazi is a flexible database application to capture and organise collections of information with a particular focus on document management. HURIDOCS started Uwazi and is supporting dozens of human rights organisations globally to use the tool.

Uwazi | HURIDOCS

Read the user guide

Installation guide

• Dependencies
• Production
• Development

Dependencies

Before anything else you will need to install the application dependencies:

• NodeJs >= 18.16.1 For ease of update, use nvm: https://github.com/creationix/nvm.
• ElasticSearch 7.17.6 https://www.elastic.co/downloads/past-releases/elasticsearch-7-17-6 Please note that ElasticSearch requires Java. Follow the instructions to install the package manually, you also probably need to disable ml module in the ElasticSearch config file: xpack.ml.enabled: false
• ICU Analysis Plugin (recommended) installation Adds support for number sorting in texts and solves other language sorting nuances. This option is activated by setting the env var USE_ELASTIC_ICU=true before running the server (defaults to false/unset).
• MongoDB 4.2 https://docs.mongodb.com/v4.2/installation/ . If you have a previous version installed, please follow the instructions on how to upgrade here. The new mongosh dependency needs to be added installation.
• Yarn https://yarnpkg.com/en/docs/install.
• pdftotext (Poppler) tested to work on version 0.86 but it's recommended to use the latest available for your platform https://poppler.freedesktop.org/. Make sure to install libjpeg-dev if you build from source.

Production

Install/upgrade procedure

Development

If you want to use the latest development code:

$ git clone https://github.com/huridocs/uwazi.git
$ cd uwazi
$ yarn install
$ yarn blank-state

If you want to download the Uwazi repository and also download the included git submodules, such as the uwazi-fixtures, which is used for running the end-to-end testing:

$ git clone --recurse-submodules https://github.com/huridocs/uwazi.git
$ cd uwazi
$ yarn install

If the main Uwazi repository had already been cloned/downloaded and now you want to load its sub-modules, you can run

$ git submodule update --init

There may be an issue with pngquant not running correctly. If you encounter this issue, you are probably missing the library libpng-dev. Please run:

$ sudo rm -rf node_modules
$ sudo apt-get install libpng-dev
$ yarn install

Docker

Infrastructure dependencies (ElasticSearch, ICU Analysis Plugin, MongoDB, Redis and Minio (S3 storage) can be installed and run via Docker Compose. ElasticSearch container will claim 2Gb of memory so be sure your Docker Engine is alloted at least 3Gb of memory (for Mac and Windows users).

$ ./run start

Development Run

$ yarn hot

This will launch a webpack server and nodemon app server for hot reloading any changes you make.

Webpack server

$ yarn webpack-server

This will launch a webpack server. You can also pass --analyzeto get detailed info on the webpack build.

Testing

Unit and Integration tests

We test using the JEST framework (built on top of Jasmine). To run the unit and integration tests, execute

$ yarn test

This will run the entire test suite, both on server and client apps.

Some suites need MongoDB configured in Replica Set mode to run properly. The provided Docker Compose file runs MongoDB in Replica Set mode and initializes the cluster automatically, if you are using your own mongo installation Refer to MongoDB's documentation for more information.

End to End (e2e)

For End-to-End testing, we have a full set of fixtures that test the overall functionality. Be advised that, for the time being, these tests are run ON THE SAME DATABASE as the default database (uwazi_developmet), so running these tests will DELETE any existing data and replace it with the testing fixtures. DO NOT RUN ON PRODUCTION ENVIRONMENTS!

Running end to end tests requires a running Uwazi app.

Running tests with Nightmare

$ yarn hot

On a different console tab, run

$ yarn e2e

Running tests with Puppeteer

$ DATABASE_NAME=uwazi_e2e INDEX_NAME=uwazi_e2e yarn hot

On a different console tab, run

$ yarn e2e-puppeteer

Note that if you already have an instance running, this will likely throw an error of ports already been used. Only one instance of Uwazi may be run in the same port at the same time.

Default login

The application's default login is admin / change this password now

Note the subtle nudge ;)

System Requirements

• For big files with a small database footprint (such as video, audio and images) you'll need more HD space than CPU or RAM
• For text documents you should consider some decent RAM as ElasticSearch is pretty greedy on memory for full text search

The bare minimum you need to be able to run Uwazi on-prem without bottlenecks is:

• 4 GB of RAM (reserve 2 for Elastic and 2 for everything else)
• 2 CPU cores
• 20 GB of disk space

For development:

• 8GB of RAM (depending on whether the services are running)
• 4 CPU cores
• 20 GB of disk space

uwazi's Issues

Add delete reference functionality

Add date as metadata building block

Properly setup code coverage for ES6

Properly setup code coverage for ES6 to include server tests as well.

Display highlight snippets in full text search results

Test code review: replace switch statements with object literals

@daneryl @konzz

Testing how we can use issues for code review. So far I'm not happy with the inline comments (blame). Github's interface for this sucks big time. Comments get lost, there is no way of tracking what conversations are active etc. The official suggestion is to use pull requests, which I think is a total overkill unless you are conducting a full code review.

Code style suggestion, replace switch statements with object literals: https://toddmotto.com/deprecating-the-switch-statement-for-object-literals.

https://github.com/huridocs/UwaziDocs/blob/9fcc2009263a36f4885b0e0bcc7bd70dfef2c973/app/react/Templates/components/FilterSuggestions.js

getTypeIcon(type) {
    let icon;
    switch (type) {
    case 'checkbox':
      icon = 'fa fa-check-square-o';
      break;
    case 'select':
      icon = 'fa fa-sort';
      break;
    case 'list':
      icon = 'fa fa-list';
      break;
    case 'date':
      icon = 'fa fa-calendar';
      break;
    default:
      icon = 'fa fa-font';
    }

    return icon;
  }

Download original file

Password change feature is broken

Password is not stored with proper encryption.

Help text on uploads form when no document types exist

Blank state on uploads form when no document types exist

Advanced search

Take search to the next level where users can even choose how the want filters to stack (AND/OR) and other options TBD.

Have search in the uploads section same as in the library

Show error messages feedback if server responded with an error or if the server didn't repond

Documents with long titles overflow the card

Redesign login page

Make elasticsearch give more relevance to the keywords in the title so its easier to quickly locate documents based on the title

Redesign admin page

• Password change workflow
• Edit "Collection name"

Show alerts in fixed position instead of absolute

When the user is not int he top of the window he doesn't see the alerts.

When saving metada elements show a feedback message on success

Test issue

This is a test issue

HowToImproviseJazz.pdf

Improve workflow for reference creation

@danicatalan please upload the design asset in https://github.com/huridocs/uwazi-assets

Fix deprecated express-session warnings

App stays in eternal loading loop when accessing a URL of a document that does not exist

When trying to load a document id that does not exist the app stays loading forever we need to detect it and show a 404 ?
This might happen if a user accidentally changes the URL of a document.

Lint the backend code and add the step to the CI server

Lint the backend code and add the step to the CI server

Documents with status "upload failed" in the upload section shouldn't be publishable

Displaying dates in the library cards beside the document type is misleading

The date displayed in the document card beside the document type is the upload date. This is misleading and can easily be read as ie "Sentence 25-07-2009", as in being a court sentence issued in 25-07-2009.

We need to create a better design for this.

Analytics and user session tracking

Can't rename templates

Maintain toggle state on filters when selecting a document type with common filters

Come to a solution for font loading for the converted documents

ATM we have deactivated font loading. These fonts take up to 1MB per font, making document rendering and loading too heavy. We need to come to a usable solution for this problem.

Add logout functionality

Display document properties in cards in search results

Add delete document functionality

Code duplication in filter icon selection

Code duplication? this could even lead to displaying different icons if the code is changed separately.

https://github.com/huridocs/UwaziDocs/blob/9fcc2009263a36f4885b0e0bcc7bd70dfef2c973/app/react/Templates/components/MetadataProperty.js

let iconClass = 'fa fa-font';
    if (this.props.type === 'select') {
      iconClass = 'fa fa-sort';
    }

    if (this.props.type === 'list') {
      iconClass = 'fa fa-list';
    }

    if (this.props.type === 'date') {
      iconClass = 'fa fa-calendar';
    }

    if (this.props.type === 'checkbox') {
      iconClass = 'fa fa-check-square-o';
    }

https://github.com/huridocs/UwaziDocs/blob/9fcc2009263a36f4885b0e0bcc7bd70dfef2c973/app/react/Templates/components/FilterSuggestions.js

getTypeIcon(type) {
    let icon;
    switch (type) {
    case 'checkbox':
      icon = 'fa fa-check-square-o';
      break;
    case 'select':
      icon = 'fa fa-sort';
      break;
    case 'list':
      icon = 'fa fa-list';
      break;
    case 'date':
      icon = 'fa fa-calendar';
      break;
    default:
      icon = 'fa fa-font';
    }

    return icon;
  }

Can't copy text from selected text in the document view

You shouldn't be able to save templates, dictionaries and relationship types with an existing name (duplicates)

Redux-form library use cases and refactor

Watch out for links into the converted content

Sometimes PDFs already have links. We should be careful with PDF hacks, such as storing XSS baits in the content etc.

Add checkbox (boolean) as metadata building block

Page titles are not displayed properly

Hitting save button several times on metadata elements leads to creation of duplicates

When saving matadata elements store locally the revision sent by couchdb so on resaving the element is not duplicated.

Import dictionary options from CSV

Add pagination to library

Have filtering in the uploads section the same as in library

Search bar: remove native autocomplete suggestions

Users need an easy way to spot documents that don't have properties existing in the template

Specially when a document template has been updated adding a new property, users need an easy way of keeping track of documents lacking that new property.

Configure webpack to minify the build file

Display breadcrumbs

Let users know exactly where they are, ie:

metadata > create new
metadata > edit > decision
library > view > Some document title
uploads > view > Some document title
library > view > Some document title > create reference to a paragraph > Some other document title

Improved filtering (faceted browsing style)

• Display amount of results and refine the search filters.

ie. In faceted browsing we can see upfront the amount of results per filter, as in a country filter:
Argentina (5)
Peru (1)
Ecuador (0) -> This one may even not show up if it has zero results.

Upon selection, other "subfilters" should be updated to reflect the actual results if combined with the active filters.

Display loader in document viewer

Hey guys, the login is not working

Slow typing in document type title

When creating a document type in the metadata section, is some sort of realtime duplicate validation going on? We should add a time trigger for that otherwise is unusable for slow connections.