GithubHelp home page GithubHelp logo

opendronemap / webodm Goto Github PK

View Code? Open in Web Editor NEW
2.7K 144.0 898.0 107.66 MB

User-friendly, commercial-grade software for processing aerial imagery. 🛩

Home Page: https://www.opendronemap.org/webodm/

License: GNU Affero General Public License v3.0

Python 36.04% CSS 1.78% JavaScript 45.85% HTML 7.59% Shell 3.06% Ruby 0.10% Dockerfile 0.40% SCSS 5.16% Batchfile 0.01%
drone photogrammetry point-cloud maps uav

webodm's Introduction

WebODM

Build Status Version Translated

A user-friendly, commercial grade software for drone image processing. Generate georeferenced maps, point clouds, elevation models and textured 3D models from aerial images. It supports multiple engines for processing, currently ODM and MicMac.

image

image

image

Getting Started

Windows and macOS users can purchase an automated installer, which makes the installation process easier.

There's also a cloud-hosted version of WebODM available from webodm.net.

To install WebODM manually on your machine:

  • Install the following applications:

  • Windows users should install Docker Desktop and 1) make sure Linux containers are enabled (Switch to Linux Containers...), 2) give Docker enough CPUs (default 2) and RAM (>4Gb, 16Gb better but leave some for Windows) by going to Settings -- Advanced, and 3) select where on your hard drive you want virtual hard drives to reside (Settings -- Advanced -- Images & Volumes).

  • From the Docker Quickstart Terminal or Git Bash (Windows), or from the command line (Mac / Linux), type:

git clone https://github.com/OpenDroneMap/WebODM --config core.autocrlf=input --depth 1
cd WebODM
./webodm.sh start 
  • If you face any issues at the last step on Linux, make sure your user is part of the docker group:
sudo usermod -aG docker $USER
exit
(restart shell by logging out and then back-in)
./webodm.sh start
  • Open a Web Browser to http://localhost:8000 (unless you are on Windows using Docker Toolbox, see below)

Docker Toolbox users need to find the IP of their docker machine by running this command from the Docker Quickstart Terminal:

docker-machine ip
192.168.1.100 (your output will be different)

The address to connect to would then be: http://192.168.1.100:8000.

To stop WebODM press CTRL+C or run:

./webodm.sh stop

To update WebODM to the latest version use:

./webodm.sh update

Manage Processing Nodes

WebODM can be linked to one or more processing nodes that speak the NodeODM API, such as NodeODM, NodeMICMAC or ClusterODM. The default configuration includes a "node-odm-1" processing node which runs on the same machine as WebODM, just to help you get started. As you become more familiar with WebODM, you might want to install processing nodes on separate machines.

Adding more processing nodes will allow you to run multiple jobs in parallel.

You can also setup a ClusterODM node to run a single task across multiple machines with distributed split-merge and process dozen of thousands of images more quickly, with less memory.

If you don't need the default "node-odm-1" node, simply pass --default-nodes 0 flag when starting WebODM:

./webodm.sh restart --default-nodes 0.

Then from the web interface simply manually remove the "node-odm-1" node.

Enable MicMac

WebODM can use MicMac as a processing engine via NodeMICMAC. To add MicMac, simply run:

./webodm.sh restart --with-micmac

This will create a "node-micmac-1" processing node on the same machine running WebODM. Please note that NodeMICMAC is in active development and is currently experimental. If you find issues, please report them on the NodeMICMAC repository.

Enable SSL

WebODM has the ability to automatically request and install a SSL certificate via Let’s Encrypt, or you can manually specify your own key/certificate pair.

  • Setup your DNS record (webodm.myorg.com --> IP of server).
  • Make sure port 80 and 443 are open.
  • Run the following:
./webodm.sh restart --ssl --hostname webodm.myorg.com

That's it! The certificate will automatically renew when needed.

If you want to specify your own key/certificate pair, simply pass the --ssl-key and --ssl-cert option to ./webodm.sh. See ./webodm.sh --help for more information.

Note! You cannot pass an IP address to the hostname parameter! You need a DNS record setup.

Where Are My Files Stored?

When using Docker, all processing results are stored in a docker volume and are not available on the host filesystem. There are two specific docker volumes of interest:

  1. Media (called webodm_appmedia): This is where all files related to a project and task are stored.
  2. Postgres DB (called webodm_dbdata): This is what Postgres database uses to store its data.

For more information on how these two volumes are used and in which containers, please refer to the docker-compose.yml file.

For various reasons such as ease of backup/restore, if you want to store your files on the host filesystem instead of a docker volume, you need to pass a path via the --media-dir and/or the --db-dir options:

./webodm.sh restart --media-dir /home/user/webodm_data --db-dir /home/user/webodm_db

Note that existing task results will not be available after the change. Refer to the Migrate Data Volumes section of the Docker documentation for information on migrating existing task results.

Common Troubleshooting

Symptoms Possible Solutions
Run out of memory Make sure that your Docker environment has enough RAM allocated: MacOS Instructions, Windows Instructions
While starting WebODM you get: 'WaitNamedPipe','The system cannot find the file specified.' 1. Make sure you have enabled VT-x virtualization in the BIOS.
2. Try to downgrade your version of Python to 2.7
On Windows, docker-compose fails with Failed to execute the script docker-compose Make sure you have enabled VT-x virtualization in the BIOS
Cannot access WebODM using Microsoft Edge on Windows 10 Try to tweak your internet properties according to these instructions
Getting a No space left on device error, but hard drive has enough space left Docker on Windows by default will allocate only 20GB of space to the default docker-machine. You need to increase that amount. See this link and this link
Cannot start WebODM via ./webodm.sh start, error messages are different at each retry You could be running out of memory. Make sure you have enough RAM available. 2GB should be the recommended minimum, unless you know what you are doing
While running WebODM with Docker Toolbox (VirtualBox) you cannot access WebODM from another computer in the same network. As Administrator, run cmd.exe and then type "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" controlvm "default" natpf1 "rule-name,tcp,,8000,,8000"
On Windows, the storage space shown on the WebODM diagnostic page is not the same as what is actually set in Docker's settings. From Hyper-V Manager, right-click “DockerDesktopVM”, go to Edit Disk, then choose to expand the disk and match the maximum size to the settings specified in the docker settings. Upon making the changes, restart docker.

Images Missing from Lightning Assets

When you use Lightning to process your task, you will need to download all assets to your local instance of WebODM. The all assets zip does not contain the images which were used to create the orthomosaic. This means that, although you can visualise the cameras layer in your local WebODM, when you click on a particular camera icon the image will not be shown.

The fix if you are using WebODM with Docker is as follows (instructions are for MacOS host):

  1. Ensure that you have a directory which contains all of the images for the task and only the images;
  2. Open Docker Desktop and navigate to Containers. Identify your WebODM instance and navigate to the container that is named worker. You will need the Container ID. This is a hash which is listed under the container name. Click to copy the Container ID using the copy icon next to it.
  3. Open Terminal and enter docker cp <sourcedirectory>/. <dockercontainerID>:/webodm/app/media/project/<projectID>/task/<taskID>. Paste the Container ID to replace the location titled <dockercontainerID>. Enter the full directory path for your images to replace <sourcedirectory>;
  4. Go back to Docker Desktop and navigate to Volumes in the side bar. Click on the volume called webodm_appmedia, click on project, identify the correct project and click on it, click on task and identify the correct task.
  5. From Docker Desktop substitute the correct <projectID> and <taskID> into the command in Terminal;
  6. Execute the newly edited command in Terminal. You will see a series of progress messages and your images will be copied to Docker;
  7. Navigate to your project in your local instance of WebODM;
  8. Open the Map and turn on the Cameras layer (top left);
  9. Click on a Camera icon and the relevant image will be shown

Have you had other issues? Please report them so that we can include them in this document.

Backup and Restore

If you want to move WebODM to another system, you just need to transfer the docker volumes (unless you are storing your files on the file system).

On the old system:

mkdir -v backup
docker run --rm --volume webodm_dbdata:/temp --volume `pwd`/backup:/backup ubuntu tar cvf /backup/dbdata.tar /temp
docker run --rm --volume webodm_appmedia:/temp --volume `pwd`/backup:/backup ubuntu tar cvf /backup/appmedia.tar /temp

Your backup files will be stored in the newly created backup directory. Transfer the backup directory to the new system, then on the new system:

ls backup # --> appmedia.tar  dbdata.tar
./webodm.sh down # Make sure WebODM is down
docker run --rm --volume webodm_dbdata:/temp --volume `pwd`/backup:/backup ubuntu bash -c "rm -fr /temp/* && tar xvf /backup/dbdata.tar"
docker run --rm --volume webodm_appmedia:/temp --volume `pwd`/backup:/backup ubuntu bash -c "rm -fr /temp/* && tar xvf /backup/appmedia.tar"
./webodm.sh start

In case when recovery .tar is missed, or corrupted you can conduct Hard Recovery

Reset Password

If you forgot the password you picked the first time you logged into WebODM, to reset it just type:

./webodm.sh start && ./webodm.sh resetadminpassword newpass

The password will be reset to newpass. The command will also tell you what username you chose.

Manage Plugins

Plugins can be enabled and disabled from the user interface. Simply go to Administration -- Plugins.

Update

If you use docker, updating is as simple as running:

./webodm.sh update

If you are running WebODM natively, these commands should do it:

cd /webodm
sudo su odm # Only in case you are running WebODM with a different user
git pull origin master
source python3-venv/bin/activate # If you are running a virtualenv
npm install
pip install -r requirements.txt
webpack --mode production
python manage.py collectstatic --noinput
python manage.py migrate

Recommended Machine Specs

To run a standalone installation of WebODM (the user interface), including the processing component (NodeODM), we recommend at a minimum:

  • 100 GB free disk space
  • 16 GB RAM

Don't expect to process more than a few hundred images with these specifications. To process larger datasets, add more RAM linearly to the number of images you want to process. A CPU with more cores will speed up processing, but can increase memory usage. GPU acceleration is also supported. To make use of your CUDA-compatible graphics card, make sure to pass --gpu when starting WebODM. You need the nvidia-docker installed in this case, see https://github.com/NVIDIA/nvidia-docker and https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker for information on docker/NVIDIA setup.

WebODM runs best on Linux, but works well on Windows and Mac too. If you are technically inclined, you can get WebODM to run natively on all three platforms.

WebODM by itself is just a user interface (see below) and does not require many resources. WebODM can be loaded on a machine with just 1 or 2 GB of RAM and work fine without NodeODM. You can then use a processing service such as the lightning network or run NodeODM on a separate, more powerful machine.

Customizing and Extending

Small customizations such as changing the application colors, name, logo, or adding custom CSS/HTML/Javascript can be performed directly from the Customize -- Brand/Theme panels within WebODM. No need to fork or change the code.

More advanced customizations can be achieved by writing plugins. This is the preferred way to add new functionality to WebODM since it requires less effort than maintaining a separate fork. The plugin system features server-side signals that can be used to be notified of various events, a ES6/React build system, a dynamic client-side API for adding elements to the UI, a built-in data store, an async task runner, a GRASS engine, hooks to add menu items and functions to rapidly inject CSS, Javascript and Django views.

For plugins, the best source of documentation currently is to look at existing code. If a particular hook / entrypoint for your plugin does not yet exist, request it. We are adding hooks and entrypoints as we go.

To create a plugin simply copy the plugins/test plugin into a new directory (for example, plugins/myplugin), then modify manifest.json, plugin.py and issue a ./webodm.sh restart.

API Docs

See the API documentation page.

Roadmap

We follow a bottom-up approach to decide what new features are added to WebODM. User feedback guides us in the decision making process and we collect such feedback via improvement requests.

Don't see a feature that you want? Open a feature request or help us build it.

Sometimes we also prioritize work that has received financial backing. If your organization is in the position to financially support the development of a particular feature, get in touch and we'll make it happen.

Getting Help

We have several channels of communication for people to ask questions and to get involved with the community:

We also have a Gitter Chat, but the preferred way to communicate is via the OpenDroneMap Community Forum.

Support the Project

There are many ways to contribute back to the project:

  • Help us test new and existing features and report bugs and feedback.
  • Share your aerial datasets.
  • Help answer questions on the community forum and chat.
  • ⭐️ us on GitHub.
  • Help us translate WebODM in your language.
  • Help us classify point cloud datasets.
  • Spread the word about WebODM and OpenDroneMap on social media.
  • While we don't accept donations, you can purchase an installer, a book or a sponsor package.
  • You can pledge funds for getting new features built and bug fixed.
  • Become a contributor 🤘

Translations

It's easy to translate WebODM in a different language!

If you want to preview your translation work, start WebODM in developer mode:

./webodm.sh restart --dev

Then edit the LOCALES file to include your translation locale code. Finally, visit the Developer Tools panel from WebODM's dashboard and press the Download and Replace Translation Files button:

image

The latest translation files from weblate.org will be downloaded and applied to the installation of WebODM.

Become a Contributor

The easiest way to get started is to take a look at our list of outstanding issues and pick one. You can also fix/improve something entirely new based on your experience with WebODM. All ideas are considered and people of all skill levels are welcome to contribute.

You don't necessarily need to be a developer to become a contributor. We can use your help to write better documentation and improve the user interface texts and visuals.

If you know how to code, we primarily use Python (Django), Javascript (React), HTML and SCSS. See the Development Quickstart and Contributing documents for more information.

To make a contribution, you will need to open a pull request (here's how). To make changes to WebODM, make a clone of the repository and run ./webodm.sh start --dev.

If you have questions visit us on the forum and we'll be happy to help you out with your first contribution.

Architecture Overview

The OpenDroneMap project is composed of several components.

  • ODM is a command line toolkit that processes aerial images. Users comfortable with the command line are probably OK using this component alone.
  • NodeODM is a lightweight interface and API (Application Program Interface) built directly on top of ODM. Users not comfortable with the command line can use this interface to process aerial images and developers can use the API to build applications. Features such as user authentication, map displays, etc. are not provided.
  • WebODM adds more features such as user authentication, map displays, 3D displays, a higher level API and the ability to orchestrate multiple processing nodes (run jobs in parallel). Processing nodes are simply servers running NodeODM.

webodm

WebODM is built with scalability and performance in mind. While the default setup places all databases and applications on the same machine, users can separate its components for increased performance (ex. place a Celery worker on a separate machine for running background tasks).

Architecture

A few things to note:

  • We use Celery workers to do background tasks such as resizing images and processing task results, but we use an ad-hoc scheduling mechanism to communicate with NodeODM (which processes the orthophotos, 3D models, etc.). The choice to use two separate systems for task scheduling is due to the flexibility that an ad-hoc mechanism gives us for certain operations (capture task output, persistent data and ability to restart tasks mid-way, communication via REST calls, etc.).
  • If loaded on multiple machines, Celery workers should all share their app/media directory with the Django application (via network shares). You can manage workers via ./worker.sh

Run the docker version as a Linux Service

If you wish to run the docker version with auto start/monitoring/stop, etc, as a systemd style Linux Service, a systemd unit file is included in the service folder of the repo.

This should work on any Linux OS capable of running WebODM, and using a SystemD based service daemon (such as Ubuntu 16.04 server for example).

This has only been tested on Ubuntu 16.04 server and Red Hat Enterprise Linux 9.

The following pre-requisites are required:

  • Requires odm user
  • Requires docker installed via system (ubuntu: sudo apt-get install docker.io)
  • Requires 'screen' package to be installed
  • Requires odm user member of docker group
  • Required WebODM directory checked out/cloned to /opt/WebODM
  • Requires that /opt/WebODM is recursively owned by odm:odm
  • Requires that a Python 3 environment is used at /opt/WebODM/python3-venv

If all pre-requisites have been met, and repository is checked out/cloned to /opt/WebODM folder, then you can use the following steps to enable and manage the service:

First, to install the service, and enable the services to run at startup from now on:

sudo systemctl enable /opt/WebODM/service/webodm-docker.service

To manually start/stop the service:

sudo systemctl stop webodm-docker
sudo systemctl start webodm-docker

To manually check service status:

sudo systemctl status webodm-docker

For the adventurous, the repository can be put anyplace you like by editing the ./WebODM/service/webodm-docker.service file before enabling the service the reflect your repository location, and modifying the systemctl enable command to that directiory.

Run it natively

WebODM can run natively on Windows, MacOS and Linux. We don't recommend to run WebODM natively (using docker is easier), but it's possible.

Ubuntu 16.04 LTS users can refer to this community script to install WebODM natively on a new machine.

To run WebODM, you will need to install:

  • PostgreSQL (>= 9.5)
  • PostGIS 2.3
  • Python 3.6
  • GDAL (>= 3)
  • Node.js (>= 6.0)
  • Nginx (Linux/MacOS) - OR - Apache + mod_wsgi or Waitress (Windows)
  • Redis (>= 2.6)
  • GRASS GIS (>= 7.8)

On Linux, make sure you have:

apt-get install binutils libproj-dev gdal-bin nginx

On Windows use the OSGeo4W installer to install GDAL. MacOS users can use:

brew install postgres postgis

Then these steps should be sufficient to get you up and running:

git clone --depth 1 https://github.com/OpenDroneMap/WebODM

Create a WebODM/webodm/local_settings.py file containing your database settings:

DATABASES = {
    'default': {
        'ENGINE': 'django.contrib.gis.db.backends.postgis',
        'NAME': 'webodm_dev',
        'USER': 'postgres',
        'PASSWORD': 'postgres',
        'HOST': 'localhost',
        'PORT': '5432',
    }
}

From psql or pgadmin, connect to PostgreSQL, create a new database (name it webodm_dev), connect to it and set the postgis.enable_outdb_rasters and postgis.gdal_enabled_drivers settings:

ALTER SYSTEM SET postgis.enable_outdb_rasters TO True;
ALTER SYSTEM SET postgis.gdal_enabled_drivers TO 'GTiff';

Start the redis broker:

redis-server

Then:

pip install -r requirements.txt
sudo npm install -g webpack
sudo npm install -g webpack-cli
npm install
webpack --mode production
python manage.py collectstatic --noinput
chmod +x start.sh && ./start.sh --no-gunicorn

Finally, start at least one celery worker:

./worker.sh start

The start.sh script will use Django's built-in server if you pass the --no-gunicorn parameter. This is good for testing, but bad for production.

In production, if you have nginx installed, modify the configuration file in nginx/nginx.conf to match your system's configuration and just run start.sh without parameters.

Windows users should refer to this guide to install Apache + mod_wsgi and run gunicorn:

gunicorn webodm.wsgi --bind 0.0.0.0:8000 --preload

If you are getting a rt_raster_gdal_warp: Could not create GDAL transformation object for output dataset creation, make sure that your PostGIS installation has PROJ support:

SELECT PostGIS_Full_Version();

You may also need to set the environment variable PROJSO to the .so or .dll projection library your PostGIS is using. This just needs to have the name of the file. So for example on Windows, you would in Control Panel -> System -> Environment Variables add a system variable called PROJSO and set it to libproj.dll (if you are using proj 4.6.1). You'll have to restart your PostgreSQL service/daemon after this change. http://postgis.net/docs/manual-2.0/RT_ST_Transform.html

If you are using Windows and are unable to go past the pip install -r requirements.txt command because of an error regarding zlib and Pillow, manually edit the requirements.txt file, remove the Pillow requirement and run:

easy_install pillow
pip install -r requirements.txt

On Windows make sure that all of your PATH environment variables are set properly. These commands:

python --version
pip --version
npm --version
gdalinfo --version
redis-server --version

Should all work without errors.

Run it on the cloud (Google Compute, Amazon AWS)

12 steps, to have WebODM running on a cloud instance.

These steps are for Google Cloud, but can also be used for Amazon AWS, and other cloud platforms with small modifications:

  1. Launch a Google Cloud instance of Ubuntu 18.0 LTS.
  2. Open the SSH terminal - Google offers SSH via the website.
  3. Run sudo apt-get update
  4. Run sudo apt-get upgrade
  5. Run sudo apt-get install docker-compose
  6. Run sudo apt-get install python-pip
  7. Run git clone https://github.com/OpenDroneMap/WebODM --config core.autocrlf=input --depth 1
  8. cd WebODM (Linux is case sensitive)
  9. sudo ./webodm.sh start
  10. You now can access webodm via the public IP address for your google instance. Remember the default port of 8000.
  11. Check that your instance's firewall is allowing inbound TCP connections on port 8000! If you forget this step you will not be able to connect to WebODM.
  12. Open http://GooglepublicIPaddressforyourinstance:8000

To setup the firewall on Google Cloud, open the instance, on the middle of the instance settings page find NIC0. Open it, and then add the TCP Port 8000 for ingress, and egress on the Firewall.

License

WebODM is licensed under the terms of the GNU Affero General Public License v3.0.

Trademark

See Trademark Guidelines

webodm's People

Contributors

abizem avatar bluecamel avatar chris-bateman avatar danbjoseph avatar dependabot[bot] avatar diegoaces avatar gromain avatar hedo88th avatar holycowmp3 avatar islamheggy avatar kathenae avatar kikislater avatar manand881 avatar mateo3d avatar mojodna avatar nchamo avatar okaluza avatar patrickelectric avatar pierotofy avatar pmumby avatar projectbarks avatar pyup-bot avatar saijin-naib avatar saricicekibrahim avatar smathermather avatar tariqislam avatar usplm avatar vesnikos avatar vinsonliux avatar w-toguchi83 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

webodm's Issues

API

WebODM should implement the following endpoints:

  • authorization
  • list projects (Project) ☑️
    • [filtered by User] ☑️
    • [filtered by Group]
    • [filtered by geographic extent]
  • list tasks (Task) ☑️
  • create tasks (Task) ☑️
  • update tasks (Task) ☑️
  • list/add/change processing nodes ☑️
  • ability to start/stop processing a Task☑️
  • get project info ☑️
    • TileJSON / equivalent
    • 3D model stuff
  • create project ☑️
  • create mission
  • get mission info
  • ...

3D Model Display

ODM-generated 3D models should be browsable using a Cesium-like interface.

Needs display and manipulation of the following:

  • Point clouds
  • Textured Mesh

Switch to Python 3

Once library requirements have been assessed, change Python version from 2 to 3.

Install - step 13 run curl failure

Ran into this issue while running docker-compose up

Step 13 : RUN curl --silent --location https://deb.nodesource.com/setup_6.x | sudo bash -
 ---> Running in 6537e09cc0d5
/bin/sh: 1: sudo: not found
ERROR: Service 'webapp' failed to build: The command '/bin/sh -c curl --silent --location https://deb.nodesource.com/setup_6.x | sudo bash -' returned a non-zero code: 127

My specs:

  • Ubuntu 14.04 server
dmb2@OpenDroneMap:~/WebODM$ docker version
Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 22:00:36 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 22:00:36 2016
 OS/Arch:      linux/amd64
dmb2@OpenDroneMap:~/WebODM$ docker-compose --version
docker-compose version 1.8.1, build 878cff1

Ability to overwrite default settings.

An ability to overwrite the default settings is usefull in numerous occasions.

This can be implemented easily by appending :

try:
from .local_settings import *
except ImportError:
pass

ofcourse local_settings.py should be added in .gitignore as well

Volumetric Measurements

It would be cool to have the ability to measure volumes directly from the map.

I think this could be approached in the following way:

  1. Load 3D points, cast them to polygonZ collection, use http://postgis.net/docs/ST_DelaunayTriangles.html to generate a TIN
  2. Create stored procedure that given a list of 2D points:

Should also add the possibility to specify a "Cut" or "Fill" property for concave/convex cases.

Task stuck on running

Processing the odm_data_bellus dataset with --resize-to 1800 option, the task remains stuck on running. node-OpenDroneMap successfully processed the images.

GDALRaster.transform increases file size

Geotiffs created using GDALRaster.transform result in large files because of lack of compression. Perhaps where necessary we should call gdal_translate -co compress=DEFLATE input.tif output.tif (or the equivalent using the python API).

Use React

The dashboard is going to be a client side app using React that uses the API.

Show image previews

Once a task is done being processed, it would be nice for the user to see a preview and orthophoto extent information directly on the dashboard panel without opening the map.

Error messages on install

Installed the Docker image on Windows 10 64-bit. In the terminal window are repeated error messages:

[##mwebapp exit with code 127 (where ## is an integer in the range 31 through 36)
No such file or directory[0m /usr/ bin/env: bash
No such file or directory/usr/ bin/env: bash
No such file or directory/usr/ bin/env: bash
(repeated a total of 10 times)

Looking at the webapp container log in Kitematic, the "No such file or directory/usr/ bin/env: bash" also shows up 11 times.

Took a while, but figured out how to open up the WebApp. Am processing a simple task, and it seems to be working. So are those error messages of concern?

Unit Test nodeodm/client_api.py

Suggested approach since it involves communication with a third party component (node-OpenDroneMap).

  1. Create python server stub from node-OpenDroneMap's OpenAPI definition (swagger.json).
  2. Call client_api.py methods to test results.

Create Docker Startup Script

Some people are experiencing issues using Docker when rebuilding images.

A script should be created to handle the following tasks:

  • Start WebODM
  • Stop WebODM
  • Rebuild From Scratch Images, ignoring cache and removing the proper volumes.

Should Celery be used here?

In regard to a discussion which began here: #40 (review)

Is scheduler.py code reinventing the wheel (functionality that already exists in Celery)?

In essence it [the code] is replicating some of the Celery features (it is a task processing queue). What is driving me to re-implement this mechanism is that so far, the task processing involves more than a single processing step (think input images in ---> assets out). Maybe I'm over-thinking it, but so far I found these cases where Celery does not seem like it would fare so well.

  1. Input images in --> assets are not what I expected. I should be able to restart the process without re-sending the images, just by changing the processing options

  2. Console output: how to retrieve real-time console output from the opendronemap process?

  3. How to protect processing nodes (authentication) without making it difficult to setup? Ideally a processing node should be (optionally) protected by a static token, or a userID/password combination that generates a temporary token. The docs here hint at using public/private key exchange for security: http://docs.celeryproject.org/en/latest/userguide/security.html (I'm not sure I'd want to go down that route)

System Messaging

WebODM should keep a log of operations, such as when a task started getting processed, whether a task succeeded, whether a task failed and display these information in the menu bar.

Python 2 or 3?

Before the project proceeds any further I would like to ask the developers which version of python the would consider using?

I would opt for python 3 as in my experience I see python 2 being slowly phased out in favor of 3 and virtually all geospatial libraries now they providing support for it.

Thoughts?

Mission Planning

WebODM should facilitate the planning of missions.

Input:

  • area(s) to cover (polygon) - if multiple areas, treat as multiple missions?
  • angle(s) to take photos
  • altitude
  • hazards (is this a thing handled by mission planning software?)
  • camera specs
  • desired overlap
  • ...

Output:

  • 3D linestring representing flight path
  • [flight planning output]
  • ...

Add default node-OpenDroneMap instance in Docker compose

Many people are experiencing issues setting up a processing node manually using docker (because of the intricacies of docker).

I will add a docker-compose file and supporting configuration to setup automatically a processing node as part of the docker compose up command.

2D Model Display

ODM-generated orthophotos should be browsable using a Leaflet-like interface.

Needs:

  • TMS endpoint as TileJSON or a standard URL template
  • geographic extent (potentially in TileJSON)
  • zoom extent (potentially in TileJSON)
  • ...

Worker Process

Need to create a thread / cronjob that performs the following:

  • Checks for tasks that need to be processed, and starts processing
  • Updates processing node info

Elevation Model

It would be cool to have the ability to display an automatically generated elevation model along with the orthophoto.

@smathermather how would you go about generating an elevation model from the current outputs of opendronemap (lastools come to mind, but they aren't OSS)?

Image Upload + Project/Task creation

Somewhat linked to #21, there's going to be a button for uploading images directly from the dashboard. This will in turn automatically create the first project and task. These are all shortcuts for the sake of ease of use, and a more in depth project and task management UI will be created.

postgresql-9.5-postgis-2.3 installation failed

Hi,

I upgraded my system to Ubuntu 16.10, and I can't get WebODM to run.
I'm getting this error :
E: Version '2.3.0+dfsg-2.pgdg80+1' for 'postgresql-9.5-postgis-2.3' was not found E: Version '2.3.0+dfsg-2.pgdg80+1' for 'postgresql-9.5-postgis-2.3-scripts' was not found E: Version '2.3.0+dfsg-2.pgdg80+1' for 'postgis' was not found ERROR: Service 'db' failed to build: The command '/bin/sh -c apt-get update && apt-get install -y --no-install-recommends postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR=$POSTGIS_VERSION postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR-scripts=$POSTGIS_VERSION postgis=$POSTGIS_VERSION && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100

Should I change something in the db/Dockerfile ?

Thank you.

Keeping track of state history

Leaving the dashboard and pressing the back button on the browser currently loses the application's state. This should be fixed with something like react-history.

Task Processing

WebODM should facilitate the creation and management of tasks.

Tasks represent processing steps required to convert a stack of photos (or video) into various ODM outputs.

Tasks contain these pieces of metadata:

  • ...
  • created timestamp
  • updated timestamp

Free Processing Nodes Volunteer Network

At some point it could be beneficial to create a basic infrastructure for allowing people to donate server resources to process maps.

It could be as simple as having volunteers install the processing software, having a sign-up form, a web page with the list of volunteers, and some code to check which nodes are online.

Communication with node-OpenDroneMap

In the next few weeks I'll dedicate some time to write the code necessary to create new ProcessingNode instances and have them communicate with https://github.com/pierotofy/node-OpenDroneMap. I will likely spend some time tweaking node-OpenDroneMap to take care of authorization issues and to make the process of setting up new `ProcessingNode instances easy.

After careful evaluation, I think I will not add Celery as discussed in chat to the mix; the additional complexity added by Celery does not seem to simplify much the problem of distributing tasks to multiple nodes. This is particularly true because the number of processing tasks that we can expect are not many, so I think a simpler cron-like lib, with some custom logic, a priority queue and basic error handling/retry will serve the purpose (hopefully) better.

Docker

I'm currently working to add support for initializing a working environment in Docker. This will make it easier for people to contribute and will allow users to get started more quickly.

Will also add instructions in the readme.

User Interface

The WebODM user interface needs to surface the following functionality:

  • task management - creation and status
  • model display - allow orthophotos + 3D models to be browsed
  • cluster management - ???
  • mission planning - is there existing FOSS for this?
  • documentation

@pierotofy can you clarify what you're thinking around cluster management?

I'm also thinking that model display and mission planning might be better suited as their own applications, integrating with the WebODM API according to their needs.

Thoughts?

Newbie Help

Hi! Such a great work! I'm very interested in the project, however I can't managed to make it work yet. When I launch the web interface and add a task, system says "There are no processing nodes available. Make sure at least one of them is reachable.". How can I create a processing node? I thought this was already included with this interface, but I was mistaken. Is there any tutorial available or walktrough process from the image loading to the final orthomosaic? A little help would be nice. Thanks in advance! Cheers!

Remove Fixtures, create base test class

Fixtures are a little problematic due to the fact that we catch signals and have first run initialization commands that do not play well with fixtures.

  1. Remove test_projects and test_users
  2. Create base class for tests that calls boot code
  3. Make all tests inherit from such base class
  4. Create test data programmatically via .create()

Processing error

Hi there
I am not sure if this error is related to webODM or the processig node

My setup is two different AWS instance (t2.medium) one running WebODM and the second one running Node-ODM, both installed with Dockers

Killed
[ERROR]   quitting cause: 
	/code/SuperBuild/install/bin/pmvs2 /var/www/data/813e7f99-16a8-4ec7-b5d8-edaa958957aa/pmvs/recon0/ option-0000
returned with code 35072.

any help will be very helpful

Processing of lots of images fails in timeout

File "C:\Users\piero\Desktop\WebODM\nodeodm\api_client.py", line 64, in new_task
data={'name': name, 'options': json.dumps(options)}).json()
File "C:\Users\piero\webodm\lib\site-packages\requests\api.py", line 110, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Users\piero\webodm\lib\site-packages\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\piero\webodm\lib\site-packages\requests\sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\piero\webodm\lib\site-packages\requests\sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "C:\Users\piero\webodm\lib\site-packages\requests\adapters.py", line 473, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond a

requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond', None, 10060, None))

One Click Installer

Although this is something to do toward the end, the Docker command line is still too confusing for many GIS technicians; there's going to be a need for a graphical installer.

Upload Issue

if the upload is quicker than you are at providing a name / clicking "Save" then you are stuck where you cannot submit the job anymore ...

Check CSS styling for admin area

Some formatting issues are still present in the administration area (due to collision with the bootstrap theme and the default admin CSS).

The admin area should be checked and fixes should be applied where necessary.

Fail when run start.sh

Ciao Piero,

Iḿ trying run WebODM from start.sh, but the following errors are listed :

emilio@emilio-laptop:/opt/WebODM$ ./start.sh
Building asssets...
Hash: bcdca8dde46cb9609cf5
Version: webpack 1.13.3
Time: 52662ms
Asset Size Chunks Chunk Names
main-bcdca8dde46cb9609cf5.js 1.9 MB 0 [emitted] main
css/main.css 41.3 kB 0 [emitted] main
[0] multi main 28 bytes {0} [built]
+ 255 hidden modules

WARNING in .//leaflet-measure/dist/leaflet-measure.js
Critical dependencies:
1:113-120 This seems to be a pre-built javascript file. Though this is possible, it's not recommended. Try to require the original source to get better results.
@ ./
/leaflet-measure/dist/leaflet-measure.js 1:113-120
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 6 hidden modules
Child extract-text-webpack-plugin:
+ 5 hidden modules
Child extract-text-webpack-plugin:
+ 14 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Child extract-text-webpack-plugin:
+ 2 hidden modules
Running migrations
Traceback (most recent call last):
File "manage.py", line 22, in
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 367, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 341, in execute
django.setup()
File "/usr/local/lib/python2.7/dist-packages/django/init.py", line 27, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 108, in populate
app_config.import_models(all_models)
File "/usr/local/lib/python2.7/dist-packages/django/apps/config.py", line 199, in import_models
self.models_module = import_module(models_module_name)
File "/usr/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
File "/opt/WebODM/app/models.py", line 23, in
from nodeodm.models import ProcessingNode
File "/opt/WebODM/nodeodm/models.py", line 10, in
from .api_client import ApiClient
File "/opt/WebODM/nodeodm/api_client.py", line 9, in
from urllib.parse import urlunparse
ImportError: No module named parse
Unhandled exception in thread started by <function wrapper at 0x7fbf61b15848>
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/django/utils/autoreload.py", line 226, in wrapper
fn(args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/runserver.py", line 113, in inner_run
autoreload.raise_last_exception()
File "/usr/local/lib/python2.7/dist-packages/django/utils/autoreload.py", line 249, in raise_last_exception
six.reraise(
_exception)
File "/usr/local/lib/python2.7/dist-packages/django/utils/autoreload.py", line 226, in wrapper
fn(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/init.py", line 27, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 108, in populate
app_config.import_models(all_models)
File "/usr/local/lib/python2.7/dist-packages/django/apps/config.py", line 199, in import_models
self.models_module = import_module(models_module_name)
File "/usr/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
File "/opt/WebODM/app/models.py", line 23, in
from nodeodm.models import ProcessingNode
File "/opt/WebODM/nodeodm/models.py", line 10, in
from .api_client import ApiClient
File "/opt/WebODM/nodeodm/api_client.py", line 9, in
from urllib.parse import urlunparse
ImportError: No module named parse

Grazie mille.

Emilio Bruno

GCP interface integration

I currently skipped the parts of code to handle GCP files. This will need to be added at some point.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.