BenchBot is a tool for seamlessly testing & evaluating semantic scene understanding tools in both realistic 3D simulation & on real robots
License: BSD 3-Clause "New" or "Revised" License
Hi, btalb
After installing benchbot, i run the command of benchbot_run as this, "benchbot_run --env miniroom:1 --task semantic_slam:active:ground_truth -f". There is a error as following,
################################################################################
################# CHECKING FOR BENCHBOT SOFTWARE STACK UPDATES #################
################################################################################
Skipping ...
################################################################################
##################### STARTING THE BENCHBOT SOFTWARE STACK #####################
################################################################################
Running the BenchBot system with the following settings:
Selected environment: miniroom:1
Selected task: semantic_slam:active:ground_truth
Actions set: active
Observations set: ground_truth
Maps: "/Game/AI_vol3_03_base/Maps/AI_vol3_scene_03"
(starting @ [0.7,0,0,-0.7,1.2,1.5,0.3])
a825ecb8729beabb70cbe2ee715030ff7c769e1574f00a35c8662cac32695f9a
Created benchbot_network.
non-network local connections being added to access control list
089e4f7b7a74ea670a21cd23eaa0b6bcdb3b12ae5caa6022ac9c986300a2938f
3e31dc23b4deec436f60c0963b3fde2ec9d3b281c036b0ebc020c25518d03087
9623b25144d5ce4c618049e6ea9e5a091101385a33f7fc8c3ddc782758d267ac
58f7fd8f431424c902762112df5c5877deeae7a2d05be2c98926244296c7dd1c
non-network local connections being removed from access control list
################################################################################
################### BENCHBOT IS RUNNING (Ctrl^C to exit) ... ###################
################################################################################
Initialising supervisor
Configuring the supervisor...
Starting a supervisor with the following configuration:
{'actions': ['move_distance', 'move_angle'],
'environment_names': ['miniroom_1'],
'observations': ['image_depth',
'image_depth_info',
'image_rgb',
'image_rgb_info',
'laser',
'poses'],
'robot': {'image_depth': {'callback_api': 'api_callbacks.decode_jsonpickle',
'callback_supervisor': 'supervisor_callbacks.encode_depth_image',
'connection': 'ros_to_api',
'ros_topic': '/camera/depth/image_raw',
'ros_type': 'sensor_msgs/Image'},
'image_depth_info': {'callback_api': 'api_callbacks.decode_jsonpickle',
'callback_supervisor': 'supervisor_callbacks.encode_camera_info',
'connection': 'ros_to_api',
'ros_topic': '/camera/depth/camera_info',
'ros_type': 'sensor_msgs/CameraInfo'},
'image_rgb': {'callback_api': 'api_callbacks.decode_color_image',
'callback_supervisor': 'supervisor_callbacks.encode_color_image',
'connection': 'ros_to_api',
'ros_topic': '/camera/color/image_raw',
'ros_type': 'sensor_msgs/Image'},
'image_rgb_info': {'callback_api': 'api_callbacks.decode_jsonpickle',
'callback_supervisor': 'supervisor_callbacks.encode_camera_info',
'connection': 'ros_to_api',
'ros_topic': '/camera/color/camera_info',
'ros_type': 'sensor_msgs/CameraInfo'},
'laser': {'callback_api': 'api_callbacks.decode_jsonpickle',
'callback_supervisor': 'supervisor_callbacks.encode_laserscan',
'connection': 'ros_to_api',
'ros_topic': '/scan_laser',
'ros_type': 'sensor_msgs/LaserScan'},
'move_angle': {'callback_supervisor': 'supervisor_callbacks.move_angle',
'connection': 'api_to_ros',
'ros_topic': '/cmd_vel',
'ros_type': 'geometry_msgs/Twist'},
'move_distance': {'callback_supervisor': 'supervisor_callbacks.move_distance',
'connection': 'api_to_ros',
'ros_topic': '/cmd_vel',
'ros_type': 'geometry_msgs/Twist'},
'move_next': {'callback_supervisor': 'supervisor_callbacks.move_next',
'connection': 'api_to_ros',
'ros_topic': '/cmd_vel',
'ros_type': 'geometry_msgs/Twist'},
'poses': {'callback_api': 'api_callbacks.decode_jsonpickle',
'callback_supervisor': 'supervisor_callbacks.create_pose_list',
'connection': 'ros_to_api'}},
'task_name': 'semantic_slam:active:ground_truth'}
Traceback (most recent call last):
File "/benchbot/ros_ws/src/benchbot_supervisor/nodes/benchbot_supervisor", line 10, in
s.run()
File "", line 374, in run
TypeError: 'module' object is not callable
################################################################################
###################### CLEANING UP ALL BENCHBOT REMNANTS #######################
################################################################################
Stopped the following containers:
58f7fd8f4314
089e4f7b7a74
Deleted the following containers:
Deleted Containers:
58f7fd8f431424c902762112df5c5877deeae7a2d05be2c98926244296c7dd1c
9623b25144d5ce4c618049e6ea9e5a091101385a33f7fc8c3ddc782758d267ac
3e31dc23b4deec436f60c0963b3fde2ec9d3b281c036b0ebc020c25518d03087
089e4f7b7a74ea670a21cd23eaa0b6bcdb3b12ae5caa6022ac9c986300a2938f
74d7f9453579d005a0cb3f47ee255ada11bd83512ab4131e06d697c0f0b987ac
030e5d4de5e9d61e1f513241b98e37771eeced8e75503162cae0c00239fffc1f
f7f1370cbd81b6b40faa13b10995bdfe19254ad17350772826f9a383a65cef02
97a1410d0d12c17d5a06ce3f89691c38ee9f943ae6f16d2753a0d20b50a92e40
f7da8869792111d43c3891dfe5d0a8bf44b5c0079e2c002c80def09ad897c104
9a9ac8bfbe590eafe00c395ad90375666d99401d3894effe63ab9623a5914370
Deleted Networks:
benchbot_network
Total reclaimed space: 228.5MB
Finished cleaning!
Hello,
I would like to use the benchbot, but, currently, I don't have access to a system with the necessary hardware requirements.
I was wondering, is it possible to run benchbot in a headless server with no graphical interface and connect to the simulation remotely?
Thanks
Currently we use the NVIDIA_DEFAULT variable to pick which driver should be installed if they're missing.
Maintaining this manually is bad. We're always going to go out of date.
Commit 4c8bf4f moves us to 510, but we need a solution that selects the latest automatically.
We have gone with conservative controller gains for now, but they could be more aggressive.
Well-tuned gains would mean the robot would move between poses quicker, while still not suffering from overshoot or instabilities.
Currently dirty simulator detection isn't detected in the Omniverse version of BenchBot.
We need to add in the logic, and push the event into the BenchBot ecosystem.
There are instances (at least in the visualization) where the RGB and depth images appear to be out of sync.
It appears to be an issue of synchronization rather than rgb and depth coming from cameras with different intrinsics. This is proved by the below gif of the rgb and depth after a move_distance 0.0 command, showing the images are perfectly aligned.
The severity of this effect appears to be amplified by how quickly the robot was moving before it stops as after some motions, there is still perfect overlap. This is shown below with the results of a 90 degree rotation where the robot smoothly slowed down to a stop:
The depth image appears to be ahead of the rgb image (at least in my couple of tests) as shown below.
Rotating anticlockwise (+ve 20)
Rotating clockwise (-ve 20)
This issue is not limited to rotation commands as after a large distance command, due to the robot motion in the simulator, sharp motion may be required to correct orientation.
Move distance 0.7
Note that because of the correction being a very sharp rotational movement, the difference is more severe than in previous examples wherein the robot still moved quite smoothly.
Again I cannot confirm yet whether this is an issue with the visualizer or the entire system of sending messages but it should be looked into so we can guarantee users are provided with perfectly aligned RGBD information.
Hi,
As we reported earlier we obtained 0.13 OMQ from benchbot evaluation of miniroom:1 on default votenet. Now after updating the repo, the OMQ improved to 0.21. The eval.ai shows your baseline as 0.41. What changed in the evaluation from before?
Hi there,
I have just installed the benchbot successfully on a machine with RTX2080 8GB, 32GB ram, i7-9700K CPU.
But when i tried to run
benchbot_run --robot carter --env miniroom:1 --task semantic_slam:passive:ground_truth
I keep getting a robot controller error. (small snippet below and the full log in the file attached.)
I'm wondering if you guys ever encountered this.
Thanks!
Chen.
...
Supervisor is now available @ 'http://0.0.0.0:10000' ...
Waiting until a robot controller is found @ 'http://benchbot_robot:10000' ...
Found
Sending environment data & robot config to controller ...
Ready
################################################################################
####################### BENCHBOT ROBOT CONTROLLER ERROR ########################
################################################################################
ERROR: The BenchBot Robot Controller container has exited unexpectedly. This
should not happen under normal operating conditions. Please see the complete
log below for a dump of the crash output:
Robot controller is now available @ 'http://0.0.0.0:10000' ...
Waiting to receive valid config data...
172.20.0.102 - - [2021-03-22 15:04:04] "GET // HTTP/1.1" 200 152 0.000542
172.20.0.102 - - [2021-03-22 15:04:05] "POST //configure HTTP/1.1" 200 137 0.066839
Starting the requested real robot ROS stack ...
THE PROCESS STARTED BY THE FOLLOWING COMMAND HAS CRASHED:
sed -i "0,/"pose":/{s/("pose": )(.)/\1[0.7, 0, 0, -0.7, 1.2, 1.5, 0.3]/}" /benchbot/isaac_sdk/apps/carter/carter_sim/bridge_config/carter_full_config.json && perl -0777 -i -pe 's/"static_mesh".?]/"static_mesh":[{"name": "bottle"}, {"name": "cup"}, {"name": "knife"}, {"name": "bowl"}, {"name": "wine glass"}, {"name": "fork"}, {"name": "spoon"}, {"name": "banana"}, {"name": "apple"}, {"name": "orange"}, {"name": "cake"}, {"name": "potted plant"}, {"name": "mouse"}, {"name": "keyboard"}, {"name": "laptop"}, {"name": "cell phone"}, {"name": "book"}, {"name": "clock"}, {"name": "chair"}, {"name": "table"}, {"name": "couch"}, {"name": "bed"}, {"name": "toilet"}, {"name": "tv"}, {"name": "microwave"}, {"name": "toaster"}, {"name": "refrigerator"}, {"name": "oven"}, {"name": "sink"}, {"name": "person"}]/s' /benchbot/isaac_sdk/apps/carter/carter_sim/bridge_config/carter_full_config.json && cd "/benchbot/addons/benchbot_addons/benchbot-addons/envs_isaac_develop/environments" && .sim_package/IsaacSimProject.sh "/Game/AI_vol3_03_base/Maps/AI_vol3_scene_03" -isaac_sim_config_json= "/benchbot/isaac_sdk/apps/carter/carter_sim/bridge_config/carter_full.json" -windowed -ResX=960 -ResY=540 -vulkan -game
Hi, although we solved issue #13, we got another problem. The stimulation window appears and then disappears immediately. Any check I can do to debug it? Thanks for your help.
i run the 'my_semantic_slam/my_semantic_slam.py' and i get some bug. the steps of installing votenet is successfully down.
and i can run the demo of votenet successfully.
here is the details of the bug:
Traceback (most recent call last):
File "/home/turtlepc-04/my_semantic_slam/my_semantic_slam.py", line 24, in
BenchBot(agent=MyAgent()).run()
File "/home/turtlepc-04/anaconda3/lib/python3.8/site-packages/benchbot_api/benchbot.py", line 326, in run
scene_fn()
File "/home/turtlepc-04/anaconda3/lib/python3.8/site-packages/benchbot_api/benchbot.py", line 321, in scene_fn
action, action_args = self.agent.pick_action(
File "/home/turtlepc-04/my_semantic_slam/my_semantic_slam.py", line 14, in pick_action
results = votenet_detection(self._votenet, observations)
File "/home/turtlepc-04/my_semantic_slam/votenet_benchbot.py", line 205, in votenet_detection
w_T_c = compute_world_T_camera(camera_pose['rotation_xyzw'],
File "/home/turtlepc-04/my_semantic_slam/votenet_benchbot.py", line 165, in compute_world_T_camera
rotation_matrix[0:3][:, 0:3] = rot.as_dcm().T
AttributeError: 'scipy.spatial.transform.rotation.Rotation' object has no attribute 'as_dcm'
Submission failed with result error code: 1
details of my_semantic_slam.py:
from benchbot_api import ActionResult, Agent, BenchBot
from votenet_benchbot import votenet_build, votenet_detection # new imports!
class MyAgent(Agent):
def init(self):
self._votenet = votenet_build()
def is_done(self, action_result):
return action_result != ActionResult.SUCCESS
def pick_action(self, observations, action_list):
results = votenet_detection(self._votenet, observations)
print("Detected %d objects in the frame: %s" %
(len(results), ",".join(r['class'] for r in results)))
return 'move_next', {}
def save_result(self, filename, empty_results, results_formant_fns):
pass # No results to save yet
if name == 'main':
print("Welcome to my Semantic SLAM solution!")
BenchBot(agent=MyAgent()).run()
Python packages (benchbot_addons, benchbot_api, & benchbot_eval) are installed by:
pip install -e .This means when we update the packages in a future benchbot_install, the files the pip package points to update, but the actual versions don't change in pip list | grep benchbot.
There is no actual effect from this bug, but it should be fixed by swapping the commands to pip install --upgrade -e . if possible.
Hello!
I successfully run a Semantic SLAM example on develop_1 env batch, got results in json format and calculated OMQ for every environment. But I achieved an overall score just about 0.013, instead 0.13, that you mention in the Tutorial. Is this is a typo in Tutorial or I have some problems on my side?
Thank you.
I was wondering if evaluation performance only works with object detection? Is it possible to just focus on SLAM without object detection to evaluate the results?
Example: benchbot_batch --robot carter --task semantic_slam:active:ground_truth --envs miniroom:1 --native python my_solution.py
the benchbot have somthing wrong so i try to reinstall the banchbot. i have the bug which i have never met before.
here the details:
Step 8/36 : RUN sudo rosdep init && rosdep update && mkdir -p ros_ws/src && source /opt/ros/melodic/setup.bash && pushd ros_ws && catkin_make && source devel/setup.bash && popd
---> Running in 633b142b6ebb
ERROR: cannot download default sources list from:
https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/sources.list.d/20-default.list
Website may be down.
The command '/bin/bash -c sudo rosdep init && rosdep update && mkdir -p ros_ws/src && source /opt/ros/melodic/setup.bash && pushd ros_ws && catkin_make && source devel/setup.bash && popd' returned a non-zero code: 4
i guess it is because the internet, but i have tried too many ways but just cannot fix it.
While running the docker command, pip could not find skbuild.
Looks like skbuild is replaced with scikit-build.
Looks like it's coming from OpenCV.4.3.0 build during benchbot_api build.
Important, this is inherent in opencv-python install. Not the problem of this repo.
Just leaving here for reference.
Hi, I want to get imu data from isaac.imu.ImuSim through adding nodes and edges in benchbot_simulator.app.json, like this,
**
nodes[
{
"name": "imu",
"components": [
{
"name": "message_ledger",
"type": "isaac::alice::MessageLedger"
},
{
"name": "imusim",
"type": "isaac::imu::ImuSim"
}
]
}
]
edges
{
{
"source": "RigidBodySubscriber/isaac.alice.TcpSubscriber/bodies",
"target": "imu/imusim/bodies"
},
{
"source": "imu/imusim/imu_raw",
"target": "imu_publisher/imu_publisher/robot_imu"
}
}**
Meanwhile, I have added two files,including imu_publisher.hh and imu_publisher.cc .
benchbot_simulator.zip
The following error is thrown when import benchbot_api is run in a terminal without an active display (i.e. environment variable DISPLAY is unset):
benchbot@submissions:~$ python3
Python 3.6.9 (default, Jul 17 2020, 12:50:27)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import benchbot_api
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/benchbot/benchbot/api/benchbot_api/__init__.py", line 4, in <module>
from . import tools
File "/home/benchbot/benchbot/api/benchbot_api/tools.py", line 6, in <module>
import matplotlib.pyplot as plt
File "/home/benchbot/.local/lib/python3.6/site-packages/matplotlib/pyplot.py", line 2319, in <module>
switch_backend(rcParams["backend"])
File "/home/benchbot/.local/lib/python3.6/site-packages/matplotlib/pyplot.py", line 270, in switch_backend
newbackend, required_framework, current_framework))
ImportError: Cannot load backend 'TkAgg' which requires the 'tk' interactive framework, as 'headless' is currently running
The error stems from the matplotlib backend requiring an active display when matplotlib is imported. The availability of matplotlib is not crucial to the BenchBot API (it is only used in the auxiliary observation visualiser). Given this, matplotlib issues should not block access to the entire API.
Importing matplotlib in the API needs to be adjusted to gracefully fail when there is no active display, so that users can run the API on headless systems as expected.
Hello,
We tried to reinstall the benchbot after updating the repo. We are getting an error at Benchbot core installation.
OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused:
Running hook #1:: error running hook: exit status 1, stdout: , stderr: invalid CUDA version: 11.2.2-1: unknown
We had uninstalled both nvidia and cuda drivers to let the installation script to install them. Both versions are chosen by the benchbot_install script.
Full log is attached. benchbot_install.txt
Currently collisions aren't detected in the Omniverse version of BenchBot.
We need to add in the logic, and push the event into the BenchBot ecosystem.
Hello, Thank you for this exciting challenge. I encountered an issue when It was trying to install Benchbot evaluation, the error is requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/simple/benchbot-addons/. FUll log is shown in the screenshot. Hope that will hear from you soon
I want to make 3D semantic map just like the below images. Can you explan me how I can make them? Is there any addons or code to visualize them?
I've done the environment setting and evaluation, but got stuck in the part to visualize the result. I want to see what's the problem in 3D map view. Thanks for a great software. 👍 :D
Hi, could you tell me the reason about modification of parent_T_camera ？
"CarterLeftCamera": {
"CameraComponent": {
"parent_actor": "carter_1",
"parent_T_camera": [1, 0, 0, 0, 0.11, 0.06, 0.31],
"parent_T_camera": [1, 0, 0, 0, 0.11, 0.06, 0.81],
"cols": 960,
"rows": 540,
"fov": 90.0,
"CarterRightCamera": {
"CameraComponent": {
"parent_actor": "carter_1",
"parent_T_camera": [1, 0, 0, 0, 0.11, -0.06, 0.31],
"parent_T_camera": [1, 0, 0.2, 0, -1, 0.0, 1.0],
"cols": 960,
"rows": 540,
"fov": 90.0,
I found something wrong when I was trying to run the example of "benchbot_submit --example hello_passive". i have finished the second step and I have gatten the pic1. but when i went to third step, it told me "Could not find a Benchbot supervisor"(pic2). how can i solve this problem? it has confused me for a long time. here are the details:
File ".hello_passive/hello_passive", line 11, in
BenchBot(agent=GuidedAgent()).run()
File "/home/turtlepc-04/.local/lib/python3.6/site-packages/benchbot_api/benchbot.py", line 84, in init
self.start()
File "/home/turtlepc-04/.local/lib/python3.6/site-packages/benchbot_api/benchbot.py", line 368, in start
"Are you sure it is available?" % self.supervisor_address)
requests.exceptions.ConnectionError: Could not find a BenchBot supervisor @ 'http://benchbot_supervisor:10000/'. Are you sure it is available?
Hi,
It seems the ISAAC 2019.2 SDK is not supported anymore by Nvidia.
There is no "Archive" folder on their download page.
(cf https://forums.developer.nvidia.com/t/downloading-isaac-sdk-2020-2/195227)
Is there any workaround?
Can Benchbot compile with the new ISAAC 2021.1 SDK?
I was wondering why
404: Not Found
We are trying to evaluate the baseline votenet (with pre-trained SUNRGBD) detection performance for semantic_slam:passive:ground_truth task. The solution script is unmodified code from Part IV: Creating an Object-based Semantic Map from Detections in this repo.
The command we used to generate results:
benchbot_batch --robo carter --task semantic_slam:passive:ground_truth --envs miniroom:1,miniroom:3,house:5 --native python my_solution.py
And the command used for evalulation:
benchbot_eval --method omq -o my_json_scores batch_0.json batch_1.json batch_2.json`
The result JSONs and score are here results.zip.
For all 3 envs, we are getting an OMQ of 0.0. We were expecting OMQ to be > 0 because there are overlaps between GT and the detections (see the bed in Figure_1- Miniroom:1). Light colors are GTs and dark colors are detections.
Can you please explain why the OMQ is 0.0?
Hi,
I have tried to install Benchbot from a Google Cloud Platform VM with GPU (Nvidia Quadro virtual workstation).
All the Nvidia drivers are mounted correctly and visible from nvidia-smi. However the Benchbot install script
fails because it cannot access the graphics card:
Is there any alternative way to install Benchbot without direct access to the physical card?
Thanks!
hi, I am an undergraduate. i have gotten the picture (first pic), but i run the eaxample "hello_passive" it showed that "Could not find a BenchBot supervisor"(pic).
Hi btalb, I canot run simulator. It is always waiting to establish connection to a running simulator ...
I excute two commands ,one is benchbot_run, the other is benchbot_submit,as following,
Originally posted by @liucsg in #2 (comment)
~/votenet/pointnet2$ python3 setup.py install
giving me an error on installation is there another way I caninstall the cuda layers manually?
Hi, Could you tell me how to solve the error occurred while installing , as follows，
/benchbot
/benchbot/isaac_sdk /benchbot
Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Hit:4 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease
Hit:5 http://packages.ros.org/ros/ubuntu bionic InRelease
Get:6 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Hit:7 https://packages.lunarg.com/vulkan bionic InRelease
Ign:2 https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu1804/x86_64 InRelease
Hit:8 https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu1804/x86_64 Release
Get:10 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1161 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1365 kB]
Fetched 2778 kB in 36s (76.4 kB/s)
Reading package lists...
[INFO] Installing packages - git git-lfs git-review build-essential g++
debconf: delaying package configuration, since apt-utils is not installed
[INFO] Successfully installed packages - git git-lfs git-review build-essential g++
[INFO] Installing packages - pkg-config zip zlib1g-dev unzip curl
[INFO] Successfully installed packages - pkg-config zip zlib1g-dev unzip curl
[INFO] Installing packages - uuid-dev
[INFO] Successfully installed packages - uuid-dev
[INFO] Installing packages - python python-dev
[INFO] Successfully installed packages - python python-dev
[INFO] Installing packages - libx11-dev libxcursor-dev libxrandr-dev libxinerama-dev
[INFO] Successfully installed packages - libx11-dev libxcursor-dev libxrandr-dev libxinerama-dev
[INFO] Installing packages - sox
debconf: delaying package configuration, since apt-utils is not installed
[INFO] Successfully installed packages - sox
[INFO] Installing packages - gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
debconf: delaying package configuration, since apt-utils is not installed
[INFO] Successfully installed packages - gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
[INFO] Installing packages - libvpx5 libtwolame-dev libtheora-bin libspeex1
debconf: delaying package configuration, since apt-utils is not installed
[INFO] Successfully installed packages - libvpx5 libtwolame-dev libtheora-bin libspeex1
[INFO] Installing packages - libopus0 libmp3lame0 libvdpau1 openexr libmpg123-0
debconf: delaying package configuration, since apt-utils is not installed
[INFO] Successfully installed packages - libopus0 libmp3lame0 libvdpau1 openexr libmpg123-0
[INFO] Installing packages - python3-pip latexmk texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended texlive-luatex texlive-xetex
debconf: delaying package configuration, since apt-utils is not installed
[INFO] Successfully installed packages - python3-pip latexmk texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended texlive-luatex texlive-xetex
[INFO] Installing packages - libudev-dev
debconf: delaying package configuration, since apt-utils is not installed
[INFO] Successfully installed packages - libudev-dev
[INFO] Installing packages - libi2c-dev
debconf: delaying package configuration, since apt-utils is not installed
[INFO] Successfully installed packages - libi2c-dev
[INFO] Installing packages - libhidapi-libusb0 libturbojpeg
debconf: delaying package configuration, since apt-utils is not installed
[INFO] Successfully installed packages - libhidapi-libusb0 libturbojpeg
[INFO] Installing packages - python3-dev libssl-dev
[INFO] Successfully installed packages - python3-dev libssl-dev
[INFO] Installing packages - python-tk libopenmpi-dev
[INFO] Successfully installed packages - python-tk libopenmpi-dev
[INFO] Installing packages - python3-sphinx python3-sphinx-rtd-theme
debconf: delaying package configuration, since apt-utils is not installed
[INFO] Successfully installed packages - python3-sphinx python3-sphinx-rtd-theme
Collecting pip
Downloading https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
Successfully installed pip-20.0.2
Collecting launchpadlib
Downloading launchpadlib-1.10.10.tar.gz (196 kB)
ERROR: Exception:
Traceback (most recent call last):
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 425, in _error_catcher
yield
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 507, in read
data = self._fp.read(amt) if not fp_closed else b""
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 62, in read
data = self.__fp.read(amt)
File "/usr/lib/python3.6/http/client.py", line 459, in read
n = self.readinto(b)
File "/usr/lib/python3.6/http/client.py", line 503, in readinto
n = self.fp.readinto(b)
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.6/ssl.py", line 1012, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.6/ssl.py", line 874, in read
return self._sslobj.read(len, buffer)
File "/usr/lib/python3.6/ssl.py", line 631, in read
v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
status = self.run(options, args)
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 331, in run
resolver.resolve(requirement_set)
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
discovered_reqs.extend(self._resolve_one(requirement_set, req))
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
abstract_dist = self.preparer.prepare_linked_requirement(req)
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
hashes=hashes,
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 287, in unpack_url
hashes=hashes,
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 159, in unpack_http_url
link, downloader, temp_dir.path, hashes
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 303, in _download_http_url
for chunk in download.chunks:
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/utils/ui.py", line 160, in iter
for x in it:
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/network/utils.py", line 39, in response_chunks
decode_content=False,
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 564, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 529, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/lib/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 430, in _error_catcher
raise ReadTimeoutError(self._pool, None, "Read timed out.")
pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
Collecting setuptools
Downloading setuptools-46.1.3-py3-none-any.whl (582 kB)
Installing collected packages: setuptools
WARNING: The scripts easy_install and easy_install-3.6 are installed in '/home/benchbot/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed setuptools-46.1.3
Collecting tensorflow-gpu==1.13.1
Downloading tensorflow_gpu-1.13.1-cp36-cp36m-manylinux1_x86_64.whl (345.2 MB)
ERROR: Exception:
Traceback (most recent call last):
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 425, in _error_catcher
yield
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 507, in read
data = self._fp.read(amt) if not fp_closed else b""
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 62, in read
data = self.__fp.read(amt)
File "/usr/lib/python3.6/http/client.py", line 459, in read
n = self.readinto(b)
File "/usr/lib/python3.6/http/client.py", line 503, in readinto
n = self.fp.readinto(b)
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.6/ssl.py", line 1012, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.6/ssl.py", line 874, in read
return self._sslobj.read(len, buffer)
File "/usr/lib/python3.6/ssl.py", line 631, in read
v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
status = self.run(options, args)
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 331, in run
resolver.resolve(requirement_set)
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
discovered_reqs.extend(self._resolve_one(requirement_set, req))
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
abstract_dist = self.preparer.prepare_linked_requirement(req)
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
hashes=hashes,
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 287, in unpack_url
hashes=hashes,
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 159, in unpack_http_url
link, downloader, temp_dir.path, hashes
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 303, in _download_http_url
for chunk in download.chunks:
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/utils/ui.py", line 160, in iter
for x in it:
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_internal/network/utils.py", line 39, in response_chunks
decode_content=False,
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 564, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 529, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/lib/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/home/benchbot/.local/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 430, in _error_catcher
raise ReadTimeoutError(self._pool, None, "Read timed out.")
pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
Collecting baselines
Cloning git://github.com/openai/baselines.git (to revision a071fa7630288ee00c86e140c5e674061f64690d) to /tmp/pip-install-6bz3t95w/baselines
Running command git clone -q git://github.com/openai/baselines.git /tmp/pip-install-6bz3t95w/baselines
Error: Building benchbot backend return a non zero error code: 1
There are some evaluation results after excuting benchbot_run and benchbot_submit, as following,
################################################################################
################# Running evaluation on '/tmp/benchbot_result' #################
################################################################################
Evaluating the performance from results in '/tmp/benchbot_result':
/home/kinco2/anaconda3/lib/python3.6/site-packages/benchbot_eval/evaluator.py:224: UserWarning: No 'class_list' field provided; assuming results have used our default class list
**Evaluation results:
{'environment_details': {'name': 'miniroom', 'numbers': ['1']},
'scores': {'OMQ': 0.0,
'avg_fp_quality': 1.0,
'avg_label': 0.0,
'avg_pairwise': 0.0,
'avg_spatial': 0.0},
'task_details': {'control_mode': 'passive',
'localisation_mode': 'ground_truth',
'type': 'semantic_slam'}}
Done.
I am not sure if the evalution result is normal.
I am trying to setup your simulator on my machine. After a successful install, I ran benchbot_run --robot carter_omni --env miniroom:1 --task semantic_slam:active:ground_truth but the command crashes moments after waiting for robot controller. The log is attached
here
Specs:
OS: Ubuntu 20.04 LTS
GPU: NVIDIA RTX A6000
CUDA: v11.6
Hi,
I am coming from issue 18, now I am using nomachine to connect to the remote machine;
I encountered a new error when I run
<benchbot_run --robot carter --env miniroom:1 --task semantic_slam:passive:ground_truth>
There is no "robot controller error" & vulkan crash log like before but unfortunately now I have this new error :
################################################################################
################### BENCHBOT IS RUNNING (Ctrl^C to exit) ... ###################
################################################################################
Initialising supervisor...
Configuring the supervisor...
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/benchbot/.local/lib/python3.6/site-packages/benchbot_supervisor/main.py", line 27, in
addons_path=args.addons_path)
File "/home/benchbot/.local/lib/python3.6/site-packages/benchbot_supervisor/benchbot_supervisor.py", line 61, in init
self.load()
File "/home/benchbot/.local/lib/python3.6/site-packages/benchbot_supervisor/benchbot_supervisor.py", line 82, in load
{} if 'results_format' not in self.config['task'] else
TypeError: argument of type 'NoneType' is not iterable
################################################################################
###################### CLEANING UP ALL BENCHBOT REMNANTS #######################
################################################################################\
I wonder if anyone has encountered the same problem as well?
Many Thanks!
High RAM usage in company (~50GB) causes company runs to be killed when the system doesn't have enough RAM.
Setting up a machine with sufficient swap space will allow the environment to run, but we want to get that RAM usage down so BenchBot can comfortably run on a system with 32GB RAM.
Startup times of each simulation are extremely poor currently (anywhere from 3 minutes to 9 minutes).
We use the on-disk cache currently, which improves subsequent runs.
We need to link the in-memory caching provided by Omniverse Cache into our Docker containers. This should offer significant improvements (we've seen demonstrations of ~30s startup times when working).
I try running the script : benchbot_run --robot carter --env miniroom:1 --task semantic_slam:active:ground_truth
But don't get in image?
The issac-sim docker pull fails at round 1.74GB. I've tried this multiple times at different times. What could possible be the issue here?
PS: I'm running this on a 90 GB partition.
`
sha256:059811a8ca6017b38b060975cacf598979aa4326921f1e6f7584da1278935a68 1.74GB / 3.74GB
FROM nvcr.io/nvidia/isaac-sim:2021.2.1@sha256:4a9a1cfadc276c72dc15b64fde4b08c1030ca92005f480605950e54c324c0708
error: failed to solve: failed commit on ref "layer-sha256:059811a8ca6017b38b060975cacf598979aa4326921f1e6f7584da1278935a68": "layer-sha256:059811a8ca6017b38b060975cacf598979aa4326921f1e6f7584da1278935a68" failed size validation: 1742733312 != 3743189127: failed precondition
ERROR: Building BenchBot simulator 'sim_omni' returned a non-zero error code: 1
Ensure that Docker has been installed correctly AND that you can run Docker
WITHOUT root access (there is no need to ever run Docker with root). See
https://docs.docker.com/install/linux/linux-postinstall/ for details on how to
fix this.
If the error is more generic, please contact us so that we can update our
pre-install host system checks.
`
When I run benchbot_run --robot carter --env miniroom:1 --task semantic_slam:active:ground_truth
I get the following error
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
The nvidia-smi output is as follows
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro RTX 8000 On | 00000000:1A:00.0 Off | Off |
| 33% 35C P8 17W / 260W | 6MiB / 48601MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 Quadro RTX 8000 On | 00000000:67:00.0 On | Off |
| 34% 45C P8 22W / 260W | 360MiB / 48598MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
Hi again,
At the PART 5: INSTALLING BENCHBOT ADD-ONS
When the installer tried to connect to cloudstor.aarnet.edu.au, it return back with the ERROR 503: Service Temporarily Unavailable.
Full log:
Created '/home/vuong/Code/rvc/benchbot/addons/benchbot_addons/.local/my_addons/{batches,environments,evaluation_methods,examples,formats,ground_truths,robots,tasks}'.
Installing add-ons based on the request string 'benchbot-addons/ssu':
Installing addon 'benchbot-addons/ssu' in '/home/vuong/Code/rvc/benchbot/addons/benchbot_addons':
Found install path './benchbot-addons/ssu'.
No action - latest already installed.
Installing addon 'benchbot-addons/tasks_ssu' in '/home/vuong/Code/rvc/benchbot/addons/benchbot_addons':
Found install path './benchbot-addons/tasks_ssu'.
No action - latest already installed.
Installing addon 'benchbot-addons/formats_object_map' in '/home/vuong/Code/rvc/benchbot/addons/benchbot_addons':
Found install path './benchbot-addons/formats_object_map'.
No action - latest already installed.
Installing addon 'benchbot-addons/envs_isaac_develop' in '/home/vuong/Code/rvc/benchbot/addons/benchbot_addons':
Found install path './benchbot-addons/envs_isaac_develop'.
No action - latest already installed.
Found remote content to install to 'environments': https://cloudstor.aarnet.edu.au/plus/s/ZqgsznfVNyxnbxV/download
Remote content is new. Fetching ...
--2021-04-29 13:46:57-- https://cloudstor.aarnet.edu.au/plus/s/ZqgsznfVNyxnbxV/download
Resolving cloudstor.aarnet.edu.au (cloudstor.aarnet.edu.au)... 202.158.207.20
Connecting to cloudstor.aarnet.edu.au (cloudstor.aarnet.edu.au)|202.158.207.20|:443... connected.
HTTP request sent, awaiting response... 503 Service Temporarily Unavailable
2021-04-29 13:46:59 ERROR 503: Service Temporarily Unavailable.
Fetching of remote content FAILED!!!
Installing addon 'benchbot-addons/robots_isaac' in '/home/vuong/Code/rvc/benchbot/addons/benchbot_addons':
Found install path './benchbot-addons/robots_isaac'.
No action - latest already installed.
Installing addon 'benchbot-addons/envs_isaac_challenge' in '/home/vuong/Code/rvc/benchbot/addons/benchbot_addons':
Found install path './benchbot-addons/envs_isaac_challenge'.
No action - latest already installed.
Found remote content to install to 'environments': https://cloudstor.aarnet.edu.au/plus/s/aSjCoQxLjVNUuwR/download
Remote content is new. Fetching ...
--2021-04-29 13:47:00-- https://cloudstor.aarnet.edu.au/plus/s/aSjCoQxLjVNUuwR/download
Resolving cloudstor.aarnet.edu.au (cloudstor.aarnet.edu.au)... 202.158.207.20
Connecting to cloudstor.aarnet.edu.au (cloudstor.aarnet.edu.au)|202.158.207.20|:443... connected.
HTTP request sent, awaiting response... 503 Service Temporarily Unavailable
2021-04-29 13:47:02 ERROR 503: Service Temporarily Unavailable.
Fetching of remote content FAILED!!!
I think that issue led to there is no simulated environment on my computer.
benchbot_run --list-environments
The following environments are supported in your BenchBot installation:
NONE!
benchbot_run --list-robots
The following robot targets are supported in your BenchBot installation:
carter
carter_remote
What should I do in this case?
Thank you for your time and waiting for your answers!
There errors with migrating the lighting across from the Unreal Engine version of Isaac Sim. This is particularly noticeable in the night time environments (variants 3 and 5 in each).
We need to re-configure these settings in Omniverse post-migration.
When I run "sudo ./install"
I get the follow error in building benchbot backend docker images
BUILDING BENCHBOT BACKEND DOCKER IMAGE:
Sending build context to Docker daemon 110.6MB
Step 1/36 : FROM benchbot/core:base
---> 82f8cca1412a
Step 2/36 : ENV ROS_WS_PATH="/benchbot/ros_ws"
---> Using cache
---> 9da9cdb2ad3d
Step 3/36 : RUN echo "deb http://packages.ros.org/ros/ubuntu bionic main" > /etc/apt/sources.list.d/ros-latest.list && apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654 && apt update && apt install -y ros-melodic-desktop-full python-rosdep python-rosinstall python-rosinstall-generator python-wstool python-catkin-tools python-pip build-essential
---> Using cache
---> ad477e4329e3
Step 4/36 : RUN wget -qO - http://packages.lunarg.com/lunarg-signing-key-pub.asc | apt-key add - && wget -qO /etc/apt/sources.list.d/lunarg-vulkan-bionic.list http://packages.lunarg.com/vulkan/lunarg-vulkan-bionic.list && apt update && DEBIAN_FRONTEND=noninteractive apt install -yq vulkan-sdk
---> Using cache
---> 8ad96ddad140
Step 5/36 : RUN useradd --create-home --password "" benchbot && passwd -d benchbot && apt update && apt install -yq sudo && usermod -aG sudo benchbot && usermod -aG root benchbot && mkdir /benchbot && chown benchbot:benchbot /benchbot
---> Using cache
---> a9c2c02eb71f
Step 6/36 : USER benchbot
---> Using cache
---> 8bd97047c382
Step 7/36 : WORKDIR /benchbot
---> Using cache
---> 8ad0a705edcf
Step 8/36 : RUN sudo rosdep init && rosdep update && mkdir -p ros_ws/src && source /opt/ros/melodic/setup.bash && pushd ros_ws && catkin_make && source devel/setup.bash && popd
---> Running in 2c2ce8c1ac5f
ERROR: cannot download default sources list from:
https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/sources.list.d/20-default.list
Website may be down.
The command '/bin/bash -c sudo rosdep init && rosdep update && mkdir -p ros_ws/src && source /opt/ros/melodic/setup.bash && pushd ros_ws && catkin_make && source devel/setup.bash && popd' returned a non-zero code: 4
ERROR: Building BenchBot backend returned a non-zero error code: 1
Ensure that Docker has been installed correctly AND that you can run Docker
WITHOUT root access (there is no need to ever run Docker with root). See
https://docs.docker.com/install/linux/linux-postinstall/ for details on how to
fix this.
If the error is more generic, please contact us so that we can update our
pre-install host system checks.
I encounter the same problem in installing ROS. It work fine after changing the proxy config in my PC,but in Docker it seems to be still wrong.
How can I solve this problem, change proxy settings of docker or installation instructions?
We have observed significant performance issues internally with the observation visualiser when using matplotlibversion 3.2+. On applying new observation data, the visualiser blocks for an indeterminate amount of time and even requires the mouse to be moved on the window before it will allow BenchBot to continue with the next action. An example is shown below:
We have observed this issue with matplotlib versions (we believe it may be related to matplotlib/matplotlib#17974):
The issue does not seem to occur with matplotlib versions:
If you are having issues running the examples, or your own code that uses the visualiser, install a working version of matplotlib on your system via:
pip install matplotlib==3.1.3
Note: Dockerfiles in the examples won't work until this is resolved as they use the latest version of matplotlib. We believe the best solution is to work around the matplotlib issues by creating an implementation that works with all versions rather than restrict all users to specific versions of a library.
Hello!
In the window of Isaac Sim I see, that the camera installed below LiDAR, but when I look at the observations dict returned by benchbot api I see, that the pose of camera_left have approx z = 1.3m, and lidar z = 0.86. Is it ok?
Also, I'm using ROS for development and create simple visualization of the data in RViz, captured from benchbot (in container benchbot_roscore).
In the picture below you can see frames (map, base_link, camera_right, lidar, camera_left). Height of camera_right looks ok, but camera_left is higher, than expected.
Also, green boxes - are boxes taken from gt file - addons/benchbot_addons/benchbot-addons/ground_truths_bear_develop/ground_truths/miniroom_1.json. It can be seen, that all of them below real objects in camera data (look at alarm clock, flowerpot).
Looks like, there some problem in transformations, probably caused by migration to omniverse.
This should not be interactive; it has been left from debugging:
Line 349 in f473fac
This should be removed in a future release.
Use the same fixes done in qcr/benchbot_robot_controller@60d0aaa.
hi, I am trying to save the data of "image_rgb", "image_ depth", "laser". but i have tried many ways, it doesn't work. what's more, the terminal didn't show anything is wrong.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.