BiBiGrid is a tool for an easy cluster setup inside a cloud environment.
License: Apache License 2.0
Web application framework mainly for bioinformatic developers to publish their tools with an user-friendly web interface.
https://github.com/BiBiServ/Development-Guidelines
dependency: >= Maven 3.3.9
Run
mvn packageWe decided that all modules should have the same version as the parent module. By using the below command in the project root you can update all child modules at once.
mvn versions:set -DnewVersion=<version>where
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
@maitai and I worked on a developer guide draft:
Bibigrid development should follow the github flow
which means that features or fixes should be pushed into a separate branch and merged into the master or development branch by creating a pull request.
This should allow us to organize code reviews and automated tests before updating the development/master branch.
Pattern for branch names:
'refactor/name'
'feature/name'
'fix/name'
Feature branches should only be merged into the developer branch.
The master branch contains the code of the latest release.
The release version names and tags should follow semantic versioning
Commit messages should follow the angular.js guide which allows to create automatically release summaries.
https://github.com/angular/angular.js/blob/master/CONTRIBUTING.md#commit
Hi,
I'm trying to setup bibigrid and have problems creating the images. I found bibigrid-image on the linked page. Sadly it doesn't work. It won't get past fetching the keys. I think it is a problem of the firewall as the first key is successfully received. The second fails, but if I modify the keyserver flag in the displayed command (keyserver.ubuntu.com ==> hkp://keyserver.ubuntu.com:80) I can download the key manually. I have limited control over the firewall.
(If this is too much off-topic for this repository, I would appreciate a redirection)
Currently the bibigrid terminate action supports only one argument (= cluster id to terminate). Sometimes it seems to be useful to terminate more than one cluster with a single call (e.g. clean up after a workshop)
Instead of looping through the ssh and sftp connect process, poll if ssh is available
Like username with accidental space at the end. Trim all parameters loaded from cmd or properties files.
This bug currently affects only OpenStack
Currently it is possible to execute startup scripts on master node only.
In case of testing a new software / modules, executing of startup scripts on slave nodes is needed.
For better automation it should be possible to define the script in openstack.properties file.
Evaluate the idea for a user friendly web interface
When terminate a cluster using the -t option it could be happend that the corresponding SG is not removed.
Until now only predefined network and subnets could be used to start an Openstack Cluster. However in the next release BiBigrid should setup a new network (if no one is given in the configuration) and should create a new subnet for each initialized cluster to enhance the have natural separation between the cluster instances.
The option sg (security groups) currently replaces a bibigrid specific security group. It isn't possible
to allow using additional security groups. This could be useful if a BiBiGrid cluster is used by another cloud instance/framework ...
When using -v and -o together, i got following exception :
java -jar target/BiBiGrid-1.0.jar -o ~/.bibigrid.properties.giessen -c -v
ERROR: Error while parsing the commandline arguments: The option 'v' was specified but an option from this group has already been selected: 'o' [StartUp:146]
This is definitely a bug ...
Since BiBiGrid now supports standard Ubuntu cloud images, we must stop/disable apt-daily.service The apt-daily.service is started by cloud-init during initialisation. The services itself is triggered immediately and prevents other apt processes to run until they are finished. In our case a running apt-daily.service prevents BiBiGrid to install ansible and run the BiBigrid specific playbooks.
A possible solution is discussed here. It seem that only a sledge-hammer helps :
systemctl stop apt-daily.service
systemctl disable apt-daily.service
systemctl kill --kill-who=all apt-daily.service
# wait until `apt-get updated` has been killed
while ! (systemctl list-units --all apt-daily.service | fgrep -q dead)
do
sleep 1;
done
Using the -V flag provides null or an error message in stead of the version number.
Ok : Master (ID: 665bf9aa-ddf8-4ce1-9d45-ef53c3281e4a) started
Ok : FloatingIP XXX assigned to Master(ID: 665bf9aa-ddf8-4ce1-9d45-ef53c3281e4a)
Ok : Master (ID: 665bf9aa-ddf8-4ce1-9d45-ef53c3281e4a) network configuration finished
Ok : Waiting for slave instances ready ...
Ok : [0/5] Instance bibigrid-slave-1-dVeR6OzoTsevLUC is active !
Ok : [1/5] Instance bibigrid-slave-4-dVeR6OzoTsevLUC is active !
Ok : [2/5] Instance bibigrid-slave-5-dVeR6OzoTsevLUC is active !
Ok : [3/5] Instance bibigrid-slave-3-dVeR6OzoTsevLUC is active !
Ok : [4/5] Instance bibigrid-slave-1-dVeR6OzoTsevLUC is active !
Ok : [5/5] Instance bibigrid-slave-4-dVeR6OzoTsevLUC is active !
Ok : [6/5] Instance bibigrid-slave-5-dVeR6OzoTsevLUC is active !
Ok : [7/5] Instance bibigrid-slave-3-dVeR6OzoTsevLUC is active !
Ok : [8/5] Instance bibigrid-slave-1-dVeR6OzoTsevLUC is active !
Ok : [9/5] Instance bibigrid-slave-4-dVeR6OzoTsevLUC is active !
Ok : [10/5] Instance bibigrid-slave-5-dVeR6OzoTsevLUC is active !
Ok : [11/5] Instance bibigrid-slave-3-dVeR6OzoTsevLUC is active !
Ok : [12/5] Instance bibigrid-slave-1-dVeR6OzoTsevLUC is active !
Ok : [13/5] Instance bibigrid-slave-4-dVeR6OzoTsevLUC is active !
Ok : [14/5] Instance bibigrid-slave-5-dVeR6OzoTsevLUC is active !
Ok : [15/5] Instance bibigrid-slave-3-dVeR6OzoTsevLUC is active !
Ok : [16/5] Instance bibigrid-slave-1-dVeR6OzoTsevLUC is active !
Ok : [17/5] Instance bibigrid-slave-4-dVeR6OzoTsevLUC is active !
Ok : [18/5] Instance bibigrid-slave-5-dVeR6OzoTsevLUC is active !
We have currently the problem that bibigrid starts with blank Ubuntu images. Everything needed is installed during the ansible run. That causes two problems:
We can avoid this, when using prepared images. Workflow to create prepared images:
BibiGrid throws NullPointer
Message:
ERROR: null [CreateClusterOpenstack:336]
ERROR: Aborting operation. Instances already running. I will try to shut them down but in case of an error they might remain running. Please check manually afterwards. [CreateIntent:41]
Further debugging shows that NullPointer is throws in this line:
by accessing network addresses:
server.getAddresses().getAddresses(conf.getNetworkname())The startup time could be increased when all slave instances are started in parallel ...
Supporting different flavors for slave instances cloud be useful, when using BiBiGrid for a worfklow that have very different resource requirements for its single tasks. Not sure how to implement this in a useful manner but it is interesting idea and challenge for the future ...
The current layout does not fit our needs anymore. We should separate the model and the general parts from the framework from the cloud specific implementations (AWS, Google, Openstack, ... ). The general configuration of the grid is currently hard coded, which is fast (while executed) but awful to extend and fix. This will be replaced by an Ansible setup in near future ( issue #47)
Here is a 1st idea for a preliminary structure:
/ansible
/model
/exceptions
/intents
/aws
/google
/openstack
We should implement unit tests for all external functions and use an continuous testing/integration framework.
Currently the image creator scripts are located in an extra project available on hg.cebitec.uni-bielefeld.de. Since BiBiGrid can't be used without prepared images, the image creator scripts should be part of bibigrid project.
Using docker on a large bibigrid cluster can cause a high internet network traffic when (larger) images must be loaded from a public docker hub. The (outside) network bandwidth can be reduced when the master acts as registry cache for all running slave instances. However the [Docker Registry](Docker v2 Registry) can be easily configured to be a registry cache ( see entry from blog.docker ).
BiBiGrid images still bases on the Ubuntu 14.04 LTS release. We should switch to the latest LTS Release (16.04).
We have to increase the user feedback during the cluster configuration done by Ansible. Depending on the chosen configuration and the number of slaves the installation and configuration can take a while without having any feedback in non verbose mode.
Link Downloads section in
https://wiki.cebitec.uni-bielefeld.de/bibiserv-1.25.2/index.php/BiBiGrid
points to jobproxy and not to the bibigrid github repository.
The project documentation is outdated (since the BiBiGrid is moved to GitHub) and must be adapted to the latest version.
some combination of instance and snaphots in spot requests result in error:
State cancelled: snapshotId can only be modified on EBS devices (400 response code)
Status bad-parameters: Your Spot request failed due to bad parameters.
Instance type(s) c4.8xlarge
I tried to mount a snapshot of a high-throughput volume type
Implement private mode without public ips started from an instance already present in the subnet/security group.
@jkrue As discussed before, it's maybe a good idea to move the early execute scripts from the metadata of the instance to the ansible process.
This would...
Also in this context, there are currently three options...
When starting several BiBiGrids in parallel (e.g. workshop environment) it happens that an available floating ip is assigned to more than BiBiGrid master instance.
Currently only existing router/network/subnetworks but no security groups can be used when initialise a new BiBiGrid cluster. In some situations (e.g. workshops, demo sessions) it makes sense to use an existing security group for initialise.
In my opinion the current properties file should be split in configuration.properties and credentials.properties.
This should make it easier to share property files without the fear of exposing the password.
I propose the following parameter:
-cr credentials.properties | [--username <arg>, --password <arg>]where
username=user
password=xxxxxx-osp,--openstack-password <arg>
-osu,--openstack-username <arg>
-a,--aws-credentials-file <arg> If we agree on this, I can set up a PR.
If there is already a way to separate configuration details from credentials, then please close this issue.
While the HDFS is working like a charm, parts of the general Hadoop configuration via Yarn are incomplete.
User name is 'unknown' if not explicitly by cmdline option. Should be this host system user name ...
Example:
ERROR: null [CreateClusterOpenstack:336]
ERROR: Aborting operation. Instances already running. I will try to shut them down but in case of an error they might remain running. Please check manually afterwards. [CreateIntent:41]
Originates from:
We have to adjust our documentation to the latest features:
When build BiBiGrid using mvn package -Dmaven.test.skip=true the packaged Java archive fails, e.g.:
$ java -jar BiBiGrid-Main-1.1.jar -o /Users/jkrueger/.bibigrid.properties -l
Ok : Alternative config file /Users/jkrueger/.bibigrid.properties will be used.
Exception in thread "main" java.lang.NoSuchFieldError: INSTANCE
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.<clinit>(SSLConnectionSocketFactory.java:146)
at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:964)
at org.openstack4j.connectors.resteasy.executors.ApacheHttpClientEngine.create(ApacheHttpClientEngine.java:75)
at org.openstack4j.connectors.resteasy.HttpCommand.initialize(HttpCommand.java:46)
at org.openstack4j.connectors.resteasy.HttpCommand.create(HttpCommand.java:40)
at org.openstack4j.connectors.resteasy.HttpExecutorServiceImpl.invoke(HttpExecutorServiceImpl.java:52)
at org.openstack4j.connectors.resteasy.HttpExecutorServiceImpl.execute(HttpExecutorServiceImpl.java:31)
at org.openstack4j.core.transport.internal.HttpExecutor.execute(HttpExecutor.java:51)
at org.openstack4j.openstack.internal.OSAuthenticator.authenticateV3(OSAuthenticator.java:187)
at org.openstack4j.openstack.internal.OSAuthenticator.invoke(OSAuthenticator.java:74)
at org.openstack4j.openstack.client.OSClientBuilder$ClientV3.authenticate(OSClientBuilder.java:172)
at org.openstack4j.openstack.client.OSClientBuilder$ClientV3.authenticate(OSClientBuilder.java:129)
at de.unibi.cebitec.bibigrid.openstack.OpenStackUtils.buildOSClientV3(OpenStackUtils.java:36)
at de.unibi.cebitec.bibigrid.openstack.OpenStackUtils.buildOSClient(OpenStackUtils.java:16)
at de.unibi.cebitec.bibigrid.openstack.ListIntentOpenstack.<init>(ListIntentOpenstack.java:30)
at de.unibi.cebitec.bibigrid.openstack.ProviderModuleOpenstack.getListIntent(ProviderModuleOpenstack.java:35)
at de.unibi.cebitec.bibigrid.StartUp.runIntent(StartUp.java:140)
at de.unibi.cebitec.bibigrid.StartUp.main(StartUp.java:94)
Running the same setup directly from an IDE works ...
We should replace the manually ssh (post) configuration by ansible. Using Ansible playbooks makes our implementation much more flexible. Configuration changes could be make easible by the end user without changing the code base.
Sometimes it could happen that a started cluster instance can't be reached for some reason. In that case the master post configuration hangs forever ... we should integrate a timeout !
... with following exception:
Ok : Upload Ansible playbook to master instance.
Ok : Connecting sftp channel...
ERROR: SFTP: {} [CreateCluster:300]
com.jcraft.jsch.SftpException: No such file
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846) ~[bibigrid-openstack-2.0.jar:na]
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:753) ~[bibigrid-openstack-2.0.jar:na]
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:700) ~[bibigrid-openstack-2.0.jar:na]
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:694) ~[bibigrid-openstack-2.0.jar:na]
at de.unibi.cebitec.bibigrid.core.intents.CreateCluster.uploadAnsibleToMaster(CreateCluster.java:289) [bibigrid-openstack-2.0.jar:na]
at de.unibi.cebitec.bibigrid.core.intents.CreateCluster.configureMaster(CreateCluster.java:244) [bibigrid-openstack-2.0.jar:na]
at de.unibi.cebitec.bibigrid.core.intents.CreateCluster.launchClusterInstances(CreateCluster.java:124) [bibigrid-openstack-2.0.jar:na]
at de.unibi.cebitec.bibigrid.StartUp.runIntent(StartUp.java:158) [bibigrid-openstack-2.0.jar:na]
at de.unibi.cebitec.bibigrid.StartUp.main(StartUp.java:93) [bibigrid-openstack-2.0.jar:na]
ERROR: Master instance configuration failed! [CreateCluster:255]
The same action initiated by the IDE (Intellj) works like a charm ...
The reverse proxy entry for the Apache contains a bug that prevents the Webserver to show the Spark service page correctly.
when starting a cluster in project A, bibigrid tries to intergrate a running instance from project B into the SGE of cluster A
ubuntu@host-192-168-0-36:~$ qhost
global - - - - - - -
host-192-168-0-36.openstacklocal lx26-amd64 12 0.40 177.1G 1.3G 0.0 0.0
host-192-168-0-37.openstacklocal lx26-amd64 6 0.28 88.5G 721.2M 0.0 0.0
host-192-168-0-38.openstacklocal lx26-amd64 6 0.28 88.5G 727.3M 0.0 0.0
host-192-168-0-39.openstacklocal lx26-amd64 6 0.18 88.5G 718.5M 0.0 0.0
host-192-168-0-40.openstacklocal lx26-amd64 6 0.29 88.5G 720.6M 0.0 0.0
host-192-168-10-45.openstacklocal lx26-amd64 12 - 177.1G - 0.0 -
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.