ispyb / ispyb-database

ISPyB Project

1. Installing
2. Database creation and update
3. Database schema

Installing

1. Clone the ISPyB repository:

   git clone https://github.com/ispyb/ISPyB.git
   

2. ISPyB needs the third-party libraries provided in the dependencies directory. These don't exist in a public repository, so install them to the local Maven repository so that Maven can find them:

   cd dependencies && mvn initialize
   

3. Build ISPyB using Maven:

   mvn clean install
   

   By default, ISPyB builds for the GENERIC site and the development environment. These can be changed with the ispyb.site and ispyb.env system properties. For example, to build for the ESRF site and the test environment:

   mvn -Dispyb.site=ESRF -Dispyb.env=test clean install
   

   If the build succeeds, a summary message will be printed like this:

   [INFO] Reactor Summary:
   [INFO]
   [INFO] ispyb-parent ...................................... SUCCESS [0.251s]
   [INFO] ispyb-ejb3 ........................................ SUCCESS [10.243s]
   [INFO] ispyb-ws .......................................... SUCCESS [1.751s]
   [INFO] ispyb-ui .......................................... SUCCESS [7.212s]
   [INFO] ispyb-ear ......................................... SUCCESS [5.048s]
   [INFO] ispyb-bcr ......................................... SUCCESS [2.217s]
   [INFO] ispyb-bcr-ear ..................................... SUCCESS [1.806s]
   

Database creation and update

Run the following creation scripts from the ispyb-ejb module (note that this requires the pxadmin database user to exist and have full permissions):

1. ispyb-ejb/db/scripts/pyconfig.sql: This corresponds to the menu options and contains both structure and data.

2. ispyb-ejb/db/scripts/pydb.sql: This corresponds to the ISPyB metadata and contains only the database structure.

3. ispyb-ejb/db/scripts/schemastatus.sql: This corresponds to the SchemaStatus table and contains both structure and data. The entries indicate which update scripts have been run.

4. ispyb-ejb/db/scripts/ispybAutoprocAttachment.sql: This corresponds to the type and names of different autoPROC attachments.

The creation scripts are normally updated for each tag, but if you are using the master branch, you may have to run the update scripts in ispyb-ejb/db/scripts/ahead.

Check the entries in the SchemaStatus table to know which scripts to execute. The scripts already run for the current tag are in ispyb-ejb/db/scripts/passed.

Creating an update script

The first line must be:

insert into SchemaStatus (scriptName, schemaStatus) values ('2017_06_06_blabla.sql','ONGOING');

Then comes the actual updates of the script:

-- SQL statements here

And the last line must be:

update SchemaStatus set schemaStatus = 'DONE' where scriptName = '2017_06_06_blabla.sql';

The last line updates the SchemaStatus table to mark the script as having been run.

Database schema

A patch or commit that changes the database schema must be accompanied by a corresponding change to the schema file to keep it up to date. This file can be edited using MySQL Workbench (a free tool from MySQL).

ispyb-database's Issues

Evaluate changes required to merge dls schema

Bring pydb up to date

There are a series of changes which are agreed in the collaboration, and update scripts exist, but they dont seem to be applied to the production pydb:

• https://github.com/ispyb/ISPyB/blob/master/ispyb-ejb/db/scripts/passed/2016/2016_05_17_DBMerging.sql

https://github.com/ispyb/ISPyB/tree/master/ispyb-ejb/db/updates

-- BLSession
-- https://github.com/ispyb/ISPyB/blob/master/ispyb-ejb/db/scripts/ahead/2019_03_29_BLSession_archived.sql
ALTER TABLE BLSession
    ADD archived boolean DEFAULT False 
        COMMENT 'The data for the session is archived and no longer available on disk';

-- DataCollectionComment
-- https://github.com/ispyb/ISPyB/blob/master/ispyb-ejb/db/scripts/passed/2017/2017_05_02_Merge_with_DLS_create.sql
-- Not in pydb.sql
DROP TABLE IF EXISTS `DataCollectionComment`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `DataCollectionComment` (
  `dataCollectionCommentId` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `dataCollectionId` int(11) unsigned NOT NULL,
  `personId` int(10) unsigned NOT NULL,
  `comments` varchar(4000) DEFAULT NULL,
  `createTime` timestamp NOT NULL DEFAULT current_timestamp(),
  `modTime` date DEFAULT NULL,
  PRIMARY KEY (`dataCollectionCommentId`),
  KEY `dataCollectionComment_fk1` (`dataCollectionId`),
  KEY `dataCollectionComment_fk2` (`personId`),
  CONSTRAINT `dataCollectionComment_fk1` FOREIGN KEY (`dataCollectionId`) REFERENCES `DataCollection` (`dataCollectionId`) ON DELETE CASCADE ON UPDATE CASCADE,
  CONSTRAINT `dataCollectionComment_fk2` FOREIGN KEY (`personId`) REFERENCES `Person` (`personId`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

-- https://github.com/ispyb/ISPyB/blob/master/ispyb-ejb/db/scripts/passed/2017/2017_05_02_Merge_with_DLS_create.sql
-- Not in pydb.sql
DROP TABLE IF EXISTS `CourierTermsAccepted`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `CourierTermsAccepted` (
  `courierTermsAcceptedId` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `proposalId` int(10) unsigned NOT NULL,
  `personId` int(10) unsigned NOT NULL,
  `shippingName` varchar(100) DEFAULT NULL,
  `timestamp` datetime DEFAULT current_timestamp(),
  `shippingId` int(11) unsigned DEFAULT NULL,
  PRIMARY KEY (`courierTermsAcceptedId`),
  KEY `CourierTermsAccepted_ibfk_1` (`proposalId`),
  KEY `CourierTermsAccepted_ibfk_2` (`personId`),
  KEY `CourierTermsAccepted_ibfk_3` (`shippingId`),
  CONSTRAINT `CourierTermsAccepted_ibfk_1` FOREIGN KEY (`proposalId`) REFERENCES `Proposal` (`proposalId`),
  CONSTRAINT `CourierTermsAccepted_ibfk_2` FOREIGN KEY (`personId`) REFERENCES `Person` (`personId`),
  CONSTRAINT `CourierTermsAccepted_ibfk_3` FOREIGN KEY (`shippingId`) REFERENCES `Shipping` (`shippingId`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COMMENT='Records acceptances of the courier T and C';
/*!40101 SET character_set_client = @saved_cs_client */;

-- DewarRegistry
-- https://github.com/ispyb/ISPyB/blob/master/ispyb-ejb/db/scripts/passed/2017/2017_05_02_Merge_with_DLS_create.sql
-- Not in pydb.sql
DROP TABLE IF EXISTS `DewarRegistry`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `DewarRegistry` (
  `facilityCode` varchar(20) NOT NULL,
  `proposalId` int(11) unsigned NOT NULL,
  `labContactId` int(11) unsigned NOT NULL,
  `purchaseDate` datetime DEFAULT NULL,
  `bltimestamp` datetime NOT NULL DEFAULT current_timestamp(),
  PRIMARY KEY (`facilityCode`),
  KEY `DewarRegistry_ibfk_1` (`proposalId`),
  KEY `DewarRegistry_ibfk_2` (`labContactId`),
  CONSTRAINT `DewarRegistry_ibfk_1` FOREIGN KEY (`proposalId`) REFERENCES `Proposal` (`proposalId`) ON DELETE CASCADE,
  CONSTRAINT `DewarRegistry_ibfk_2` FOREIGN KEY (`labContactId`) REFERENCES `LabContact` (`labContactId`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

Add User Portal Sync permission

Regenerate 1_tables.sql on update pr