hicservices / rdmp

Research Data Management Platform (RDMP) is an open source application for the loading,linking,anonymisation and extraction of datasets stored in relational databases.

Research Data Management Platform

• Demo Video
• Releases
• Quickstart
• FAQ
• User Manual
• Glossary
• Changelog
• Libraries
• New Developers

RDMP is a free, open source software application for cohort building, loading, linking, anonymisation and extraction of datasets stored in relational databases (SQL Server, MySQL, Postgres and Oracle). It was designed from the bottom up to support with data provenance, preserving domain knowledge and configuration management workflows.

RDMP does not require your data be moved or transformed prior to processing and is integrates into existing SQL based extraction practices.

Install

Signed release binaries for the RDMP client and Command Line Interface (CLI) are in the available in the GitHub releases section.

Windows Install Guide

Download and unzip rdmp-client.zip from the GitHub releases section and run ResearchDataManagementPlatform.exe. This will take you to an installation/setup screen which will guide you through the rest of the initial setup process.

Linux CLI Install Guide

The following steps can be used to install the RDMP CLI and start an SqlServer docker container to install into. Make sure to set the version (e.g. 7.0.14) to the latest and set a custom password if desired.

wget https://github.com/HicServices/RDMP/releases/download/v7.0.14/rdmp-cli-linux-x64.zip

unzip -d rdmp-cli ./rdmp-cli-linux-x64.zip

sudo docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=<YourStrong@Passw0rd>" \
   -p 1433:1433 --name sql1 --hostname sql1 \
   -d mcr.microsoft.com/mssql/server:2019-latest

cd rdmp-cli

cat > Databases.yaml << EOF
CatalogueConnectionString: Server=localhost;Database=RDMP_Catalogue;User ID=SA;Password=<YourStrong@Passw0rd>;Trust Server Certificate=true;
DataExportConnectionString: Server=localhost;Database=RDMP_DataExport;User ID=SA;Password=<YourStrong@Passw0rd>;Trust Server Certificate=true;
EOF

chmod +x ./rdmp

./rdmp install localhost RDMP_ -e -D -u SA -p "<YourStrong@Passw0rd>"
./rdmp gui

Build

Building on Windows

You can build RDMP from the command line using dotnet build or through an IDE e.g. Visual Studio or Visual Studio Code (Requires latest dotnet SDK).

The Windows client:

dotnet build
cd Application\ResearchDataManagementPlatform\bin\Debug\net6.0-windows\win-x64
./ResearchDataManagementPlatform.exe

The console client:

dotnet build
cd Tools\rdmp\bin\Debug\net6.0\
./rdmp.exe --help

Building on Linux

Only the console client can be built/run in Linux

cd Tools/rdmp
dotnet build
cd bin/Debug/net6.0
./rdmp --help

Tests

To run tests you will need an instance of SQL Server. These instructions use LocalDb which is an optional installable package in Visual Studio. See Tests.md for more information on setting up LocalDb, testing on linux or using an alternate server as well as information on the types of tests included and how to write new ones.

dotnet test ./scripts/run-all-tests.proj -c Release -p:BuildInParallel=false

Contributing

We welcome all contributions including:

• Issues and bug reports
• Code Reviews
• Translations
• Documentation
• Code Contributions

rdmp's Issues

Bad NLog.config xml

https://github.com/HicServices/RDMP/blob/develop/Tools/rdmp/NLog.config#L13

All <target> tags should appear in the <targets> block (not outside it)

Save on Close

Describe the bug
When editing an ExtractionInformation and other times when you make changes to an object in a tab then close the tab it does not offer to save

To Reproduce
Steps to reproduce the behavior:

1. Open an ExtractionInformation
2. Change the SQL
3. Close

Expected behavior
A dialog box to appear offering to save changes before closing

RDMP Version
v4.1.5

Cloning a pipeline with broken components

Describe the bug
Cloning seems to silently fail for broken pipelines

To Reproduce
Steps to reproduce the behavior:

1. Create a new pipeline with at least 1 middle component
2. Edit the database so the middle component has an illegal Type name
3. Clone the pipeline

Expected behavior
A clone of the original including all broken components and their arguments

RDMP Version
v4.1.5

Allow identifiable extractions

Is your feature request related to a problem? Please describe.
When data is already de-identified or de-identification happens later down the road e.g. as a file operation the ReleaseId substitution shouldn't be enforced

Describe the solution you'd like
Add a setting that allows identifiablee extractions to take place (where Private and Release ID are the same column)

rdmp-cli: blank lines printed to console when running dle file load

Describe the bug

When running e.g. ./rdmp dle --command run -l 12, many blank lines are printed to the console. Potentially could be one line for each row loaded? Or per log statement, but for some reason nothing is shown.

To Reproduce
Steps to reproduce the behavior:

1. From the NSH system, run the above command when there is a file available for loading
2. Observe output

Expected behavior

Not this.

Screenshots

N/A

RDMP Version

rdmp-cli v4.1.8

Error with Stack Trace

N/A

Database Engine

MySQL

Additional context

N/A

initial column view on low res monitors

If a tree view collection e,g, Data Export is shown on a low res monitor then the Name field can be invisible due to auto sizing which looks very confusing

Invalid dates in LoadProgress

Describe the bug
When using a LoadProgress with a DLE having No days to load and/or the OriginDate after the ProgressDate should be illegal

To Reproduce
Steps to reproduce the behavior:

1. Create a LoadMetadata
2. Add a LoadProgress
3. Set the Progress date to the future, or to yesterday
4. Set the OriginDate today
5. Add a RemoteTableAttacher
6. Runt he data load

Expected behavior

• The LoadProgressUI should tell you there is a problem with the dates it has (for future dates)
• The DLE should fail checks for the RemoteTableAttacher and/or generally looking around the load pre module evaluations

Screenshots

RDMP Version
v4.1.5

Error with Stack Trace

If applicable, paste the entire stack trace here, leave the triple quotes (```)

Database Engine
NA

Additional context
Issue raised by @shorban-uod

Add Pipeline used to extraction logs

Is your feature request related to a problem? Please describe.
Add Pipeline used to the logs of extractions

Describe the solution you'd like
A log of an info message at the start of the extraction containing the Name and ID of the pipeline used would be helpful

Additional context

Running "Checks" on Extraction creates a DataLoadRun with empty progress

Get rid of Microsoft.Office.Interop.Excel dependency

We should look into npoi as an alternative to interop since interop only works where Excel is installed which is bad for servers etc.

Could use NPOI (handles xls and xlsx)
https://github.com/tonyqus/npoi/tree/master/

Could use EPPlus (handles xlsx only - I think)
https://github.com/JanKallman/EPPlus

Increase timeout of CountDISTINCTCohortInDatabase

This method and others like it need sensible long timeouts. They are used as part of lengthy extrractions and when there are many cohorts or the cohort is based on a view then it could be quite slow to execute.

Expose Favourite/Unfavourite on context menu

Is your feature request related to a problem? Please describe.
Sometimes the Favourite column is not visible. This makes it impossible to favourite some things e.g. AggregateConfigurations that are part of a CohortIdentificationConfiguration

Describe the solution you'd like
A clear and concise description of what you want to happen.

When combined with #143 this would allow the user to identify the most used aggregates and easily deploy them into new cics

Include 'RunAfterStaging' option in RDMP

Is your feature request related to a problem? Please describe.
When a data load fails after loading RAW (a long time e.g. 100 million records)

Describe the solution you'd like
The ability to resume the data load from the 'RAW is already populated' stage

Interactive CLI mode crashes when insufficient arguments are provided for AssociateCatalogueWithLoadMetadata

Describe the bug

Enter Command (or 'exit')
Command:
AssociateCatalogueWithLoadMetadata 
2020-10-22 07:57:54.3253 ERROR Expected parameter at index 0 to be a Rdmp.Core.Curation.Data.DataLoad.LoadMetadata (for parameter 'loadMetadata') but it was Missing . 
2020-10-22 07:57:54.3253 INFO Fatal error occurred so returning -1 . System.Exception: Expected parameter at index 0 to be a Rdmp.Core.Curation.Data.DataLoad.LoadMetadata (for parameter 'loadMetadata') but it was Missing
   at Rdmp.Core.CommandExecution.CommandInvoker.ExecuteCommand(ConstructorInfo constructorInfo, CommandLineObjectPicker picker) in /home/thomas/RDMP/Rdmp.Core/CommandExecution/CommandInvoker.cs:line 196
   at Rdmp.Core.CommandExecution.CommandInvoker.ExecuteCommand(Type type, CommandLineObjectPicker picker) in /home/thomas/RDMP/Rdmp.Core/CommandExecution/CommandInvoker.cs:line 148
   at Rdmp.Core.CommandLine.Runners.ExecuteCommandRunner.RunCommand(String command) in /home/thomas/RDMP/Rdmp.Core/CommandLine/Runners/ExecuteCommandRunner.cs:line 74
   at Rdmp.Core.CommandLine.Runners.ExecuteCommandRunner.RunCommandExecutionLoop(IRDMPPlatformRepositoryServiceLocator repositoryLocator) in /home/thomas/RDMP/Rdmp.Core/CommandLine/Runners/ExecuteCommandRunner.cs:line 110
   at Rdmp.Core.CommandLine.Runners.ExecuteCommandRunner.Run(IRDMPPlatformRepositoryServiceLocator repositoryLocator, IDataLoadEventListener listener, ICheckNotifier checkNotifier, GracefulCancellationToken token) in /home/thomas/RDMP/Rdmp.Core/CommandLine/Runners/ExecuteCommandRunner.cs:line 64
   at Rdmp.Core.Program.Run(RDMPCommandLineOptions opts) in /home/thomas/RDMP/Tools/rdmp/Program.cs:line 192
   at Rdmp.Core.Program.RunCmd(ExecuteCommandOptions opts) in /home/thomas/RDMP/Tools/rdmp/Program.cs:line 166
   at Rdmp.Core.Program.<>c.<HandleArguments>b__3_8(ExecuteCommandOptions opts) in /home/thomas/RDMP/Tools/rdmp/Program.cs:line 101
   at Rdmp.Core.Program.HandleArguments(String[] args) in /home/thomas/RDMP/Tools/rdmp/Program.cs:line 78

To Reproduce
Steps to reproduce the behavior:

1. Launch rdmp CLI ./rdmp cmd
2. Type AssociateCatalogueWithLoadMetadata

Expected behavior
Command should prompt for inputs (e.g. enter a Catalogue, enter a LoadMetadata). This would be consistent with commands like NewObject

At the very least it shouldn't crash out of the command loop!

RDMP Version
4.2.0

Database Engine
Sql Server

Additional context
Running under linux CLI Konsole

Allow DLE to run without requiring the backup trigger

Is your feature request related to a problem? Please describe.
The archive trigger should not be a mandatory requirement for the DLE to run

Describe the solution you'd like
Add an option in the LoadMetadata or globally to specify that backup triggers are not required

Additional context
This is useful in situations where alternative backup strategies are in place e.g. temporal tables.

• We need to make sure that some features like the _Archive table diff viewer considers this new situation too.
• We need to make sure the MERGE/INSERT parts of the DLE do not require the hic_validFrom fields or hic_dataLoadRunID field (although this might still be useful).

View Extract options should be available at filter container level aswell as filter level

Is your feature request related to a problem? Please describe.
The commands for ViewExtract, ViewAggregate etc should work at both the Filter and FilterContainer level (currently only work for individual filters).

Describe the solution you'd like

Choose Load Directory should show the current value

Is your feature request related to a problem? Please describe.
When you right click the load directory of a Load that has a value in it's LocationOfFlatFiles this should appear in there resulting dialog

CoumnSwapper with mixed int/string datatypes

Reported by Magalie

Describe the bug
When lookup table has string values in it and you try to swap values read from a flat file.

To Reproduce
Steps to reproduce the behavior:

1. Create a lookup table with varchar(10) column type for the MappingFromColumn
2. Create a flat file with some numbers in it
3. Run ColumnSwapper component with Delimited source

Swapper complains about not finding a mapping for the values read from the file

Expected behavior
Regardless of input datatype the mapping should work

Screenshots
If applicable, add screenshots to help explain your problem.

RDMP Version
4.1.9

Database Engine
Sql Server

There is no way to order STAGING=>LIVE table migration in DLE

This creates problems if you have foreign key constraints and the Parent table is chosen to be MERGE first.

This can be reproduced by modifying the test DLELoadTwoTables and flipping the order of the lines:

TableInfo childTableInfo = Import(childTbl, lmd, logManager);
TableInfo parentTableInfo = Import(parentTbl,lmd,logManager);

Right click menu shows big red X after prolonged use on VDI

Reported By: Magalie

Describe the bug
When running in virtual desktop environment after a prolonged period the context menu starts showing up as a red x (see screenshot). The app still runs but no menu works. The only way to solve the problem is to restart RDMP.

Screenshots

RDMP Version
4.2.0

RemoteTableAttacher doesn't respond to Abort request in DataLoad

Describe the bug
When running a data load which contains a RemoteTableAttacher, the SQL provided was very inefficient and possibly invalid. This meant that the database hanged when trying to run the query, and when trying to abort the data load RDMP was not able to terminate the query.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
I expected RDMP to terminate the query on the database and abort the data load successfully.

Screenshots
N/A

RDMP Version
Displayed in the task bar of the software e.g. v4.1.9

Error with Stack Trace

N/A

Database Engine
Sql Server

Additional context
Can provide the code which caused the database to hang for reproducibility if necessary

Custom Metadata report $foreach Catalogue

Is your feature request related to a problem? Please describe.
Custom templates should include things like:

<table border=1>
<tr><th>ID</th><th>Clinical Datasets </th><th>Description</th><th>Time Period</th></tr>
$foreach Catalogue
<tr><td>$ID</td><td>$Name</td><td></td></tr>
$end
</table>

Currently when running in OneFile mode the output file contains the entire template repeated over and over with no option for a head/tail to the document that doesn't get repeated.

Describe the solution you'd like

There is support for $foreach CatalogueItem already. So a feature using the same syntax and only valid when running in one file mode and for Catalogues.

Describe alternatives you've considered
Being able to reference head and tail template files would also work but be more restrictive and cumbersome

RDMP keeps trying to close windows

Describe the bug
New Catalogue etc seems to trigger CloseAllWindows.

Lets get rid of all calls to closing windows without good reason. Especially now that we have sessions.

PopulateRAW VerifyExistenceOfRawData includes tables not in the load

This method can pass where loads share the same RAW database and can actually crash in corner cases where the table enumeration includes one that is droped before the count statement can execute (in a parallel processing environment).

RDMP/Rdmp.Core/DataLoad/Engine/LoadExecution/Components/PopulateRAW.cs

Line 95 in f85f1c7

if (rawDbInfo.DiscoverTables(false).All(t=>t.IsEmpty()))

Build warning caused by NuDoc 0.5.0

👋
The HIC.RDMP.Plugin project (and projects which depend on it) has a dependency on NuDoc/NuDoq 0.5.0 which produces the following warning on builds: "Warning NU1602 NuDoc 0.5.0 does not provide an inclusive lower bound for dependency NuDoq. An approximate best match of NuDoq 1.2.5 was resolved". It also appears the package hasn't been updated since 2015, and does not fully support .NET Core, so maybe it's use could be reviewed? Thanks.

Add IsView to ITableInfo

Please add this property to interface

rdmp cmd list Catalogue fails if there are no Catalogues

1. Create new RDMP platform database servers
2. run rdmp cmd list Catalogue

2020-02-04 10:19:48.0941 INFO Setting yaml config value for CatalogueConnectionString . 

2020-02-04 10:19:48.1260 INFO Setting yaml config value for DataExportConnectionString . 

Bad parameter.  Expected Type 'MapsDirectlyToDatabaseTable.IMapsDirectlyToDatabaseTable[]' for index 0.  Error was:Object reference not set to an instance of an object.

2020-02-04 10:19:49.6264 ERROR Expected parameter at index 0 to be a MapsDirectlyToDatabaseTable.IMapsDirectlyToDatabaseTable[] (for parameter 'toList') but it was Catalogue . 

2020-02-04 10:19:49.6264 INFO Fatal error occurred so returning -1 . System.Exception: Expected parameter at index 0 to be a MapsDirectlyToDatabaseTable.IMapsDirectlyToDatabaseTable[] (for parameter 'toList') but it was Catalogue

   at Rdmp.Core.CommandExecution.CommandInvoker.ExecuteCommand(ConstructorInfo constructorInfo, CommandLineObjectPicker picker) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Rdmp.Core\CommandExecution\CommandInvoker.cs:line 199

   at Rdmp.Core.CommandLine.Runners.ExecuteCommandRunner.Run(IRDMPPlatformRepositoryServiceLocator repositoryLocator, IDataLoadEventListener listener, ICheckNotifier checkNotifier, GracefulCancellationToken token) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Rdmp.Core\CommandLine\Runners\ExecuteCommandRunner.cs:line 58

   at Rdmp.Core.Program.Run(RDMPCommandLineOptions opts) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Tools\rdmp\Program.cs:line 155

   at Rdmp.Core.Program.HandleArguments(String[] args) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Tools\rdmp\Program.cs:line 73

The command "./rdmp-cli/rdmp cmd list Catalogue" exited with 255.
cache.2

Importing Catalogue descriptions overwrites Acronym/CatalogueFolder

Commands for importing Catalogue (and CatalogueItem) descriptions should not overwrite non description fields.

Commands are in namespace:
CatalogueManager.CommandExecution.AtomicCommands.Sharing

When Catalogues have null LiveLoggingServer_ID

Describe the bug
Create a DLE load which targets a Catalogue. Update your database to have null in the field LiveLoggingServer_ID. Run load checks. This causes a null reference check error but should instead offer to populate it with the default if one is configured

Log CacheProgress when doing cache based Data Load Engine executions

Is your feature request related to a problem? Please describe.
Log the current progress and window date/times for a load to the logs

CLI command Set should give suggestions

Is your feature request related to a problem? Please describe.
When using the Set command and unsure about the exact property name there should be suggestions like there are with command names

./rdmp cmd Set LoadMetadata:1 Location /home/thomas/temp/
Environment:4.0.30319.42000
2020-10-22 08:07:26.6701 INFO Setting yaml config value for CatalogueConnectionString . 
2020-10-22 08:07:26.7301 INFO Setting yaml config value for DataExportConnectionString . 
Command Impossible:Unknown Property 'Location'
2020-10-22 08:07:29.1342 INFO Exiting with code 0 . 

Describe the solution you'd like
Suggestions of properties should be presented like when you get a command name wrong:

thomas@thomas-CF-31WBLAHLM:~/RDMP/Tools/rdmp/bin/Debug/netcoreapp2.2/linux-x64$ ./rdmp cmd Catalogue
Environment:4.0.30319.42000
2020-10-22 08:08:52.9159 INFO Setting yaml config value for CatalogueConnectionString . 
2020-10-22 08:08:52.9770 INFO Setting yaml config value for DataExportConnectionString . 
2020-10-22 08:08:55.1286 ERROR Unknown or Unsupported Command 'Catalogue', use ListSupportedCommands to see available commandsSimilar commands include:
AddCatalogueToCohortIdentificationAsPatientIndexTable
AddCatalogueToGovernancePeriod
AssociateCatalogueWithLoadMetadata
ImportCatalogueItemDescription
ImportCatalogueItemDescriptions
LinkCatalogueItemToColumnInfo
MakeCatalogueItemExtractable
MakeCatalogueProjectSpecific
MakeProjectSpecificCatalogueNormalAgain

Ability to set ProcessTaskArguments from CLI more easily

Is your feature request related to a problem? Please describe.
Yes, if you have 2+ process tasks with the same argument it is difficult to set the value to something on only one of them (from command line).

Further more it is difficult to set values that relate to objects e.g. "1,2,3" where those are object IDs

Describe the solution you'd like
Add a new command ExcuteCommandSetArgument which takes a process task and describes the name of the argument you want set. It should call SetValue to support hard typed objects e.g. "Catalogue:mycata" instead of having to know the ID

Describe alternatives you've considered
If you know the IDs you can call ExecuteCommandSet but it has limitations

Additional context
See Integration repo

BinderWithErrorProviderFactory does not respect ProblemProvider._ignoreBadNamesFor

Describe the bug
Create a filter with a name that includes illegal characters e.g. '>='. Although it does not appear red when you edit it in ExtractionFilterUI you get an error icon saying the name is bad

Screenshots

RDMP Version
v4.1.9

Additional context
We need to update the following line to respect the ProblemProvider._ignoreBadNamesFor field (which would need to become public and static).

RDMP/Rdmp.UI/Rules/BinderWithErrorProviderFactory.cs

Line 44 in f85f1c7

if(dataMember.Equals("Name") && databaseObject is INamed && !(databaseObject is ExternalDatabaseServer))

Ability to override date/time patterns in AnySeperatorFileAttacher

Is your feature request related to a problem? Please describe.
When flat files are in formats like YYYYMMDD then you have to do RAW column type high jinks

Describe the solution you'd like
Add a property ExplicitDateTimeFormat to AnySeperatorFileAttacher or it's base class

TableVarcharMaxer when table names have spaces

Issue reported by Zsolt

In the meantime there seems to be another issue in DLE.
TableVarcharMaxer throws error when trying to alter columns with space in their names.
To reproduce:

Create a table with column names containing space
Import the table as catalogue
Create a load with a TableVarcharMaxer
Run the load

View INSERTS/UPDATES with MySql DBMS

Describe the bug
View INSERTS/UPDATES not working with MySql

Success 16/11/2020 12:52:13 Table user is attempting to view updates/inserts for is called someDb.SomeTable
Success 16/11/2020 12:52:13 The runtime name of the table is SomeTable
Success 16/11/2020 12:52:13 The following TableInfos were given to us to pick from someDb.SomeTable
Fail 16/11/2020 12:52:13 Could not find an appropriate TableInfo from those mentioned above

Database Engine
MySql

Additional context
v4.1.8

AnySeparatorFileAttacher with database `int` datatype and big int values for insertion

Attach failed on job Rdmp.Core.DataLoad.Engine.Job.DataLoadJob Attacher was of type AnySeparatorFileAttacher see InnerException for specifics:

 Error processing file ePrescribe_historical_extract_2018_2019.csv
Error reading file ''.  Problem loading value nnnnnnnnnnnnnnn into data table (on Line number 5) the header we were trying to populate was ePR DMD Current Unique Reference and was of datatype System.Int64
Could not parse string value 'nnnnnnnnnnnnnnn' with Decider Type:IntTypeDecider
'nnnnnnnnnnnnnnn' (System.String) is not convertible to 'System.Int32'.

Find/Update similar filters

Is your feature request related to a problem? Please describe.
When you save changes to a CatalogueItem description it brings up a propagate dialog. It would be nice if a similar thing happened when you made changes to ExtractionFilter objects too (where the filter SQL previously exactly matched other catalogue level filters).

Describe the solution you'd like
This filter exists for each modality, if you change one it should ask if you want to update all of them

Additional context
Should not propogate when looking at filters in CohortIdentificationConfigurations or ExtractionConfigurations. Could ask Yes/No to propogate from a Catalogue level filter (ExtractionFilter) down to unfrozen cics.

The CatalogueItem one looks like:

Sessions

Is your feature request related to a problem? Please describe.
The Favourites toolbox shows all objects the user has favourited. But often a user wants to do several actions at once with a cluster of related objects. The ability to create a new toolbox with one or more root objects as a "Session" toolbox would help here.

The favourites toolbox lets you have arbitrary objects but there is only ever one Favourites toolbox.

Describe the solution you'd like

1. Click New Session
2. Select one or more objects
3. Persist the window through restarts of RDMP

Easiest way to do this would be to define a special prefix for sessions and just reuse the Favourites table, they would then be shared with other users and persist and generally integrate well with the rest of the codebase.

As an additional bonus Session/Favourites should be available to rapidly find in search windows (there are 2 version so fthe search window btw).

Dotnet Core 3.1 Support

Currently the RDMP CLI supports net core 2.2 while the GUI Client targets Dotnet Framework 4.6.1. Dotnet core 3.0 introduced support for windows forms which is a route for upgrading the GUI client. I would like to go direct to 3.1 but there are a number of barriers:

Microsoft have deleted ContextMenu/MenuItem:
dotnet/winforms#2614

Current UI dependencies include:

• Scintilla.Net
• AutoCompleteMenu-ScintillaNET
• DockPanelSuite (See: dockpanelsuite/dockpanelsuite#616)
• ObjectListView
• Squirrel.Windows

Current framework references include:

• aa6b957#diff-3254677a7917c6c01f55212f86c57fbfR10

This is being tackled on feature branch: feature/dotnetcore3.1

Drag and Drop AggregateConfiguration into a CohortIdentificationConfigurationContainer should be supported

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
In the above image the system should offer Import Import (Copy of) Cohort Set into container

Easier access to DataExtraction logs

Is your feature request related to a problem? Please describe.
Accessing the logs of data extractions is not as easy as for loads

Describe the solution you'd like
Add an option to the ExtractionConfiguration objects that takes you to a screen like with loads

Describe alternatives you've considered
The current way to access DataExtract logs across all projects is via the Log Viewer (View...=>Log Viewer) and searching for DataExtraction.

Additional context

Logs of loads are easy to access

Bulk copy descriptions between Catalogues

Is your feature request related to a problem? Please describe.
Importing descriptions from similar/exact named columns in other Catalogues would be nice.

Describe the solution you'd like

Describe alternatives you've considered
You can export the descriptions to file and then import again into the new catalogue but that is a slow workaround.

Additional context
Most useful when building many Catalogues that are mostly the same (e.g. extraction views, same data in different zones etc).

When you type in corrupted connection strings in GUI client

Describe the bug
If you enter an invalid connection string (i.e. one that is semantically invalid not just inaccessible/non existant) you can get stuck in a loop while trying to get to startup screen

Screenshots
If applicable, add screenshots to help explain your problem.

RDMP Version
4.1.9

Add GoTo filter=>Master filter

Is your feature request related to a problem? Please describe.
When an extraction or cohort building filter is cloned from a Catalogue it would be nice to have a GoTo that takes you to the master filter

Describe the solution you'd like
Add GoTo here

.Net Core 3.1

.Net Core 2.2 is still used for the CLI components, but is now deprecated by Microsoft and no longer available for installation on some systems.

Long term, migrating the GUI components beyond .Net Framework as per #41 is ideal, but in the shorter term the CLI components can be updated without encountering the same blocking issues that apply to the GUI.

Wrong platform database listed on startup, plus increased startup time

Hi, I have a set of platform databases setup on a local vm scr1 which works fine. On startup though, the client displays that it is connecting to localhost scr2, and then waits for some timeout before properly connecting to the remote platform database scr3. Closing the dialogue in scr2 also results in the remote platform databases being loaded.

RDMP v4.1.6.

The given assembly name or codebase was invalid

Describe the bug
When setting a ProcessTaskArgument from CLI to an array of objects.

To Reproduce
Steps to reproduce the behavior:

1. Run on command line: ./rdmp cmd set ProcessTaskArgument:TablesToIsolate Value 1,2,3,4,5,6

Expected behavior
Value should be set to TableInfo objects 1-6

Screenshots
https://travis-ci.com/github/SMI/Integration/builds/178383562#L1737

RDMP Version
v4.1.5

Error with Stack Trace

Problem line was 'set ProcessTaskArgument:TablesToIsolate Value 1,2,3,4,5,6':The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047) . 

2020-08-04 09:52:25.1594 INFO Fatal error occurred so returning -1 . System.Exception: Error executing script.  Problem line was 'set ProcessTaskArgument:TablesToIsolate Value 1,2,3,4,5,6':The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047) ---> System.IO.FileLoadException: The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)

   at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMarkHandle stackMark, IntPtr pPrivHostBinder, Boolean loadTypeFromPartialName, ObjectHandleOnStack type, ObjectHandleOnStack keepalive)

   at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean loadTypeFromPartialName)

   at System.RuntimeType.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark)

   at System.Type.GetType(String typeName)

   at Rdmp.Core.Repositories.MEF.GetType(String type)

   at Rdmp.Core.CommandLine.Interactive.Picking.PickObjectByName.IsLegitType(String possibleTypeName, Type& t)

   at Rdmp.Core.CommandLine.Interactive.Picking.PickObjectByName.IsMatch(String arg, Int32 idx)

   at System.Linq.Enumerable.WhereEnumerableIterator`1.ToArray()

   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)

   at Rdmp.Core.CommandLine.Interactive.Picking.CommandLineObjectPicker.ParseValue(String arg, Int32 idx)

   at System.Linq.Enumerable.SelectIterator[TSource,TResult](IEnumerable`1 source, Func`3 selector)+MoveNext()

   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)

   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

   at Rdmp.Core.CommandLine.Interactive.Picking.CommandLineObjectPicker..ctor(IEnumerable`1 args, IRDMPPlatformRepositoryServiceLocator repositoryLocator)

   at Rdmp.Core.CommandLine.Runners.ExecuteCommandRunner.GetCommandAndPickerFromLine(String command, CommandLineObjectPicker& picker, IRDMPPlatformRepositoryServiceLocator repositoryLocator)

   at Rdmp.Core.CommandLine.Runners.ExecuteCommandRunner.RunScript(RdmpScript script, IRDMPPlatformRepositoryServiceLocator repositoryLocator)

   --- End of inner exception stack trace ---

   at Rdmp.Core.CommandLine.Runners.ExecuteCommandRunner.RunScript(RdmpScript script, IRDMPPlatformRepositoryServiceLocator repositoryLocator)

   at Rdmp.Core.CommandLine.Runners.ExecuteCommandRunner.Run(IRDMPPlatformRepositoryServiceLocator repositoryLocator, IDataLoadEventListener listener, ICheckNotifier checkNotifier, GracefulCancellationToken token)

   at Rdmp.Core.Program.Run(RDMPCommandLineOptions opts)

   at Rdmp.Core.Program.HandleArguments(String[] args)

DecimalTypeDecider does not specify which source column caused error

Describe the bug
When using a RemoteTableAttacher, the DecimalTypeDecider failed due to the value '.' in a certain column, however RDMP does not specify which column the value '.' originates from.

To Reproduce
Steps to reproduce the behavior:

1. Set up a data load to a target table containing a single column of the float data type
2. Create a DatabaseAttacher which uses the code "select '1';"
3. Run the data load

Expected behavior
Either:

1. DecimalTypeDecider chooses that '.' is an invalid decimal and sets the column to NULL
2. (Preferred) Add the source column to the error message so that the user can investigate and decide how to fix the problem in the RemoteTableAttacher code

Screenshots

RDMP Version
v4.2.0

Error with Stack Trace
Stack trace for screenshot 1:

Could not parse string value '.' with Decider Type:DecimalTypeDecider
   at TypeGuesser.Deciders.DecideTypesForStrings`1.Parse(String value)
   at FAnsi.Discovery.BulkCopy.ConvertStringTypesToHardTypes(DataTable dt)
   at FAnsi.Discovery.BulkCopy.Upload(DataTable dt)
   at Rdmp.Core.DataLoad.Engine.Pipeline.Destinations.SqlBulkInsertDestination.SubmitChunk(DataTable chunk, IDataLoadEventListener job) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Rdmp.Core\DataLoad\Engine\Pipeline\Destinations\SqlBulkInsertDestination.cs:line 67
   at Rdmp.Core.DataLoad.Engine.Pipeline.Destinations.SqlBulkInsertDestination.ProcessPipelineData(DataTable toProcess, IDataLoadEventListener listener, GracefulCancellationToken cancellationToken) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Rdmp.Core\DataLoad\Engine\Pipeline\Destinations\SqlBulkInsertDestination.cs:line 173
   at Rdmp.Core.DataFlowPipeline.DataFlowPipelineEngine`1.ExecuteSinglePass(GracefulCancellationToken cancellationToken) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Rdmp.Core\DataFlowPipeline\DataFlowPipelineEngine.cs:line 220
   at Rdmp.Core.DataFlowPipeline.DataFlowPipelineEngine`1.ExecutePipeline(GracefulCancellationToken cancellationToken) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Rdmp.Core\DataFlowPipeline\DataFlowPipelineEngine.cs:line 110
'.' (System.String) is not convertible to 'System.Decimal'.
   at TB.ComponentModel.TypeConverter.Convert(Object value, Type destinationType, CultureInfo culture)
   at TB.ComponentModel.TypeConverter.ConvertTo[T](Object value, CultureInfo culture)
   at TypeGuesser.Deciders.DecideTypesForStrings`1.ParseImpl(String value)
   at TypeGuesser.Deciders.DecideTypesForStrings`1.Parse(String value)

Stack trace for screenshot 2:

Data Flow Pipeline Crashed
   at Rdmp.Core.DataFlowPipeline.DataFlowPipelineEngine`1.ExecutePipeline(GracefulCancellationToken cancellationToken) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Rdmp.Core\DataFlowPipeline\DataFlowPipelineEngine.cs:line 182
   at Rdmp.Core.DataLoad.Modules.Attachers.RemoteTableAttacher.Attach(IDataLoadJob job, GracefulCancellationToken cancellationToken) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Rdmp.Core\DataLoad\Modules\Attachers\RemoteTableAttacher.cs:line 385
   at Rdmp.Core.DataLoad.Engine.LoadExecution.Components.Runtime.AttacherRuntimeTask.Run(IDataLoadJob job, GracefulCancellationToken cancellationToken) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Rdmp.Core\DataLoad\Engine\LoadExecution\Components\Runtime\AttacherRuntimeTask.cs:line 49
Could not parse string value '.' with Decider Type:DecimalTypeDecider
   at TypeGuesser.Deciders.DecideTypesForStrings`1.Parse(String value)
   at FAnsi.Discovery.BulkCopy.ConvertStringTypesToHardTypes(DataTable dt)
   at FAnsi.Discovery.BulkCopy.Upload(DataTable dt)
   at Rdmp.Core.DataLoad.Engine.Pipeline.Destinations.SqlBulkInsertDestination.SubmitChunk(DataTable chunk, IDataLoadEventListener job) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Rdmp.Core\DataLoad\Engine\Pipeline\Destinations\SqlBulkInsertDestination.cs:line 67
   at Rdmp.Core.DataLoad.Engine.Pipeline.Destinations.SqlBulkInsertDestination.ProcessPipelineData(DataTable toProcess, IDataLoadEventListener listener, GracefulCancellationToken cancellationToken) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Rdmp.Core\DataLoad\Engine\Pipeline\Destinations\SqlBulkInsertDestination.cs:line 173
   at Rdmp.Core.DataFlowPipeline.DataFlowPipelineEngine`1.ExecuteSinglePass(GracefulCancellationToken cancellationToken) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Rdmp.Core\DataFlowPipeline\DataFlowPipelineEngine.cs:line 220
   at Rdmp.Core.DataFlowPipeline.DataFlowPipelineEngine`1.ExecutePipeline(GracefulCancellationToken cancellationToken) in D:\Jenkins\workspace\RDMP\HIC.RDMP.GitHub.Release\Rdmp.Core\DataFlowPipeline\DataFlowPipelineEngine.cs:line 110
'.' (System.String) is not convertible to 'System.Decimal'.
   at TB.ComponentModel.TypeConverter.Convert(Object value, Type destinationType, CultureInfo culture)
   at TB.ComponentModel.TypeConverter.ConvertTo[T](Object value, CultureInfo culture)
   at TypeGuesser.Deciders.DecideTypesForStrings`1.ParseImpl(String value)
   at TypeGuesser.Deciders.DecideTypesForStrings`1.Parse(String value)

Database Engine
Sql Server

Additional context
Add any other context about the problem here.

Filter Parameter prompt

Is your feature request related to a problem? Please describe.
When adding a filter to a CohortIdentificationConfiguration or SelectedDatasets the filter edit tab opens up. If a filter has parameters it should instead prompt you to enter values for those parameters and not open.

Describe the solution you'd like
Use GetText to prompt for a value with the default text of the default value for the parameter

Additional context
Where there are known good values to pick from this should not occur