erikbra / grate Goto Github PK

Record all scripts as having been run, but don't actually run them.

Create installers in the installers/msi folder.

WIX is possibly the best alternative:

• https://wixtoolset.org/
• https://www.firegiant.com/wix/tutorial/

PROs: Free, command-line
CONs: .NET 4.x dependent (maybe harder to run in GitHub actions?

Ref chucknorris/roundhouse#424

• What does grate do if an Anytime/Everytime script is changed, and re-run?
• What does grate do if we change a one-time script, and we run with --warnononetimescriptchanges
• On the next run, make sure we use the latest run of the script to compare the hash, not a random one.

When setting up the grate repo for local development, I noted that (specifically with MS SQL Server), many of the test runs resulted in failed connection attempts:

However, the last few tests connected successfully, and the final test report doesn't report any test failures. I'd like to assume that the failed connect attempts were deliberate (I haven't verified by checking the unit test code yet), but I'd also assume there would only be a few of them. There were many during the initial run of "dotnet test" after cloning the repo.

Opening this ticket for tracking/investigation purposes.

Relevant Tooling:

• Windows 10
• Docker Desktop with WSL2

There seems to be a bug, when creating the migrations folder in the container. I get the following:

Unhandled exception: System.IO.IOException: The file '/app/grate' already exists.
   at System.IO.FileSystem.CreateDirectory(String )
   at System.IO.Directory.CreateDirectory(String )
   at grate.Migration.GrateMigrator.CreateChangeDropFolder(String folder)
   at grate.Migration.GrateMigrator.Migrate()
   at grate.Commands.MigrateCommand.<>c__DisplayClass0_0.<<-ctor>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Invocation.CommandHandler.GetExitCodeAsync(Object value, InvocationContext context)
   (...)

Repro:

• Copy your database scripts to the ./my_scripts folder.
• Create an environment variable, CONNECTIONSTRING, on the host,with your connection string.
• Create a simple Dockerfile like this:

FROM erikbra/grate:latest
COPY ./my_scripts ./db
ENTRYPOINT ./grate -f=db --connectionstring "$CONNECTIONSTRING"

• build and run it like this:

 docker build -t my-db-image .
 docker run --rm -e CONNECTIONSTRING my-db-image

Observe the error.

I got it in the winget repo!

First "external" package manager after NuGet.

❯ winget search grate
Name  Id            Version Match                  Source
---------------------------------------------------------
grate erikbra.grate 0.9.4                          winget
Volta Volta.Volta   1.0.5   Command: volta-migrate winget
❯ winget install erikbra.grate
Found grate [erikbra.grate] Version 0.9.4
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
Downloading https://github.com/erikbra/grate/releases/download/0.9.4/grate-0.9.4.msi
  ██████████████████████████████  11.6 MB / 11.6 MB
Successfully verified installer hash
Starting package install...
Successfully installed
❯ get-command grate

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     grate.exe                                          0.0.0.0    C:\Users\erikb\AppData\Local\erikbra\grate\grate.exe
❯ 

(microsoft/winget-pkgs#29982)

//cc @wokket

After using Winget or downloading the latest Release, how could a first time user with a dacpac(?) use Grate to connect to a client SQL Database, publish schema changes, and execute custom SQL data updates?

I have thought a bit about lowering the level of logs on the "skipped" lines to "Debug", so that only what is actually applied is output by default, how do we feel about that? We would get the full list if setting the verbosity to Debug on the command line.
I think it would make it easier to see what wat actually run in the output.

Thoughts?

How could one use grate in a Kubernetes setting?

• Init container?
• etc

Maybe @wokket s examples are enough. Verify 😄

Warn on one time script changes, bug ignore them, do not run the changed script.

Do we want/need this?

Example output:

grate v1.0.0.0 has grated your database (hpWAJepVEbKOUSD)! You are now at version a.b.c.d. All changes and backups can be found at 
"/home/runner/work/grate/grate/grate.unittests/bin/Release/net6.0/output/migrations/hpWAJepVEbKOUSD/localhost,49153/2021-09-22T20_07_36".

If the server name of the connection string contains a comma, replace that from the file name. I'm not sure every OS likes a comma in a file.

Since grate will suppress console output for skipped files by default, it would make sense to skip output of the extra dividers and whitespace to trim unnecessary content as well. This will help to eliminate noise when verifying that scripts are detected and run appropriately.

Perhaps showing something like:

Looking for scripts in xxxx
--no new scripts--

and/or checking for a directory before output to enable simplifying the message to

Skipping runAfterFirstUp, xxxx folder does not exist

Original comment posted by @docrinehart in #140 (comment)

I'm seeking clarification on the use of the --version option mentioned in #8

If I use the example script powershell:

grate `
--files=.\db `
--env=Local `
--connstring="Server=(localdb)\MSSQLLocalDB;Integrated Security=true;Database=grate_test" `
--version=1.0 

It all works fine.

If I run it a 2nd time with the env=Test, it works as described and only the 'Test' script from the 'up' folder is executed, but...

Change the --version=1.1 and it runs all the 'one time' scripts again.

How should the semantic version control work?

Do script files have a version number added into the file name like the ```ENV```` keyword?

- [ ] Set up domain (github pages?)

• Make the 'docs' folder publish there (or something similar)

aka silent. Make it run, but shut up about it.

Describe the bug
While running "dotnet grate" with only a connection string, the migrations run successfully but the skipped scripts are not listed in the tool output.

To Reproduce

• Create new up script
• Execute grate with connection string
• After initial execution succeeds, verify database is created/updated
• Re-execute grate
• Observe that the up script is not listed in the tool output

Expected behavior
Tool output should include skipped migration scripts in the console output to confirm script presence and evaluation.

Screenshots

Desktop (please complete the following information):

• Windows 10, Powershell 7.2,0
• grate version v0.11.0.0 (via dotnet install to project)

Additional context
I was able to verify that the initial run shows the script names

I started migrating an existing project over, and found there's currently no support for --drop.

We use this as part of our integration tests against localdb to drop and recreate the database from scratch for each run.

Is this a feature you're intentionally keeping out of scope for grate? If it just hasn't been got to yet I'd like to add it to the (long) list of features that need doing :)

To make the package presentation on nuget.org better, make user a README file is included in the NuGet package, as described here: https://docs.microsoft.com/en-us/nuget/nuget-org/package-readme-on-nuget-org

As the .msi is unsigned Windows prompts with the blue screen re: unknown publisher.

This is both a usability thing and may well prevent the installer being run in some environments which require signed installers on servers.

Pre-release should be fine to begin with.

This is both a blocker for #41 and also a pre-req for me to start running grate against real-world projects for testing.

Option to avoid storing the whole SQL text in the ScriptsRun table. Especially useful if running in small/embedded databases, where the grate tables can become a substantial part of the database size.

Ref.

• chucknorris/roundhouse#246
• chucknorris/roundhouse#374

Previously RH used --version for this, but the .Net CommandLine libraries claim this for the application version.

I'm suggesting we add this using --dbversion instead to avoid conflicts if that's ok?

• Add flag
• Update Tests
• Update (create?) Migration Docco

We need to set up a build pipeline. Try GitHub actions, as it's "close to the code".

• Build single-file binaries for all platforms
  • win-x64
  • mac-x64
  • linux-x64
  • and probably more variants of the above
• Build .net core global tool
  - [ ] Build docker image
• Run all tests (with databases in docker)
  - [ ] Create release/publish to chocolatey, nuget, apt-source, homebrew or others?

https://github.com/chucknorris/roundhouse/wiki/TokenReplacement

I need this for alter database {{DatabaseName}} ... scripts as we use different database names in each environment.

I'm suggesting we only do the baked in tokens initially, and do UserTokens later...

We publish some 100s of MBs of artifacts on each build, and use a loooot of storage. Would it make sense to only publish artifacts on "release" builds? (from a release/* branch, from tags, etc?)

In preference to publishing a first class docker image (or just as a prelude to doing so) Build a small example showing how to build and publish a docker image for migrations.

I wish dotnet had support for --example like rust :(

• basic dockerfile (based on a for-real RH project)
• Get grate published as a dotnet-tool
• Build docker-compose file to provide a dbase and migration image that talk to each other.
• Update to use grate not RH
• Get the example to actually run... (waiting on #64 and release)

https://wiki.alpinelinux.org/wiki/Creating_an_Alpine_package

(See also #60 , probably some lessons to be learned there).

Maybe a good idea to build this in a container, as there are so many environment requirements to building an alpine package

G'day guys,

I just wanted to open a general discussion around where you want to head with this, your priorities, things we want to defer until later etc (and fair warning I haven't looked at the code yet, so Erik may have already ticked these boxes.

I've spent a bit of time pondering this and have some general ideas, but am obviously only one voice, and fully expect others to have had different experiences/use cases with RoundhousE so feel free to shoot me down here with your own experiences :)

I agree that we're more likely to make progress if we're working on features we actually use, so here's a quick brain dump on the features I personally use and love:

• The general directory structure based suite of RunOnce, RunAlways, RunIfChanged scripts
• Environment specific scripts
• Versioning info stored in the database (does anyone use something other than in-database versioning?)
• Version number sourced from a provided .DLL
• GO batch separation support
• SQL Server (on prem and Azure) only
  • Improving the support for connecting to Azure SQL would be nice (tokens, managed identity etc)
• DropCreate and Update modes. I have used RestoreAndUpdate in other environments too, but it feels less useful in a Cloud-First world to me.
• I have used tokens in the past for limited scenarios so I can see it's value.
• I have used the AdminConnection feature in the past for server-level setup of the database, but also ran into bugs around transaction scoping etc.
• .Net Global Tool installer (awesome!)
  • I currently build a docker image (based on Alpine) that uses .net tool installer to get Roundhouse on... Rather than having a formal docker image maybe we can just have some docco/a sample showing how to build a working docker based migration so there's less things to maintain?

wrt features I don't use, and I would be comfortable leaving until later (perhaps forever):

• MSbuild support. .Net is increasingly moving away from MSBuild, and I'd rather see support for an AzDo/GitHub task, or Docker-based deploys etc
• Oracle support. As above, I don't back myself to get it right, and have no means (or desire to learn) how to test it.
• MySQL had some funky keyword generation stuff going on back in RoundhousE,... It'd be nice to stay simple.

In other general thoughts (with no idea of how involved it would actually be):

• If we can somehow do 'pluggable' database support, so someone using SQL Server doesn't need to pull a binary with (say) Oracle or Postgres dlls. This would both be a smaller/faster install and save any (future) licencing issues for those that don't need it? No idea how we'd make it work yet...
• Separate the actual migration/execution engine from the Console app that's running it? Would this get us any benefits re: hosting roundhouse inside your application for those people that want in-app migrations?
• Maintaining Back-Compat with Roundhosue for the versioning info would make migrating to grate easier
  • Same file diffing logic
  • Sample showing how to point grate at the Roundhouse schema ?

Anyway that's some early thinking from my side, I'm off to look at some code :)

Can use one of the self-contained binaries - no .NET needed on base images.

• Document how to work with volumes
• Document how to use in a k8s scenario(?)

I won't let this slide just yet. The blocking barrier was just connecting to an Oracle database (in a container) without having to have TNSNAMES.ORA files, but this should be possible. I won't give up! :)

Try these:

• https://stackoverflow.com/a/6239416/25338
• https://hub.docker.com/r/oracleinanutshell/oracle-xe-11g (look at the recipe there)

• There is always a new entry in the tables on each run
• Do we ever update the modified_date, or is it always equal to entry_date?
• If we don't ever update it, could we think of a scenario where we should have?

Just return a true/false of whether the database is up to date or not. No output, just return true on up to date, and false if not.

Implies --disableoutput

https://docs.chocolatey.org/en-us/create/create-packages

I have taken an existing Roundhouse project that targets MSSQL, installed Grate as a DotNet Tool, adjusted the parameters that were incompatible with Grate, and then run the grate command and I am seeing the following error:

Unhandled exception: Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the login process. (provider: SSL Provider, error: 0 - The certificate chain was issued by an authority that is not trusted.)
---> System.ComponentModel.Win32Exception (0x80090325): The certificate chain was issued by an authority that is not trusted.
at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions) at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)
at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
at Microsoft.Data.SqlClient.SqlConnection.InternalOpenAsync(CancellationToken cancellationToken)

To Reproduce
I am deploying a pretty simple database that contains tables, SP, and scripts in the up folder that populate data and grant execute permissions. These are the config settings I am using:

grate.exe --databasetype=sqlserver --files=C:\POC.Grate\Scripts --connectionstring=Database=TestDb;Server=localhost\sqlexpress;Trusted_Connection=True; --version=1.0.0.0 --silent --schema=RoundhousE

Expected behavior
I would expect the database to be deployed as it was before with Roundhouse.

Desktop (please complete the following information):

• OS: Windows 10

The options are currently in a pretty random order (mainly because I keep tacking things on the end!).

We should get them added in some sort of logical order (Most important first? Alphabetical? The same as RH?)

Just print out what would have been done, don't do it.

Is it as simple as adding another parameter to this method?

async Task LogAndRunSql()
            {
                _logger.LogInformation(" Running {scriptName} on {serverName} - {databaseName}.", scriptName, Database.ServerName, Database.DatabaseName);
                await RunTheActualSql(sql, scriptName, migrationType, versionId, connectionType);
                theSqlRun = true;
            }

Tasks:

• Do not copy the scripts to output folder on dryrun
• Do not run the scripts
• Do not record the scripts as run
• DO output the scripts as run
• DO compare the scripts with existing, perform all the logic, etc

Get a build and test CI action (no publish/release) running on push to feature branches so we know how broken things are before raising a PR,

Demonstrate how to use grate in a pipeline, e.g. Azure Devops pipelines or github actions

1. Get artifacts (db scripts) from somewhere
2. Run grate with --dryrun
3. Create a manual "verification" step
4. Run grate without --dryrun

Should provide for a nice workflow in DevOps (the method of working, not the MS Tool itself) organizations.
Maybe just create a whole separate example repo, with a build .yaml and everything? Or put it in the "examples" folder

Since this one is a blocker for people at my org giving grate a try, I figured I'd write up an issue to get this tracked. This would mimic the RestoreAndRun mode defined here that Roundhouse has by using some similar restore configuration options

I have already started down the path of implementing this feature, but if you have any thoughts around this @erikbra I am open to discussing it further to make sure it lines up with your ideas around this project before any kind of PR is opened.

RH doesn't run a .ENV script unless the environment is specifically specified (https://github.com/chucknorris/roundhouse/blob/master/product/roundhouse.core/environments/DefaultEnvironment.cs#L26) but grate is running those scripts by default (https://github.com/erikbra/grate/blob/main/grate/Migration/DbMigrator.cs#L164)

Ref. chucknorris/roundhouse#343

• Username/password
• AD username/password
• AD username/password with MFA
• Token
• others?

Once we get a dockerfile added to the repo (via #62) get it published to DockerHub by the release pipeline.

Start here:

• https://docs.microsoft.com/en-us/windows/package-manager/package/manifest
• https://github.com/microsoft/winget-create
• https://www.infoworld.com/article/3611365/getting-started-with-winget-the-windows-package-manager.html

System.Commandline accepts response files containing the command line arguments. Document how to use them for grate.

See https://github.com/dotnet/command-line-api/blob/main/docs/Features-overview.md#response-files

Run all anytime scripts on the run