erikbra / grate Goto Github PK
View Code? Open in Web Editor NEWgrate - the SQL scripts migration runner
License: MIT License
grate - the SQL scripts migration runner
License: MIT License
Record all scripts as having been run, but don't actually run them.
Create installers in the installers/msi
folder.
WIX is possibly the best alternative:
PROs: Free, command-line
CONs: .NET 4.x dependent (maybe harder to run in GitHub actions?
Ref chucknorris/roundhouse#424
--warnononetimescriptchanges
When setting up the grate repo for local development, I noted that (specifically with MS SQL Server), many of the test runs resulted in failed connection attempts:
However, the last few tests connected successfully, and the final test report doesn't report any test failures. I'd like to assume that the failed connect attempts were deliberate (I haven't verified by checking the unit test code yet), but I'd also assume there would only be a few of them. There were many during the initial run of "dotnet test" after cloning the repo.
Opening this ticket for tracking/investigation purposes.
Relevant Tooling:
There seems to be a bug, when creating the migrations folder in the container. I get the following:
Unhandled exception: System.IO.IOException: The file '/app/grate' already exists.
at System.IO.FileSystem.CreateDirectory(String )
at System.IO.Directory.CreateDirectory(String )
at grate.Migration.GrateMigrator.CreateChangeDropFolder(String folder)
at grate.Migration.GrateMigrator.Migrate()
at grate.Commands.MigrateCommand.<>c__DisplayClass0_0.<<-ctor>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Invocation.CommandHandler.GetExitCodeAsync(Object value, InvocationContext context)
(...)
Repro:
./my_scripts
folder.FROM erikbra/grate:latest
COPY ./my_scripts ./db
ENTRYPOINT ./grate -f=db --connectionstring "$CONNECTIONSTRING"
docker build -t my-db-image .
docker run --rm -e CONNECTIONSTRING my-db-image
Observe the error.
First "external" package manager after NuGet.
❯ winget search grate
Name Id Version Match Source
---------------------------------------------------------
grate erikbra.grate 0.9.4 winget
Volta Volta.Volta 1.0.5 Command: volta-migrate winget
❯ winget install erikbra.grate
Found grate [erikbra.grate] Version 0.9.4
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
Downloading https://github.com/erikbra/grate/releases/download/0.9.4/grate-0.9.4.msi
██████████████████████████████ 11.6 MB / 11.6 MB
Successfully verified installer hash
Starting package install...
Successfully installed
❯ get-command grate
CommandType Name Version Source
----------- ---- ------- ------
Application grate.exe 0.0.0.0 C:\Users\erikb\AppData\Local\erikbra\grate\grate.exe
❯
//cc @wokket
After using Winget or downloading the latest Release, how could a first time user with a dacpac(?) use Grate to connect to a client SQL Database, publish schema changes, and execute custom SQL data updates?
I have thought a bit about lowering the level of logs on the "skipped" lines to "Debug", so that only what is actually applied is output by default, how do we feel about that? We would get the full list if setting the verbosity to Debug on the command line.
I think it would make it easier to see what wat actually run in the output.
Thoughts?
How could one use grate in a Kubernetes setting?
Maybe @wokket s examples are enough. Verify 😄
Warn on one time script changes, bug ignore them, do not run the changed script.
Do we want/need this?
Example output:
grate v1.0.0.0 has grated your database (hpWAJepVEbKOUSD)! You are now at version a.b.c.d. All changes and backups can be found at
"/home/runner/work/grate/grate/grate.unittests/bin/Release/net6.0/output/migrations/hpWAJepVEbKOUSD/localhost,49153/2021-09-22T20_07_36".
If the server name of the connection string contains a comma, replace that from the file name. I'm not sure every OS likes a comma in a file.
Since grate will suppress console output for skipped files by default, it would make sense to skip output of the extra dividers and whitespace to trim unnecessary content as well. This will help to eliminate noise when verifying that scripts are detected and run appropriately.
Perhaps showing something like:
Looking for scripts in xxxx
--no new scripts--
and/or checking for a directory before output to enable simplifying the message to
Skipping runAfterFirstUp, xxxx folder does not exist
Original comment posted by @docrinehart in #140 (comment)
I'm seeking clarification on the use of the --version option mentioned in #8
If I use the example script powershell:
grate `
--files=.\db `
--env=Local `
--connstring="Server=(localdb)\MSSQLLocalDB;Integrated Security=true;Database=grate_test" `
--version=1.0
It all works fine.
If I run it a 2nd time with the env=Test, it works as described and only the 'Test' script from the 'up' folder is executed, but...
Change the --version=1.1
and it runs all the 'one time' scripts again.
How should the semantic version control work?
Do script files have a version number added into the file name like the ```ENV```` keyword?
- [ ] Set up domain (github pages?)
aka silent. Make it run, but shut up about it.
Describe the bug
While running "dotnet grate" with only a connection string, the migrations run successfully but the skipped scripts are not listed in the tool output.
To Reproduce
Expected behavior
Tool output should include skipped migration scripts in the console output to confirm script presence and evaluation.
Desktop (please complete the following information):
Additional context
I was able to verify that the initial run shows the script names
I started migrating an existing project over, and found there's currently no support for --drop.
We use this as part of our integration tests against localdb to drop and recreate the database from scratch for each run.
Is this a feature you're intentionally keeping out of scope for grate? If it just hasn't been got to yet I'd like to add it to the (long) list of features that need doing :)
To make the package presentation on nuget.org better, make user a README file is included in the NuGet package, as described here: https://docs.microsoft.com/en-us/nuget/nuget-org/package-readme-on-nuget-org
As the .msi is unsigned Windows prompts with the blue screen re: unknown publisher.
This is both a usability thing and may well prevent the installer being run in some environments which require signed installers on servers.
Pre-release should be fine to begin with.
This is both a blocker for #41 and also a pre-req for me to start running grate against real-world projects for testing.
Option to avoid storing the whole SQL text in the ScriptsRun table. Especially useful if running in small/embedded databases, where the grate tables can become a substantial part of the database size.
Previously RH used --version
for this, but the .Net CommandLine libraries claim this for the application version.
I'm suggesting we add this using --dbversion
instead to avoid conflicts if that's ok?
We need to set up a build pipeline. Try GitHub actions, as it's "close to the code".
https://github.com/chucknorris/roundhouse/wiki/TokenReplacement
I need this for alter database {{DatabaseName}} ...
scripts as we use different database names in each environment.
I'm suggesting we only do the baked in tokens initially, and do UserTokens later...
We publish some 100s of MBs of artifacts on each build, and use a loooot of storage. Would it make sense to only publish artifacts on "release" builds? (from a release/* branch, from tags, etc?)
In preference to publishing a first class docker image (or just as a prelude to doing so) Build a small example showing how to build and publish a docker image for migrations.
I wish dotnet
had support for --example
like rust :(
https://wiki.alpinelinux.org/wiki/Creating_an_Alpine_package
(See also #60 , probably some lessons to be learned there).
Maybe a good idea to build this in a container, as there are so many environment requirements to building an alpine package
G'day guys,
I just wanted to open a general discussion around where you want to head with this, your priorities, things we want to defer until later etc (and fair warning I haven't looked at the code yet, so Erik may have already ticked these boxes.
I've spent a bit of time pondering this and have some general ideas, but am obviously only one voice, and fully expect others to have had different experiences/use cases with RoundhousE so feel free to shoot me down here with your own experiences :)
I agree that we're more likely to make progress if we're working on features we actually use, so here's a quick brain dump on the features I personally use and love:
GO
batch separation supportDropCreate
and Update
modes. I have used RestoreAndUpdate
in other environments too, but it feels less useful in a Cloud-First world to me.AdminConnection
feature in the past for server-level setup of the database, but also ran into bugs around transaction scoping etc.wrt features I don't use, and I would be comfortable leaving until later (perhaps forever):
In other general thoughts (with no idea of how involved it would actually be):
Roundhouse
schema ?Anyway that's some early thinking from my side, I'm off to look at some code :)
Can use one of the self-contained binaries - no .NET needed on base images.
I won't let this slide just yet. The blocking barrier was just connecting to an Oracle database (in a container) without having to have TNSNAMES.ORA files, but this should be possible. I won't give up! :)
Try these:
Just return a true/false of whether the database is up to date or not. No output, just return true on up to date, and false if not.
Implies --disableoutput
I have taken an existing Roundhouse project that targets MSSQL, installed Grate as a DotNet Tool, adjusted the parameters that were incompatible with Grate, and then run the grate command and I am seeing the following error:
Unhandled exception: Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the login process. (provider: SSL Provider, error: 0 - The certificate chain was issued by an authority that is not trusted.)
---> System.ComponentModel.Win32Exception (0x80090325): The certificate chain was issued by an authority that is not trusted.
at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource
1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions) at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource
1 retry, DbConnectionOptions userOptions)
at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
at Microsoft.Data.SqlClient.SqlConnection.InternalOpenAsync(CancellationToken cancellationToken)
To Reproduce
I am deploying a pretty simple database that contains tables, SP, and scripts in the up folder that populate data and grant execute permissions. These are the config settings I am using:
grate.exe --databasetype=sqlserver --files=C:\POC.Grate\Scripts --connectionstring=Database=TestDb;Server=localhost\sqlexpress;Trusted_Connection=True; --version=1.0.0.0 --silent --schema=RoundhousE
Expected behavior
I would expect the database to be deployed as it was before with Roundhouse.
Desktop (please complete the following information):
The options are currently in a pretty random order (mainly because I keep tacking things on the end!).
We should get them added in some sort of logical order (Most important first? Alphabetical? The same as RH?)
Just print out what would have been done, don't do it.
Is it as simple as adding another parameter to this method?
async Task LogAndRunSql()
{
_logger.LogInformation(" Running {scriptName} on {serverName} - {databaseName}.", scriptName, Database.ServerName, Database.DatabaseName);
await RunTheActualSql(sql, scriptName, migrationType, versionId, connectionType);
theSqlRun = true;
}
Get a build and test CI action (no publish/release) running on push to feature branches so we know how broken things are before raising a PR,
Demonstrate how to use grate in a pipeline, e.g. Azure Devops pipelines or github actions
Should provide for a nice workflow in DevOps (the method of working, not the MS Tool itself) organizations.
Maybe just create a whole separate example repo, with a build .yaml and everything? Or put it in the "examples" folder
Since this one is a blocker for people at my org giving grate a try, I figured I'd write up an issue to get this tracked. This would mimic the RestoreAndRun mode defined here that Roundhouse has by using some similar restore configuration options
I have already started down the path of implementing this feature, but if you have any thoughts around this @erikbra I am open to discussing it further to make sure it lines up with your ideas around this project before any kind of PR is opened.
RH doesn't run a .ENV script unless the environment is specifically specified (https://github.com/chucknorris/roundhouse/blob/master/product/roundhouse.core/environments/DefaultEnvironment.cs#L26) but grate is running those scripts by default (https://github.com/erikbra/grate/blob/main/grate/Migration/DbMigrator.cs#L164)
Ref. chucknorris/roundhouse#343
Once we get a dockerfile added to the repo (via #62) get it published to DockerHub by the release pipeline.
System.Commandline accepts response files containing the command line arguments. Document how to use them for grate.
See https://github.com/dotnet/command-line-api/blob/main/docs/Features-overview.md#response-files
Run all anytime scripts on the run
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.