GithubHelp home page GithubHelp logo

Comments (64)

clement911 avatar clement911 commented on September 9, 2024 6

This would probably be an investment and cost initially but it would probably pay off in the long term.
@kburtram any thoughts?
We've been waiting patiently for 3.5 years 😢

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024 5

Are there plans to port https://www.nuget.org/packages/Microsoft.SqlServer.TransactSql.ScriptDom/ to .net core?

from sqltoolsservice.

shueybubbles avatar shueybubbles commented on September 9, 2024 4

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024 3

I see.
Well I hope that the relevant team can get the ball rolling in this direction.
I think it would open up a number of very interesting scenario that would empower the sql server community.
For us, we want to pre-process our scripts with macros and also format consistently but I imagine there would be other cool scenarios such as linters and code analyzers.

from sqltoolsservice.

shueybubbles avatar shueybubbles commented on September 9, 2024 3

The SqlToolsService used by ADS is itself open source. It gets all its dependencies from nuget.
Checking its PackageReference (see https://github.com/microsoft/sqltoolsservice/blob/master/src/Microsoft.SqlTools.ServiceLayer/Microsoft.SqlTools.ServiceLayer.csproj) for DacFx, we find:

<PackageReference Include="Microsoft.SqlServer.DacFx" Version="150.4534.2-preview" />

Installing that package, we find the following netstandard2.0 DLLs:

 Directory of C:\Users\username\.nuget\packages\microsoft.sqlserver.dacfx\150.4534.2-preview\lib\netstandard2.0

09/11/2019  02:09 PM    <DIR>          .
09/11/2019  02:09 PM    <DIR>          ..
09/05/2019  09:05 PM        11,212,920 Microsoft.Data.Tools.Schema.Sql.dll
09/05/2019  09:04 PM           257,144 Microsoft.Data.Tools.Utilities.dll
09/05/2019  09:04 PM           329,848 Microsoft.SqlServer.Dac.dll
09/05/2019  09:04 PM           752,760 Microsoft.SqlServer.Dac.Extensions.dll
09/05/2019  09:06 PM         4,559,128 Microsoft.SqlServer.TransactSql.ScriptDom.dll
09/05/2019  09:07 PM            22,296 Microsoft.SqlServer.Types.dll

from sqltoolsservice.

srutzky avatar srutzky commented on September 9, 2024 2

Just FYI: Another parsing bug that is still affecting SSMS (and likely also SQLCMD) is related to the odd (and probably not well known) ability of SQL Server to allow nested block comments. Splitting up batches based on the GO separator has issues with nested block comments, which I detailed in the following ticket:

"GO" in 2nd half of nested block comments breaks batch parsing in SSMS and SQLCMD

from sqltoolsservice.

kevcunnane avatar kevcunnane commented on September 9, 2024 2

@sherland that's not an official Microsoft package. The only official, NetStandard-compatible way to get ScriptDom is to reference the DacFx nuget package. I hear you on it being inconvenient to pull in all the DLLs if you just need ScriptDom - @shueybubbles or @pensivebrian can likely comment on priority of separating it into its own package.

from sqltoolsservice.

dmarkle avatar dmarkle commented on September 9, 2024 1

Ugh, the pre-release package has been unlisted from nuget:

"The owner has unlisted this package. This could mean that the package is deprecated or shouldn't be used anymore."

Does anyone see if there is an update, maybe if MS is shipping this somewhere else now?

from sqltoolsservice.

FlorianGrimm avatar FlorianGrimm commented on September 9, 2024 1

The VS SQL Analyzer uses an modified / improved version.
Please give use the code and so the opportunity to build tools around the t-sql code.

from sqltoolsservice.

Charles-Gagnon avatar Charles-Gagnon commented on September 9, 2024 1

@masonwheeler Could you please open a new issue? This repo is good enough for now, but finding specific issues buried in unrelated threads is difficult 😄

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

Not going to happen? We use the parser in our app but it's always scary to grab a dll from the GAC and just copy it to our app.
If it was properly open sourced, it would give us confidence that this library is not going to be removed under us!

from sqltoolsservice.

kburtram avatar kburtram commented on September 9, 2024

@clement911 SQLParser is the primary components used for SSMS, SQL Ops Studio and vscode-mssql language services. It is safe to assume the parser will continue to be maintained for the foreseeable future. There should be a redist you can use which is the recommend way to install this component.

The SQL Parser component is currently build as part of the SSMS internal repository. SSMS (and therefore SQL Parser) is built using a custom build environment that we'd need to update to a more standard environment prior to open sourcing. Also, there are various internal procedures we need to figure out, such as how best to handle vNext changes prior to public disclosure, etc.

We have backlog items for open-sourcing Parser, SMO, DacFx and other tools components, but there are quite a bit of details we need to work through prior to publishing these components that weren't originally developed as OSS. Unfortunately, I don't have an ETA for this work.

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

There should be a redist you can use
Which redist do you mean? The dll in the sql server folder?

from sqltoolsservice.

jzabroski avatar jzabroski commented on September 9, 2024

@clement911 Microsoft has started publishing the SqlParser as a nuget package, which is a step in the right direction: https://www.nuget.org/packages/Microsoft.SqlServer.SqlParser/140.17279.0-xplat

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

Great to hear.
Here is my user voice entry if more people would like to vote:
https://feedback.azure.com/forums/908035-sql-server/suggestions/34741105-open-source-microsoft-sqlserver-sqlparser-dll-re

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

@jzabroski I found a few bugs in the parser from Microsoft.SqlServer.SqlParser.
OVER clauses and NULLIF are not parsed correctly.
Do you know where I can report bugs?

from sqltoolsservice.

jzabroski avatar jzabroski commented on September 9, 2024

@kburtram works for Microsoft and would be better to answer that. Unless it's open source I doubt these issues get resolved. I anecdotally believe a lot of bugs have crept into SqlParser over the last 3 releases. I have done very odd things where I reverse the string manually in the text, assign it to @sqlcommand, reverse it again in memory in TSQL, and execute it using EXEC (@sqlcommand). Then it executes fine. Most of the issues have to do with privileged commands like sp_dropuser. I posted about it on StackOverflow.

Maybe try the same trick..

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

Thanks @jzabroski , although I'm not really sure what you mean by 'reversing the string'.
We're just trying to parse some sql code.

@kburtram I'm calling on to you again then :)
We're taking a pretty big dependency on the sql parser nuget package. We're really glad it is now on nuget!
We are relying on the parser to provide us with an AST that we can visit to inspect various things about
our TSQL code and even change on the fly via pseudo macros.
It is very powerful and pretty cool but the problem is that the AST is often not full fidelity to the code given. It seems that certain code paths are not fully parsed, such as OVER bodies, NULLIF parameters, and more.
Is there anything we can do to facilitate the open sourcing a parser?
I'm sure it would open up some really cool possibilities such as linters, analyzer, editors, macros, etc...
It's pretty frustrating because it looks like the parsing works 95% great, but there are just a few issues.

If this is a dead end, are there any other TSQL parser out there you know of? We need a parser capable of producing a full fidelity AST...

from sqltoolsservice.

jzabroski avatar jzabroski commented on September 9, 2024

from sqltoolsservice.

kburtram avatar kburtram commented on September 9, 2024

@clement911 I don't think we have plans to open source the parser within the next few months. We'd probably want to do that as part of opening a larger body of code, such as SMO, which will take a while.

@shueybubbles is the owner of the Parser Nuget package (and SMO) and may have additional context. If you provide example SQL that the parser isn't correctly handling I can open an internal bug against the SQL Parser component.

In the near-term it's possible something like the parser that @jzabroski mentioned may work better for you. Though we certainly would want to fix any significant bugs in our language service SQL parser (and eventually open-up that codebase).

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

I haven't looked at that other parser no. If we get desperate we may have to look at it...

I will get some minimal bug repros and post them here shortly.

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

So here are the 2 bugs I found so far.
I parse and inspect the AST with Parser.Parse(sqlQuery).Script.Xml

Bug 1)
Compare the two ASTs for the two following queries.

Query 1: SELECT NULLIF([col], 0) FROM [Table]
Query 2: SELECT ISNULL([col], 0) FROM [Table]

The queries are identical except that one is using NULLIF and the other is using ISNULL.
Yet, the AST of the query with ISNULL includes the parameters within the function call whereas the AST of the query with NULLIF doesn't.

nullif_bug

Bug 2)

The OVER() clause seems to be swallowed up in the SUM aggregate function call and is not present at all in the AST.

over_bug

from sqltoolsservice.

jzabroski avatar jzabroski commented on September 9, 2024

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

Of course we know that there is at least one parser that is comprehensive, and that is the parser used by Sql Server itself. If only that was available...

On the programming language side, the situation has become ideal, with C# and Typescript both coming with a language service as a first class citizen, which is powered by the same parser that the compilers use. The tooling has become much better thanks to that...

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

I just came across another parser.

https://www.nuget.org/packages/Microsoft.SqlServer.TransactSql.ScriptDom/
https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2010/dd194286(v=vs.100)

This just gives me hope.
I will investigate further but for now I'm confused.
What is the relationship between this project and Microsoft.SqlServer.SqlParser?

from sqltoolsservice.

kburtram avatar kburtram commented on September 9, 2024

@clement911 we use Microsoft.SqlServer.Management.SqlParser for our language service, so it's optimized around those types of tasks (e.g., IntelliSense suggestions, QuickInfo tooltips, diagnostic errors).

I think ScriptDom is more around working with the AST, which may work better for your scenario. You may also need to bring in DacFx to work with ScriptDom, and it's not available on .Net Core AFAIK.

@kevcunnane do you have additional context on ScriptDom vs. SqlParser?

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

Good news!
Microsoft.SqlServer.TransactSql.ScriptDom.TsqlParser is able to parse those test cases I posted above!
This is looking very promising for us. The API is different but at least the AST covers the full syntax.

@kburtram I see what you're saying, although it seems sharing a single parser would be worthwhile given the complexity of the task...

FYI it is also available as a nuget package and the only dependency is .Net 4.0.
https://www.nuget.org/packages/Microsoft.SqlServer.TransactSql.ScriptDom/

from sqltoolsservice.

FlorianGrimm avatar FlorianGrimm commented on September 9, 2024

Is it possible to publish your ANTLR definition?
I'm using currently ASP.NET Core with .NET Framework. And for ASP.Net Core 3.0 ....

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

Hi @FlorianGrimm ,
Could you share your scenario? What are you trying to achieve?

I'm not sure if you checked out Microsoft.SqlServer.TransactSql.ScriptDom.TsqlParser but for us it worked much better. As I understand it, it is the same parser as the one used by Sql server itself, but exposed as a .net dll.

from sqltoolsservice.

FlorianGrimm avatar FlorianGrimm commented on September 9, 2024

I currently testing Microsoft.SqlServer.TransactSql.ScriptDom.TsqlParser, but it doesn't have a a .Net Core assembly.
I saw the ANTLR classes in it. ANTLR.org has a grammer file, but I don't know how much close it is.
I have to do some type analysis.
Basically I want to know if i call a SQL Statement what will the result look like Name and Type - for more than one SELECT Statement. No GUI no external exe.
It tried some libraries. Even for "simple" things I want behave like the server e.g. nested comments /* \n--*/ \n */.
I guess I saw 3 different SQL Parser from Microsoft. TsqlParser works well, but it's only available for .Net 4.x.
The ANTRL 4 runtime now has a return value for visitors which can be really useful - IMHO.
That's why I asked for the grammer files

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

Got it.
I see your points. It would be great if the team could update Microsoft.SqlServer.TransactSql.ScriptDom.TsqlParser to be available on .net core.
Asp.net core 3.0 won't be available on .net so we will definitely have to upgrade to .net core and we do use TsqlParser...
It's frustrating, there seems to be no way to reach the relevant teams...

With respect to your scenario have you considered calling this stored proc:
https://docs.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sp-describe-first-result-set-transact-sql?view=sql-server-2017

from sqltoolsservice.

MarcusKohnert avatar MarcusKohnert commented on September 9, 2024

@FlorianGrimm This pre-release package is working well for our scenarios on .NET Core.

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

@MarcusKohnert , is there a difference between this package and https://www.nuget.org/packages/Microsoft.SqlServer.TransactSql.ScriptDom/ ?

from sqltoolsservice.

MarcusKohnert avatar MarcusKohnert commented on September 9, 2024

@clement911 I don't know your referenced package so I don't know if there are any differences. Microsoft.SqlServer.SqlParser/150.18068.0-xplat is packaged by Microsoft at least which is a plus for me. ;)

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

Have you considered using https://www.nuget.org/packages/Microsoft.SqlServer.TransactSql.ScriptDom/?

from sqltoolsservice.

dmarkle avatar dmarkle commented on September 9, 2024

Oh. That's awesome. I've been waiting for that to go on Nuget forever. It's like a dream come true, and SqlParser from SMO works great (except SMO doesn't really like installing on my dev workstation with tons of other assemblies already installed). But I can get around that.... Next thing you know, they'll be porting SQL AMO to .Net Core (ha)

from sqltoolsservice.

ststeiger avatar ststeiger commented on September 9, 2024

Anybody that needs an ANTLR4 version for .NET Core:

https://github.com/ststeiger/SqlParser

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

Checking back again to see if Microsoft.SqlServer.TransactSql.ScriptDom might be ported to .Net core?
We want to migrate to .net core but can't because of this dependency.

from sqltoolsservice.

ststeiger avatar ststeiger commented on September 9, 2024

Both ScriptDom and SqlParser are ported to netstandard 2.0, which you can run on .NET Core.

Just download SQL-Ops Studio (latest-version) there you find it in subdirectory resources\app\extensions\mssql\sqltoolsservice\Windows\<VERSION>

But it uses ANTLR behind the scenes, so you might as well use ANTLR yourselfs.
It would be great if they published the full T-SQL antlr grammar file for each version, though.

If you need to make any fixes, you can download ILSpy v5, decompile the netstandard dll, and with 5 small fixes it compiles again. Then you can fix the bugs yourselfs.

But again, instead of that, just download the ANTLR 4 T-SQL grammar, and create your own ANTLR parser. Fix the grammar file, if you need better T-SQL support.

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

As far as I can see the latest ScriptDom only target .net framework.
See https://www.fuget.org/packages/Microsoft.SqlServer.TransactSql.ScriptDom/
Am I missing something?

With respect to the correct grammar, are you speaking of SqlParser specifically ?
It is my understanding that Microsoft.SqlServer.TransactSql.ScriptDom is maintained by the sql server team and is already a full-fidelity parser.

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

See below. Only one target framework moniker, for net40

image

from sqltoolsservice.

ststeiger avatar ststeiger commented on September 9, 2024

NetStandard 2.0
Show NetStandard 2.0

Build output
Show build output

.NET Core 2.1
Show .NET Core

And it's running:
ScriptDom on NetCore

from sqltoolsservice.

ststeiger avatar ststeiger commented on September 9, 2024

What you're doing wrong is taking the DLL from nuget, instead of taking the one that AzureDataStudio uses. (AzureDataStudio runs on Linux)

https://github.com/microsoft/azuredatastudio

Download the zip archive, and get the NetStandard2.0 dlls from subfolder
resources\app\extensions\mssql\sqltoolsservice\Windows\<VERSION>

With respect to "correct grammar", you only need the grammar file if you build your own parser with ANTLR. If you use the SqlParser provided by MS, you don't need any grammar file at all - the corresponding code has already been created and is part of ScriptDom/SqlParser.

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

Oh I see. Great.
Are there any plans to publish this Net standard dll in Nuget, or update the existing nuget package to use this dll?
It would mean we can easily update to the latest version moving forward, as opposed to referencing the dll manually.

from sqltoolsservice.

shueybubbles avatar shueybubbles commented on September 9, 2024

The Microsoft.SqlServer.DacFx nuget package has an appropriate license for ScriptDom, which it includes.

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

@shueybubbles Microsoft.SqlServer.DacFx target Net framework though. I want to use .net core.

from sqltoolsservice.

ststeiger avatar ststeiger commented on September 9, 2024

@clement911: I have no idea what license Microsoft.SqlServer.TransactSql.ScriptDom comes with, so I'd rather not publish it.

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

Referencing the dll from Azure Data Studio works. Great!
Hopefully it can be packaged in nuget.

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

Me neither. I guess if the right people at Microsoft could get together maybe they could sort this out.
At least there is a work around for now.

from sqltoolsservice.

jzabroski avatar jzabroski commented on September 9, 2024

Thanks, David Shiflet! @shueybubbles

from sqltoolsservice.

ststeiger avatar ststeiger commented on September 9, 2024

What do you use Dac for ?

Anybody knows what Microsoft.Data.Tools.Schema.Sql.dll and Microsoft.Data.Tools.Utilities.dll can do ?

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

@shueybubbles oh I see, so it's a preview package.
I was looking at Microsoft.SqlServer.DacFx.x64...

I don't want to be a pain here, but in our project, we only need Microsoft.SqlServer.TransactSql.ScriptDom.dll
The package Microsoft.SqlServer.DacFx has a LOT of dependencies and we'd love to see small package for Microsoft.SqlServer.TransactSql.ScriptDom.dll

from sqltoolsservice.

sherland avatar sherland commented on September 9, 2024

This is rather anoying ;-)
https://www.nuget.org/packages/Microsoft.SqlServer.TransactSql.ScriptDom/ should be netstandard compatilbe, and Microsoft.SqlServer.DacFx should have a dependency on the nuget, instead of embedding its own copy.
As an example: Which dll will be used in projects that reference both Microsoft.SqlServer.TransactSql.ScriptDom (could be indirecty) and Microsoft.SqlServer.DacFx?

from sqltoolsservice.

duncansmart avatar duncansmart commented on September 9, 2024

I've just stumbled across this. Sorry about the Microsoft.SqlServer.TransactSql.ScriptDom nuget package that I started uploading years ago causing all the confusion recently, but I'm struggling to see how I could have made things much clearer 🤷‍♂
image

Good to see .NET Core/netstandard support in Microsoft.SqlServer.DacFx!

from sqltoolsservice.

jzabroski avatar jzabroski commented on September 9, 2024

@duncansmart The "Microsoft." prefix is supposed to be reserved for Microsoft projects. On top of that, "Microsoft.SqlServer.Management" is literally the prefix for Smo. I know you feel you made things obvious, but how do you think developers were confused?

from sqltoolsservice.

duncansmart avatar duncansmart commented on September 9, 2024

@jzabroski I've been getting emails from people asking for a .NET Core version of that package and people here seem to think it's an official release. I added it years before the official DacFx packages were added. Naming it after the assembly it packaged seemed sensible in allowing people to be able to find it, indeed this was pretty common practice in the early days of NuGet when some parts of Microsoft were slow in releasing official NuGet packages.

from sqltoolsservice.

shueybubbles avatar shueybubbles commented on September 9, 2024

from sqltoolsservice.

jzabroski avatar jzabroski commented on September 9, 2024

@duncansmart Either way, your naming of the NuGet package isn't desirable today given Nuget package prefix id's, specified here: https://github.com/NuGet/Home/wiki/NuGet-Package-Identity-Verification#what-does-a-reserved-package-id-prefix-do - This RFC went into effect in 2017. In general, since the 1990s and Java engineers using "reverse domain name" for namespaces, e.g. com.java.*, it's well-established not to use somebody else's company name as a prefix for a code artifact, whether it be an assembly name, package name or namespace.

You're in a strange spot right now, because permanently deleting a package is not supported by Nuget, yet that would be the right thing to do.

from sqltoolsservice.

WeihanLi avatar WeihanLi commented on September 9, 2024

Any progress?

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

Coming back to the original question. Still no progress on open sourcing the parser?

from sqltoolsservice.

clement911 avatar clement911 commented on September 9, 2024

Any progress on open sourcing the parser Microsoft.SqlServer.Management.SqlParser?
What's the real blocker here?

from sqltoolsservice.

jzabroski avatar jzabroski commented on September 9, 2024

Just a guess: $$$ :)

from sqltoolsservice.

masonwheeler avatar masonwheeler commented on September 9, 2024

This thread seems to be the closest thing we have to a place to report bugs in Microsoft.SqlServer.Management.SqlParser for the moment. (If there's a better place, please let me know.)

I just ran across an annoying problem in the current NuGet build: calls to string_agg are parsed as a SqlBuiltinScalarFunctionCallExpression node, but string_agg is not a scalar function; it's an aggregate function. So this call should be represented by a SqlAggregateFunctionCallExpression node.

from sqltoolsservice.

masonwheeler avatar masonwheeler commented on September 9, 2024

@Charles-Gagnon Done

from sqltoolsservice.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.