jokedst / csvquery

Plugin for Notepad++ that treats CSV files as (read only) SQL tables

CSV Query - Notepad++ plugin

A plugin to Notepad++ to parse different types of CSV files and display them in a table. The data is stored in a in-memory SQLite database (or MSSQL if configured), so you can write SQL queries against the data.

It tries to auto-detect separators, headers and column types. If it fails it asks you for it.

Use modern SQL

CsvQuery has a built-in SQLite engine. This is quite old, 3.7.7.1, so new features like windowing functions doesn't work. There is no plans on updating this (unless someone else makes a new C# port of SQLite).

However, CsvQuery can also use SQL Server as backend. Then all features in SQL Server can be used right from CsvQuery.

To use SQL Server you must first install SQL Server locally. Any version should work (there are free ones).
Make sure "Integrated security" is checked during install of SQL Server (it is by default).

Then go into CsvQuery settings (Plugins -> CsvQuery -> Settings)

• Set "StorageProvider" to "MSSQL"
• Set "Database" to "CsvQueryDB"

Now click "Read File" again, and it will use SQL Server syntax in CsvQuery.

Another option is to use an external SQLite tool with a more modern SQLite engine.
To do this, change the CsvQuery setting "Database" to a filename instead of ":memory:". The next time you click "Read file" this file will be created as a standard SQLite file, and can be queried with the external tool.

License

This package as a whole is licensed under the GPL v3. See gpl-3.0.txt

The CsharpSqlite code is licensed under MIT license (which is apparently GPL v3 compatible). See CsharpSqlite\License-CsharpSqlite-MIT.txt

Planned features

• Support for more types of CSV files (more test cases, basically)
• Optimizations

Auto-detection

CSV Query detects the separator by calculating the variance in occurrence of characters on each line, then chose the one that seems best, preferring one of comma, semicolon, pipe or tab. It's certainly not perfect, but it handles all the files I regularly work with, which is why I wrote the plugin in the first place.

If the first line is "significantly different" from the rest it assumes the first line is a header and use it for the column names in the database.

Used Libraries

NppPlugin.NET v0.6-0.7 by UFO-Pu55y and later by kbilsted

Base for making Notepad++ plugins in C#

https://github.com/kbilsted/NotepadPlusPlusPluginPack.Net Licensed under GPL v3

CSharpSQLite

SQLite database rewritten in C#. I basically copied the whole codebase into CSV Query to get a single DLL (and to mess around with it a bit).

https://code.google.com/p/csharp-sqlite/
MIT Licence

csvquery's Issues

Inprove query autocomplete

Remember queries between restarts, at least the last 100 or so.

Perhaps provide a way to clear this cache... And/or a setting to disable it.

Support for column names with spaces

CSVQuery doesn't support querying columns whose names contain spaces. Export your contacts in Outlook csv format for a sample data set and try this query:
select First Name as FN from THIS;
You get a "Could not execute query" error dialog

As bad as having spaces in column name is, its worth being supported for such datasets.

Package isnt trusted by Notepad++ Plugin Manager.

Hi,
I can install the plugin but I saw an warning that the plugin is not trusted bu Notepad++ plugin manager.
Is it possible to trust it please ?

Thanks.

CSV Query windows only appear when Notepad++ is run as Administrator

I am running Notepad++ 8.6.4 x64 on Windows 10 Enterprise v21H2. The plugin version is 1.2.9. Although some of the menu items seem to work (e.g. CvsQuery > Manual Parse Settings), when I select "Toggle CSV Query Window" nothing happens. When I run Notepad++ as Administrator, everything works flawlessly.

Unfortunately, this is a work computer and I do not have Admin rights. I was able to test with temporary Admin access but this is not something I can do going forward. Is there a way to set permissions so the plugin will run as a non-elevated user?

Make Plugin Multiple screen aware

I'd like to learn enough about this plug-in to be able to work on one of my own.

To this goal, I noticed that when I use the menus (under the Plugins Menu choosing CVSQuery then choosing one of the side menu choices, like "About" or "Settings" or "Manual Parse Settings", this CvsQuery plugin pops up a window on the first monitor, and not on the second monitor where notepad++ is running.

It does handle showing the "on-the-side" menu options on the second monitor, just the new windows it creates.

How do I learn how to find where the code "knows" to use the first monitor instead of the current monitor?

Thanks,
Dave Whitten
713-870-3834
[email protected]

CR newline not supported

Files with CR newline were not properly parsed, and instead gave some sort of error. The bug was in CsvQuery/Csv/CsvSettings.cs.

Doesn't seem like this was a bug per se, since you apparently made the conscious choice not to support this.

That said, I have worked with CSV files that use CR newline, and I figured I'd fix this.

I'll open a PR.

Add "has header row" to Manual Parse Settings

CsvQuery not installing/working on version v8.3.3

1. the CsvQuery is not showing in the plugin manager

2. When i try to install the CsvQuery v1.2.8-64 thru settings > Import > import plugins, it says it has installed the plugin and it will show after i restart the application. When i do this the plugin is not show up (opened notepad++ as administrator) .

n

Debug info
Notepad++ v8.3.3 (64-bit)
Build time : Mar 13 2022 - 17:20:02
Path : C:\Program Files\Notepad++\notepad++.exe
Command Line :
Admin mode : OFF
Local Conf mode : OFF
Cloud Config : OFF
OS Name : Windows 11 (64-bit)
OS Version : 2009
OS Build : 22000.556
Current ANSI codepage : 1252
Plugins : mimeTools.dll NppConverter.dll NppExport.dll NppSaveAsAdmin.dll XMLTools.dll

How can i fix this?

Add "Manual parse settings" button to main UI

Add a button next to "Read File" where you go to the "Manual parse settings" immediately.
Preferably just a gear icon or something.

Also set the separator in that dialog by analyzing the file so it might be right. If analysiz fails, set it to "last used value" if any, if not, comma.

Table Name in File Tab and Drop Table Button

Would it be possible that in the file tab on notepad++ to display an icon when the file has been read into a database (and possibly show the table name). I know this might be problematic for those of use that run update and delete functions against the table, because then the file and the table are no longer the same, perhaps an icon to indicate that as well?

---

After working with a lot of different files I noticed that the number of tables keeps going up.
There is no easy way to remove a table from the database after you no longer need it.

I'm not suggesting the table automatically be dropped when the file is closed, I recognize there is a use case for having them stay.

Maybe add a button to drop the files associated table from the SQLite database? Right now you have to list the files, then find the table that corresponds to the file you want and drop it manually.

write protected portable installation: error on exit

I created a portable version of NPP with a selection of plugins. This portable version should be write protected to avoid users to change the configuration. When a user who has no access to the folder exits the application, he gets the following error.
Error message:
CSVQuery.dll just crashed in
notify(SCNotification *notification) :
notification -> nmhdr.code == 1009
notification -> nmhdr.hwndFrom == 00000000001403EA
notification -> nmhdr.idFrom == 0

A user having access to the folder does not get this message.

feature request: do query on more than one Notepad++ tab.

Suppose I have two opened CSV files, on tabs New 1 and New 2.

It would be great when a query like this could work:

SELECT *
FROM "New 1" as n1
LEFT JOIN "New 2" as n2 on n2.i = n1.i

(provided, of course, that columns i do exist in both CSV's)

Things I can think of that needs attention:

1. The tablename THIS should be rename to that name of the current tab.
2. Care should be taken when the name of the tab changes (save as...)
3. ...... ?

Show Line Number in Query Results

Request for an option to show the original line number in the CSV file (instead of the sequence row numbers).

embedded comma in double quoted string causes failures

Using this example:
"ControlID","Status","FeatureName","ResourceGroupName","ResourceName","ChildResourceName","ControlSeverity","IsBaselineControl","SupportsAutoFix","Description","Recommendation","DetailedLogFile"
"test123","test123","test123","test123","test123","test123","test123","test123","test123","test123","comma goes here, ","test123"

I get the error above. It doesn't prompt to manually parse, but if i manually parse with default settings, file is read correctly.

v1.2.5

Thanks!

Update in NotePad++ 7.8.5 dont work.

Hi,
I am tried to update csvquery in NotePad++ v7.8.5 thru the module admin.
The module admin downloaded the update and NPP restarted.

csvquery is still displayed in the tab updates.
It doesnt look like NPP updated it.

Can you fix it please ?

Crash when sorting on some columns

Had ran this query:
select sum(belopp), substr(transdatum,1,5) from this group by substr(transdatum,1,5) order by 2

Tried to sort by second column by clicking on header
Got this exception:

************** Exception Text **************
System.ArgumentException: [substr(transdatum isn't a valid Sort string entry.
at System.Data.DataTable.ParseSortString(String sortString)
at System.Data.DataView.CheckSort(String sort)
at System.Data.DataView.set_Sort(String value)
at System.Data.DataView.System.ComponentModel.IBindingList.ApplySort(PropertyDescriptor property, ListSortDirection direction)
at System.Windows.Forms.DataGridView.DataGridViewDataConnection.Sort(DataGridViewColumn dataGridViewColumn, ListSortDirection direction)
at System.Windows.Forms.DataGridView.SortInternal(IComparer comparer, DataGridViewColumn dataGridViewColumn, ListSortDirection direction)
at System.Windows.Forms.DataGridView.Sort(DataGridViewColumn dataGridViewColumn, ListSortDirection direction)
at System.Windows.Forms.DataGridView.OnColumnHeaderMouseClick(DataGridViewCellMouseEventArgs e)
at System.Windows.Forms.DataGridView.OnMouseClick(MouseEventArgs e)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.DataGridView.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

************** Loaded Assemblies **************
CsvQuery
Assembly Version: 0.1.5782.17473
Win32 Version: 0.1.5782.17473
CodeBase: file:///C:/Program%20Files%20(x86)/Notepad++/plugins/CsvQuery.dll

feature request: add option to not reset the sql query on action "Read file"

My use case is that I often need to reload a file or load a new file with the same kind of contents.
I need to use the same sql query on these files.
When I press "Read File", the sql query is reset to "SELECT * FROM THIS"
And I have to set it back to the query I need to use.
I would like an option that disables the reset of the query, either via an option in the Settings dialog, or via a checkbox next to the "Read File" button.

Add support for horizontal scroll

By saying horizontal scroll, I mean wheel left/right or shift+wheel up/down
Thanks~

multiple file tabs for query joins?

Hi, is it possible to do joins against other files or file tabs?

eg.

SELECT *
FROM Customer.csv
INNER JOIN Orders.csv
ON Customer.csv.ID = Orders.csv.ID

I know this is getting a little complex but it would be a nice feature.

Multiline value has no delimiters between items

Multiline value has no delimiters.

Please find attached screenshot.
First column of first line (except header line) contains multiline value in double quotes:

a;b;c
"You should see this
multiline
text
in
one row
of
column a";Text for b;Text for c

Words before newline delimiter has no space after, f.e. words text and in are joined to single textin and so on.

Export to CSV

Add ability to export the query result to a new (or update an existing?) CSV file.

Preferably the user should be able to set separator.

A column named 'Something' already belongs to this DataTable

When running a query that results in two columns having the same name (e.g. a self-join with select *) will cause an error popup.

Example data:

12,hej
32,san
99,foo

Run the query SELECT * FROM THIS A JOIN THIS B ON A.Col1 = B.Col1
Results in the error

Error when executing an action: A column named 'Col1' already belongs to this DataTable

Proposed fix:
Make sure all columns has unique name when filling up the DataTable with the query results

ROW_NUMBER not supported

Hi! Is the ROW_NUMBER function actually supported? Any alternatives? Thanks!

Would be great to have generic ColN fieldnames when first line is not header

Hi,
This is potentially a great tool for me (and will probably still be quite useful even without the feature I'm talking about).

Only problem for me is that the first line of my CSV file is not a columns def and I don't want to have to add one to my file each time.
It would be really useful for me if I could have ColN as the fieldname if the first row is not a column def.

Invisible icon in toolbar when using new fluent icon style (npp v8 onwards)

Using Default Style:

Using new style:

Saw below comment on the compare plugin page:

specifically, plugin authors need to call NPPM_ADDTOOLBARICON_FORDARKMODE instead of NPPM_ADDTOOLBARICON if Notepad++ v8.0 or newer. (Actually, I don't think it hurts to call both, to keep the plugin compatible with older versions...)

[Suggestions] CsvQuery Combo box optimizations

I'd like to advance the following proposal about the behavior of the CSV Query combo box where the user types the sql to execute

• Every time the query is executed by hitting ENTER on the keyboard or clicking on the Execute button, the Box used to write the query looses the focus and must be clicked again to regain focus and to correct the query or type a one, Such behavior is not much good: IMHO would be better to let the focus and the caret into the field unless the user hits TAB or clicks elsewhere
• Please make the box multi-line capable so that the typed sql can be subdivided into multiple lines and be more comprehensible to write and analyse, especially on complex long queries one liner can be more prone to errors and more difficult to correct: indeed a query typing like this
  SELECT * FROM THIS WHERE FIELD1 = 'FILTERCRITERIA' SORT BY FIELD2 ASC
  is far less comprehensible than the same query typed like this
  SELECT * FROM THIS
  WHERE FIELD1 = 'FILTERCRITERIA'
  SORT BY FIELD2 ASC
• On the left side of the grid view Show row numbers by default
• Implement a feature to allow direct editing of the data contained into cells just by double-clicking on a target cell: after exiting the cell the csv file has to be updated accordingly in the usual notepad view.
• Implement a feature to allow update of data by using UPDATE queries
• Make CsvQuery use the SQLLite.dll library as separated file to just coping / installing it into the same directory of the CsvQuery.dll so in case of need can be updated on its own just replacing the old sqllite.dlll file with the updated one.

Thank you.

Not Sorting

v1.2.9

Simple data set

Not sorting

feature request: make sqlquery field combo box instead of text box

The text box for the sql query has autocomplete, which is nice, but I would prefer not to type the same thing over again so I can select it from an autocomplete list.
Changing the text box to a combo box, that has all previous queries AND autocomplete, would be nice.

CSVQuery not working on Notepad++ 8.4.8

Hi

CSVQuery 1.2.9 is not working on Notepad++ 8.4.8. When I select Toggle CSV Query Window, nothing happens. There is no error but the split screen parsing into columns no longer happens.

Latest Notepad++ 8.2.2 dev builds crash

Using the latest Notepad 8.2.2 development build and opening the CSV Query window causes Notepad++ to crash. See this thread for info on other plugins that are affected as well.

It may be as simple as recompiling with the latest headers from Notepad++ project:

• notepad-plus-plus\scintilla\include\Scintilla.h
• notepad-plus-plus\scintilla\include\Sci_Position.h

Cheers.

OutOfMemoryException when parsing large files

CsvQuery gets OutOfMemoryException even though the files are not that big (e.g. 50MB file).

The problem is the conversion of text from Scintilla to C# - right now the rather inefficient Encoding.GetString(byte[]) is used, and it first copies the memory from N++ into a byte[] (which in turn makes N++ to first reorganize it's memory so it's linear).

Crash dump:
OutOfMemory.txt

Possible ways to fix it:

Chunked reading - read the text in smaller chunks.
Easy to implement, just make sure to remember that UTF-8 chars might span the border between chunks.

Unmanaged reading using pointers
Scintilla can give us a pointer to the text instead of copying it into a byte[]. However, that's a pointer into UTF-8 data, which C# (or Windows) doesn't have any good functions to read.
Pro: Probably much faster
Cons: Not managed to find a way to actually do it without copying the data several times (ideally only a single copy from the Scintilla pointer to the C# String internal widechar data should be enough).

Update Command Not Supported

Error configuring the SQLite database

After plugin installation, and open notepad++, this dialog pop up. I guess a library is missing or version mismatch. Is there any advise? Any help would be much appreciated. Thanks.

Platform: Win 7 x64
Notepad++ version: 7.5.8 x64
Plugin release version: 1.2.6 x64

SQL newline and whitespace support, plans for newer sqlite libraries

Hi, CSV Query is an excellent tool. I now use it almost daily.

I just have a few questions about possible enhancements:

• The current text box for executing the queries is just a little too small when utilising long statements without its support for newlines and whitespace. It is a little difficult to view recent statements. Has there been any thought to extend this functionality to include whitespace support? And somewhat related, is there any way to execute a query to the running sqlite instance apart from using the provided text area and execute button?
• Is there any plan or appetite to incorporate a newer build of SQLite into CSVQuery? I note that the C# SQLite library you use has not been updated in a while and I imagine that changing libraries would be difficult.

Thanks for this tool, it is very useful and versatile.

Tom

CsvQuery-v1.2.9-x64 : does not load dll on Npp 8.4.8 x64

Tried manual install, or from pluginManager.

I get a message :

Failed to load

CsvQuery.dll not compatible with current version of Notepad++
Do you want to remove [...]

I also tried on a fresh install.

enhancement request - allow user specified field wrapper

Settings (and Manual Parse) allows user to specify the delimiter, but not the text/field wrapper applied to the file. For example, text file might use | as the column separator but each field is inside its own field wrapper of ^ - so file reads:
^field1^|^field2^|^field3^
Ability to specify this would be very useful in some scenarios (like mine) and can be disregarded by others if not applicable to them

Error when trying to open NPP w/ CSVQuery plugin

I have tried NPP 7.5.1 and 7.5.4, both threw the same error.

When I try opening NPP I get this error message. I tried installing by dropping the CSVQuery.dll into the plugins folder and also tried via the plugins manager. I tried the newest test version and 1.2.5 and 1.2.0 versions of CSVQuery.

CSV Query Error
Error configuring the SQLite database ':memory:': Method not found: '!!0[] System.Array.Empty()'. Falling back to in-memory SQLite

After this message I get a popup from NPP saying that CsvQuery.dll is not compatible with the current version of Notepad++.

feature request: Saving file settings

Saving file settings.
or for files with a specific extension, let's say * .txtsqlite
Many times called the team: "Плаги&ны->CsvQuery->Manual parse settings"
And many times mistaken: (

Сохранение настроек дла файла
или для файлов с определенным расширением, допустим *.txtsqlite
Много раз вызывал команду "Плаги&ны->CsvQuery->Manual parse settings"
И много раз ошибался :(

Columns in table is sometimes out of order

When you first view a query results with columns in one order, then do a query where you want them in another order, the previous order is remembered.

E.g. run SELECT Col1,Col2 FROM THIS
Then run SELECT Col2,Col1 FROM THIS
Expected result: Col2 should be first, then Col1
Actual result: Col1 is first, then Col2

Workaround: Run "SELECT 1 FROM THIS", then the new query. Now they are in correct order.

Possible cause: I don't tear down the grid properly between queries.

Add "Show Column Number" & "Show Column Excel Letter"

Hello,

This module is wonderfull !

Just some features missing for me, add two new entries in right-click popup menu :
"Show Column Number" starting from "0" (I'm a coder)
"Show Column Excel Letters"

Perhaps add in settings panel :

• start row count from 0 or 1 (with 1 per default to keep version compatibility)
• start column count from 0 or 1

Thank you and sorry for my bad english.

Do you want a maintainer?

Because it looks to me like you don't want to continue maintaining this repo, and I'd be (tentatively) interested in taking over.
If you are still interested in being the primary maintainer, I'm sorry for making assumptions.
I don't know that I could do much with the MSSQL aspect of this plugin, because I've only ever used the SQLite aspect, but I figure one maintainer is better than zero, so... 🤷‍♂️

Better detect header row if many rows are equal

If a file with >50 rows has one or more columns where the value is the same all the time except for the first row, or if all distinct values exist on many (>20?) rows except the value on the first row, we can assume the first row is a header.

How do I stipulate tab separator in Manual Parse Settings?

I have a TSV file where the headers are on the 5th line of the text file. The plugin is not parsing correctly. I thought I could help by stipulating that the separator was not a comma. I tried typing the tab key but that just pushes me to "Quote inside text prefix". I tried Alt9 or Alt0009 (I am on Windows 10). Nothing worked. Can you help me?

Love what your plugin does.