AlaSQL is an open source project used on more than two million page views per month - and we appreciate any and all contributions we can get. Please help out.
Have a question? Ask on Stack Overflow using the "alasql" tag.
AlaSQL - ( à la SQL ) [ælæ ɛskju:ɛl] - is an open source SQL database for JavaScript with a strong focus on query speed and data source flexibility for both relational data and schemaless data. It works in the web browser, Node.js, and mobile apps.
This library is designed for:
- Fast in-memory SQL data processing for BI and ERP applications on fat clients
- Easy ETL and options for persistence by data import / manipulation / export of several formats
- All major browsers, Node.js, and mobile applications
We focus on speed by taking advantage of the dynamic nature of JavaScript when building up queries. Real-world solutions demand flexibility regarding where data comes from and where it is to be stored. We focus on flexibility by making sure you can import/export and query directly on data stored in Excel (both .xls
and .xlsx
), CSV, JSON, TAB, IndexedDB, LocalStorage, and SQLite files.
The library adds the comfort of a full database engine to your JavaScript app. No, really - it's working towards a full database engine complying with most of the SQL-99 language, spiced up with additional syntax for NoSQL (schema-less) data and graph networks.
/* create SQL Table and add data */
alasql("CREATE TABLE cities (city string, pop number)");
alasql("INSERT INTO cities VALUES ('Paris',2249975),('Berlin',3517424),('Madrid',3041579)");
/* execute query */
var res = alasql("SELECT * FROM cities WHERE pop < 3500000 ORDER BY pop DESC");
// res = [ { "city": "Madrid", "pop": 3041579 }, { "city": "Paris", "pop": 2249975 } ]
var data = [ {a: 1, b: 10}, {a: 2, b: 20}, {a: 1, b: 30} ];
var res = alasql('SELECT a, SUM(b) AS b FROM ? GROUP BY a',[data]);
// res = [ { "a": 1, "b": 40},{ "a": 2, "b": 20 } ]
// file is read asynchronously (Promise returned when SQL given as array)
alasql(['SELECT * FROM XLS("./data/mydata") WHERE lastname LIKE "A%" and city = "London" GROUP BY name '])
.then(function(res){
console.log(res); // output depends on mydata.xls
}).catch(function(err){
console.log('Does the file exist? There was an error:', err);
});
alasql("CREATE TABLE example1 (a INT, b INT)");
// alasql's data store for a table can be assigned directly
alasql.tables.example1.data = [
{a:2,b:6},
{a:3,b:4}
];
// ... or manipulated with normal SQL
alasql("INSERT INTO example1 VALUES (1,5)");
var res = alasql("SELECT * FROM example1 ORDER BY b DESC");
console.log(res); // [{a:2,b:6},{a:1,b:5},{a:3,b:4}]
If you are familiar with SQL it should come as no surprise that proper use of indexes on your tables is essential to get good performance.
npm install --save alasql # node
bower install --save alasql # bower
npm install -g alasql # global installation for command line tools
For the browser: include alasql.min.js
<script src="https://cdn.jsdelivr.net/npm/[email protected]"></script>
See the "Getting started" section of the wiki
More advanced topics are covered in other wiki sections like "Data manipulation" and in questions on Stack Overflow
Other links:
-
Documentation: Github wiki
-
Library CDN: jsDelivr.com
-
Feedback: Open an issue
-
Try online: Playground
-
Website: alasql.org
All contributions are extremely welcome and greatly appreciated(!) - The project has never received any funding and is based on unpaid voluntary work: We really (really) love pull requests
AlaSQL project is very young and still in an active development phase, therefore it may have bugs.
Please, submit any bugs and suggestions as an issue.
AlaSQL uses Semantic Versioning so please note that the major version is zero (0.y.z) and the API can not be considered 100% stable. Consider this before using the library in production and please check out the limitations of the library
AlaSQL is designed for speed, and we have applied common SQL engine optimizations:
- Queries are cached as compiled functions
- Joined tables are pre-indexed
WHERE
expressions are pre-filtered for joins
See more performance-related info on the wiki
Use "good old" SQL on your data with multiple levels of: JOIN
, VIEW
, GROUP BY
, UNION
, PRIMARY KEY
, ANY
, ALL
, IN
, ROLLUP()
, CUBE()
, GROUPING SETS()
, CROSS APPLY
, OUTER APPLY
, WITH SELECT
, and subqueries. The wiki lists supported SQL statements and keywords.
You can use all benefits of SQL and JavaScript together by defining your own custom functions. Just add new functions to the alasql.fn object:
alasql.fn.myfn = function(a,b) {
return a*b+1;
};
var res = alasql('SELECT myfn(a,b) FROM one');
You can also define your own aggregator functions (like your own SUM(...)
). See more in the wiki
var ins = alasql.compile('INSERT INTO one VALUES (?,?)');
ins(1,10);
ins(2,20);
See more in the wiki
Group your JavaScript array of objects by field and count number of records in each group:
var data = [{a:1,b:1,c:1},{a:1,b:2,c:1},{a:1,b:3,c:1}, {a:2,b:1,c:1}];
var res = alasql('SELECT a, COUNT(*) AS b FROM ? GROUP BY a', [data] );
See more ideas for creative data manipulation in the wiki
AlaSQL extends "good old" SQL to make it closer to JavaScript. The "sugar" includes:
- Write Json objects -
{a:'1',b:@['1','2','3']}
- Acesss object propertires -
obj->property->subproperty
- Access Ooject and arrays elements -
obj->(a*1)
- Access JavaScript functions -
obj->valueOf()
- Format query output with
SELECT VALUE, ROW, COLUMN, MATRIX
- ES5 multiline SQL with
var SQL = function(){/*SELECT 'MY MULTILINE SQL'*/}
and pass instead of SQL string (will not work if you compress your code)
You can import from and export to CSV, TAB, TXT, and JSON files. File extensions can be omitted. Calls to files will always be asynchronous so multi-file queries should be chained:
var tabFile = 'mydata.tab';
alasql.promise([
"SELECT * FROM txt('MyFile.log') WHERE [0] LIKE 'M%'", // parameter-less query
[ "SELECT * FROM tab(?) ORDER BY [1]", [tabFile] ], // [query, array of params]
"SELECT [3] AS city,[4] AS population FROM csv('./data/cities')",
"SELECT * FROM json('../config/myJsonfile')"
]).then(function(results){
console.log(results);
}).catch(console.error);
AlaSQL can read (but not write) SQLite data files using SQL.js library:
<script src="alasql.js"></script>
<script src="sql.js"></script>
<script>
alasql([
'ATTACH SQLITE DATABASE Chinook("Chinook_Sqlite.sqlite")',
'USE Chinook',
'SELECT * FROM Genre'
]).then(function(res){
console.log("Genres:",res.pop());
});
</script>
sql.js
calls will always be asynchronous.
The node module ships with an alasql
command-line tool:
$ npm install -g alasql ## install the module globally
$ alasql -h ## shows usage information
$ alasql "SET @data = @[{a:'1',b:?},{a:'2',b:?}]; SELECT a, b FROM @data;" 10 20
[ 1, [ { a: 1, b: 10 }, { a: 2, b: 20 } ] ]
$ alasql "VALUE OF SELECT COUNT(*) AS abc FROM TXT('README.md') WHERE LENGTH([0]) > ?" 140
// Number of lines with more than 140 characters in README.md
More examples are included in the wiki
AlaSQL plays nice with d3.js and gives you a convenient way to integrate a specific subset of your data with the visual powers of D3. See more about D3.js and AlaSQL in the wiki
AlaSQL can export data to both Excel 2003 (.xls) and Excel 2007 (.xlsx) formats with coloring of cells and other Excel formatting functions.
Meteor is amazing. You can query directly on your Meteor collections with SQL - simple and easy. See more about Meteor and AlaSQL in the wiki
Angular is great. In addition to normal data manipulation, AlaSQL works like a charm for exporting your present scope to Excel. See more about Angular and AlaSQL in the wiki
Pinpointing data on a map should be easy. AlaSQL is great to prepare source data for Google Maps from, for example, Excel or CSV, making it one unit of work for fetching and identifying what's relevant. See more about Google Maps and AlaSQL in the wiki
AlaSQL can query data directly from a Google spreadsheet. A good "partnership" for easy editing and powerfull data manipulation. See more about Google Spreadsheets and AlaSQL in the wiki
Take charge and add your idea or vote for your favorite feature to be implemented:
Please be aware that AlaSQL has bugs. Beside having some bugs, there are a number of limitations:
-
AlaSQL has a (long) list of keywords that must be escaped if used for column names. When selecting a field named
key
please writeSELECT `key` FROM ...
instead. This is also the case for words like`value`
,`read`
,`count`
,`by`
,`top`
,`path`
,`deleted`
,`work`
and`offset`
. Please consult the full list of keywords. -
It is OK to
SELECT
1000000 records or toJOIN
two tables with 10000 records in each (You can use streaming functions to work with longer datasources - see test/test143.js) but be aware that the workload is multiplied soSELECT
ing from more than 8 tables with just 100 rows in each will show bad performance. This is one of our top priorities to make better. -
Limited functionality for transactions (supports only for localStorage) - Sorry, transactions are limited, because AlaSQL switched to more complex approach for handling
PRIMARY KEY
s /FOREIGN KEY
s. Transactions will be fully turned on again in a future version. -
A
(FULL) OUTER JOIN
andRIGHT JOIN
of more than 2 tables will not produce expected results.INNER JOIN
andLEFT JOIN
are OK. -
Please use aliases when you want fields with the same name from different tables (
SELECT a.id AS a_id, b.id AS b_id FROM ?
). -
At the moment AlaSQL does not work with JSZip 3.0.0 - please use version 2.x.
-
JOIN
ing a sub-SELECT
does not work. Please use awith
structure (Example here) or fetch the sub-SELECT
to a variable and pass it as an argument (Example here). -
AlaSQL uses the FileSaver.js library for saving files locally from the browser. Please be aware that it does not save files in Safari 8.0.
There are probably many others. Please help us fix them by submitting an issue. Thank you!
ETL example:
alasql([
'CREATE TABLE IF NOT EXISTS geo.country',
'SELECT * INTO geo.country FROM CSV("country.csv",{headers:true})',
'SELECT * INTO XLSX("asia") FROM geo.country WHERE continent_name = "Asia"'
]).then(function(res){
// results from the file asia.xlsx
});
AlaSQL can run in a Web Worker. Please be aware that all interaction with AlaSQL when running must be async.
From the browser thread, the browser build alasql-worker.min.js
automagically uses Web Workers:
<script src="alasql-worker.min.js"></script>
<script>
var arr = [{a:1},{a:2},{a:1}];
alasql([['SELECT * FROM ?',[arr]]]).then(function(data){
console.log(data);
});
</script>
The standard build alasql.min.js
will use Web Workers if alasql.worker()
is called:
<script src="alasql.min.js"></script>
<script>
alasql.worker();
alasql(['SELECT VALUE 10']).then(function(res){
console.log(res);
}).catch(console.error);
</script>
From a Web Worker, you can import alasql.min.js
with importScripts
:
importScripts('alasql.min.js');
When targeting the browser, several code bundlers like Webpack and Browserify will pick up modules you might not want.
Here's a list of modules that AlaSQL may require in certain environments or for certain features:
- Node.js
- fs
- net
- tls
- request
- path
- React Native
- react-native
- react-native-fs
- react-native-fetch-blob
- Vertx
- vertx
- Agonostic
- XLSX/XLS support
- cptable
- jszip
- xlsx
- cpexcel
- es6-promise
- XLSX/XLS support
There are several ways to handle AlaSQL with Webpack:
Ideal when you want to control which modules you want to import.
var IgnorePlugin = require("webpack").IgnorePlugin;
module.exports = {
...
// Will ignore the modules fs, path, xlsx, request, vertx, and react-native modules
plugins:[new IgnorePlugin(/(^fs$|cptable|jszip|xlsx|^es6-promise$|^net$|^tls$|^forever-agent$|^tough-cookie$|cpexcel|^path$|^request$|react-native|^vertx$)/)]
};
As of AlaSQL 0.3.5, you can simply tell Webpack not to parse AlaSQL, which avoids all the dynamic require
warnings and avoids using eval
/clashing with CSP with script-loader.
Read the Webpack docs about noParse
...
//Don't parse alasql
{module:noParse:[/alasql/]}
If both of the solutions above fail to meet your requirements, you can load AlaSQL with script-loader.
//Load alasql in the global scope with script-loader
import "script!alasql"
This can cause issues if you have a CSP that doesn't allow eval
.
Read up on excluding, ignoring, and shimming
Example (using excluding)
var browserify = require("browserify");
var b = browserify("./main.js").bundle();
//Will ignore the modules fs, path, xlsx
["fs","path","xlsx", ... ].forEach(ignore => { b.ignore(ignore) });
For some frameworks (lige Vue) alasql cant access XLSX by it self. We recommend handeling it by including AlaSQL the following way:
import XLSX from 'xlsx';
alasql.utils.isBrowserify = false;
alasql.utils.global.XLSX = XLSX;
Please remember to send the original event, and not the jQuery event, for elements. (Use event.originalEvent
instead of myEvent
)
You can use JSON objects in your databases (do not forget use == and !== operators for deep comparision of objects):
alasql> SELECT VALUE {a:'1',b:'2'}
{a:1,b:2}
alasql> SELECT VALUE {a:'1',b:'2'} == {a:'1',b:'2'}
true
alasql> SELECT VALUE {a:'1',b:'2'}->b
2
alasql> SELECT VALUE {a:'1',b:(2*2)}->b
4
Try AlaSQL JSON objects in Console [sample](http://alasql.org/console?drop table if exists one;create table one;insert into one values {a:@[1,2,3],c:{e:23}}, {a:@[{b:@[1,2,3]}]};select * from one)
Useful stuff, but there might be dragons
AlaSQL is a multi-paradigm database with support for graphs that can be searched or manipulated.
// Who loves lovers of Alice?
var res = alasql('SEARCH / ANY(>> >> #Alice) name');
console.log(res) // ['Olga','Helen']
See more in the wiki
You can use browser localStorage and DOM-storage as a data storage. Here is a sample:
alasql('CREATE localStorage DATABASE IF NOT EXISTS Atlas');
alasql('ATTACH localStorage DATABASE Atlas AS MyAtlas');
alasql('CREATE TABLE IF NOT EXISTS MyAtlas.City (city string, population number)');
alasql('SELECT * INTO MyAtlas.City FROM ?',[ [
{city:'Vienna', population:1731000},
{city:'Budapest', population:1728000}
] ]);
var res = alasql('SELECT * FROM MyAtlas.City');
Try this sample in jsFiddle. Run this sample two or three times, and AlaSQL store more and more data in localStorage. Here, "Atlas" is the name of localStorage database, where "MyAtlas" is a memory AlaSQL database.
You can use localStorage in two modes: SET AUTOCOMMIT ON
to immediate save data
to localStorage after each statement or SET AUTOCOMMIT OFF
. In this case, you need
to use COMMIT
statement to save all data from in-memory mirror to localStorage.
AlaSQL supports plugins. To install a plugin you need to use the REQUIRE
statement. See more in the wiki
Yes, you can even use AlaSQL as a very simple server for tests.
To run enter the command:
$ alaserver
then open http://127.0.0.1:1337/?SELECT%20VALUE%20(2*2) in your browser
Warning: Alaserver is not multi-threaded, not concurrent, and not secured.
AlaSQL currently has over 1200 regression tests, but they only cover of the codebase.
AlaSQL uses mocha
for regression tests. Install mocha
and run
$ npm test
or open test/index.html for in-browser tests (Please serve via localhost with, for example, http-server
).
You can use AlaSQL's ASSERT operator to test the results of previous operation:
CREATE TABLE one (a INT); ASSERT 1;
INSERT INTO one VALUES (1),(2),(3); ASSERT 3;
SELECT * FROM one ORDER BY a DESC; ASSERT [{a:3},{a:2},{a:1}];
AlaSQL uses SQLLOGICTEST
to test its compatibility with SQL-99. The tests include about 2 million queries and statements.
The testruns can be found in the testlog.
If you want to try the most recent development version of the library please download this file or visit the testbench to play around in the browser console.
MIT - see MIT licence information
AlaSQL is an OPEN Open Source Project. This means that:
Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.
We appreciate any and all contributions we can get. If you feel like contributing, have a look at CONTRIBUTING.md
Many thanks to:
- Zach Carter for Jison parser-generator
- Andrew Kent for JS SQL Parser
- Eli Grey for FileSaver.js
- SheetJS for JS XLSX Library
and other people for useful tools, which make our work much easier.
- AlaX - Export to Excel with colors and formats
- WebSQLShim - WebSQL shim over IndexedDB (work in progress)
- AlaMDX - JavaScript MDX OLAP library (work in progress)
- Other similar projects - list of databases on JavaScript
© 2014-2018, Andrey Gershun ([email protected]) & Mathias Rangel Wulff ([email protected])
alasql's People
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.