xo / usql

Universal command-line interface for SQL databases

xo

xo is a command-line tool to generate idiomatic code for different languages code based on a database schema or a custom query.

Installing | Building | Using | Releases

Supported languages

At the moment, xo only supports Go. Support for other languages will come soon.

How it works

In schema mode, xo connects to your database and generates code using Go templates. xo works by using database metadata and SQL introspection queries to discover the types and relationships contained within a schema, and applying a standard set of base (or customized) Go templates against the discovered relationships.

Currently, xo can generate types for tables, enums, stored procedures, and custom SQL queries for PostgreSQL, MySQL, Oracle, Microsoft SQL Server, and SQLite3 databases.

Note: While the code generated by xo is production quality, it is not the goal, nor the intention for xo to be a "silver bullet," nor to completely eliminate the manual authoring of SQL / Go code.

In query mode, xo parses your query to generate code from Go templates. It finds related tables in your database to ensure type safety.

Database Feature Support

The following is a matrix of the feature support for each database:

	PostgreSQL	MySQL	Oracle	Microsoft SQL Server	SQLite
Models	✅	✅	✅	✅	✅
Primary Keys	✅	✅	✅	✅	✅
Foreign Keys	✅	✅	✅	✅	✅
Indexes	✅	✅	✅	✅	✅
Stored Procs	✅	✅	✅	✅	✅
Functions	✅	✅	✅	✅	✅
ENUM types	✅	✅
Custom types	✅

Installing

xo can be installed via Release, via Homebrew, via AUR, via Scoop or via Go:

Installing via Release

1. Download a release for your platform
2. Extract the xo or xo.exe file from the .tar.bz2 or .zip file
3. Move the extracted executable to somewhere on your $PATH (Linux/macOS) or %PATH% (Windows)

Installing via Homebrew (macOS and Linux)

Install xo from the xo/xo tap in the usual way with the brew command:

# install
$ brew install xo/xo/xo

Installing via AUR (Arch Linux)

Install xo from the Arch Linux AUR in the usual way with the yay command:

# install
$ yay -S xo-cli

Alternately, build and install using makepkg:

# clone package repo and make/install package
$ git clone https://aur.archlinux.org/xo-cli.git && cd xo-cli
$ makepkg -si
==> Making package: xo-cli 0.4.4-1 (Sat 11 Nov 2023 02:28:28 PM WIB)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
...

Installing via Scoop (Windows)

Install xo using Scoop:

# Optional: Needed to run a remote script the first time
> Set-ExecutionPolicy RemoteSigned -Scope CurrentUser

# install scoop if not already installed
> irm get.scoop.sh | iex

# install xo with scoop
> scoop install xo

Installing via Go

Install xo in the usual Go fashion:

# install latest xo version
$ go install github.com/xo/xo@latest

Quickstart

The following is a quick overview of using xo on the command-line:

# Make an output directory for generated code.
$ mkdir -p models

# Generate code from your Postgres schema. (Default output folder is models)
$ xo schema postgres://user:pass@host/dbname

# Generate code from a Microsoft SQL schema using a custom template directory (see notes below)
$ mkdir -p mssqlmodels
$ xo schema mssql://user:pass@host/dbname -o mssqlmodels --src custom/templates

# Generate code from a custom SQL query for Postgres
$ xo query postgres://user:pass@host/dbname -M -B -2 -T AuthorResult << ENDSQL
SELECT
  a.name::varchar AS name,
  b.type::integer AS my_type
FROM authors a
  INNER JOIN authortypes b ON a.id = b.author_id
WHERE
  a.id = %%authorID int%%
LIMIT %%limit int%%
ENDSQL

# Build generated code - verify it compiles
$ go build ./models/
$ go build ./mssqlmodels/

Command Line Options

The following are xo's command-line commands, arguments, and options:

$ xo --help-long
usage: xo [<flags>] <command> [<args> ...]

Flags:
      --help     Show context-sensitive help (also try --help-long and
                 --help-man).
  -v, --verbose  enable verbose output
      --version  display version and exit

Commands:
  help [<command>...]
    Show help.


  query [<flags>] <DSN>
    Generate code for a database custom query from a template.

    -s, --schema=<name>            database schema name
    -t, --template=go              template type (createdb, dot, go, json, yaml;
                                   default: go)
    -f, --suffix=<ext>             file extension suffix for generated files
                                   (otherwise set by template type)
    -o, --out=models               out path (default: models)
    -a, --append                   enable append mode
    -S, --single=<file>            enable single file output
    -D, --debug                    debug generated code (writes generated code
                                   to disk without post processing)
    -Q, --query=""                 custom database query (uses stdin if not
                                   provided)
    -T, --type=<name>              type name
        --type-comment=""          type comment
    -F, --func=<name>              func name
        --func-comment=""          func comment
    -M, --trim                     enable trimming whitespace
    -B, --strip                    enable stripping type casts
    -1, --one                      enable returning single (only one) result
    -l, --flat                     enable returning unstructured values
    -X, --exec                     enable exec (no introspection performed)
    -I, --interpolate              enable interpolation of embedded params
    -L, --delimiter=%%             delimiter used for embedded params (default:
                                   %%)
    -Z, --fields=<field>           override field names for results
    -U, --allow-nulls              allow result fields with NULL values
    -d, --src=<path>               template source directory
    -2, --go-not-first             disable package comment (ie, not first
                                   generated file)
        --go-int32=int             int32 type (default: int)
        --go-uint32=uint           uint32 type (default: uint)
        --go-pkg=<name>            package name
        --go-tag="" ...            build tags
        --go-import="" ...         package imports
        --go-uuid=<pkg>            uuid type package
        --go-custom=<name>         package name for custom types
        --go-conflict=Val          name conflict suffix (default: Val)
        --go-initialism=<val> ...  add initialism (i.e ID, API, URI)
        --go-esc=none ...          escape fields (none, schema, table, column,
                                   all; default: none)
    -g, --go-field-tag=<tag>       field tag
        --go-context=only          context mode (disable, both, only; default:
                                   only)
        --go-inject=""             insert code into generated file headers
        --go-inject-file=<file>    insert code into generated file headers from
                                   a file
        --go-legacy                enables legacy v1 template funcs
        --go-enum-table-prefix     enables table name prefix to enums
        --json-indent="  "         indent spacing
        --json-ugly                disable indentation

  schema [<flags>] <DSN>
    Generate code for a database schema from a template.

    -s, --schema=<name>            database schema name
    -t, --template=go              template type (createdb, dot, go, json, yaml;
                                   default: go)
    -f, --suffix=<ext>             file extension suffix for generated files
                                   (otherwise set by template type)
    -o, --out=models               out path (default: models)
    -a, --append                   enable append mode
    -S, --single=<file>            enable single file output
    -D, --debug                    debug generated code (writes generated code
                                   to disk without post processing)
    -k, --fk-mode=smart            foreign key resolution mode (smart, parent,
                                   field, key; default: smart)
    -i, --include=<glob> ...       include types (<type>)
    -e, --exclude=<glob> ...       exclude types/fields (<type>[.<field>])
    -j, --use-index-names          use index names as defined in schema for
                                   generated code
    -d, --src=<path>               template source directory
        --createdb-fmt=<path>      fmt command (default:
                                   /home/ken/.npm-global/bin/sql-formatter)
        --createdb-fmt-opts=<opts> ...
                                   fmt options (default: -u, -l={{ . }}, -i=2,
                                   --lines-between-queries=2)
        --createdb-constraint      enable constraint name in output (postgres,
                                   mysql, sqlite3)
        --createdb-escape=none     escape mode (none, types, all; default: none)
        --createdb-engine=""       mysql table engine (default: InnoDB)
        --createdb-trim-comment    trim leading comment from views and procs
                                   (--no-createdb-trim-comment)
        --dot-defaults="" ...      default statements (default: node
                                   [shape=none, margin=0])
        --dot-bold                 bold header row
        --dot-color=""             header color (default: lightblue)
        --dot-row=""               row value template (default: {{ .Name }}: {{
                                   .Type.Type }})
        --dot-direction            enable edge directions
    -2, --go-not-first             disable package comment (ie, not first
                                   generated file)
        --go-int32=int             int32 type (default: int)
        --go-uint32=uint           uint32 type (default: uint)
        --go-pkg=<name>            package name
        --go-tag="" ...            build tags
        --go-import="" ...         package imports
        --go-uuid=<pkg>            uuid type package
        --go-custom=<name>         package name for custom types
        --go-conflict=Val          name conflict suffix (default: Val)
        --go-initialism=<val> ...  add initialism (i.e ID, API, URI)
        --go-esc=none ...          escape fields (none, schema, table, column,
                                   all; default: none)
    -g, --go-field-tag=<tag>       field tag
        --go-context=only          context mode (disable, both, only; default:
                                   only)
        --go-inject=""             insert code into generated file headers
        --go-inject-file=<file>    insert code into generated file headers from
                                   a file
        --go-legacy                enables legacy v1 template funcs
        --go-enum-table-prefix     enables table name prefix to enums
        --json-indent="  "         indent spacing
        --json-ugly                disable indentation
        --postgres-oids            enable postgres OIDs

  dump [<flags>] <out>
    Dump internal templates to path.

    -t, --template=go   template type (createdb, dot, go, json, yaml; default:
                        go)
    -f, --suffix=<ext>  file extension suffix for generated files (otherwise set
                        by template type)

About Base Templates

xo provides a set of generic "base" templates for each of the supported databases, but it is understood these templates are not suitable for every organization or every schema out there. As such, you can author your own custom templates, or modify the base templates available in the xo source tree, and use those with xo by a passing a directory path via the --src flag.

For non-trivial schemas, custom templates are the most practical, common, and best way to use xo (see below quickstart and related example).

Custom Template Quickstart

The following is a quick overview of copying the base templates contained in the xo project's templates/ directory, editing to suit, and using with xo:

# Create a working directory
$ mkdir -p my-tpl

# Dump an embedded template to disk
$ xo dump -t createdb my-tpl

# edit base template files
$ vi my-tpl/*.go.tpl

# see command line options for the template
$ xo schema --src my-tpl --help

# generate a schema using the custom template
$ xo schema --src my-tpl -o models postgres://user:pass@host/db

See the Custom Template example below for more information on adapting the base templates in the xo source tree for use within your own project.

Storing Project Templates

Ideally, custom templates for your project/schema should be stored alongside your project. and generated as part of an automated build pipeline using go generate:

# Add to custom xo command to go generate:
$ tee -a gen.go << END
package mypackage

//go:generate xo postgres://user:pass@host/db -o models --src templates
END

# Run go generate
$ go generate

# Add custom templates and gen.go to project
$ git add templates gen.go && git commit -m 'Adding custom xo templates for models'

Note: via the --template/-t parameter of xo dump you can generate other templates with xo. The default template is the go template.

Template Language/Syntax

xo templates are standard Go text templates. Please see the documentation for Go's standard text/template package for information concerning the syntax, logic, and variable use within Go templates.

Template Context and File Layout

The contexts (ie, the . identifier in templates) made available to custom templates can be found in templates/types.go (see below table for more information on which file uses which type).

Each language, has its own set of templates for $TYPE and are available in the templates/.

Template File	Type	Description
hdr.xo.*.tpl		Base template. Executed with content for a template.
db.xo.*.tpl		Package level template with base types and interface data. Generated once per package.
schema/enum.xo.*.tpl	Enum	Template for schema enum type definitions. Generates types and related methods.
schema/foreignkey.xo.*.tpl	ForeignKey	Template for foreign key relationships. Generates related method.
schema/index.xo.*.tpl	Index	Template for schema indexes. Generates related method.
schema/proc.xo.*.tpl	Proc	Template to generate functions to call defined stored procedures in the db.
schema/typedef.xo.*.tpl	Type	Template for schema table/views.
query/custom.xo.*.tpl	Query	Template for custom query execution.
query/typedef.xo.*.tpl	Type	Template for custom query's generated type.

For example, Go has templates/gotpl/schema/foreignkey.xo.go.tpl which defines the template used by xo for generating a function to get the foreign key type in Go. The templates are designed to be Database agnostic, so they are used for both PostgreSQL and Microsoft SQL the same, and all other supported database types. The template is passed a different instance of templates.ForeignKey instance (for each foreign key in a table). To get the Name field in from ForeignKey, the template can use {{ .Data.Name }}, or any other field similarly.

Examples

Example: End-to-End

Please see the booktest example for a full end-to-end example for each supported database, showcasing how to use a database schema with xo, and the resulting code generated by xo.

Additionally, please see the northwind and django for a demonstration of running xo against larger schema and against databases from other frameworks. Please note that these examples are works in progress, and may not work properly in all scenarios.

Example: Ignoring Fields

Sometimes you may wish to have the database manage the values of columns instead of having them managed by code generated by xo. As such, when you need xo to ignore fields for a database schema, you can use the -e or --exclude flag. For example, a common use case is to define a table with created_at and/or modified_at timestamps fields, where the database is responsible for setting column values on INSERT and UPDATE, respectively.

Consider the following PostgreSQL schema where a users table has a created_at and modified_at field, where created_at has a default value of now() and where modified_at is updated by a trigger on UPDATE:

CREATE TABLE users (
  id          SERIAL PRIMARY KEY,
  name        text NOT NULL DEFAULT '' UNIQUE,
  created_at  timestamptz   default now(),
  modified_at timestamptz   default now()
);

CREATEOR REPLACE FUNCTION update_modified_column() RETURNS TRIGGER AS $$
BEGIN
    NEW.modified_at= now();
RETURN NEW;
END;
$$language 'plpgsql';

CREATE TRIGGER update_users_modtime BEFORE UPDATE ON users
  FOR EACH ROW EXECUTE PROCEDURE update_modified_column();

We can ensure that these columns are managed by PostgreSQL and not by the application logic but by xo by passing the --exclude or -e flag:

# Ignore special fields
$ xo schema postgres://user:pass@host/db -e users.created_at -e users.modified_at
# or, To ignore these fields in all tables
$ xo schema postgres://user:pass@host/db -e *.created_at -e *.modified_at

Example: Custom Template -- adding a GetMostRecent lookup for all tables (Go)

Often, a schema has a common layout/pattern, such as every table having a created_at and modified_at field (as in the PostgreSQL schema in the previous example). It is then a common use-case to have a GetMostRecent lookup for each table type, retrieving the most recently modified rows for each table (up to some limit, N).

To accomplish this with xo, we will need to create our own set of custom templates, and then add a GetMostRecent lookup to the .type.go.tpl template.

First, we dump the base xo Go template:

$ mkdir -p my-tpl

$ xo dump my-tpl

We can now modify the templates to suit our specific schema, adding lookups, helpers, or anything else necessary for our schema.

To add a GetMostRecent lookup, we edit our copy of the typedef.xo.go.tpl template:

$ vi templates/gotpl/schema/typedef.xo.go.tpl

And add the following templated GetMostRecent func at the end of the file:

// GetMostRecent{{ $type.Name }} returns n most recent rows from '{{ $table }}',
// ordered by "created_at" in descending order.
func GetMostRecent{{ $type.Name }}(ctx context.Context, db DB, n int) ([]*{{ $type.Name }}, error) {
    const sqlstr = `SELECT ` +
        `{{ $type.Fields "created_at" "modified_at" }}` +
        `FROM {{ $table }} ` +
        `ORDER BY created_at DESC LIMIT $1`

    rows, err := db.QueryContext(ctx, sqlstr, n)
    if err != nil {
        return nil, logerror(err)
    }
    defer rows.Close()

    // load results
    var res []*{{ $type.Name }}
    for rows.Next() {
        {{ $short }} := {{ $type.Name }}{
        {{- if $type.PrimaryKey }}
            _exists: true,
        {{ end -}}
        }
        // scan
        if err := rows.Scan({{ fieldnames $type.Fields (print "&" $short) }}); err != nil {
            return nil, logerror(err)
        }
        res = append(res, &{{ $short }})
    }
    return res, nil
}

We can then use the templates in conjunction with xo to generate our "model" code:

$ xo schema postgres://user:pass@localhost/dbname --src templates/

There will now be a GetMostRecentUsers func defined in models/user.xo.go, which can be used as follows:

db, err := dburl.Open("postgres://user:pass@localhost/dbname")
if err != nil { /* ... */ }

// retrieve 15 most recent items
mostRecentUsers, err := models.GetMostRecentUsers(context.Background(), db, 15)
if err != nil { /* ... */ }
for _, user := range users {
    log.Printf("got user: %+v", user)
}

Using SQL Drivers

Please note that the base xo templates do not import any SQL drivers. It is left for the user of xo's generated code to import the actual drivers. For reference, these are the expected drivers to use with the code generated by xo:

Database (driver)	Package
PostgreSQL (postgres)	github.com/lib/pq
SQLite3 (sqlite3)	github.com/mattn/go-sqlite3
MySQL (mysql)	github.com/go-sql-driver/mysql
Microsoft SQL Server (mssql)	github.com/microsoft/go-mssqldb
Oracle (ora)	github.com/sijms/go-ora/v2

Additionally, please see below for usage notes on specific SQL database drivers.

MySQL (mysql)

If your schema or custom query contains table or column names that need to be escaped using any of the --escape-* options, you must pass the sql_mode=ansi option to the MySQL driver:

$ xo --escape-all 'mysql://user:pass@host/?parseTime=true&sql_mode=ansi' -o models

And when opening a database connection:

db, err := dburl.Open("mysql://user:pass@host/?parseTime=true&sql_mode=ansi")

Additionally, when working with date/time column types in MySQL, one should pass the parseTime=true option to the MySQL driver:

$ xo schema 'mysql://user:pass@host/dbname?parseTime=true' -o models

And when opening a database connection:

db, err := dburl.Open("mysql://user:pass@host/dbname?parseTime=true")

SQLite3 (sqlite3)

While not required, one should specify the loc=auto option when using xo with a SQLite3 database:

$ xo schema 'file:mydatabase.sqlite3?loc=auto' -o models

And when opening a database connection:

db, err := dburl.Open("file:mydatabase.sqlite3?loc=auto")

About Primary Keys

For row inserts xo determines whether the primary key is automatically generated by the DB or must be provided by the application for the table row being inserted. For example a table that has a primary key that is also a foreign key to another table, or a table that has multiple primary keys in a many-to-many link table, it is desired that the application provide the primary key(s) for the insert rather than the DB.

xo will query the schema to determine if the database provides an automatic primary key and if the table does not provide one then it will require that the application provide the primary key for the object passed to the Insert method. Below is information on how the logic works for each database type to determine if the DB automatically provides the PK.

PostgreSQL Auto PK Logic

• Checks for a sequence that is owned by the table in question.

MySQL Auto PK Logic

• Checks for an autoincrement row in the information_schema for the table in question.

SQLite Auto PK Logic

• Checks the SQL that is used to generate the table contains the AUTOINCREMENT keyword.
• Checks that the table was created with the primary key type of INTEGER.

If either of the above conditions are satisfied then the PK is determined to be automatically provided by the DB. For the case of integer PK's when you want to override that the PK be manually provided then you can define the key type as INT instead of INTEGER, for example as in the following many-to-many link table:

  CREATE TABLE site_contacts (
  contact_id	INT NOT NULL,
  site_id	INT NOT NULL,
  PRIMARY KEY(contact_id,siteid),
  FOREIGN KEY(contact_id) REFERENCES contacts (contact_id),
  FOREIGN KEY(site_id) REFERENCES sites (site_id)
)

SQL Server Auto PK Logic

• Checks for an identity associated with one of the columns for the table in question.

Oracle Auto PK Logic

ALWAYS GENERATED types will be parsed as Auto PK types for Oracle.

About xo: Design, Origin, Philosophy, and History

xo can likely get you 99% "of the way there" on medium or large database schemas and 100% of the way there for small or trivial database schemas. In short, xo is a great launching point for developing standardized packages for standard database abstractions/relationships, and xo's most common use-case is indeed in a code generation pipeline, ala stringer.

Design

xo is NOT designed to be an ORM or to generate an ORM. Instead, xo is designed to vastly reduce the overhead/redundancy of (re-)writing types and funcs for common database queries/relationships -- it is not meant to be a "silver bullet".

History

xo was originally developed while migrating a large application written in PHP to Go. The schema in use in the original app, while well-designed, had become inconsistent over multiple iterations/generations, mainly due to different naming styles adopted by various developers/database admins over the preceding years. Additionally, some components had been written in different languages (Ruby, Java) and had also accumulated significant drift from the original application and accompanying schema. Simultaneously, a large amount of growth meant that the PHP/Ruby code could no longer efficiently serve the traffic volumes.

In late 2014/early 2015, a decision was made to unify and strip out certain backend services and to fully isolate the API from the original application, allowing the various components to instead speak to a common API layer instead of directly to the database, and to build that service layer in Go.

However, unraveling the old PHP/Ruby/Java code became a large headache, as the code, the database, and the API, all had significant drift -- thus, underlying function names, fields, and API methods no longer coincided with the actual database schema, and were named differently in each language. As such, after a round of standardizing names, dropping cruft, and adding a few relationship changes to the schema, the various codebases were fixed to match the schema changes. After that was determined to be a success, the next target was to rewrite the backend services in Go.

In order to keep a similar and consistent workflow for the developers, the previous code generator (written in PHP and Twig templates) was modified to generate Go code. Additionally, at this time, but tangential to the story, the API definitions were ported from JSON to Protobuf to make use of its code generation abilities as well.

xo is the open source version of that code generation tool, and is the fruits of those development efforts. It is hoped that others will be able to use and expand xo to support other databases -- SQL or otherwise -- and that xo can become a common tool in any Go developer's toolbox.

Goals

Part of xo's goals is to avoid writing an ORM, or an ORM-like in Go, and to instead generate static, type-safe, fast, and idiomatic Go code across languages and databases. Additionally, the xo developers are of the opinion that relational databases should have proper, well-designed relationships and all the related definitions should reside within the database schema itself: ie, a "self-documenting" schema. xo is an end to that pursuit.

Related Projects

• dburl - a Go package providing a standard, URL style mechanism for parsing and opening database connection URLs
• usql - a universal command-line interface for SQL databases

Other Projects

The following projects work with similar concepts as xo:

Go Generators

• ModelQ
• sqlgen
• squirrel
• scaneo
• acorn and rootx (read overview here)

Go ORM-likes

• sqlc

usql's Issues

Can I ask you a favor?

Hi Ken

I am the Program Manager in the Microsoft Big Data Team who manages the U-SQL Language project.

I have noticed that you are using the name usql for a cool project providing a unified SQL command line tool.

Unfortunately, we have a SQL like language called U-SQL (often hashtagged with #USQL since twitter cannot handle – in references) for which we have a trademark. Both these names are in the same general space, and based on the comments on hacker news, it seems that it can lead to confusion (besides the search collisions on GitHub).

Can I kindly ask you to change the name of your project? If you do, I am more than happy to help with spreading the news about the tool and its new name on my blog and twitter account.

Thanks
Michael

Can I enable transaction by default?

Very scared of accidentally deleting wrong data

Henplus has rc file to set not to auto commit, hope this does as well.

[Feature Request] Output as json

Is it possible to output rowsets as json instead of a table?

It would be useful when running cli statements piping to other commands.

Dependency Manager

Why not use a dependency Manager? It can help to have future problems of update packages that you use.

• godep (https://github.com/tools/godep)

Connected with driver * should be written in stderr

I think that would be useful to write anything that is not a SQL result to STDERR.

This is useful when you are writing scripts like:
echo "show tables;" | ./usql mysql://user:pass@localhost:port/db 2> /dev/null

I woul like to get just the query results.

Avatica SQL driver has moved

Hey,

I am the author of the avatica database/sql driver. The driver has been donated to the Apache Calcite project (under the Apache Foundation). The new repository is now at https://github.com/apache/calcite-avatica-go.

I don't have enough bandwidth to make a PR at the moment, but would appreciate if you can update the import paths in usql.

Cheers,
Francis

Oracle query output is getting garbled

Hello
I am observing a strange behavior with usql. I am running usql on ubuntu 16.04 LTS. I am connecting to an oracle instance. I issue a simple query and the output comes as expected, clear and clean. I issue the same query, and typically first two columns are garbled. portion of second column appears as appended to first column. If I execute the query again 3-4 times, display becomes OK but again gets garbled.

It is making the output unreliable as it becomes garbage.

I have syntax highlight set to False. For your reference I am attaching the screenshot. Please let me know if you need anything else.

Will appreciate a quick response so that I can continue using this useful tool.

regards
Kishore

Cannot to connect to local Postgres (ssl required)

[pavel@localhost ~]$ ./usql pg://pavel@localhost/template1
Type "help" for help.

pg:pavel@localhost/template1=> select 10;
error: pq: SSL is not enabled on the server

help suggest \h for SQL help, but it is an invalid command

Look:

pg:...=> help
You are using usql, the universal command-line interface for SQL databases.
Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help with usql commands
       \g or terminate with semicolon to execute query
       \q to quit
pg:...=> \h
Invalid command \h. Try \? for help.
pg:...=>  

Killed: 9 on macOS

I tried compiling it from source and I was able to compile it without errors.
However when executed it just outputs: Killed 9.

$ usql mysql://root@localhost/test
Killed: 9
$ usql
Killed: 9

os version:

System Version: macOS 10.12.4 (16E195)

go version

$ go version
go version go1.8 darwin/amd64

$ go get -u -v github.com/knq/usql
github.com/knq/usql (download)
github.com/alexflint/go-arg (download)
github.com/knq/dburl (download)
github.com/chzyer/readline (download)
github.com/mattn/go-isatty (download)
github.com/olekukonko/tablewriter (download)
github.com/mattn/go-runewidth (download)
github.com/denisenkom/go-mssqldb (download)
Fetching https://golang.org/x/crypto/md4?go-get=1
Parsing meta tags from https://golang.org/x/crypto/md4?go-get=1 (status code 200)
get "golang.org/x/crypto/md4": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/md4?go-get=1
get "golang.org/x/crypto/md4": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto?go-get=1
Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code 200)
golang.org/x/crypto (download)
Fetching https://golang.org/x/net/context?go-get=1
Parsing meta tags from https://golang.org/x/net/context?go-get=1 (status code 200)
get "golang.org/x/net/context": found meta tag main.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/context?go-get=1
get "golang.org/x/net/context": verifying non-authoritative meta tag
Fetching https://golang.org/x/net?go-get=1
Parsing meta tags from https://golang.org/x/net?go-get=1 (status code 200)
golang.org/x/net (download)
github.com/go-sql-driver/mysql (download)
github.com/lib/pq (download)
github.com/knq/xoutil (download)
github.com/mattn/go-sqlite3 (download)

Error when trying to login to Oracle

$ ./bin/usql or://user:pass@MYDB
error: ora: ORA-00942: table or view does not exist

My guess is it fails at this query:

SELECT version FROM V$INSTANCE;

Any ideas?

Oracle On The Roadmap?

Just curious to know if a handler for Oracle is on the roadmap.

Brew formula

Hello!
Thanks for the tool. Are you planning to create homebrew formula to install it on mac?
If not really, would you mind if I add it mysqlf to my nettools repo ?

Common Table Expressions

I could be wrong, but I don't think usql will work with https://technet.microsoft.com/en-us/library/ms190766(v=sql.105).aspx since those statements do not start with an INSERT or SELECT. Am I wrong?

variable assignment while NOT in interactive mode

I need to run a script using -f option in non-interactive mode at command line. This script needs to have some value for query filters. I tried the following

driverFile.sql
\prompt filterValue 'Filter Value for Query '
\i actualSQL.sql

actualSQL.sql
select count(*) from someTable where filterColumn = :'filterValue'

And when I execute it, following error is generated
error: not interactive

How do I use a variable and make script execution more dynamic using usql with -f and -o command line arguments?

[feature] Cassandra support

Hi, I saw Cassandra in future plan and I'd like to give it a shot, I got some questions before start writing the code.

• The cassandra driver gocql does not implement database/sql interface, there is a thin wrapper. I think it might be better to write the wrapper in usql to reduce dependency
• Cassandra's CQL has some differences with SQL, is there any SQL specific logic I need to be aware of (completion etc.)?
  • Cassandra does not have transaction, it does have batch, which is BEGIN, APPLY but no ROLLBACK
• testing, it seems test is also in todo list

Connect to local host with local user by default

Feature request: Mirror the psql behaviour of connection defaults:

• "psql will connect via a Unix-domain socket to a server on the local host"
• "The default user name is your operating-system user name, as is the default database name"

This makes it really easy to fire up psql to do quick testing and prototyping, and I think it would be a useful feature to have in usql.

typed text isn't visible (need a way to select dark-background or light-background palettes)

The typed text "help" below wasn't visible in the terminal.

$ usql sqlite:/var/db/pkg/local.sqlite
Connected with driver sqlite3 (SQLite3 3.21.0)
Type "help" for help.

sq:/var/db/pkg/local.sqlite=> help
You are using usql, the universal command-line interface for SQL databases.
Type:  \copyright for distribution terms
       \? for help with usql commands
       \g or terminate with semicolon to execute query
       \q to quit
sq:/var/db/pkg/local.sqlite=>  

Add support for interactive password input.

Some passwords may contain ':', '@', '/' and other special characters. Also, passwords in url may cause secure issues.

It would be nice to have an option like -W in psql to input passwords interactively.

defining a variable and using it inside the script

I have a SQL script which I want to run using usql. The script runs a query where a comparison value for a condition should be user supplied like

select * from listing where city = 'someUSERsuppliedValue'

A preferred approach will be to have a variable defined at the command line and use the variable in the script.

How can I accomplish this in usql?

Appreciate the help very much

U-SQL Unit Test for Stored Procedures

Hi,
I am trying to create U-SQL unit tests for some U-SQL stored procedures, do you have a simple example that I can follow?
Many thanks.

Support to open last command in $EDITOR

Hi I'm starting to use usql I love it. Thank you.

It would be great if you could open the last entered command in an editor, save the command and execute it. Will look into it later this week.

Allow Postgres without SSL?

In my local dev setup, SSL is not enabled in PG. It throws this error, in this case:

usql pg://localhost/agoranhs
Type "help" for help.

pg:localhost/agoranhs=> select * from events limit 2;
error: pq: SSL is not enabled on the server
pg:localhost/agoranhs=> ^D

variable assignment is not working in sql server (issue #45)

Using usql to connect to a sql server instance and would like to use a variable. But not able to get the value of the variable passed as single quoted string. The following is the output:

[Connected with driver mssql (Microsoft SQL Server 11.0.2100.60, RTM, Enterprise Edition: Core-based Licensing (64-bit))
Type "help" for help.

ms:[email protected]:1433/sfstage=> select 'someValue' colName;
   colName   
+-----------+
  someValue  
(1 rows)

ms:[email protected]::1433/sfstage=> \set userSupplied 'someValue'
ms:[email protected]::1433/sfstage=> \set
SHOW_HOST_INFORMATION = 'false'
SYNTAX_HL = 'false'
SYNTAX_HL_FORMAT = 'terminal256'
SYNTAX_HL_OVERRIDE_BG = 'true'
SYNTAX_HL_STYLE = 'monokai'
TIME_FORMAT = '2006-01-02T15:04:05.999999999Z07:00'
userSupplied = 'someValue'
ms:[email protected]::1433/sfstage=> select ':userSupplied' colName;
     colName     
+---------------+
  :userSupplied  
(1 rows)

The second select should output exactly same as the first one.

What am I doing wrong

How do I turn off the row count output from a select statement

How do I turn off the row count output

[Connected with driver mssql (Microsoft SQL Server 11.0.2100.60, RTM, Enterprise Edition: Core-based Licensing (64-bit))
Type "help" for help.

ms:[email protected]:1433/sfstage=> select 'someValue' colName;
   colName   
+-----------+
  someValue  
(1 rows)

would like to turn off the "(1 rows)" feedback line.

Provide executable for platforms

Hi,

Thank you for an excellent tool and sharing with the community, would you be open to providing an executable for windows and other platforms. I would like to use it without having to install the Go language on the host.

Can't get oracle to work?

nevermind, looks like golang errors fucking silently when package dependencies fail to install? god that's terrible.

find the broken shit dependencies after you run go get -u -tags oracle github.com/knq/usql:
go list -json ... | grep "Err"

failed dependencies and their fixes:
github.com/alexbrainman/odbc

apt install unixodbc unixodbc-dev

sqlite: schema?

Doesn't seem to work:

~/Documents $ sqlite3 mail.db  
SQLite version 3.24.0 2018-06-04 19:24:41
Enter ".help" for usage hints.
sqlite> .schema alias
CREATE TABLE IF NOT EXISTS "alias" (
  "domain_id" integer NOT NULL,
  "recepient" varchar(256) NOT NULL,
  "username" varchar(128) NOT NULL,
  "updated_at" timestamp NOT NULL DEFAULT(CURRENT_TIMESTAMP),
  "created_at" timestamp NOT NULL
);
sqlite> 
~/Documents $ usql mail.db 
Connected with driver sqlite3 (SQLite3 3.23.1)
Type "help" for help.

sq:mail.db=> .schema alias
sq:mail.db->  

[Feature Request] Autocomplete

It would be nice to see some autocomplete when introducing files after \i
I would ask for autocomplete sql keywords, table names and column names (as in psql), but it seems like a lot to do 😄

ascii hosting provider isn't mobile friendly

i know this isn't ur problem at all, this is just the 5th really cool repo i've seen recently using this ascii video provider instead of a gif (or something like youtube). and it really just sucks on mobile. had to say something.

odbc text driver issues

I'm having trouble getting an ODBC text driver to work:

usql "odbc://Microsoft+Text+Driver+(.txt+.csv)/?Dbq=c:/temp"
error: adodb: Exception occurred. ([Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified)

usql "odbc://Microsoft+Access+Text+Driver+(.txt,+.csv)/?Dbq=c:/temp"
error: adodb: Exception occurred. ([Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified)

From: https://www.connectionstrings.com/microsoft-text-odbc-driver/

Any assistance would be greatly appreciated

Binary Oracle driver?

Is the Oracle driver available as an external binary download? I am not able to build usql with Oracle support...

usql: cannot execute binary file

Hi,
First, thank you for coming up with a great idea like universal sql command line tool. I love it.

I am trying to install/setup it on Amazon Linux on AWS. After install and set hte $PATH, i tried to run a command against postgresql as document suggested.
I ran into this error: usql: cannot execute binary file

Can you provide steps to overcome it for CentOs linux?

Thanks in advance
Rajesh C

not able to login to an oracle instance using .usqlpass entry...

I am able to login to an oracle instance using usql and dburl at the command line. But if I put the password entry in ~/.usqlpass file, it complains about incorrect password. My assumption is that may be oracle entry in .usqlpass file has a different format

Following is the format of the oracle password entry in .usqlpass

or:ORACLE_HOST:SQLNET_PORT:1521:ORACLE_SID:USER:PASSWORD

my usql command line is

$ usql or://USERi@ORACLE_HOST:SQLNET_PORT/ORACLE_SID

And it returns the following error message

error: ora: ORA-01017: invalid username/password; logon denied
Enter password:

What am I missing? (I have doubly checked the .usqlpass entry for typos)

Add support to "-v, --variable" argument

From https://www.postgresql.org/docs/7.0/static/app-psql.htm :

Performs a variable assignment, like the \set internal command. Note that you must separate name and value, if any, by an equal sign on the command line. To unset a variable, leave off the equal sign. These assignments are done during a very early state of startup, so variables reserved for internal purposes might get overwritten later.

I.e: useful to pass variables to sql scripts.

Add support for mssql instances.

Mssql can have multiple instances on one server. In that case every instance waiting for connections on its own port and you can determine port you need via 1434/udp port.

"github.com/denisenkom/go-mssqldb" driver support this feature, but when i try
usql "ms://user:pass@mssql-server\INSTANCENAME/dbname"
I got error: could not connect to database: parse ms://user:pass@mssql-server\INSTANCENAME: invalid character "\\" in host name.

go-get on Mac does install usql

I did the following:

go get -u github.com/knq/usql
usql --help #returns command not found

Buggy dollar quoting

Hello,

dollar quoting seems to be buggy in 0.6.0. Given either script:

DO
$$
BEGIN
PERFORM 1;
END
$$;

or

DO
$do$
BEGIN
PERFORM 1;
END
$do$;

With usql, see how statement ends at the first ;:

usql 'postgres://url' -f dollar-bug.sql
Connected with driver postgres (PostgreSQL 9.6.6)
error: pq: 42601: unterminated dollar-quoted string at or near "$do$
BEGIN
PERFORM 1;"
error: pq: 42601: unterminated dollar-quoted string at or near "$do$;"

But with psql, all is fine:

psql 'postgres://url' -f dollar-bug.sql                                                                                                                    DO

Vertical/Expanded row display

Can I view records vertically with column names along side data?
It's like '\X' flag for `Expanded Display' in psql(PostgreSQL).
I am exploring Sql Server 2008 database

Support context

If you use the context.Context methods that database/sql provides, you can allow the user to cancel in-flight queries or transactions in your user-interface.

Problem with installing oracle driver

I installed usql with command
$ go get -u -tags 'oracle odbc' github.com/xo/usql

It looks successfully installed and works fine with mysql.

$ usql mysql://tmp:tmp@localhost:3306
Connected with driver mysql (5.7.20-log)
Type "help" for help.

my:tmp@localhost:3306=> ^D

however when i tried to connect to oracle databases, i got following error message

$ usql oracle://[email protected]:1521/tdb
error: ora: driver not available

try:

  go get -u -tags oracle github.com/knq/usql

my environments

OS: Mac OS 10.11
go: go1.8.3 darwin/amd64

cat oci8.pc
prefix=/Users/usr0200557/bin/oracle-sdk/instantclient_11_2
libdir=${prefix}
includedir=${prefix}/sdk/include/

Name: oci8
Description: Oracle Instant Client
Version: 11.2
Libs: -L${libdir} -lclntsh
Libs.private:
Cflags: -I${includedir}

Thank you for reading!

Using usql as a library

First of all, props on an amazing project!

I'd like to utilize some of the awesome here in my own golang app. The idea is that I could wrap around usql and utilize it to drop me into a sql prompt.

Is this possible, short of shelling out to the usql binary? Which functions/methods could I use?

Add support for "all" drivers in packaged releases

The release page contains links to ready made releases of usql ( which is awesome ) however they do not contain all drivers, namely Oracle.

This was only tested on the linux tar.bz2 file, but I suspect the osx version is the same. Please update the binary releases so that they contain all drivers.

P.S. Thanks for the awesome program, I'm looking forward to using it.

Problems with building oracle version on Windows

Here is the complete setup I used to build u-sql exe on Windows with oracle support:

Environment

• Windows 10
• Powershell 5

Build

1. Installed golang and gcc stuff with chocolatey:

#iwr https://chocolatey.org/install.ps1 -UseBasicParsing | iex
choco install golang mingw pkgconfiglite
refreshenv

2. Downloaded Instant Client Downloads for Microsoft Windows (x64). All zip files extracted into the same directory c:\oracle\instantclient_12_1

3. Created file c:\oracle\oci8.pc file with the following content:

prefix=/devel/target/1.0
exec_prefix=${prefix}
libdir=C:/oracle/instantclient_12_1/sdk/lib/msvc
includedir=C:/oracle/instantclient_12_1/sdk/include
oralib=C:/oracle/instantclient_12_1/sdk/lib/msvc
orainclude=C:/oracle/instantclient_12_1/sdk/include
gcclib=c:/tools/mingw64/lib
gccinclude=c:/tools/mingw64/lib
glib_genmarshal=glib-genmarshal
gobject_query=gobject-query
glib_mkenums=glib-mkenums
Name: oci8
Version: 12.1
Description: oci8 library
Libs: -L${oralib} -L${gcclib} -loci
Libs.private:
Cflags: -I${orainclude} -I${gccinclude}

4. Run the following build command:

$Env:PKG_CONFIG_PATH="c:\oracle"  #location of oci8.pc
go get -u -tags oracle github.com/knq/usql

It builds without problems and produces ~ 13MB file in $HOME\go\bin\usql.exe
When I run it nothing happens - no error, no output, no nothing. When I build non oracle binary everything works as expected.

Using dependency walker I found out that many of the DLLs are missing:

       Module                                                        File Time Stamp   Link Time Stamp   File Size   Attr.  Link Checksum  Real Checksum  CPU  Subsystem  Symbols     Preferred Base      Actual Base  Virtual Size  Load Order  File Ver        Product Ver     Image Ver    Linker Ver  OS Ver  Subsystem Ver
-----  ------------------------------------------------------------  ----------------  ----------------  ----------  -----  -------------  -------------  ---  ---------  ----------  ------------------  -----------  ------------  ----------  --------------  --------------  -----------  ----------  ------  -------------
[ ? ]  API-MS-WIN-CORE-APIQUERY-L1-1-0.DLL                           Error opening file. The system cannot find the file specified (2).
[ ? ]  API-MS-WIN-CORE-APPCOMPAT-L1-1-1.DLL                          Error opening file. The system cannot find the file specified (2).
[ ? ]  API-MS-WIN-CORE-APPINIT-L1-1-0.DLL                            Error opening file. The system cannot find the file specified (2).
[ ? ]  API-MS-WIN-CORE-ATOMS-L1-1-0.DLL                              Error opening file. The system cannot find the file specified (2).
[ ? ]  API-MS-WIN-CORE-CRT-L1-1-0.DLL                                Error opening file. The system cannot find the file specified (2).
[ ? ]  API-MS-WIN-CORE-CRT-L2-1-0.DLL                                Error opening file. The system cannot find the file specified (2).
[ ? ]  API-MS-WIN-CORE-DATETIME-L1-1-2.DLL                           Error opening file. The system cannot find the file specified (2).
[ ? ]  API-MS-WIN-CORE-DELAYLOAD-L1-1-1.DLL                          Error opening file. The system cannot find the file specified (2).
[ ? ]  API-MS-WIN-CORE-ENCLAVE-L1-1-0.DLL                            Error opening file. The system cannot find the file specified (2).
[ ? ]  API-MS-WIN-CORE-ERRORHANDLING-L1-1-3.DLL                      Error opening file. The system cannot find the file specified (2).
[ ? ]  API-MS-WIN-CORE-FIBERS-L2-1-1.DLL                             Error opening file. The system cannot find the file specified (2).
[ ? ]  API-MS-WIN-CORE-FILE-L1-2-2.DLL                               Error opening file. The system cannot find the file
...

Looks like vcredist but I have all installed:
...

choco list -l vcredist
Chocolatey v0.10.3
vcredist140 14.10.25008.0
vcredist2008 9.0.30729.6161
vcredist2010 10.0.40219.2
vcredist2012 11.0.61031
vcredist2015 14.0.24215.20170201
5 packages installed.
...

I would appreciate any help.

Does usql support authenticating via domain to MS Sql?

One thing I hate is trying to SSMS to come up correctly when I'm connecting with Domain Auth to a domain that I don't have a trust with. Does Usql have the capability to auth via domain to a domain with no trust?

pg_dump functionality

Hi!

one killer feature of this would be to provide dump functionality. it wouldn't (at first) need to output portable SQL, as there's really no such thing: Postgres and MySQL might have some common set (especially for INSERT) but it will break down for indexes, table creations, stored procedures and whatnot.

still it would be nice to have a way to export the content of a database, even if it means reusing db-specific instructions... in my use case, i'm working with an embedded device where usql is one of the very select few commandline clients that works at all (thanks to the excellent golang cross-compilation features), so it would allow me to export (sqlite) databases from those devices and make a/b comparisons.

thanks!

oracle: number type shown in scientific notation

Example:

Column type:   NUMBER(22)
Value:         43856197
Shown by usql: 4.3856197e+07

Installation issue

Hey there! Unfortunately, current installation as it recommended in the documentation doesn't work due to (as it seems to me) dependencies issue e.g. when I do:

go get -u -tags most github.com/knq/usql

As a result I get this error message:

# cd /Users/username/.go/src/github.com/akrennmair/goauth; git pull --ff-only
You are not currently on a branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.

    git pull <remote> <branch>

package github.com/akrennmair/goauth: exit status 1

Not sure is it something I can fix locally or you could do it on your side, but currently for me the default installation just doesn't do the trick.

Just in case, my current system and env:

ProductName: Mac OS X
ProductVersion: 10.12.4
BuildVersion: 16E195
Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
go version go1.8 darwin/amd64

What is necessary to be able to implement completion?

Hello.

I just recently came to this project, and I found it to be very interesting.

I'm looking forward to the Autocomplete feature, and I intend contributing to the project to implement it.

What would be the general guidelines to have something like this implemented?

Describe command doesn't work for mysql on ubuntu 14.04

Hi --

Thanks for this nice sql tool. In general it's lightyears faster than phpmyadmin and other GUI based applications which is why I'm seeking to utilize it more in my workflows.

I've noticed the DESCRIBE command seems to be not providing useful data from my MySQL database.

I figured maybe describe was MySQL specific, but I don't see it listed here:
https://dev.mysql.com/doc/refman/5.7/en/extensions-to-ansi.html

The example test queries I'm executing are:

my:dev_user@localhost/_drupal=> describe papers_diff_06262017_prepared_inline_content;
DESCRIBE
my:dev_user@localhost/_drupal=> DESCRIBE papers_diff_06262017_prepared_inline_content;
DESCRIBE
my:dev_user@localhost/_drupal=> DESC papers_diff_06262017_prepared_inline_content;
DESC

Other system information:

my:dev_user@localhost/_drupal=> \conninfo
Connected with driver mysql (dev_user:dev@tcp(localhost:3306)/_drupal?loc=Local&parseTime=true&sql_mode=ansi)

$ ./usql --version
usql 0.5.0

$ mysql --version
mysql  Ver 14.14 Distrib 5.5.57, for debian-linux-gnu (x86_64) using readline 6.3