Podman/Docker image for tokenize identifiers and/or literals of SQL queries (MySQL 8 dialect). This allows you to get a SQL query string with unique literals and/or unique identifiers replaced with tokens, which can be useful in cases of further analysis of the SQL query.
ANTLR Project antlr4 4.12.0 using BSD-3-clause license
Python Software Foundation Python 3.11.3 using PSF license
Positive Technologies MySQL grammar using MIT License
Clone the repository to your environment using git:
git clone https://github.com/vreitech/mysqltokenizer.git
cd mysqltokenizer
Build mysqltokenizer image using buildah:
buildah bud -t mysqltokenizer
Run mysqltokenizer contaner using podman and tokenize SQL query by 4 different ways:
echo 'select * from `some_table` order by desc limit 10;' > ./example.sql
# Tokenize everything (identifiers and literals)
cat ./example.sql | podman run -i --rm localhost/mysqltokenizer
# Tokenize, but left identifiers untouched
cat ./example.sql | podman run -i --rm localhost/mysqltokenizer -i
# Tokenize, but left literals untouched
cat ./example.sql | podman run -i --rm localhost/mysqltokenizer -l
# Tokenize, but left identifiers and literals untouched
# (i. e. just capitalize ther rest of tokens and remove line feeds and extra spaces)
cat ./example.sql | podman run -i --rm localhost/mysqltokenizer -il
Run mysqltokenizer contaner using podman and tokenize SQL query by 4 different ways:
echo 'select * from `some_table` order by desc limit 10;' > ./example.sql
# Tokenize everything (identifiers and literals)
cat ./example.sql | podman run -i --rm docker.io/vreitech/mysqltokenizer
# Tokenize, but left identifiers untouched
cat ./example.sql | podman run -i --rm docker.io/vreitech/mysqltokenizer -i
# Tokenize, but left literals untouched
cat ./example.sql | podman run -i --rm docker.io/vreitech/mysqltokenizer -l
# Tokenize, but left identifiers and literals untouched
# (i. e. just capitalize ther rest of tokens and remove line feeds and extra spaces)
cat ./example.sql | podman run -i --rm docker.io/vreitech/mysqltokenizer -il
If you decided to pull the image from GitHub package storage you can replace docker.io
onto ghcr.io
in image URI.