tdslite / tdslite

Lightweight, platform independent, embedded-ready Microsoft SQL Server (MSSQL) Connector written in pure C++11 that can work with just 2kB of SRAM!

---

Lightweight, platform independent Microsoft SQL Server (MSSQL) connector (MS-TDS implementation) written in pure C++11 that can work with just 2kB of SRAM!

---

Built around C++'s zero cost / only pay for what you use mantra. The implementation is based on Microsoft's MS-TDS: Tabular Data Stream Protocol technical specification (revision number 33).

---

Getting started

Take a look into examples below to get your hands dirty!

➤ 01 - Initialize Library

Example code illustrating how to initialize the library.

➤ 02 - CRUD with tdslite

The four basic operations: Create, read, update, delete.

➤ 03 - Read rows from table

Retrieve a result set with a query (a.k.a SELECTing rows)

➤ 04 - Read info/error messages

How to use info callback function to read info/error messages sent by the server.

➤ 05 - Execute query with parameters

Learn how to use and bind parameters in queries with tdslite.

➤ 06 - Custom memory allocator functions

Learn how to set user-defined malloc/free functions for tdslite's memory allocation.

➤ ESP WiFi example

tdslite, but with WiFiClient instead of EthernetClient.

➤ Fiddle with minimal-sql-shell

It's a playground to try sql commands. Compile the project, and just run ./build/bin/tdslite.examples.minimal executable.

➤ Sketches

Verificaton sketches for the library

➤ Board-specific examples

• Arduino Uno WiFi rev.2: Uses WiFiNINA & tdslite to perform INSERT & SELECT.

---

Key features

• Pure C++11, header-only library
• Zero external dependencies
• Dead simple, intuitive API
• Permissive open-source license (MIT)
• Suitable for embedded development
• Low memory footprint
• Decoupled networking design that can be easily extended
• Callback based design that reduces the memory consumption
• Supports networking interfaces that implement Arduino's EthernetClient.
• Supports:
  • ... query execution driver.execute_query(...)
  • ... queries with parameters driver.execute_rpc(...)
  • ... reading result sets

---

Installing the library

Import the tdslite-<version>.zip into your libraries. See releases.

... via Platform IO

    pio pkg install --library "mustafakemalgilor/tdslite"

See the official PlatformIO registry page for details.

... via the Arduino IDE library manager

Just search for tdslite, and install it.

Also, take a look at ardu-badge page for a detailed installation guide.

... via the Arduino IDE, manually

Sketch -> Include Library -> Add .ZIP Library... and then select the downloaded tdslite-<version>.zip file.

The goal

The main design goal of the project is to enable a wide variety of constrained devices (e.g. embedded, IoT) to talk to a TDS server (e.g. Microsoft SQL Server). The aim is not to implement the TDS standard down to every single letter. Most of the use cases do not require many legacy features that the TDS standard carries along, so the remaining feature set that TDS offer will be implemented on an as-needed basis, rather than being implemented upfront.

tdslite is:

• a lightweight, production-ready implementation that can work in embedded environments
• agnostic from any particular architecture, platform, environment or standard library implementation (*except for a few basic things).
• not embedded-centric, but embedded-suitable

The design

tdslite is made from two main parts: the tdslite library, and the networking implementation. They're located at tdslite and tdslite-net folders respectively.

The main user-facing class of tdslite is the tdslite::driver<T> class, where the T is the networking implementation of choice. tdslite does not depend on any concrete networking implementation and can work with anything that satisfies the constraints defined in the network_io_contract class.

The tdslite-net library provides two of such networking implementations; one of them is based on Boost.ASIO (tdslite-net-asio), and the other one being Arduino EthernetClient interface compatible one(tdslite-net-arduino). ASIO-based networking implementation is used for the integration tests and the example sql shell application. tdslite-net-arduino can work with anything that implements the EthernetClient interface (e.g. EthernetClient, WiFiClient):

    #include <Ethernet.h>
    #include <WiFi.h>
    #include <tdslite.h>

    tdsl::uint8_t net_buf[1024] = {};

    /* You can use the arduino_driver with EthernetClient: */
    tdsl::arduino_driver<EthernetClient> driver_e{net_buf};

    /* ... or with WiFiClient: ... */
    tdsl::arduino_driver<WiFiClient> driver_w{net_buf};

    /* ... or, even with your own, custom type that implements the EthernetClient interface: */
    struct my_client {
        int connect(const char * host, unsigned short port) ;
        bool connected();
        int available();
        void flush();
        void stop();
        tdsl::size_t write(const unsigned char * buf, tdsl::size_t len);
        int read(unsigned char * buf, unsigned long amount);
        tdsl::uint16_t localPort();
        tdsl::uint16_t remotePort();
        IPAddress localIP();
        IPAddress remoteIP();       
    };
    /* use your own custom type */
    tdsl::arduino_driver<my_client> driver_c{net_buf};

---

Runtime dependencies

tdslite is a self-contained library with no external dependencies, and it does not rely on C++ standard library to function. Instead, needed functionality from C++ standard library is implemented from scratch as needed (e.g., type_traits, span). Therefore, the only requirement to use this library is to have a decent, C++11 compliant compiler.

---

The tech stack

The below are the tools and libraries that used for developing this project:

• CMake: build system generator for the project
• Docker: The development environment containers
• googletest: unit and integration tests
• Conan: used as dependency manager, fetches dependencies for tests (e.g. Boost, googletest)
• Boost: Used for the ASIO-based test network implementation
• clang-format: Code formatting
• pio: Platform.IO CLI, used for HW CI tests

---

Testing

Tests are under the tests folder. The tests are primarily separated to four categories:

• unit Unit tests.
• integration tdslite - MSSQL integration tests. These tests use the internal MSSQL server.
• cxxcompat All unit tests and integrations tests curated and compiled with C++11/14/17/20 modes.
• sketches Verification sketches for real hardware tests

The release process

There are a few simple criterias before releasing the new version of the tdslite.

• All unit, integration and cxxcompat tests must be green.
• The CI pipeline must be green.
• The sketches* must run stable for at least 3 days straight in the device farm**, with
  • No freezes
  • No restarts
  • No memory leaks

Sketches

sketches that are intended for real hardware testing. These sketches are used for performing stability testing before the releases. The sketches and the used hardware list is as follows.

• Arduino sketch:
  • Arduino Mega
  • Arduino Nano
  • Arduino Uno
  • Arduino Uno WiFi Rev. 2
  • Arduino Portenta H7
• ESP sketch
  • NodeMCU-32 (ESP32S)
  • NodeMCU V3 (ESP8266MOD)

We're intending to make this grow larger with more hardware and different kinds of tests. You can donate real hardware or a few bucks in order to make this list grow bigger.

Contributing

See the contributing guide for detailed instructions on how to get started with our project.

We accept different types of contributions, including some that don't require you to write a single line of code.

---

Q&A

Q: My ethernet library is not supported. What can I do?

A: Most of the mainstram third-party ethernet libraries are compatible with EthernetClient interface. For the ethernet libraries that are not compatible with EthernetClient interface you may:

• a) roll your own wrapper class
• b) suggest third-party library author to implement a compatible interface
• c) suggest tdslite inclusion

The option c may be feasible if it's a widely adopted Ethernet library.

---

Built with ❤︎ by mkg and

tdslite's Issues

Incompatibility with WiFiNina library

There are two problems, one of them is pretty trivial and the other requires a small change in how we pull data from the underlying client.

The first problem is, WiFiNina does not implement the following functions in its client:

• .localPort()
• .localIP()

So, the sketches would not compile when TDSL_DEBUG_* macros are defined. This is due to some diagnostics output we put into tdsl_netimpl_arduino.hpp, such as:

TDSL_DEBUG_PRINTLN("... connected, %d --> %d.%d.%d.%d:%d ...",client.localPort(), client.remoteIP() [0],client.remoteIP() [1], client.remoteIP() [2],client.remoteIP() [3], client.remotePort());

As for the second one, the client.read(buf, size) function works a little bit differently in WiFiNina than the official Arduino EthernetClient or ESP's WiFiClient. I mistakenly assumed that the client.read(buf,size) function would be implemented pretty much the same for all, but WiFiNina uses a 64-byte intermediate buffer for AVR and 1500 bytes for the rest. So, the WiFiNina currently enforces the read(buf,size) API users to pull the data in at most 64-byte chunks, regardless of the size of the actual available data in the hardware RX buffer. This of course interferes with the current mode of operation of tdslite's network implementation. The current implementation is roughly as follows:

• Read the TDS header, which has a fixed size of 8
• Read the total packet size from the header, so we know how many bytes we should expect
• Request exactly packet size bytes from the client via calling read(buf, packet_size)

This change is introduced with the following commit:

arduino-libraries/WiFiNINA@ea1a366

... and it seems that there's no way to disable it, so we have to revise how we pull data.

do_connect problem

This is not a problem when I connect to the wifi that have the sql server, but when I switch to my mobile hotspot this function will still return success as it will return 0 (Success code) even if cr is 0 which crash the code. So i added if(cr == 0) return 1; to prevent the problem

template <typename T>
        TDSL_SYMBOL_VISIBLE int do_connect(T target, tdsl::uint16_t port) {
            // Disconnect if already connected
            do_disconnect();

            int cr      = 0;
            int retries = 5;

            // There may be residue data in network buffer
            // Reset it to ensure a clean start
            this->network_buffer.get_writer()->reset();

            tdsl::char_view destination_host{};

            // Given that @p target can be a progmem string,
            // we need to ensure that it is in SRAM before passing
            // it to `connect` function. For that, instead of allocating
            // a new buffer, we use the network buffer since it's sufficiently
            // large free space not yet in use for anything.

            if (target.size_bytes() <= this->network_buffer.get_writer()->remaining_bytes()) {

                auto w = this->network_buffer.get_writer();
                for (auto c : target) {
                    auto wr = w->write(c);
                    (void) wr;
                }
                destination_host = w->inuse_span().template rebind_cast<const char>();
            }
            else {
                return -99; // FIXME: proper error code
            }

            // Retry up to MAX_CONNECT_ATTEMPTS times.
            while (retries--) {
                TDSL_DEBUG_PRINTLN("... attempting to connect to %s:%d, %d retries remaining ...",
                                   destination_host.data(), port, retries);
                cr = client.connect(destination_host.data(), port);
                if (cr == 1) {
                    TDSL_DEBUG_PRINTLN("... connected, %d --> %d.%d.%d.%d:%d ...",
                                       client.localPort(), client.remoteIP() [0],
                                       client.remoteIP() [1], client.remoteIP() [2],
                                       client.remoteIP() [3], client.remotePort());
                    break;
                }

                TDSL_DEBUG_PRINTLN("... connection attempt failed (%d) ...", cr);
                delay(1000);
            }

            // Reset the network buffer so it's ready to
            // use by its real purpose
            this->network_buffer.get_writer()->reset();

            if (cr == 1) {
                return 0;
            }
            if(cr == 0) return 1; // Added this then good
            return cr;
        }

Asking how to send variable along with a SQL query.

When I'm trying to put a variables into write SQL query after convert them to strings, I got a compile error in the below attached arduino file.
Arduino code

tdslite sketch for ESP8266 NodeMcu V3 - Errors

Arduino IDE 2.1.0
Mode MCU 1.0 (ESP-12E Module)
tdslite 0.2.0

Depending on the ESP8266 Community version (2.7.4 or 3.1.2) we have a compilation error for the 2.7.4 version and a runtime error for the 3.1.2 version (as per attachments).

25.06.2023_08.25.23_REC.mp4

ESP8266 Community 2_7_4 Error.txt

Do not blindly pick "affected_rows" from done tokens

Currently, the done token callback blindly picks affected_rows count from any done token, even if the token does not contain a valid affected_rows value:

tdslite/src/tdslite/detail/tdsl_command_context.hpp

Lines 102 to 108 in 43982a7

	tds_ctx.callbacks.done = {
	[](void * uptr, const tds_done_token & dt) noexcept -> void {
	command_context & ctx = *static_cast<command_context *>(uptr);
	ctx.qstate.result.status = dt.status;
	ctx.qstate.result.affected_rows = dt.done_row_count;
	TDSL_DEBUG_PRINT("cc: done token -- affected rows(%d)\n", dt.done_row_count);
	},

The callback must do a bitwise status value check if the status value indicates that the done token contains a valid row count (0x10: DONE_COUNT) see reference

arduino_driver makes the program stop

Discussed in #28

^{Originally posted by mustafakemalgilor August 24, 2023}
Hi folks,

Today, I got my hands on Portenta H7 and found a weird Arduino bug instead of a tdslite one. It seems like mbed EthernetClient implementation has an invalid move constructor, so moving an EthernetClient to another crashes the board:

Simple reproducer:

#include <Ethernet.h>

// --------------------------------------------------------------------------------
// remove_reference

template<typename T>
struct remove_reference {
  typedef T type;
};

template<typename T>
struct remove_reference<T &> {
  typedef T type;
};

template<typename T>
struct remove_reference<T &&> {
  typedef T type;
};


#define MOVE(...) \
  static_cast<typename remove_reference<decltype(__VA_ARGS__)>::type &&>( \
    __VA_ARGS__)

void setup() {
  EthernetClient ec = {};
  EthernetClient ec2{ MOVE(ec) };  // Crashes the board
}

void loop() {}

I opened a ticket in ArduinoCore-mbed arduino/ArduinoCore-mbed#715 to get it fixed. Meanwhile, I'll issue a temporary fix for the mbed boards.

Implement TLS support

The library is currently unusable in environments where TLS is a must.

Wifi with tdslite

Hello,

Can we use tdslite with the WiFi library in Arduino IDE ?
I use WiFi and I need tu use tdslite but I can't une ethernet and in whole example a see you use ethernet library.

I would like to make a feature request to include a timeout setting when attempting to connect.

Hello to you Mustafa,

I would like to make a feature request to include a timeout setting when attempting to connect.
Perhaps it could be an optional parameter passed via the dirver.connect command.
In my application for example, the ability to connect to the database is an environmental attribute that the program must be responsive to by facilitating a local storage option while the sensors continue to function. When connection to the remote data becomes viable, the program would then transmit it's local cache to that system.
Currently, if the remote data connection is available the tdslite library connects immediately.
If the remote data is not available the program remains unresponsive during this timeout/retry period.

Please do let me know if there are any questions regarding this request or if I may be of any assistance .

Regards

Originally posted by @bradbeshaw in #39 (comment)

Add "udev" to installed packages

The new version of VSCode Arduino extension requires "udevadm" command to be present in the environment in order to enumerate the serial ports, so related functionality is broken (e.g. serial monitor, selecting a serial port) when "udev" package is not installed, which is the case for the development environment.

Add support for reading "numeric/decimal" data types

The numeric/decimal data type currently has no internal type to be able to read it. Implement a sql_decimal/numeric data type so users can read the numeric/decimal columns.

How to Bind ''String'' parameters

As in your example code query with parameters, I need to pass timestamp as a " string " in the below code section. Please guide me how to include it.

tdslite works on ESP32, but need helps for displaying the queried data.

@mustafakemalgilor I tried tdslite on my ESP32 DeviKitC board, it works well.

However, since I am a newbie for C++, I would need your helps. I used tdslite/tests/sketches/esp/esp.cpp and modified to accommodate my WiFi SSID/PWD, SQL Server addres/username/password ....

And changed setup() and loop() to have tdslite_loop() run only once.

// --------------------------------------------------------------------------------
void setup() {
    bool r = {false};
    r      = init_serial();
    r      = r && wifi_setup();
    r      = r && tdslite_setup();
    if (not r) {
        SERIAL_PRINTLNF("... setup failed ...");
        for (;;) {
            delay(1000);
        }
    }
    SERIAL_PRINTLNF("--- setup finished ---");

    tdslite_loop();
}

// --------------------------------------------------------------------------------

void loop() {
    // tdslite_loop();
    // delay(250);
}

The Serial output from ESP32 is as below
...
received COLMETADATA token -> column count [3]
row field 0 -> []
row field 1 -> []
row field 2 -> []
row: 1 2 1701998412
received done token -> status [16] | cur_cmd [193] | done_row_count [1]
cc: done token -- affected rows(1)
network_impl_base::do_receive_tds_pdu(...) -> packet_data_cb needs 0 more bytes
netbuf: [consumed 282, inuse 0, free 8192]
netbuf: [consumed 0, inuse 0, free 8192]
'>> Report: row count [1] <<

For the output of "row: 1 2 1701998412", I believe the third column is a pointer printed as uint32. I would like the output be like "row:1 2 Lorem ipsum ... velit.".

I know I should modify the codes in row_callback(), but as a C++ newbie, I have no idea how to modify the codes

    for (const auto & field : row) {
        SERIAL_PRINTF("%d\t", field.as<tdsl::uint32_t>());
    }

Sorry for the C++ newbie question, your helps would be much appreciated.

Allow mix-and-match of progmem/non-progmem string views in connection parameters

Currently, it is not possible to mix and match the said string views. It would be more flexible for the end user if the implementation allows to do so.

Connection to Azure SQL Database

Is it possible to connect directly to an Azure SQL Database? I have been trying to use it with no result. The connection I am trying is through wifi, and I might be doing something wrong.

    decltype(driver)::progmem_connection_parameters params;
    // Server's hostname or IP address.
    params.server_name = TDSL_PMEMSTR("<server name>.database.windows.net");
    // SQL server port number
    params.port        = 1433;
    // SQL server login user
    params.user_name   = TDSL_PMEMSTR("sa");
    // SQL server login user password
    params.password    = TDSL_PMEMSTR("<password>");
    // Database name(optional)
    params.db_name     = TDSL_PMEMSTR("<DB Name>");