mparticle / smartype

Json Schema as code, autocomplete for your data model!

Overview

Smartype is a powerful code generation tool that allows you to get the most out of the time you spend organizing and defining the structure of your data.

It aims to completely eliminate an entire class of data quality bugs 🐛 by generating statically-typed API clients based on the popular JSON schema standard.

🏁 Smartype is built to translate any JSON schema into type-safe models
🏁 It gives you auto-complete for your data model in your IDE of choice
🏁 It's open source and extensible via a plugin system
🏁 And it's built for all and with ❤️ by mParticle

Example

The following JSON schema describes a coffee order with a few required parameters:

• item: An string value with a predefined set of allowed values
• quantity: A numeric value indicating how many coffees were ordered
• milk: A boolean value indicating if you want milk in your coffee

Smartype does the following with this:

• Consumes the JSON schema and generates Kotlin data classes
• Uses Kotlin Multiplatform to translate that Kotlin code into other languages and generate consumable binaries
• Surfaces an API to send and receive these "messages", which can be consumed by any analytics provider or your own system

Supported Environments

Smartype supports the following language environments today:

1. Any JVM environment, including Kotlin and Java for Android and server-side contexts
2. iOS Swift and Objective-C
3. Web browsers and React Native via TypeScript and JavaScript

mParticle Customers

Smartype is designed to be used by anyone, but support today is primarily for mParticle's Events API and SDKs by way of the mParticle Smartype receiver.

Navigate to the mParticle docs for more specific docs related to mParticle Data Plans and SDKs.

Workflow

Smartype is shipped as a CLI tool, and so a the typical workflow would be:

1. Assemble your JSON schema into a file for consumption by Smartype or download the mParticle Data Plan from the Web UI or using the mParticle CLI tool.
2. Run Smartype to generate your libraries.
3. Incorporate and use those libraries in any number of environments

Using the mParticle CLI Tool to Download the Data Plan

Rather than manually creating a JSON file, mParticle provides these files ready for use by Smartype. There are multiple ways to retrieve them, but for automation purposes using the mParticle CLI tool is the best option.

We provide a ready-to-use Github Actions workflow file here, which can be adapted to other CI systems: https://github.com/mParticle/mparticle-workflows/blob/main/.github/workflows/data-plan-fetch.yml

Individual developers can also manually download the Data Plan JSON by using the mParticle CLI directly from the command line.

Install the mParticle CLI tool

1. Install Node using your preferred method for your platform.
2. Install the mParticle CLI tool: npm install -g @mparticle/cli.

Download the Data Plan JSON

Once the mParticle CLI has been installed, the Data Plan JSON can be downloaded using a single command.

First you will need the following pieces of information:

• Data Plan ID
• Data Plan Version
• Workspace ID
• Client ID (mParticle access token)
• Client Secret (mParticle access token)

You can create and manage your mParticle access tokens for Data Planning with the API Credentials interface.

# Here we'll use environment variables, but you can simply pass the values directly to the mp command if you'd prefer
export $DATA_PLAN_ID=...
export $DATA_PLAN_VERSION=...
export $WORKSPACE_ID=...
export $CLIENT_ID=...
export $CLIENT_SECRET=...
# $OUT_FILE can be any file name you like
export $OUT_FILE=${DATA_PLAN_ID}_${DATA_PLAN_VERSION}.json

# Pull down the JSON file
mp planning:data-plan-versions:fetch --dataPlanId=$DATA_PLAN_ID --versionNumber=$DATA_PLAN_VERSION --workspaceId=$WORKSPACE_ID --clientId=$CLIENT_ID --clientSecret=$CLIENT_SECRET --outFile=$OUT_FILE

Smartype CLI Usage

Smartype is deployed as an executable jar CLI, and you can download the latest release from the Github releases page.

The CLI provides two key commands:

• init: Initialize a configuration file that's used by Smartype to generate code.
• generate: Generates strongly-type libraries based on your data model

Smartype init

Smartype init will ask you a series of questions and then create a Smartype configuration file.

java -jar smartype.jar init

Smartype generate

Smartype generate will read your configuration file and output binaries that are ready for consumption in an application.

java -jar smartype.jar generate

If this is your first call to 'generate', you may want to go grab a cup of coffee while it downloads dependencies. (It will be much faster the second time!)

Integrating Generated Code

To use the objects created by Smartype, you'll want to add the generated code to your projects. You will also want to initialize Smartype when the app starts up, and register any receivers that you would like to get notified for each message that gets logged.

The following code snippets use the mParticle receiver as an example, but receivers can be created to wrap any interface to which you want to send data, including for your own inhouse processing.

You can also (optionally) add yourself as a receiver, and then implement a receive function to get a copy of all JSON messages that are sent. See the example projects for details of how this is done per platform.

iOS

Smartype generate will create an "fat" dynamic framework that you can include directly with your projects.

• To use Smartype on iOS, start by adding smartype.xcframework to your Xcode project
• Next, import and initialize Smartype prior to use, and register any receivers
• The SmartypeApi object will surface a series of methods which each represent the top-level items in your schema
• Pass the fully constructed objects into your SmartypeApi instance for all receivers

import Smartype

...
let api = SmartypeApi()
api.addReceiver(receiver: MParticleReceiver())
api.addReceiver(receiver: self)

let chooseCustomAttributes = ChooseItemDataCustomAttributes
    .init(quantity: 5,
          milk: true,
          item: .cortado
)
let itemData = ChooseItemData.init(customAttributes: chooseCustomAttributes)
let chooseItem = api.chooseItem(data: itemData)
api.send(message: chooseItem)

Android

Smartype generate will create an aar file that you can include directly with your projects.

To use Smartype on Android, start by adding the generated smartype.aar to your project and any 3rd-party receivers that you plan on using:

dependencies {
    implementation "com.mparticle:smartype-api:1.2.4"
    implementation "com.mparticle:smartype-mparticle:1.2.4"
    implementation fileTree(dir: 'libs', include: ['**/*.aar'])
}

The Smartype API dependencies are deployed as a multiplatform library leveraging Gradle Module metadata, and in order for Android projects to resolve the right dependency, you may need to add the following to ensure debug builds use the "release" artifact.

buildTypes {
    debug {
        matchingFallbacks = ["release"]
    }
}

• Import and initialize Smartype prior to use, and register your receivers
• The SmartypeApi object will surface a series of methods which each represent the top-level items in your schema
• Pass the fully constructed objects into your SmartypeApi instance for all receivers

val api = SmartypeApi()
api.addReceiver(MParticleReceiver())
api.addReceiver(this)
val message = api.chooseItem(
    ChooseItemData(
        ChooseItemDataCustomAttributes(
            quantity = 5.0,
            milk = true,
            item = ChooseItemDataCustomAttributesItem.CORTADO
        )
    )
)
//the message object will now be sent to all receivers
api.send(message)

Web

Smartype supports both the mParticle Web SDK as well as the mParticle React Native plugin.

Smartype generate will create a set of .js and .d.ts files that you can include directly with your projects. Our example uses webpack to concatenate and minify the source files.

To use Smartype with the Web SDK or with React Native, start by adding the generated smartype-dist directory to your project and any 3rd-party receivers that you plan on using, then include the relevant files in your typescript or javascript sources:

import * as smartype from "../smartype-dist/smartype.js"

• Import and initialize Smartype prior to use, and register your receivers
• The SmartypeApi object will surface a series of methods which each represent the top-level items in your schema
• Pass the fully constructed objects into your SmartypeApi instance for all receivers

import * as smartype from "../smartype-dist/smartype.js"

var api = new smartype.SmartypeApi()
api.addReceiver(smartype.mParticleReceiver())

var message = smartype.chooseItem(
      new smartype.ChooseItemData(
        new smartype.ChooseItemDataCustomAttributes(
          1, true, new smartype.ChooseItemDataCustomAttributesItem().CORTADO()
        )
      )
    )

//the message object will now be sent to all receivers
api.send(message)

React Native

In order to enable React Native:

• Add mParticle's React Native plugin to your React project if you haven't already
• Inject the mParticle React Native plugin into your mParticleReceiver:

import MParticle from 'react-native-mparticle'

...

var api = new smartype.SmartypeApi()
var receiver = smartype.mParticleReceiver()
receiver.react = MParticle
api.addReceiver(receiver)

You will also want to exclude the generated .smartype directory from your React Project by configuring your metro.config.js file:

const exclusionList = require('metro-config/src/defaults/exclusionList');

... 

module.exports = {
  resolver: {
      blockList: exclusionList([/\.smartype\/.*/])
    },
};

Example Projects

Check out the examples here

Contributing

At mParticle, we are proud of our code and like to keep things open source. If you'd like to contribute, simply fork this repo, push any code changes to your fork, and submit a Pull Request against the master branch.

License

Apache 2.0

smartype's Issues

1.7.0 artifacts not published to maven servers - generate command fails

We appear to be having an issue running mvnx com.mparticle:smartype-generator generate and it seems possible it may be due to the fact that the various 1.7.0 dependencies have not been published.

smartype-mparticle up to 1.6.2 only

• https://mvnrepository.com/artifact/com.mparticle/smartype-mparticle
• https://central.sonatype.com/artifact/com.mparticle/smartype-mparticle/versions

smartype-api up to 1.6.2 only

• https://mvnrepository.com/artifact/com.mparticle/smartype-api
• https://central.sonatype.com/artifact/com.mparticle/smartype-api/versions

Error Output

* What went wrong:
Execution failed for task ':smartype:compileReleaseKotlinAndroid'.
> Could not resolve all files for configuration ':smartype:releaseCompileClasspath'.
   > Could not find com.mparticle:smartype-mparticle:1.7.0.
     Required by:
         project :smartype
   > Could not find com.mparticle:smartype-api:1.7.0.
     Required by:
         project :smartype

Generator doesn't work without a Data Plan?

The README says that Smartype is analytics API agnostic:

Smartype is designed to be used by anyone, but support today is primarily for mParticle's Events API and SDKs by way of the mParticle Smartype receiver.

But the docs say:

Smartype is designed to work with mParticle Data Plans. If you haven’t yet created a Data Plan, you’ll need to do so and download your data plan to work with Smartype.

So what's correct? I'm assuming the latter.

Running the generator is failing for me. And it makes me think I need to become an mParticle customer, and create a Data Plan if I want to run the Smartype generator. Is that correct?

❯ java -jar smartype-generator-1.2.0.jar init
Would you like to generate iOS? [no]: yes
Would you like to generate Android? [no]: yes
Would you like to generate Web? [no]: yes
Where should Smartype libraries be generated [smartype-dist]:
Please specify a file path containing the JSON schema that you'd like to use for generation: data.json
smartype.config.json generated successfully! Have fun :-)
❯ java -jar smartype-generator-1.2.0.jar generate
Extracting mParticle Data Plan
Exception in thread "main" java.lang.NullPointerException
	at com.mparticle.smartype.generator.adapters.MParticleDataPlanAdapter.extractSchemas(MParticleDataPlanAdapter.kt:22)
	at com.mparticle.smartype.generator.Generate.run(Generator.kt:97)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:154)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:162)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:14)
	at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:252)
	at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:249)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:267)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:290)
	at com.mparticle.smartype.generator.GeneratorKt.main(Generator.kt:236)

data.json

{
    "smartype_object_name": "doggo",
    "properties": {
        "id": {
            "type": "integer",
            "description": "The doggo UUID"
        },
        "breed": {
            "type": "string",
            "description": "The doggo breed"
        },
        "favorite": {
            "type": "boolean",
            "description": "Whether this doggo breed is a favorite."
        }
    }
}

smartype.config.json

{
    "iosOptions": {
        "enabled": true
    },
    "androidOptions": {
        "enabled": true
    },
    "webOptions": {
        "enabled": true
    },
    "binaryOutputDirectory": "smartype-dist",
    "apiSchemaFile": "data.json",
    "dedupEnums": false
}