datastreamsio / moby-tracking-sdk

Moby, the mobile SDK for Android and iOS

Moby tracking SDK

O2MC's mobile tracking SDK for collecting and measuring analytical events.

Getting started

These instructions will get you up and running on your local machine. There is an Android and iOS app available for local testing.

Prerequisites

The following tools are required depending on which mobile OS you are targeting.

Android

• Android SDK platform-tools 26

iOS

• XCode 9+
• iOS 9+

Installation

Android

Hello, World!

For a Hello-world example, please refer to this page.

Refer to the API documentation for more details on how to use the SDK.

Data specification

The data is sent as JSON data. The format contains two main properties, the device and the events property. Both properties are guaranteed to be included.

Example

{
    "device": {
        "appId": "io.o2mc.app",
        "locale": "en_US",
        "name": "Android SDK built for x86",
        "os": "android",
        "osVersion": "9"
    },
    "events": [
        {
            "name": "MainActivity created",                                                     
            "timestamp": "2018-07-25T11:19:17+02:00",
        },
        {
            "name": "Clicked button: 'Create Track Event'",                                                     
            "timestamp": "2018-07-25T11:19:17+02:00",
            "value": "Name"
        }
    ],
    "number": 1,
    "retries": 0,
    "sessionId": "6458def9-30b8-4591-9eb3-9e9881b8dc3f",
    "timestamp": "2018-07-25T11:19:18+02:00"
}

Refer to the specification documentation for more details.

Contributing

Any kind of contribution is welcome. For bug reports, feature requests, please open an issue without any hesitation. For code contributions, it's strongly suggested to open an issue for discussion first. For more details, please refer to contributing documentation.

License

MIT license.

Copyright (c) Insite Innovations and Properties B.V.

moby-tracking-sdk's Issues

iOS warnings

There are multiple build time warnings on iOS. Most of those warnings are bad practice and should be resolved.

Use logger framework

Currently there are many stack prints and such.
A logger framework should be used to allow easy filtering and so on.

Provide iOS example app using Swift

Currently there is only an example app in objective-c. A Swift app would be helpful as well since many apps are developed using Swift these days.

Comply with some GitHub standards for repository documentation

See https://github.com/O2MC/dimml-mobile-sdk/community

• Description
• README
• Contributing
• License

Javadoc in IDE

Problem

When using the option to include our SDK using the .aar file, Android Studio does not show the JavaDoc comments from the source code. This is caused because the source code is not included in the .aar file, just the compiled bytecode.

To verify this, open the .aar file --> sources.jar, notice the .class extension.

Solution

We should include the original source code in the jar -- where the comments are still included -- for the IDE to notice the JavaDoc and provide assistance to the SDK user accordingly.

So far I've researched a lot on Google only to notice there's no simple way to acquire the desired effect when manually including a .aar library. The process of including source code and thus JavaDoc seems way more effective when publishing a .aar file using BinTray and then importing it using a SLoC like implementation 'com.squareup.okhttp3:okhttp:3.10.0'. In the long term, this seems much more desirable as opposed to manually downloading the SDK anyway.

Publishing the SDK to bintray and jCenter could quite likely also fix #9. Both of these issues (this one and #9) are solved by popular solutions such as the Facebook SDK, the Picasso library, and the OkHttp library.

Some attempts to automatically push the SDK to Bintray using Gradle have been performed, but unsuccessfully, see branch feat/android-javadoc-in-aar.

Add key value tracking support

Currently iOS supports two tracking methods:

• track - no payload
• trackWithProps - dict as payload

A lot of data would probably be key value data. This would require additional tracking methods for this. Most of them are primitives.

• trackWithInt
• trackWithLong
• trackWithFloat
• trackWithDouble
• trackWithNSNumber
• trackWithString
• trackWithBool

Another solution would be to extend the trackWithProps method to support multiple values. This is a less preferred pattern for objective-c.

Fat AAR generation

A compiled AAR module doesn't link any dependencies statically by default. It would be desirable to have our dependencies statically linked into our distributed AAR module.

There are some projects that try to achieve this but all of them seem deprecated.
For example:

https://github.com/adwiv/android-fat-aar
https://stackoverflow.com/questions/49881928/android-create-uber-fat-aar-error-on-implementation
https://github.com/Vigi0303/fat-aar-plugin

Optional json properties

Problem

When sending json payload to the backend some properties might be optional(eg. event.properties and the deviceId while being anonymous, see cfdf3c9).

The backend team should be able to handle our optional json properties.

Proposed solutions

The following pseudo payload will be used as an example.

{
    "foo": "bar"
}

Set value to null

We could send the optional properties with a null value. This keeps the payload data consistent which could be useful when querying the data.

{
    "foo": null
}

Not at all

We could leave them out entirely in case the property is not set. This saves a little amount of data sent over the wire.

{

}

Restructure documentation

Tutorials, examples, etc. are placed in a little messy way.
Restructure for clarity.

Dispatcher does not retry after network error

The dispatcher does not automatically retry sending data to an end point which results in an HTTP failure.

For instance

2018-06-22 13:33:42.264194+0200 app-obj-c[69591:4850218] lenght (570) Funnel -> ( {
  "application" : {
    "osVersion" : "11.4",
    "AppId" : "app-obj-c",
    "os" : "iOS",
    "connection" : "3G",
    "ip" : "192.168.x.x",
    "device" : "Simulator"
  },
  "tracked" : {
    "alias" : [
      {
        "alias" : "eventName",
        "event" : "alias",
        "identitiy" : "",
        "time" : "2018-06-22T13:33:38+02:00"
      }
    ],
    "eventName" : [
      {
        "alias" : "39C1FEBD-D7BE-430C-89EC-93610850C86C",
        "event" : "eventName",
        "identitiy" : "",
        "time" : "2018-06-22T13:33:37+02:00"
      }
    ]
  }
} ) has been dispatched to: http://127.0.0.1:3655/events
2018-06-22 13:33:42.272314+0200 app-obj-c[69591:4855701] TIC TCP Conn Failed [2:0x60000016f840]: 1:61 Err(61)
2018-06-22 13:33:42.272911+0200 app-obj-c[69591:4855701] Task <935D156C-4AC7-455C-9765-5B5E6A36AE4A>.<0> HTTP load failed (error code: -1004 [1:61])
2018-06-22 13:33:42.273039+0200 app-obj-c[69591:4859820] NSURLConnection finished with error - code -1004

The expected behaviour would be that the SDK would retry sending the data shortly in case of a failure.

Rename iOS' framework target to a less generic name

Currently the framework's target product is named tracker. This name is too generic and should be refactored to something less generic. A possibility would be O2MTracker.

Event name sanity checks

Both platforms should do sanity checks on event names. For example; new lines and white space should be avoided any time because its error prone.

Tests (framework)

Problem

After refactoring code, everything has to be tested manually before merging back into the master branch.
This is extremely prone to human error and forgetfulness.

Solution

Create unit tests to verify existing functionality. Then, when refactoring code, just run the tests before pushing to production.

Setting up a baseline for tests also encourages easier test-driven development for future features.

Integration tests are a good expansion to this, but will be considered an enhancement for now. (integration tests in this context = sending actual http requests and verifying the response from the backend)

Edit

Code for Android has been refactored. Still, create tests to improve code quality and better practice overall for future features.

Dynamic dependency retrieval

Retrieve SDK's dependencies dynamically in install-v1.gradle instead of hard-coded.
Until then, update version numbers etc always when updating the SDK's version numbers in its build.gradle file.

Stop generating events on max retries

'Problem'

When the max retry limit has been reached, the SDK will stop trying to resend events to the backend.
This saves data usage and battery because of saved cpu calculations.
Events however, are still generated and added to the eventbus. These are minor calculations, but it's possible to optimize it in order to reduce overhead and therefore save battery usage.

Solution

Stop tracking events when we know we're not going to send them anyway.

Potential side effect

May the desired function definition for 'max retries' change into 'try it again at a later time', it would have been desirable to have tracked the events anyway.

Android studio inspector warnings

Android studio's inspector throws warnings on code inspection. We should solve and keep track of them.

The following summary has been generated by the Android Studio Inspector.

• Android Lint: Accessibility (1 warning)
• Android Lint: Correctness (3 warnings)
• Android Lint: Performance (1 warning)
• Android Lint: Security (3 warnings)
• Android Lint: Usability (1 warning)
• Class structure (6 warnings)
• Compiler issues (4 warnings)
• Declaration redundancy (103 warnings)
• Java 7 (1 warning)
• Java language level migration aids (3 warnings)
• Javadoc issues (18 warnings)
• Probable bugs (7 warnings)
• Spelling (95 typos)
• Verbose or redundant code constructs (3 warnings)

Convert domain objects to Kotlin code

Reduces amount of code by about 90% in data classes, increasing robustness and readability.

Event Timestamp

Provide Event objects with a timestamp field when generating one (in EventGenerator). This is probably valuable for analytics.

SDK crashes on unexpected URL

When configuring the the end point URL to a random port on localhost the SDK crashes the app completely.

The SDK should never crash the app.

Update Readme

It's not clear what the project actually is or does.
Make the readme more descriptive. Maybe look at an OSS readme template online.

Automatic view tracking

Problem

At the moment all events are being tracked manually. Tracking across different Android activities or iOS view controllers requires manual work.

Solution

Integrate Android and iOS tracking without requiring the user to implement everything them self.

A mapping should be created between Android and iOS.

Logging more elegantly

Problem

Differentiating between logging in production and debug currently requires to prepend the log statement with if (BuildConfig.DEBUG). It would be desirable to remove log statements in all other build types than debug in a more meta-way so that you don't have to prepend the if statement every time.

Solution

A possible solution would be to use proguard rules. Proguard for libraries seems to be a perfect fit. So far, creating an aar with a proguard.txt file in it with specified rules has been done. The rules aren't respected upon testing it in another app however. Therefore, thoroughly test this functionality to the point of production releases. A possible scenario for this could be:

1. Make appropriate changes to gradle build files and/or proguard rules.
2. Generate a .aar file from the SDK.
3. Use the .aar file in another project.
4. Build and install a signed .apk file from the other project, which implements the .aar.
5. Run the app. There should be no logging messages from the SDK in the console.

Optional (but useful): Also generate a signed .apk with our debug build type. Verify that the logs will appear in this build.

Create contributing / branching doc

It's currently unclear to new contributors as to how they'd start contributing. It's also useful to note down conventions for existing contributors.

Remove ip property

Ip address method seems dodgy and not required. Ip addresses can always be collected on the backend in case it would be required.

https://github.com/O2MC/dimml-mobile-sdk/blob/1e37441a2805d21d2d9522d1fc6e0f707d3ee701/ios/sdk/O2MC/Dispatcher.m#L75-#L102

Android kotlin example project

Would be nice to have an example kotlin app since it's an officially supported language for Android.

Move device model mapping to backend

Problem

Currently the mapping of device's model name is done within this SDK. The device's technical name will be converted to a marketing name.

Since the mapping is done in the frontend the list can be outdated and unable to map device names. The Android implementation does sync a list from an upstream party, the iOS' doesn't. The list can get outdated and adds another dependency within this SDK, thus more maintenance for parties who will implement this SDK.

Android

https://github.com/O2MC/dimml-mobile-sdk/blob/ee0fce2f220fae1269bc133b4c5a0206067a290f/android/sdk/src/main/java/io/o2mc/sdk/business/DeviceManager.java#L44

iOS

https://github.com/O2MC/dimml-mobile-sdk/blob/ee0fce2f220fae1269bc133b4c5a0206067a290f/ios/sdk/O2MC/O2MDevice.m#L20-L61

Solution

Move mapping to the backend.

Privacy aware batch identification

Problem

Events are sent to the backend as a batch. While we can assume that a batch belongs to a single device, multiple batches can't be easily identified as belonging to a single device.

Solution

Respecting the user's privacy is our main concern. Given that it's up to the implementing party the API should be able to support; no identifier, session identifier, persistent identifier(from high to low priority).

No identifier (anonymous)

The user wants to go anonymous or hasn't given any consent yet. This should be the default setting since we must respect the user's privacy.

Session identifier

A random identifier will be generated on each sdk instance generation.

Persistent identifier

This will be the least privacy respecting option. Meaning there will be a persistent identifier stored on the device which is used to identify the user across sessions.

Add system locale information

System locale are very helpful thus should be added to the device object property by default.

Android locale docs
iOS locale docs

Eventbus is not cleared after successful dispatch

Targeting Android API 28

Problem

When ran on an Android device with API level 17+, the EventDispatcher won't be able to post data over HTTP because of a more strict HTTP/HTTPS policy by Android:

E/EventDispatcher: Unable to post data: 'CLEARTEXT communication to 10.0.2.2 not permitted by network security policy'

Solution

This is expected behavior, but the error message should be more informative and come some kind of solution. Implement instructions on how to solve it, and a more informative error log.

Keep track of batch number

Before the code refactor, batches were kept track of.
Not sure if the product owner even needs/wants this functionality, leaving it open for now.

Allow dynamic end point configuration in test apps

Problem

Currently the test apps don't allow changing the end points from within the apps. This means the end user has to constantly recompile the apps.

Solution

Allow changing the end point URL from within the app. This makes the app more useful when debugging in the field and allows non-developers to also test out the app.

Currently the end point is set in the constructor. This means we have to create setters (and potentially getters) for configuring the end point to the sdk api.

Batches implementation

Use batches instead of list of events, greatly reduces network data usage at scale and generally makes more sense.

Track with props arguments should be reversed

Currently the trackWithProperties method arguments are in a reverse order. This is error prone when implementing and will result in analyzing incorrect data.

-(void)trackWithProperties:(nonnull NSString*)eventName properties:(nonnull NSDictionary*)properties;

Should be:

-(void)trackWithProperties:(nonnull NSDictionary*)properties (nonnull NSString*)eventName properties;

Empty Events

Don't send an empty list of events to the backend every interval.

Dispatcher ignores max retries on unpredicted http response

The setMaxRetries threshold is being ignored when the dispatcher's connection succeeds but doesn't get a 200 or 201 http code as response. As result the batchmanager keeps triggering the dispatcher.

Tracking with properties format difference

Problem

There is a difference between iOS and Android on how they handle tracking with properties(eg. additional data).

SDK API methods

The first difference is that Android trackWithProperties() method requires string as property value data, iOS' requires an NSDictionary as input.

Android

https://github.com/O2MC/dimml-mobile-sdk/blob/3283bd2a45bf5ffafb9550bfbf37987bc582829b/android/sdk/src/main/java/io/o2mc/sdk/O2MC.java#L190-L200

iOS

https://github.com/O2MC/dimml-mobile-sdk/blob/3283bd2a45bf5ffafb9550bfbf37987bc582829b/ios/sdk/O2MC/O2MC.h#L85-L90

Output

The output format also differs. Android returns everything in a single string and iOS parses it and returns it as json data.

Android

{
  "events": [
    {
      "name": "Clicked button: 'Create Track Event'",
      "timestamp": "2018-08-15T12:54:47+02:00",
      "value": "Name"
    }
  ]
}

iOS

{
  "events": [
    {
      "name": "eventName",
      "timestamp": "2018-08-15T12:27:35+02:00",
      "value": {
        "foo": "bar"
      },
    }
  ]
}

Solution

They are both valid ways of handling additional data. Lets discuss the pro's and con's of both.

Logging in Production

Logs should not appear in the production release.

Preferable use Proguard rules to remove the log statements compile time.