automattic / automattic-tracks-android

Client library for tracking user events for later analysis

Automattic-Tracks-Android

Client library for tracking user events for later analysis

Introduction

Tracks for Android is a client library used to help track events inside of an application. This project solely is responsible for collecting the events, storing them locally, and on a schedule send them out to the Automattic servers. Realistically this library is only useful for Automattic-based projects but the idea is to share what we've made.

Usage

repositories {
    maven {
        url 'https://a8c-libs.s3.amazonaws.com/android'
        content {
            includeGroup 'com.automattic'
            includeGroup 'com.automattic.tracks'
        }
    }
}

dependencies {
    // For 'tracks' module
    implementation 'com.automattic:Automattic-Tracks-Android:{version}'

    // For 'experimentation' module
    implementation 'com.automattic.tracks:experimentation:{version}'
}

Publishing a new version

In the following cases, the CI will publish a new version with the following format to our S3 Maven repo:

• For each commit in an open PR: <PR-number>-<commit full SHA1>
• Each time a PR is merged to trunk: trunk-<commit full SHA1>
• Each time a new tag is created: {tag-name}

License

Automattic-Tracks-Android is available under the GPL v2 license. See the LICENSE file for more info.

automattic-tracks-android's Issues

Setting `"null"` String instead of `null` for known key in `SentryEvent`

Current logic applies "null" String value for a known key before every event.

So when we have knownKeys = listOf("foobar"), and we won't provide any logic in DataProvider#provideExtrasForEvent, we'll end up with event.get("foobar") = "null" which is invalid. It should be event.get("foobar") = null

The `_lg` property should contain the Application language and not the device language

To get the application language:
properties.put(APP_LOCALE, mContext.getResources().getConfiguration().locale.toString());

Put the device language as a new device property. Current devices properties below:

            [device_info_os] => Android
            [device_info_os_version] => 5.0
            [device_info_manufacturer] => Genymotion
            [device_info_brand] => generic
            [device_info_model] => Google Nexus 6 - 5.0.0 - API 21 - 1440x2560
            [device_info_app_name] => WordPress
            [device_info_app_version] => 4.1-rc-1
            [device_info_app_version_code] => 182
            [device_info_has_nfc] => false
            [device_info_has_telephony] => true
            [device_info_display_density_dpi] => 560
            [device_info_bluetooth_version] => none
            [device_info_current_network_operator] => 
            [device_info_phone_radio_type] => gsm
            [device_info_wifi_connected] => true

/cc @astralbodies

Make sure the event is still valid before sending it to the server

On iOS we received some reports of a high number of duplicate events from a single client. Even though we fixed the underlying bug that caused those events a long time ago, they keep coming up, and we think the app might have them queued and it keeps trying to send them all.

Nothing similar reported on Android but better to change the function TracksClient.isStillValid(Event), that now returns always true, by cheking the

• retry count
• timestamp (expire after a while (1 week?))

Ref: Automattic/Automattic-Tracks-iOS#68

Enable `androidx`

Sentry SDK 4.2.0 uses androidx artifacts, we have to start using it in order to bump Sentry.

Track device orientation

Start tracking device orientation "portrait/landscape".

Add experiment list screen to :experimentation module

Add experiment list screen (as seen here) to :experimentation module

Use `tags` instead of `extra` for application context

Sentry's extra feature is softly detracted. One of its flaws is that it's not searchable. So e.g. when WCAndroid uses it to set site id, we can't search for all issues from a selected site.

Other clients usages:

• WPAndroid doesn't use extra
• as well as simplenote-android

Instead, we should use tags which should be a perfect use case.

e.g. this is sample with custom 1-application tag:

Change `trackAliasUser` by adding a 3rd parameter to it that specifies user type

Title has it.

Integrate static code analysis for Kotlin

We could enable static code analysis for Kotlin, a good candidate could be ktlint which is used in other libs, e.g. FluxC

`CrashLoggingOkHttpInterceptor` could format Jetpack related requests better

Having a WooCommerce Android app we very often use Jetpack proxy for API calls. Currently used SentryOkHttpInterceptor logs the raw value of request and we end up with such breadcrumbs:

It's helpful but we could improve readability of it.

Convert sonatype config to bintray config

Set a max limit on the events queue size

We should set a max number of items the queue can contain. In case of long offline usage of the app, or in case of errors, the queue could grow up till to 10 thousand events and slow down the client network connection.

Continuous integration setup

Maybe w could add continuous integration setup to this library, WDYT @oguzkocer @jkmassel ?

I was thinking about two simple workflows/pipelines:

1. Basic check for commit - build and test
2. Release on new tag

If it's ok, I can help with that, I believe CircleCI is our choice for every CI/CD right?

Do not immediataly retry on network error

We should add a back-off time on network request error, otherwise there is a very high possibility to fall in a loop where the network connection is retried every second.

`java.lang.NullPointerException: lastException.module must not be null`

Sometimes, exceptions that are bundled with SentryEvent can have a null value for module, type or value.

Those values are not annotated as @NonNullable in Sentry SDK. See https://github.com/getsentry/sentry-java/blob/4.3.0/sentry/src/main/java/io/sentry/protocol/SentryException.java#L88

In current implementation we will receive Null Pointer Exception and event will be sent without appending any additional data.

`Tracks` API should be in sync with current `WPAndroid` requirements

WPAndroid has its own Sentry configuration. Automattic-Tracks should cover all needed cases

• filters out the last exceptions that occurred during an EventBus event. This should be extracted to Tracks library: https://github.com/wordpress-mobile/WordPress-Android/blob/a9eeed79f6d85fd3a3a55157ff9b2932248c00cc/WordPress/src/main/java/org/wordpress/android/util/CrashLogging.kt#L44-L55

---

• allows to toggle session tracking feature https://github.com/wordpress-mobile/WordPress-Android/blob/a9eeed79f6d85fd3a3a55157ff9b2932248c00cc/WordPress/src/main/java/org/wordpress/android/util/CrashLogging.kt#L31

Update: session tracking is enabled by default: https://github.com/getsentry/sentry-java/blob/7209e3700159ca5e52c1825b7d99004edf10e8a6/sentry/src/main/java/io/sentry/SentryOptions.java#L193

---

• allows to define cache dir path https://github.com/wordpress-mobile/WordPress-Android/blob/a9eeed79f6d85fd3a3a55157ff9b2932248c00cc/WordPress/src/main/java/org/wordpress/android/util/CrashLogging.kt#L30

Update: cache dir path is already set by Sentry SDK. See: https://github.com/getsentry/sentry-java/blob/7209e3700159ca5e52c1825b7d99004edf10e8a6/sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java#L208-L212

---

• allows to append data before sending an event https://github.com/wordpress-mobile/WordPress-Android/blob/a9eeed79f6d85fd3a3a55157ff9b2932248c00cc/WordPress/src/main/java/org/wordpress/android/util/CrashLogging.kt#L68-L79

---

• Make CrashLogging not a static class (object)

---

• API for adding breadcrumbs https://github.com/wordpress-mobile/WordPress-Android/blob/a9eeed79f6d85fd3a3a55157ff9b2932248c00cc/WordPress/src/main/java/org/wordpress/android/util/CrashLogging.kt#L96-L101

---

• API for adding a tag to SentryEvent https://github.com/wordpress-mobile/WordPress-Android/blob/a9eeed79f6d85fd3a3a55157ff9b2932248c00cc/WordPress/src/main/java/org/wordpress/android/util/CrashLogging.kt#L122

Crash report: RuntimeException in com.automattic.android.tracks.DeviceInformation

Fatal Exception: java.lang.RuntimeException: Can't create handler inside thread that has not called Looper.prepare()
       at android.os.Handler.(Handler.java)
       at android.bluetooth.BluetoothAdapter$2.(BluetoothAdapter.java)
       at android.bluetooth.BluetoothAdapter.(BluetoothAdapter.java)
       at android.bluetooth.BluetoothAdapter.getDefaultAdapter(BluetoothAdapter.java)
       at com.automattic.android.tracks.DeviceInformation.isBluetoothEnabled(DeviceInformation.java)
       at com.automattic.android.tracks.DeviceInformation.getMutableDeviceInfo(DeviceInformation.java)
       at com.automattic.android.tracks.MessageBuilder.createRequestCommonPropsJSONObject(MessageBuilder.java)
       at com.automattic.android.tracks.TracksClient$2.run(TracksClient.java)
       at java.lang.Thread.run(Thread.java)

Offer `SentryOkHttpInterceptor` to the client to add automated breadcrumbs for http calls

We can add SentryOkHttpInterceptor which will automatically add breadcrumbs with information about all http calls.

https://docs.sentry.io/platforms/android/configuration/integrations/okhttp/

Blocked by wordpress-mobile/WordPress-FluxC-Android#1961

Drop `jcenter()` artifactory

Since Jetbrains moved org.jetbrains.trove4j:trove4j to mavenCentral today, we'll able to drop jcenter() artifactory from the project.

https://youtrack.jetbrains.com/issue/IDEA-261387

android.database.sqlite.SQLiteReadOnlyDatabaseException

From Fabric:

Fatal Exception: android.database.sqlite.SQLiteReadOnlyDatabaseException: attempt to write a readonly database (code 1032)
       at android.database.sqlite.SQLiteConnection.nativeExecute(SQLiteConnection.java)
       at android.database.sqlite.SQLiteConnection.execute(SQLiteConnection.java:555)
       at android.database.sqlite.SQLiteSession.execute(SQLiteSession.java:619)
       at android.database.sqlite.SQLiteStatement.execute(SQLiteStatement.java:44)
       at com.automattic.android.tracks.datasets.EventTable.insertEvent(EventTable.java:102)
       at com.automattic.android.tracks.TracksClient$1.run(TracksClient.java:117)
       at java.lang.Thread.run(Thread.java:761)

Source: 5c4ef231f8b88c2963cb9e91

Change `userHasOptOutProvider` to `crashLoggingEnabled`

Relates to: #80 (comment)

Reasoning

There are other cases for not wanting to enable crash logging. User opting out is one of them, but there are other factors like type of build. See: https://github.com/wordpress-mobile/WordPress-Android/blob/a9eeed79f6d85fd3a3a55157ff9b2932248c00cc/WordPress/src/main/java/org/wordpress/android/util/CrashLogging.kt#L163-L171

We should update method name to reflect the real behavior and let client decide when to enable/disable crash logging.

Refactor Experiment so it doesn't depend on ExPlat

We need to refactor the implementation of Experiment so it doesn't depend on ExPlat. This will help us to avoid having circular dependency problems in the clients (ExPlat depends on Set<Experiment> and Experiment depends on ExPlat).

Fix Missing recycle() calls lint warning

From Android lint report:

Prepare sample app

A sample app will help us in end-to-end, manual testing of the library.

Fix `Dependabot`

After defining version via Gradle's local property (def) in #99 , Dependabot stopped creating update PRs, even if in logs it can see that it should.

Add sonatype config to publish the artefact to a public maven repo

add sonatype config to publish the artefact to a public maven repo - reuse what we've done for other projects like here: https://github.com/wordpress-mobile/PersistentEditText-Android/blob/develop/PersistentEditText/build.gradle#L35-L89 - I see you already have the gradle.properties-example file for sonatype.

Provide `CrashLogging#initialize` method

Description

Instead constructor-based initialisation, we should provide a separate initialisation method. Constructor-based one is tricky to use - if the consumer app wants to catch unhandled exceptions but doesn't want to send custom Sentry reports, it won't inject CrashLogging anywhere and therefore - won't make SentryCrashLogging#init to run.

This way, Sentry SDK won't be initialised and exceptions won't be catched.

Adding separate initialisation method will make SDK initialisation visible and transparent.

Resolve Android lint errors

In #55 (precisely: dbc594f) lint-baseline file has been introduced, which suppress several Android lint errors. We should resolve them and remove the baseline file.

Crash report: NPE in DeviceInformation.isWifiConnected

That might be a new thing from Android 7.x

Fatal Exception: java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String android.net.wifi.WifiSsid.getHexString()' on a null object reference
       at android.os.Parcel.readException(Parcel.java:1689)
       at android.os.Parcel.readException(Parcel.java:1636)
       at android.net.IConnectivityManager$Stub$Proxy.getNetworkInfo(IConnectivityManager.java:1175)
       at android.net.ConnectivityManager.getNetworkInfo(ConnectivityManager.java:876)
       at com.automattic.android.tracks.DeviceInformation.isWifiConnected(DeviceInformation.java:237)
       at com.automattic.android.tracks.DeviceInformation.getMutableDeviceInfo(DeviceInformation.java:168)
       at com.automattic.android.tracks.MessageBuilder.createRequestCommonPropsJSONObject(MessageBuilder.java:82)
       at com.automattic.android.tracks.TracksClient$2.run(TracksClient.java:146)
       at java.lang.Thread.run(Thread.java:761)

Fix Static Field Leaks lint warning

From Android lint report:

ExPlat: Main Issue

Original issue created by @renanferrari in WordPress-Android

---

This issue centralizes all tasks and potential improvements to the ExPlat implementation on WPAndroid.

Tasks

• Initial A/A experiment implementation. (PR: wordpress-mobile/WordPress-Android#13960)
• Initial A/B experiment implementation. (PR: wordpress-mobile/WordPress-Android#14286)
• Add support for logged-out experiments.
• Add functionality to manage active experiments. (PR: wordpress-mobile/WordPress-Android#14559)
• Add UI to list active experiments (for internal testing only).
• Add functionality to manually edit assignments of active experiments (for internal testing only).
• Add support for other loading strategies (experiments that require loading screens, for example).
• Consider not fetching assignments if data collection is turned off.
• Extract implementation to a shared library (which one is yet TBD). (#114)
• :experimentation module depends on FluxC because of the classes ExperimentStore and ExperimentRestClient. We should move these classes to :experimentation module to stop depending on FluxC.
• refactor ExPlat class so we don't have to use Lazy to avoid circular dependency in the Set<Experiment> parameter. We should consider splitting the ExPlat class.

Notes

• I'm going to update the tasks with links for the specific issues and PRs once they're up.

Use the proper regexp to validate properties

We should use a slightly different regexp to validate properties. At the moment we are using one regExp for both events and properties.

Bump Sentry SDK to 4.3.0

It's necessary to bump the SDK to at least 2.x.x to have access to SentryOptions#setBeforeSend, necessary to go with encrypted logs used in WP/WCAndroid. But at this point I can't see any reason against bumping to the newest possible version of SDK which is 4.3.0

Use FluxC checkstyle configuration and configure travis to run the checks

Use FluxC checkstyle configuration and configure travis to run the checks

Get a README in this project

Create a README to describe what the project is and how to use it. Even though the audience is internal to Automattic, we should still document its usage.

android.database.sqlite.SQLiteDatabaseLockedException

It looks like there's a semi-rare bug in Tracks where it tries to access the database while it's locked.

This results in a stack trace like:

Fatal Exception: android.database.sqlite.SQLiteDatabaseLockedException: database is locked (code 5): , while compiling: PRAGMA journal_mode
#################################################################
Error Code : 5 (SQLITE_BUSY)
Caused By : The database file is locked.
	(database is locked (code 5): , while compiling: PRAGMA journal_mode)
#################################################################

This is happening in both WooCommerce (5c1a0c9ef8b88c296364bc62-fabric) and WordPress (55ec3db1f5d3a7f76bd47380-fabric)

Bump Gradle to 6.1.1, AGP to 4.0.2

Send events to the server more often

We've had our first step in comparing Mixpanel and Track numbers, and seems that Tracks numbers are always lower (5% of delta) than the ones recorded in Mixpanel.

The Android library works in this way:

• We store events in the db, when the fixed limit of 10 events is reached, all events are read from the DB and sent to the server with a batch request. If the request fails events are re-enqueued in the DB.
• When the app is sent in background, all events are sent to the server. Doesn't matter how many events are in the queue.

My hypothesis is that the app fails to push events (ie: a network error) when the "on background" event happens, and then the user doesn't reopen the app in the next 5 days.
Important note: On the server events that are 5 days old or more are rejected.

How to fix this:

1. We can create a service that runs in background and sends the events to the server, but we will have hard time justifying a background process just to send usage data over the Internet.
2. Be more aggressive, and send events to the server more often. ie: Change the 10 events limit, and lower it to 5.

Also removing the policy that 5 days old events are discarded on the server may help a lot.

cc @martinremy @astralbodies @xyu

Support Kotlin programming language

Prepare project to work with Jitpack

As we plan to host Tracks on Jitpack, we have to make sure that it'll be compatibile.

Bumping project to use Gradle 6.1.1 in #54 results with failing ./gradlew publishToMavenLocal as this task is provided currently by bintray-release plugin which does not support Gradle 6.x

Failing ./gradlew publishToMavenLocal results with failing Jitpack build as this is the way Jitpack prepares artifacts
.

Add a property that keeps track if the device is a tablet

This is a hard one, since it's not clear when a device is a tablet or not.

We can calculate the diagonal of the screen in inches, and report it in a property as double.
Also make assumptions, and add another property, boolean, like is_large_screen that is true when the screen diagonal is > 7 inches.

cc @iamthomasbishop

Specify the WPCOM user ID in addition to login name

According to @skeltoac the _ui key is required for all user types - _ul is somewhat optional but should be sent in addition to the ID.

Code: https://github.com/Automattic/Automattic-Tracks-Android/blob/develop/AutomatticTracks/src/main/java/com/automattic/android/tracks/MessageBuilder.java#L106

Additional logging data are immutable

Issue

Interface introduced in #51 has been changed from function invocations to fields. Having a sample from WCAndroid implmenetaion, applying field like:

override val currentUser: TracksUser = TracksUser(
        currentAccount?.userId.toString(),
        currentAccount?.email,
        currentAccount?.userName
    )

will result with no updating a user. The currentUser will be evaluated only once. Client's could resolve this by using Kotlin's backing fields like:

override val currentUser: TracksUser
      get() = TracksUser(
          currentAccount?.userId.toString(),
          currentAccount?.email,
          currentAccount?.userName
      )

but this is highly unpleasant API to work with and for sure error-prone.

Enhancement

Tracks offers a method CrashLogging#setNeedsDataRefresh. So in addition to updating data, clients need to signal (again) an intention to update it. I'd like to suggest that responsibility to apply updated data should be on Tracks side.

Instead of

fun setCurrentSite(site: SiteModel?) {
      this.currentSite = site
      CrashLogging.setNeedsDataRefresh()
  }

I'd like clients to work like this

fun setCurrentSite(site: SiteModel?) {
      this.currentSite = site
  }

Setup `Dependabot`

Recent bump of Sentry SDK from 1.7.16 to 4.3.0 required a lot of effort: #51 . Not bumping this dependency resulted in lack of some nice features like Sentry's breadcrumbs on clients reports.

I'd like to setup a reminder for this dependency so we can react more quickly to any changes of Sentry SDK.

Crash report wpandroid: SQLiteException in TracksDatabaseHelper.getReadableDb

Fatal Exception: android.database.sqlite.SQLiteException: Failed to change locale for db '/data/user/0/org.wordpress.android/databases/tracks.db' to 'en_AU'.
       at android.database.sqlite.SQLiteConnection.setLocaleFromConfiguration(SQLiteConnection.java:393)
       at android.database.sqlite.SQLiteConnection.open(SQLiteConnection.java:216)
       at android.database.sqlite.SQLiteConnection.open(SQLiteConnection.java:191)
       at android.database.sqlite.SQLiteConnectionPool.openConnectionLocked(SQLiteConnectionPool.java:463)
       at android.database.sqlite.SQLiteConnectionPool.open(SQLiteConnectionPool.java:185)
       at android.database.sqlite.SQLiteConnectionPool.open(SQLiteConnectionPool.java:177)
       at android.database.sqlite.SQLiteDatabase.openInner(SQLiteDatabase.java:806)
       at android.database.sqlite.SQLiteDatabase.open(SQLiteDatabase.java:791)
       at android.database.sqlite.SQLiteDatabase.openDatabase(SQLiteDatabase.java:694)
       at android.app.ContextImpl.openOrCreateDatabase(ContextImpl.java:581)
       at android.content.ContextWrapper.openOrCreateDatabase(ContextWrapper.java:275)
       at android.database.sqlite.SQLiteOpenHelper.getDatabaseLocked(SQLiteOpenHelper.java:223)
       at android.database.sqlite.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:163)
       at com.automattic.android.tracks.datasets.TracksDatabaseHelper.getDatabase(TracksDatabaseHelper.java:35)
       at com.automattic.android.tracks.datasets.TracksDatabaseHelper.getReadableDb(TracksDatabaseHelper.java:49)
       at com.automattic.android.tracks.datasets.EventTable.getEventsCount(EventTable.java:56)
       at com.automattic.android.tracks.TracksClient$2.run(TracksClient.java:137)
       at java.lang.Thread.run(Thread.java:818)
Caused by android.database.sqlite.SQLiteCantOpenDatabaseException: unable to open database file (code 14)
       at android.database.sqlite.SQLiteConnection.nativeExecute(SQLiteConnection.java)
       at android.database.sqlite.SQLiteConnection.execute(SQLiteConnection.java:555)
       at android.database.sqlite.SQLiteConnection.setLocaleFromConfiguration(SQLiteConnection.java:369)
       at android.database.sqlite.SQLiteConnection.open(SQLiteConnection.java:216)
       at android.database.sqlite.SQLiteConnection.open(SQLiteConnection.java:191)
       at android.database.sqlite.SQLiteConnectionPool.openConnectionLocked(SQLiteConnectionPool.java:463)
       at android.database.sqlite.SQLiteConnectionPool.open(SQLiteConnectionPool.java:185)
       at android.database.sqlite.SQLiteConnectionPool.open(SQLiteConnectionPool.java:177)
       at android.database.sqlite.SQLiteDatabase.openInner(SQLiteDatabase.java:806)
       at android.database.sqlite.SQLiteDatabase.open(SQLiteDatabase.java:791)
       at android.database.sqlite.SQLiteDatabase.openDatabase(SQLiteDatabase.java:694)
       at android.app.ContextImpl.openOrCreateDatabase(ContextImpl.java:581)
       at android.content.ContextWrapper.openOrCreateDatabase(ContextWrapper.java:275)
       at android.database.sqlite.SQLiteOpenHelper.getDatabaseLocked(SQLiteOpenHelper.java:223)
       at android.database.sqlite.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:163)
       at com.automattic.android.tracks.datasets.TracksDatabaseHelper.getDatabase(TracksDatabaseHelper.java:35)
       at com.automattic.android.tracks.datasets.TracksDatabaseHelper.getReadableDb(TracksDatabaseHelper.java:49)
       at com.automattic.android.tracks.datasets.EventTable.getEventsCount(EventTable.java:56)
       at com.automattic.android.tracks.TracksClient$2.run(TracksClient.java:137)
       at java.lang.Thread.run(Thread.java:818)

Offer fragment's lifecycle-aware breadcrumbs logger

Similar to this: https://github.com/getsentry/sentry-java/blob/main/sentry-android-core/src/main/java/io/sentry/android/core/ActivityLifecycleIntegration.java but for fragment.

This will have to be manually added by clients into fragments lifecycle as suggested in woocommerce/woocommerce-android#3891 (review)

Extract `crashloggin` package to a separate module

See internal discussion: p1626707401400500/1626676695.387300-slack-CC7L49W13

Drop usage of `BuildConfig.VERSION_NAME`

#51 (comment)

We'll have to do this before migration to AGP 4.1.x

Use WordPress-Utils after it has been splitted in WordPress-Utils / WordPress-Common

Use WordPress-Utils after it has been splitted in WordPress-Utils / WordPress-Common and then remove NetworkUtils, StringUtils and SqlUtils from the main repo

Extract ExPlat from WordPress-Android to Automattic-Tracks-Android

Extract ExPlat implementation from WordPress-Android to Automattic-Tracks-Android so it can also be used by woocommerce-android and other projects.