Removing QA and internal users from analytics pipeline

When developers and publishers think of operating games as a service, what comes to mind are usually live ops, content drops and analytics. Product managers pour over reports to tease out player insights to make game and content changes; while live ops and lifecycle marketing teams look for nuggets to improve their events and campaigns. This post will address an overlooked issue - increasing numbers of QA and test users.

With more content drops, lifecycle campaigns and live ops events, the number of QA and testing users will increase dramatically.

The reason is that QA always needs to simulate the first time experience of players. As such new QA and test user ids always show up in the data pipeline. Add to it the need for testing in multiple languages and the problem is further compounded.

What's the big deal?

Since data and reports are used to assess performance, it's critical that QA and test users are not included in the reports. These include UA reports where QA users tend to show up as Organic users. Monetization and retention reports are all affected since QA has to test the in-app purchase funnel. On the flip side, if QA users find a bug and developers want to reproduce the bug, it is helpful to have data report showing the exact sequence of user actions.

The bottom line is that the data pipeline has to allow for the tagging and identification of QA and test users. Such tagging is harder because QA tend to be outsourced and executed in global locations. Achieving compliance with tagging at the QA sites needs constant vigilance and validation. One way is to conduct QA user id audits at the end of each QA cycle. Comparing the number of newly created QA users in the database with the QA test plan will allow for confidence that QA users are not leaking into the reporting pipeline.

Technical challenges with tagging QA users

Most game data pipelines start by creating a user id when a player starts the game for the first time. From that first game play session or player registration, all the user's events are tied to that user id. Since QA users have to simulate the first time user experience, their QA test session will create new user ids. How then do you go about tagging these user ids as QA or test users?

One way is to ensure that the QA user can write down or save the user id for their test session. This requires both code in the game to expose the user id, and also a test plan step for QA users to record the user id used in the testing. The crucial second step involves having a process to load these QA user ids into the data pipeline and tag them as QA user ids.

Another method is to allow the QA user to tag themselves within the game test session. This would require code in the game to allow hidden tagging features. The second step here would be for the pipeline to route these tagged user ids to a separate process. This approach is less error prone, but requires upfront development.

Keep or delete QA test data

Once the data pipeline separates the QA user data from the main pipeline, a common question is whether to delete or keep the QA data. On the one hand, storing the QA data adds complexities to the data pipeline and database structure. There will be "QA" versions of all the tables in the database, and these will need regular maintenance. However these QA tables can be invaluable when debugging and reproducing issues. These tables can show the exact sequence of events that led up to a nasty bug.  These tables may also be required for QA compliance audit.

Subscribe to our monthly newsletter so you don't miss any Sonamine content.