Gathering Feedback: Collecting User Statistics

[Sewer56]

Collecting Anonymous User Statistics for Reloaded-II

Sup Nerds.
I was recently considering gathering some statistics to aid the development of Reloaded3.

Essentially I'd release an update where Reloaded-II's launcher would keep a log of the things you do to modify your games' mod setups over an extended period of time. Then some day, say 2 months later, I'd release another update that would allow you to review this information, and anonymously upload it (one time only).

Main intended goal is optimizing how I store loadouts, so I can make them smaller.
A Loadout is just a per-game profile, think Create a Class in your shooter game. In Reloaded-II, there is only one, in Reloaded3, you could have multiple.

With statistics, I could find more common patterns in how people use Reloaded-II; and better plan around how mods themselves and loadout history should be structured.

I want to know what the people's thought about this is.
So I am asking for feedback regarding the matter.

Below is a list of bullet points I made about the specifics of the idea in mind.

---

Summary

A one time event where we collect information about how Reloaded-II is used over a period of ~2 months.

Ground Rules

• No Full File Paths:
  • To avoid collecting any information that might contain usernames or other sensitive data.
• No Personally Identifying Information: No usernames, etc. Enforce complete anonymity.
• Consent-Based: Uploading the data will be done with a manual click of a button by the user.
• Transparency: Users will be able to review all data before sending.
• Public Data Option: Users can opt-in to also make their anonymous data part of a public dataset.
  • That would be posted on GitHub at the end of the experiment.

Reloaded will never upload anything to the internet without user consent.
I don't do telemetry today, and I will never do telemetry.

Events to be Collected

The following user actions would be logged.

• {X} Game was launched.
• Mod load order for {X} game changed.
• Mod {X} was updated.
• Mod config {X} changed.
• Mod {X} was enabled/disabled.
• Mod {X} was installed.
• Sort order was changed (preserve vs. non-preserve).

This info would be directly used to better design how loadout info is compressed.

Game Mod Setup Information

• Number of mods.
• Sizes of mods.
• Installed mods (all ModIds).
• File sizes per extension (grouped by mod).

Launcher & System Info

• Current launcher theme.
  • To determine how many people are using custom themes.
• Launcher language.
• OS Language
  • Find out how many people with a non-English OS use a custom language.
• Is the user running Wine? (i.e. Are you on Linux/macOS/FreeBSD).
• Number of storage disks (size & type, e.g., SSD, HDD).
• Number of added games (to Reloaded).

---

Your Feedback

Basically I want to know.

• Do you consider collecting anonymous user statistics under these conditions to be acceptable?
• Are there any additional data points you think should be included or excluded?
• Any concerns or suggestions regarding privacy and data handling?

And anything else that comes to mind.
That's all.

[MarkKoz]

Do you consider collecting anonymous user statistics under these conditions to be acceptable?

Yes because it's opt-in, explicit, and users (even non-technical ones) are able to review exactly what is being submitted

Are there any additional data points you think should be included or excluded?

I don't understand enough about the requirements of a loadout to have an idea of what data would be useful to optimise its structure.

This may increase complexity, but consider adding a way for users to choose which of the gathered data is submitted.

Any concerns or suggestions regarding privacy and data handling?

As a software author you may claim to do one thing, and ostensibly that is true. However, users need to trust the software they download, because most do not have the skill or time to audit it. Regardless of what you claim, I could see this being taken in bad faith by some users and you losing their trust.

[Sewer56]

Currently this is just an idea.

I'm still not 100% sure if I want to go through with this, buit would have been bad to throw away the plan, so I figured I'd post it to at least gather feedback.

I don't understand enough about the requirements of a loadout to have an idea of what data would be useful to optimise its structure.

The explanation in the 'About' page may be handy here.

In Reloaded3, I plan to represent each loadout as a sequence of events. i.e. What has changed.
These events can be used to reconstruct the loadout as it was in any point in time.

There are 2 main requirements:

• Reproducibility: Sending your loadout allows a developer to reproduce your setup 1:1 to help debug crashes, etc.
  • This also enables Cloud Sync-ing, for end users.
• Time Machine: Ability to roll back a loadout to an earlier point in time, for example if you made a change that broke your game.

The Problem

I'm not sure who would be willing to host the data for Cloud Sync.
Hosting it is actually very cheap, but I have a revenue of £0.
In fact, I'm around 300-400 GBP in the negative if subtracting historical bills (domain name, VPS etc.) from all donations I've ever received.

To achieve this, ideally I'd love to put Reloaded3 on GOG or Steam, and use the cloud save functionality. That would be a win-win; end users could get the software really easily, and my cloud costs would be subsidized. Alternatively, I could try to convince my employer (NexusMods) to provide some per user storage in the API; but that's not really guaranteed. These entities may ultimately turn around and ask what's in it for me?.

So if worst comes to worst, I might have to self host, otherwise the process may be inconvenient for non-technical people.
Therefore in order to minimize costs, and to also be more convincing to external entities, I have to minimize the size of cloud synchronized information.

So to be very explicit, I'm interested in:

• The statistical relative probabilities of each event.
• And which events are commonly done together.

Most of the space used by each loadout is expected to be events (Events.bin Documentation). These are very heavily bit packed; you'll notice that each event has multiple lengths. When loadouts are small, we can generate smaller sized events, and save even more space.

However there are opportunities for even more savings.

For example, a common thing a user may do is first drag a mod in their mod list, changing its load order. And then Disable or Enable a mod.
Under the current encoding that I've spent the last 2 weekends planning, this would take 2 events, likely 4 bytes each. One for moving and one for enabling/disabling. These also have timestamps (4 byte each) and a commit message (1 byte). So 18 bytes total.

However if end users perform these two actions commonly together, it may be possible to create a 'combined' action, that moves AND changes the state of a mod (1 less event), saving 4 bytes on the event, and 5 bytes on timestamp and message. This means going from 18 bytes to 9, a 50% saving.

Right now, I don't have these sorts of statistics however. Logging would help me identify usage patterns (like this one) and better compress the data.

[Note: On the live docs, timestamps aren't 4 bytes yet, that will change when I commit later]

---

As for some other statistics:

• File sizes per extension (grouped by mod).
• Number of storage disks (size & type, e.g., SSD, HDD).

I want to automatically offload mods containing music/video (stuff that doesn't need fast access speeds) to HDD on multi disk systems. To help users keeping their SSD free space empty.

• Launcher & OS Language.

I want to know if getting multi-language support in the loader should be a priority in order to help foreign users read logs.

• Sizes of mods.

I want to know how common big monolithic mods are.

I want to steer mods towards being as modular as possible. By identifying big mods that are commonly used, I can start planning and thinking how these mods can be reimagined as multiple smaller mods rather than 1 big pack.

---

Regardless of what you claim, I could see this being taken in bad faith by some users and you losing their trust.

That's what keeps me paranoid about trying.

A lot of software these days is opt-out, never tells you, phones home and gets away with it.
If I did that, chances are nobody would ever notice, but I refuse to be that person.

I want to do things the right way.
I want to do things that respect the user.

I specifically picked out data that is only relevant to the information I want to learn. Data that does not contain anything user specific, and the ability to review data before it is sent. I might let the user check what they want and don't want to submit, but I don't want to spend too much time writing the code, as ultimately all of it will be thrown away once the 2 month event ends. (And I'm also just 1 person).

[Pixelguin]

I think this would be a great idea! "Telemetry" gets a bad rap because corporations use it as an excuse to collect all sorts of data they really have no business keeping, but it's an important part of making sure community software developers focus on what's important to users.

[SirGamers]

• Do you consider collecting anonymous user statistics under these conditions to be acceptable?

yep, seems good to me, plus its opt in/out (which I will totally opt in)

• Are there any additional data points you think should be included or excluded?

not really, I think you've covered all the bases

• Any concerns or suggestions regarding privacy and data handling?

ehhhhhh, no

to solve the cloud problem, I think steam and gog is a good idea but...