Expand Stream's system design documentation?

[xeoncross]

The docs have a great intro that explains the technology buildup to arrive at inventing stream but then it stops without explaining how stream uses Cassandra + Redis (plus celery message queue?) to solve this problem. (For all I know it doesn't.)

As a developer, a quick explanation of how this framework solves the problem would be really great. High-level overviews don't answer the how.

[piperck]

I'am wondering so I see doc. But I saw doc and I'am lost.

[dwightgunning]

This is definitely something that could be expanded upon, but as you can probably appreciate it's a matter of contributor bandwidth.

There is a brief overview of (Activity Storage Strategies)[https://github.com/tschellenbach/Stream-Framework/blob/master/docs/activity.rst#activity-storage-strategies] and guidance on (Choosing a storage layer)[https://github.com/tschellenbach/Stream-Framework/blob/master/docs/choosing_a_storage_backend.rst] with guidance and pros/cons when choosing out of Redis and Cassandra.

To the question of "how stream uses Cassandra + Redis", are you able to be more specific about which aspect(s) of the architecture are of interest? Perhaps we can prioritise a list of topics to work on.

[xeoncross]

Stuff like this is a good start:

When using Cassandra as storage Stream Framework will denormalize activities; there is not an activity storage but instead every user feed will keep the complete activity

But the structure of the denormalized objects would be interesting (or why you chose to do that rather than split up the objects into multiple keys like redis often encourages).

Basically, this is built for developers but I feel like the documentation focuses on high-level overviews instead of details like "a user object consists of these 9 fields in cassandra of which X,Y,Z have indexes, but is split up into 3 hashes when using redis..."

[dwightgunning]

I can see you're quite active with several OSS projects so I'm interested in your opinion on this.

I've only recently become involved with stream-framework but it seems the focus for the docs is to help developers get started quickly and to provide guidance on how to integrate the framework. My experience with a couple communities has been that most often the discussions and details about underlying design decisions occur in threads like this, over IRC, and via email.

It'd be helpful to find some good examples of how to approach this in general, and perhaps how to structure this kind of information. Any projects you feel would offer inspiration or could be used as a reference?

[xeoncross]

It'd be helpful to find some good examples of how to approach this in general

It has occurred to me that many projects have trouble with preparing technical design documentation. Part of it stems from the fact that no-one wants to take time to record every design decision.

I believe the easiest way to begin fleshing out the nitty-gritty details is simply to paste the lowest level object breakdown you can. This would probably be a schema allowing someone to quickly assess the actual implementation. For example, MediaWiki posts a database layout but you could simple have a text page for the cassandra schema and another one for the redis layout.

Then as people have time or pickup the product they can add notes (We index FOO to help reduce load of single-user date searches). This would allow a new developer (like me) to be able to jump into the codebase already aware of the types of CRUD and object handling I should expect.