Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Analytic Azos.Log.Fact with arch log encapsulation: {type: atom, dims: {}, data: {}} #861

Closed
itadapter opened this issue May 1, 2023 · 12 comments
Assignees
Labels
- RTM - Released to manufacturing CONCEPT Needs conceptual design session PROJECT A project which is comprised of multiple tasks

Comments

@itadapter
Copy link
Contributor

itadapter commented May 1, 2023

Related:

Big Data analytics project

Schema of analytics event today is:

public Guid EmitAnalyticsEvent(Atom eventType,
                                   object dims,
                                   object metrics,
                                   int source = 0,
                                   Guid rel = default(Guid),
                                   MessageType messageType = MessageType.Info,
                                   DateTime utcTimeStamp = default)
@itadapter itadapter self-assigned this May 1, 2023
@itadapter itadapter added the CONCEPT Needs conceptual design session label May 1, 2023
@itadapter
Copy link
Contributor Author

We would still need HOST, APP, GUID,...why not use Log.Message as-is?
What benefit does a new type provide (if any) ? I think it adds bloat and complexity.

This type might be needed in ANALYSIS services, but not here.

@itadapter
Copy link
Contributor Author

We need to add Fact instead

@itadapter itadapter changed the title Add Analytic event record with arch log and processing methods: {type: atom, dims: {}, data: {}} Add Analytic Azos.Log.Fact with arch log encapsulation: {type: atom, dims: {}, data: {}} May 8, 2023
@itadapter
Copy link
Contributor Author

Azos.Log.Fact

Use Log.ArchiveConventions to form/extract facts to/from log messages

@itadapter
Copy link
Contributor Author

Need unit tests for: ArchiveConventions all methods

zhabis added a commit that referenced this issue May 10, 2023
zhabis added a commit that referenced this issue May 10, 2023
zhabis added a commit that referenced this issue May 10, 2023
zhabis added a commit that referenced this issue May 10, 2023
zhabis added a commit that referenced this issue May 10, 2023
zhabis added a commit that referenced this issue May 10, 2023
@itadapter
Copy link
Contributor Author

Need more Bixon tests

zhabis added a commit that referenced this issue May 10, 2023
zhabis added a commit that referenced this issue May 11, 2023
@itadapter
Copy link
Contributor Author

  • Add test for data documents
  • Convert DOCs to JsonDataMap using JsonDataMap ToJsonDataMap(JsonWritingOptions options = null), this way we dont need to write doc as Json
  • Add perf test JSON vs Bixon

@itadapter
Copy link
Contributor Author

itadapter commented May 11, 2023

    2023 May 11 .Net 6 Release
    --------------------------
Complex:
      JSON wrote 32,000 in 1.0 sec at 30,776 ops/sec; 1,590 chars
      JSON read 32,000 in 2.6 sec at 12,181 ops/sec
      BIXON wrote 32,000 in 0.8 sec at 41,327 ops/sec; 1,201 bytes
      BIXON read 32,000 in 0.8 sec at 41,088 ops/sec

Simple:
     JSON wrote 250,000 in 0.8 sec at 315,031 ops/sec; 307 chars
     JSON read 250,000 in 2.3 sec at 108,265 ops/sec
     BIXON wrote 250,000 in 0.8 sec at 329,386 ops/sec; 265 bytes
     BIXON read 250,000 in 0.5 sec at 541,920 ops/sec

zhabis added a commit that referenced this issue May 12, 2023
@itadapter
Copy link
Contributor Author

itadapter commented May 12, 2023

2023 May 12 .Net 6 Release
    --------------------------
Complex:
  JSON wrote 32,000 in 1.0 sec at 31,409 ops/sec; 1,590 chars
  JSON read 32,000 in 2.6 sec at 12,312 ops/sec
  BIXON wrote 32,000 in 0.9 sec at 35,354 ops/sec; 1,201 bytes 
  BIXON read 32,000 in 0.7 sec at 42,765 ops/sec

Simple:
  JSON wrote 250,000 in 0.8 sec at 308,273 ops/sec; 307 chars
JSON read 250,000 in 2.3 sec at 110,971 ops/sec
  BIXON wrote 250,000 in 0.9 sec at 272,497 ops/sec; 265 bytes
  BIXON read 250,000 in 0.5 sec at 516,437 ops/sec

@itadapter
Copy link
Contributor Author

todo:

  • add unit test for JsonReader.DocReadOptions.By.BixonDoNotMaterializeDocuments
  • check all unit tests manually with printed dumps

itadapter added a commit that referenced this issue May 17, 2023
Wip #861 Facts, Type Hints, Bixon
zhabis added a commit that referenced this issue May 17, 2023
@itadapter itadapter added the PROJECT A project which is comprised of multiple tasks label May 17, 2023
@itadapter
Copy link
Contributor Author

itadapter commented May 17, 2023

The JSON serialization for Fact should be akin to that of Message-
the EMPTY Rel guids should be excluded from json map:

{
			"FactType": "SnLoad",
			"Id": "5359ff15-dd8d-458e-a22a-d7f736bce777",
			"RelatedId": "00000000-0000-0000-0000-000000000000",//<========================
			"Gdid": "0:2:195833",
			"Channel": "ua-anl",
			"Topic": "ua",
			"Host": "Mac-Studio",
			"App": "uafolio",
			"RecordType": "PerformanceInstrumentation",
			"Source": 0,
			"UtcTimestamp": "2023-05-08T19:09:18.596Z",
			"Dimensions": {
				"scene": "Appt_Dashboard",
				"user": "4309"
			},
			"Metrics": null
		},

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
- RTM - Released to manufacturing CONCEPT Needs conceptual design session PROJECT A project which is comprised of multiple tasks
Projects
None yet
Development

No branches or pull requests

1 participant