Cannot retrieve data from key value store agent

[radry]

I'm trying to read (serialize) data with a csv agent provided by key value storage agent.

The problem is that the data is always empty.

Key Value Store Agent control target is set to the CSV Agent.
In the CSV Agent "Controllers" is set to the Key Value Store Agent.

So Data exchange should be possible.

Key Value Storage Agent:

{
  "key": "{{ created_at | date: '%s' }}",
  "variable": "price",
  "value": {
    "title": "{{title}}",
    "price": "{{price}}"
  }
}

Sample Data:

{
  "1707746837": {
    "title": "foo",
    "price": "111"
  },
  "1707747142": {
    "title": "foo",
    "price": "111"
  },
  "1707747674": {
    "title": "foo",
    "price": "111"
  }
}

CSV Agent:

{
  "mode": "serialize",
  "separator": ",",
  "data_key": "data",
  "with_header_radio": "true",
  "with_header": "true",
  "use_fields": "price",
  "output": "event_per_file",
  "data_path": "$.*"
}

I tried several different values for "data_path" and "use_field" but it's always empty (To be more precise the values it outputs are "\n\n" )

[radry]

I figured out that aparently CSV Agent can't directly read from key value store.
I then tried to use an Even Formatting Agent, which was able to read the whole content of key value store but the output was just a string and not a properly formatted json or object.

The output of the Event Formating Agent looks like this:

"data": "{\"1707833407\":{\"title\":\"foo\",\"price\":\"111\"}}"

How can I output this as proper json so that csv agent can read the indiviual values? The problem is the data in key value store doesn't have a known value "key"...

So essentially I want to transfer the content of key value store to the csv agent. If I have to use an event formatting agent it should just pass the data 1:1 without reformatting it, so what do I put into "instructions"?

Event Formatting Agent:

{
  "instructions": {
    "data": "{{price | json}}"
  },
  "mode": "clean"
}

UPDATE:

With this event formatter agent it forwards all data from the key value store agent as a new event:

{
  "mode": "clean",
  "instructions": "{{price|as_object}}"
}

Now the CSV Agent receives an Event that looks like this:

{
  "1707746837": {
    "title": "foo",
    "price": "111"
  }
}

HOWEVER the problem now is that the CSV Agent outputs this, regardless of the data_path I set it to:

{
  "data": "\"1707833407\"\n\"{\"\"title\"\"=>\"\"foo\"\", \"\"price\"\"=>\"\"111\"\"}\"\n"
}

It's impossible to read the separate values and the resulting CSV is useless.

If I set data_path to $.*.title it gives the error:
undefined method 'collect' for "foo":String

If I set data_path to $.* and Use field to price the output is:

{
  "data": "\"price\"\n\n"
}

This doesn't make sense.

[virtadpt]

I never did figure out how to get the K/VS Agent to work. I do, however, recognize the problem you're seeing now: You're getting serialized JSON and not "real" JSON.

There might be an in-Huginn fix for that but I'm using Jq Agent to deserialize the strings into JSON documents that can then be processed.

{
    "filter": "[ .data | fromjson ]"
}

..should do what you need.

The Liquid module as used by Huginn has a json filter that can serialize JSON into a string, but no matching fromjson to deserialize it for some reason. Maybe as_object?

[radry]

@virtadpt

I want to have a CSV File at the end, which I can upload via ftp. I already managed to get an object/json out of K/VS if you read my second comment.
The issue that remains is that the CSV Agent doesn't process the object/json correctly and doesn't output a correct CSV File.

[virtadpt]

I did read it. You have to de-serialize the JSON (all of the \" stuff) before Csv Agent can work with it. I ran into the same problem with one of my agent networks, and that's what solved it.

[radry]

Where do I put that filter you suggest?

The CSV Agent already receives an object (the description says it can read objects??).

This is what the CSV Agent receives:

{
  "1707746837": {
    "title": "foo",
    "price": "111"
  }
}

Update:

When I use an JQ Agent and send the Events from the Event Formatter there it says this error:

Error output from jq:
jq: error (at <stdin>:0): null (null) only strings can be parsed

[virtadpt]

Right after your Csv Agent.

[radry]

@virtadpt

Results in this error:

Error output from jq:
jq: error (at <stdin>:0): Unexpected extra JSON values (while parsing '"{""title""=>""foo"", ""price""=>""111""}"
')

Besides, what will the JQ Agent output? I need a comma separated CSV File, not a JSON.

One of the problems is that I passed an object instead of json to the CSV Agent. But even when I change the Event Formatting Agent to output proper json, the CSV Agent still doesn't produce a CSV File.

[carmatana]

I'm trying to read (serialize) data with a csv agent provided by key value storage agent.

The problem is that the data is always empty.

Key Value Store Agent control target is set to the CSV Agent. In the CSV Agent "Controllers" is set to the Key Value Store Agent.

So Data exchange should be possible.

Key Value Storage Agent:

{
  "key": "{{ created_at | date: '%s' }}",
  "variable": "price",
  "value": {
    "title": "{{title}}",
    "price": "{{price}}"
  }
}

Sample Data:

{
  "1707746837": {
    "title": "foo",
    "price": "111"
  },
  "1707747142": {
    "title": "foo",
    "price": "111"
  },
  "1707747674": {
    "title": "foo",
    "price": "111"
  }
}

CSV Agent:

{
  "mode": "serialize",
  "separator": ",",
  "data_key": "data",
  "with_header_radio": "true",
  "with_header": "true",
  "use_fields": "price",
  "output": "event_per_file",
  "data_path": "$.*"
}

I tried several different values for "data_path" and "use_field" but it's always empty (To be more precise the values it outputs are "\n\n" )

Hi,

@radry

What do you do to have the Formatting Agent able to read de KeyValue Storage agent ?

I am stuck in that step.

The KeyValue Agent cannot be a source of the Formatting Agent, or at least it does not appear in the list of sources

[radry]

@carmatana

In the EventFormatting Agent you set the KeyValue Storage under "Controllers" (NOT "Source").

[carmatana]

Thanks @radry

I have my agents as you mention but without success:

I have an event that gets data from a webpage and send it to a Key Value Storage:

as shown in the image, data is arriving ok to the KVS agent and the EventFormating Agent is set as Controller Target.

Int the EventFormating agent I have:

But when dry-running the EventFormating Agent I receive an error message:

When running in normal mode the first agent (webpage) nothing happens with the Formatting agent.

Any idea what I am missing ?

[radry]

@carmatana

I don't remember exactly how I did and I don't have the scenario in its original state anymore.

If I remember correctly dry run doesn't work in this case, that's normal. You need to use an external (manual) trigger or send an actual event. Something must trigger the Event Formatting agent, only then it will pull data from the KVS agent.

Also I suggest you enable "propagate immediately" (it makes debugging easier)

[carmatana]

@radry

Thans for both, your attention and your suggestion.

You are right, I defined a Manual Agent that has as a Receiver the EventFormatting agent and it worked. Data flows form KVS agent to Formatting agent.

However, I arrived to the same problem you reported, data comes in a non useful format.

If you know the name of the key, you can get the value, for example in my case I have in the KVS agent:

variable: {{ var }}

and one of my key-value pairs is: "MX:30"

In the eventformatting agent I can extract only the value of that key with:

{{ var.MX }} or {{ var["MX"] }}

The result is only "30" without all the other non useful characters.

But I do not know how to extract the key name itself.

In my case keys are always the same, so this is not a problem, I can do:

text: MX,{{ var.MX }}{ % line_break %}
Key2,{{ var.key2 }}{ % line_break %}
Key3,{{ var.key3 }}{ % line_break %}
etc
etc

Note: the above goes everything in one single line, without line breaks.

In case I would like to have it in CSV format.

But for that purpose I guess that it is easier to have the data generating(retrieving) agents (i.e. Website Agent) writing directly to a file with the Local File Agent, adding one line for each event with a CSV structure, without using at all the KVS agent.

After that, if needed, the CSV agent can receive that file and process line by line or all the lines in bulk.

Thanks again !!