Data format discovery

[TomWright]

It would be nice if dasel could guess the data format when a parser is not defined.

Idea

Dasel will look at the given input and discover the data format (JSON for example).

This means the parser flag will no longer be required.

How can we actually detect the file type?

When reading from a file we can look at the extension... this is already done.

This idea is more around the parsing of some unknown string of characters that is piped into dasel.

1. If it starts with < it's XML.
2. If it starts with { it's JSON.
3. If it starts with [ and ends with ] it's JSON.
4. If it starts with [ and doesn't end with ] it's TOML.
5. If the first and second lines contain the same number of comma's it's CSV.
6. Otherwise it's YAML.

Gotcha's

1. This will need to be able to deal with multi-document files (JSON and YAML).

These are some pretty basic checks which could easily be caught out, but for general usage it may be good enough.

Does anyone have any ideas for this?

[ewen-lbh]

You could also use encoding/json's Validate (I guess that packages to marshal/unmarshal other formats also have an equivalent function), it'd complement those basic checks

This is how I would see it:

1. Try to validate XML
2. Try to validate TOML
3. Try to validate YAML
4. Try to validate JSON

Then use those results as such:

• If it's valid XML, it can only be XML
• If it's valid YAML
  • and If it's valid JSON, we consider it to be JSON (well all JSON files are valid YAML files since YAML is a superset of JSON)
  • and if it's valid TOML (somehow), we consider it to be TOML
  • and only valid YAML, it's YAML.
• if it's not valid YAML, it's TOML

The obvious drawback is that performance wise, checking validity of up to 4 file formats could be too slow, but I didn't test that

nice coverage btw

[TomWright]

Thanks for your input.

That does have drawbacks of potentially unmarshalling the entire document twice - once when finding the parser and again to do the lookup.

With some re-working of the logic it's possible to only unmarshal the data once in which case this would be great.

I'll leave this here for further ideas and input before I start working on it.