Example of how to extract say the 'Template' from the dump file?

[Jibun-no-Kage]

Example of how to extract say the 'Template' from the dump file? For example one may have a dump file, but just need the device template, not the entire dump file. If one restores the entire dump file back to the device, that would be overkill when only the template is needed. This would be a typical use case for a typical user, one would think.

[curzon01]

Use JSON output format for backup instead of dmp files, this allows editing of your backup data and restoring subset of data.

Example for 'Template' related config data:

Step 1: Extract dmp data into human readable format (JSON)

decode-config.py -s config.dmp -o config.json

to reduce the JSON file before editing, the output can be limited to a command group (template is part of the Management command group):

decode-config.py -s config.dmp -o config.json -g Management

This creates an editable JSON file config.json.

Both examples work, but the second only contains the configuration data of Management cmnd and is easier to work with.

Step 2: Edit the generated JSON file to limit it to the desired data

We need the main JSON keys containing "template", all others can be deleted (delete also the complete main key "header", it contains a template subkey which is not needed).

After editing, it should look like this:

{
  "templatename": "Generic", 
  "user_template": {
    "base": 18, 
    "flag": 0, 
    "gpio": [
      65504, 
      65504, 
      65504, 
      65504, 
      65504, 
      65504, 
      65504, 
      65504, 
      65504, 
      65504, 
      65504, 
      65504, 
      65504, 
      4704
    ], 
    "name": "Generic"
  }
}

Note: Be aware to delete the last comma after the last key as described (see Hint under Restore subset of data).

Step 3: Write back data to dmp file or to a device directly

Overwrite your original dmp file 'config.dmp'

decode-config.py -s config.dmp -i config.json

Write template to device direclty

decode-config.py -s 192.168.15.9 -i config.json

Note for ESP32:

For ESP32 based device, always keep the key "config_version" within your JSON.

{
  "config_version": 1, 
  "templatename": "NeoPool", 
  "user_template_esp32": {
    "base": 1, 
    "flag": 0, 
    "gpio": [
      0, 
      0, 
      0, 
      0, 
      0, 
      0, 
      0, 
      0, 
      0, 
      0, 
      0, 
      0, 
      0, 
      0, 
      0, 
      1, 
      0, 
      1, 
      1, 
      1, 
      0, 
      0, 
      6976, 
      0, 
      0, 
      0, 
      0, 
      0, 
      7008, 
      1, 
      0, 
      0, 
      0, 
      0, 
      0, 
      0
    ], 
    "name": "Atom Lite Sugar Valley"
  }
}

If your are using group limit -g, append group 'Internal' to get this key within your JSON:

decode-config.py -s config.dmp -o config.json -g Management Internal

[Subnum12]

For what reason I cannot create a new dmp file from modified JSON file? With above description I only can update existing dump file but I need to remove options from dump file. That is required if you remove rules and add scripts to firmware.

[curzon01]

With above description I only can update existing dump file but I need to remove options from dump file

A dmp file is a binary map of the Tasmota setting structure.
Tasmota therefore always expects a complete binary file with all Tasmota settings in a dmp file, so (unlike the decode-config JSON format) you can't remove or omit structural elements from a dmp file. You can only overwrite them with new data or leaving them unchanged.

That is required if you remove rules and add scripts to firmware.

To do this, remove the "rules" key from your the JSON

  "rules": ["", "", ""], 

and add the "script" key with your script

  "script": ">D\nscript error must start with >D", 

Notes:

1. Rules and Script are special on Tasmota because both use the same memory locations therefor this is why you can't use Tasmota Rules and Script at the same time.
   You also can't not load script json settings to Tasmota which is compiled with enabled rules and you also can't load rules json settings to Tasmota which is compiled with enabled script.

2. For what reason I cannot create a new dmp file from modified JSON file?

Because it makes no sense: You can only upload a dmp file to a Tasmota device afterwards, but decode-config can also do this live to an online device (see overview image), so this unnecessary intermediate step (JSON -> dmp -> Tasmota) was omitted.

However, decode-config does not generate the left side within the overview image (decode-config parameter -s), this applies to a physical tasmota as well as to a dmp file - the source must always exist.

[Subnum12]

Thanks for detailed description.

Actually this question was cause by unsuccessful attempts to recover devices. As this issue maybe also caused by additional bug of Tasmota let me just summarize what I've tested with regards to decode-config.

a.dmp file contains "script"
b.dmp file contains "rules"

a.json:

{
    "script": ""
}

b.json

{
    "rules": ["", "", ""]
}

decode-config -s a.dmp --backup-file Out.dmp -i a.json
INFO: Load data from file 'a.dmp'
INFO: Backup successful to '.\Out.dmp' (Tasmota format)
INFO: Restore skipped, configuration data unchanged

decode-config -s a.dmp --backup-file Out.dmp -i b.json
INFO: Load data from file 'a.dmp'
INFO: Backup successful to '.\Out.dmp' (Tasmota format)
WARNING 9 (@5480): file '.\r.json' array 'rules' couldn't restore, format has changed! Restore value contains <class 'list'> but an array of size [3] is expected
Premature exit - #9 Restore data error

decode-config -s b.dmp --backup-file Out.dmp -i a.json
INFO: Load data from file 'a.dmp'
INFO: Backup successful to '.\Out.dmp' (Tasmota format)
Traceback (most recent call last):
  File "C:\Python\Python311\Scripts\decode-config.py", line 6865, in <module>
    restore(ARGS.restorefile, ARGS.backupfileformat, CONFIG)
  File "C:\Python\Python311\Scripts\decode-config.py", line 6226, in restore
    new_decode_cfg = mapping2bin(config, jsonconfig, restorefilename)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python\Python311\Scripts\decode-config.py", line 6032, in mapping2bin
    set_field(_buffer, config['info']['hardware'], name, setting_fielddef, data, addroffset=0, filename=filename)
  File "C:\Python\Python311\Scripts\decode-config.py", line 5582, in set_field
    valid = min_ <= len(value) <= max_
                    ^^^^^^^^^^
TypeError: object of type 'bool' has no len()

[curzon01]

Test 1 and 2 do what I described. Test 3 could still be intercepted at some point.
What is the aim of these tests? The results are expected

[curzon01]

Fixed error for your third test (#85)

Your second test can not be reproduced - there is no error when loading rules based json into a script based config. Your 2. example does not match input/output (used param -i b.json but screen copy warning outputs file '.\r.json'...) - sorry for such lot misstakes in #85 but a and b naming for rules and scripts totaly confused me :-)

[Subnum12]

What is the aim of these tests? The results are expected

I like to switch between firmware compiled with define USE_SCRIPT and default firmware with rules enabled. If so, I want to replace script by rules and and vice versa using -i option. Everything else should remain as it is.
Main concern here is I do not have network access to devices on that time, meaning I cannot use decode-config to directly upload config. I have to create a dmp file to upload it via web-api later on.

Now I can confirm test 2 & test 3 on decode-config works without any error. However, I cannot finally test the behavior because of Tasmota issue 20702.

What looks still unusual to me, after running test 2 and 3 even with new content in rules or scripts parameter I get this output:
INFO: Restore skipped, configuration data unchanged

Also a decode-config -s Out.dmp does not show the content of previously provided -i json file. the element name (script/rules) remains empty.

A dmp file is a binary map of the Tasmota setting structure.
Tasmota therefore always expects a complete binary file with all Tasmota settings in a dmp file, so (unlike the decode-config JSON format) you can't remove or omit structural elements from a dmp file. You can only overwrite them with new data or leaving them unchanged.

My I would not expect the element name to be changed, however the content should have changed from -i json file.

[curzon01]

What looks still unusual to me, after running test 2 and 3 even with new content in rules or scripts parameter I get this output:
INFO: Restore skipped, configuration data unchanged

This is the expected result.
Test 2 and 3 check whether the cross-ordered data are ignored so that there are no changes for each restore.

Whereas changes to the currently valid JSON keys are always accepted.

Let do the tests:

1. We use the two dmp files rules.dmp and script.dmp from the respective Tasmota with enabled rules and enabled script (I will not name it a and b :-) )

Force check of empty data:

rulesscript.json

{
    "rules": ["", "", ""],
    "script": ""
}

> /decode-config.py -s rules.dmp -i rulesscript.json
INFO: Load data from file 'rules.dmp'
INFO: Restore skipped, configuration data unchanged

> ./decode-config.py -s script.dmp -i rulesscript.json
INFO: Load data from file 'script.dmp'
INFO: Restore successful to file 'script.dmp' from './script.json'

and check a second time:

> ./decode-config.py -s rules.dmp | jq '.rules'
[
  "",
  "",
  ""
]
> ./decode-config.py -s script.dmp | jq '.script'
""

2. Make some changes in backup json

rulesscript.json

{
    "rules": ["ON Event#Test DO DELAY 1 ENDON", "", ""],
    "script": ">D\nscript error must start with >D"
}

3. Now restore it to the desired system dmp

> ./decode-config.py -s rules.dmp -i rulesscript.json -v
INFO: Load data from file 'rules.dmp'
INFO: File 'rules.dmp' is using Tasmota v13.3.0.5 on ESP82
INFO: Reading restore file './rulesscript.json' (JSON format)
INFO: Value for 'rules' changed from "" to "ON Event#Test DO DELAY 1 ENDON"
INFO: Config file contains data of Tasmota v13.3.0.5
INFO: Write new data to file 'rules.dmp' using restore file './rulesscript.json'
INFO: Restore successful to file 'rules.dmp' from './rulesscript.json'

> ./decode-config.py -s script.dmp -i rulesscript.json -v
INFO: Load data from file 'script.dmp'
INFO: File 'script.dmp' is using Tasmota v13.3.0.5 on ESP82
INFO: Reading restore file './rulesscript.json' (JSON format)
INFO: Value for 'script' changed from "" to "3f08437783fe8ca4fffcfa1a2fd4bd57fa43fb4291f8421b"
INFO: Config file contains data of Tasmota v13.3.0.5
INFO: Write new data to file 'script.dmp' using restore file './rulesscript.json'
INFO: Restore successful to file 'script.dmp' from './rulesscript.json'

(don't be surprised about the output for script changes, this is the compressed string)

and final check:

> ./decode-config.py -s rules.dmp | jq '.rules'
[
  "ON  Event#Test DO DELAY 1 ENDON",
  "",
  ""
]
> ./decode-config.py -s script.dmp | jq '.script'
">D\n script error must start with >D"

What you also see here by using a single JSON backup file is that invalid keys from the JSON are also ignored for the respective Tasmota configuration.

That's exactly how it should be.

Main concern here is I do not have network access to devices on that time, meaning I cannot use decode-config to directly upload config. I have to create a dmp file to upload it via web-api later on.

If your devices are comletely offline, that's the only possible way. If you have access via MQTT you can use decode-config to update your devices too.

[curzon01]

Two additonals to your issue:

Deleting data

My I would not expect the element name to be changed, however the content should have changed from -i json file.

I didn't say somthing about renaming elements, my comment was about your request "removing" data. The complete quote

With above description I only can update existing dump file but I need to remove options from dump file

You can only overwrite them with new data or leaving them unchanged.

removing something from a binary dmp is impossible. As already mentioned, you can overwrite values or leave them as they are. What you want and need is overwrite rules/script with an empty string and that's what decode-config does

Switching rules/script based firmware

Some hint about that as both occupy the same memory area:

Switching from rule to script based tasmota normally might be valid without an issue. Anyway if you load a rule-based Tasmota firmware into a device that previously contained a script-based Tasmota firmware and didn't previously deleted the script, more or less problems may occur.

However, save way before switching between both firmware is complete deleting rules (on rules base Tasmota) and script (on script based Tasmota) before update this device to the new firmware.