[Test Build] Upcoming X-mas build, testers wanted #4917

TD-er · 2023-12-23T12:27:02Z

We're approaching Christmas and I know I mentioned there will be a X-mas release, but I'm not 100% sure it will be ready.
This all depends on the amount of critical bugs found by all of you.
@fly74 @ghtester @chemmex @alex-kiev @iz8mbw @Wookbert @Dickesplumpel

Please test the latest build (same as this GH Actions build ) and let me know what critical issues you may find.

The ESP32 builds with "LittleFS" are all running the latest ESP-IDF 5.1 code (upto commits of a few days ago from Espressif)
New platforms are ESP32-C2 and ESP32-C6.
Those only have a 'custom' and 'normal' build right now as I plan on adding a 2-step OTA for those so these can all have a small partition for the OTA and a large partition for the build. This concept allows for all 4M boards to run a "Max" build and thus will reduce the number of build files significantly.
This will later also be made available for all other ESP32-variants including some steps to migrate existing setups.

All ESP32-builds still running SPIFFS are based on ESP-IDF 4.4

I've been working on this migration to IDF5.x for over 2 months full time, as a lot had to be changed and upstream bugs had to be fixed for Arduino and ESP-IDF.
So I would be pleasantly surprised if no critical bugs were found.

If you plan on migrating a build using SPIFFS to one using LittleFS, please keep in mind you will loose the settings.
So please backup your settings and restore them when making this migration.

There is also some extra control of persistent settings like WiFi credentials etc. which will be stored outside the file system.
So you could first update to a SPIFFS build, then check all the boxes on the "Factory Reset" page, save and wait for at least 5 minutes to let ESPEasy know the WiFi connection is considered stable. Then extra WiFI info will be stored in the NVS partition.
If you then migrate to a LittleFS build, your WiFi will be kept (and all other things checked on that factory reset page)

Dickesplumpel · 2023-12-23T16:03:00Z

Esp8266-4M1M seems to work generaly perfect but I have trouble with dummy device or with taskvalueset, even if "taskvalueset to all plugins" is enabled taskvalueset to dummy device seems not working anymore.

fly74 · 2023-12-23T17:12:12Z

Esp8266-4M1M seems to work generaly perfect but I have trouble with dummy device or with taskvalueset, even if "taskvalueset to all plugins" is enabled taskvalueset to dummy device seems not working anymore.

Would be important to me.

TD-er · 2023-12-23T18:19:50Z

Hmm sounds like a bug which needs to be fixed before any release (and thus exactly the reason I asked for testing :) )
Happy to hear about this bug :)

TD-er · 2023-12-23T18:52:37Z

That was easy to fix, or a stupid bug... ;)

When the build is ready, I will upload it to the webflasher.

TD-er · 2023-12-23T19:48:48Z

Latest build with fix is now uploaded to the web flasher.

Dickesplumpel · 2023-12-23T20:01:51Z

Taskvalueset works! 👍
Maybe tomorrow I have a closer look at ESP32-S2 and ESP32 builds.

TD-er · 2023-12-23T20:04:24Z

There is still one issue I'm looking into right now and that's when calling wifidisconnect, the ESP sometimes reboots with a crash.
At least when using LittleFS builds on the ESP32-C6.

fly74 · 2023-12-23T20:11:44Z

That was easy to fix, or a stupid bug... ;)

When the build is ready, I will upload it to the webflasher.

Looks good in my landscape.

fly74 · 2023-12-23T21:26:56Z

Houston there is a problem:

I have a

wich is filled remotely with MMddHHmm to compare in a rule, if the time is longer than 6 minutes away.

let,1,[D1#Dummy]+6

if [var#1]>%sysmonth%%sysday_0%%syshour_0%%sysmin_0%
...
else

in this release it always comes to the else path, even if D1 is not longer than 6 minutes away.

2nd if I try to change a rule, save does nothing. (MS Edge)

tonhuisman · 2023-12-23T21:44:56Z

2nd if I try to change a rule, save does nothing. (MS Edge)

What build environment is that on? (as in: Works on my ESP32 machine 😛)

fly74 · 2023-12-23T21:46:43Z

2nd if I try to change a rule, save does nothing. (MS Edge)

What build environment is that on? (as in: Works on my ESP32 machine 😛)

In MS Edge?

tonhuisman · 2023-12-23T21:48:21Z

no, ESP8266, ESP32, etc. Normal, Collection, Display?

fly74 · 2023-12-23T21:52:05Z

<html>
<body>
<!--StartFragment-->
Firmware
--
Build:⋄ | 20231223 - Mega32
System Libraries:⋄ | ESP32 SDK 4.4.6
Git Build:⋄ | HEAD_40df5ee
Plugin Count:⋄ | 48 [Normal]
Build Origin: | GitHub Actions
Build Time:⋄ | Dec 23 2023 19:15:05
Binary Filename:⋄ | ESP_Easy_mega_20231223_normal_ESP32_4M316k
Build Platform:⋄ | Linux-6.2.0-1018-azure-x86_64-with-glibc2.35
Git HEAD:⋄ | HEAD_40df5ee
System Status
Syslog Log Level: | None
Serial Log Level: | None
Web Log Level: | None
Enable Serial Port Console: | true
Console Serial Port: | HW Serial0 RX:3 TX:1 @ 115201 baud
Network Services
Network Connected: | ✔
NTP Initialized: | ✔
ESP Board
ESP Chip ID: | 16169629 (0xF6BA9D)
ESP Chip Frequency: | 240 MHz
ESP Crystal Frequency: | 40 MHz
ESP APB Frequency: | 80 MHz
ESP Chip Model: | ESP32-D0WDQ6
ESP Chip Features: | Wi-Fi bgn / BLE
ESP Chip Revision: | 1.0
ESP Chip Cores: | 2
ESP Board Name: | Espressif Generic ESP32 4M Flash ESPEasy 1810k Code/OTA 316k FS

<!--EndFragment-->
</body>
</html>

tonhuisman · 2023-12-23T22:11:20Z

And what build did that still work on?, as it seems to be related to parsing the if condition, and that hasn't recently changed, AFAICS.
if I change your rule to:

Let,1,[D1#Dummy]+6
Let,2,%sysmonth%%sysday_0%
  If [var#1]>%v2%%syshour_0%%sysmin_0%

it works as intended, though I can see what you expect

fly74 · 2023-12-23T22:12:52Z

If I could save the rule I would test it.

tonhuisman · 2023-12-23T22:13:41Z

Can you hard-refresh the browser page, so it can re-fetch the js code?

tonhuisman · 2023-12-23T22:16:42Z

or watch the log in a second browser tab (or serial log) to see if it throws an error when saving the rules file?

TD-er · 2023-12-23T22:19:23Z

What kind of "sensor type" have you used for the dummy?
This looks like the float resolution, which is essentially 6 decimals and you're using 8 decimals here.
This is exactly why I added other integer types to the dummy.

fly74 · 2023-12-23T22:22:09Z

Can you hard-refresh the browser page, so it can re-fetch the js code?

Firefox don't work too. But download and edit and upload works.
@tonhuisman The rulechange works. Thx.

So only the save would be interesting.

Browserdebug:

fly74 · 2023-12-23T22:23:56Z

What kind of "sensor type" have you used for the dummy? This looks like the float resolution, which is essentially 6 decimals and you're using 8 decimals here. This is exactly why I added other integer types to the dummy.

TD-er · 2023-12-23T22:24:31Z

What I meant is this selection on the Dumy task:

You're referring to the value of the dummy task value, which is by default a float. (Single/dual/quad etc are all float)

fly74 · 2023-12-23T22:27:39Z

What I meant is this selection on the Dumy task:

You're referring to the value of the dummy task value, which is by default a float. (Single/dual/quad etc are all float)

But Ton's rulechange works with single :/

tonhuisman · 2023-12-23T22:32:13Z

Seems to be a parsing issue, as I got an error with the original rule that didn't have the %sysmonth% part replaced

Edit:
2011416: Calculate: Unknown token input: %sysmonth%232333 = 0

TD-er · 2023-12-23T22:32:23Z

But Ton's rulechange works with single :/

Yep, it looks like he is splitting the value into separate floats.
But depending how the value is being set via remote calls, this will still fail on new updates.

If you set the new value via an event value, then you can split it with enough resolution just fine.
But if it is stored in the dummy as a single value, then a new update may not appear to change the task value.

Edit:
This has always been a problem, that's exactly why I added these other options to store int values and even int64 types if really needed.

Edit2:
Any integer value outside the range of -8388608 ... 8388607 will be stored with reduced resolution as a float simply can't represent those values without loss of resolution.

fly74 · 2023-12-23T22:39:47Z

But Ton's rulechange works with single :/

Yep, it looks like he is splitting the value into separate floats. But depending how the value is being set via remote calls, this will still fail on new updates.

If you set the new value via an event value, then you can split it with enough resolution just fine. But if it is stored in the dummy as a single value, then a new update may not appear to change the task value.

Edit: This has always been a problem, that's exactly why I added these other options to store int values and even int64 types if really needed.

Edit2: Any integer value outside the range of -8388608 ... 8388607 will be stored with reduced resolution as a float simply can't represent those values without loss of resolution.

@TD-er I can't recognize the problem. The 8 digit are set remotely to

to single.

TD-er · 2023-12-23T22:46:24Z

Hmm tested here and indeed, you're still in the proper range.
Just looked here and it seems I was off by a factor of 2 for the range.
Strange as I thought it had a range of 23 bits, but apparently it allows upto 2^24 -1.

TD-er · 2023-12-23T22:53:36Z

Just had a chat with Ton, apparently 3 variables replaced in a single line seems to work, but 4 doesn't work.
Have to dive into this a bit further...
So let me get a fresh beer and have a look :)

fly74 · 2023-12-23T22:56:12Z

Hmm tested here and indeed, you're still in the proper range. Just looked here and it seems I was off by a factor of 2 for the range. Strange as I thought it had a range of 23 bits, but apparently it allows upto 2^24 -1.

Precision limitations on integer values
Integers between 0 and 16777216 can be exactly represented (also applies for negative integers between −16777216 and 0)

That makes it clear. Int is enough to store 8 digit when the left 2 is the month 0..12.

chemmex · 2023-12-25T21:51:35Z

What's the reason for that if the plugin allows to choose the GPIO?

fly74 · 2023-12-25T21:52:22Z

Which plugin? Basic, BUSFX...

chemmex · 2023-12-25T21:53:22Z

P038, basic. I confirm it's not working on an empty build, ESP_Easy_mega_20231225_normal_ESP8266_4M1M_VCC

fly74 · 2023-12-25T21:56:28Z

It's not in the docu but it must be GPIO-2.

chemmex · 2023-12-25T22:00:58Z

I missed that discussion, maybe. Well, that's one more reason to finally switch to ESP32, but I have several hundreds of 8266 boards in the field with Neopixel wired to GPIO14. Apart from that fixed GPIO is not what could be expected from Neopixel plugin

fly74 · 2023-12-25T22:03:18Z

I missed that discussion, maybe. Well, that's one more reason to finally switch to ESP32, but I have several hundreds of 8266 boards in the field with Neopixel wired to GPIO14. Apart from that fixed GPIO is not what could be expected from Neopixel plugin

As I remember it was the same for me. The conclusion was, that the gpio settings was only for ESP32 and 8266 is alsways gpio-2. @tonhuisman correct ?

TD-er · 2023-12-25T22:03:45Z

You can check to see if there's already an issue about it as I also don't remember the specifics of this discussion anymore.
If there's no issue about it, can you make a new one so we can gather all info there as this one will be closed very soon...

chemmex · 2023-12-25T22:05:23Z

Yes, I will check that out. For now, of course, it is out of scope of current release, so congratulations!

tonhuisman · 2023-12-25T22:07:08Z

Hmm, maybe when we switched from the Adafruit_Neopixel library to the NeoPixelBus library. That only supports GPIO-2 on ESP8266 😞

TD-er · 2023-12-25T22:07:49Z

You all thanks for the true effort in detecting all these bugs in this short amount of time.

fly74 · 2023-12-25T22:09:54Z

Hmm, maybe when we switched from the Adafruit_Neopixel library to the NeoPixelBus library. That only supports GPIO-2 on ESP8266 😞

Don't break it. I have a lot of neopixel nodes.

iz8mbw · 2023-12-26T08:23:38Z

Hi @TD-er.
I'm trying to build a "custom_ESP32_4M316k_LittleFS" but I receive errors:

tonhuisman · 2023-12-26T09:07:38Z

Sometimes this is fixed by restarting the build, and running the Clean option from the pio menu might help too.
If it still fails you could clean out the C:\Users\fabio\.platformio\packages folder, so with the next build pio will get the latest platform files downloaded.

iz8mbw · 2023-12-26T09:56:33Z

Thanks Ton.
I fixed it by:

Close Visual Studio Code
Delete all inside "C:\Users\fabio.platformio\packages"

Going to test "LittleFS" on my boards.

iz8mbw · 2023-12-26T10:43:53Z

Built OK.
Just for your info, I have these:

Compiling .pio\build\custom_ESP32_4M316k_LittleFS\src\src\Globals\CPlugins.cpp.o
src/src/ESPEasyCore/ESPEasy_setup.cpp: In function 'void ESPEasy_setup()':
src/src/ESPEasyCore/ESPEasy_setup.cpp:355:27: warning: 'esp_pm_config_esp32_t' is deprecated: please use esp_pm_config_t instead [-Wdeprecated-declarations]
  355 |     esp_pm_config_esp32_t pm_config = {
      |                           ^~~~~~~~~
In file included from src/src/ESPEasyCore/ESPEasy_setup.cpp:57:
C:/Users/fabio/.platformio/packages/framework-arduinoespressif32/tools/esp32-arduino-libs/esp32/include/esp_pm/include/esp_pm.h:32:25: note: declared 
here
   32 | typedef esp_pm_config_t esp_pm_config_esp32_t   __attribute__((deprecated("please use esp_pm_config_t instead")));
      |                         ^~~~~~~~~~~~~~~~~~~~~
Compiling .pio\build\custom_ESP32_4M316k_LittleFS\src\src\Globals\CRCValues.cpp.o

Compiling .pio\build\custom_ESP32_4M316k_LittleFS\src\src\PluginStructs\P023_data_struct.cpp.o
src/src/PluginStructs/P020_data_struct.cpp: In member function 'bool P020_Task::hasClientConnected()':
src/src/PluginStructs/P020_data_struct.cpp:87:45: warning: 'WiFiClient WiFiServer::available()' is deprecated: Renamed to accept(). [-Wdeprecated-declarations]
   87 |     ser2netClient = ser2netServer->available();
      |                     ~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from C:/Users/fabio/.platformio/packages/framework-arduinoespressif32/libraries/WiFi/src/WiFi.h:38,
                 from c:\espeasy-mega\src\ESPEasy_common.h:37,
                 from c:\espeasy-mega\src\_plugin_helper.h:4,
                 from c:\espeasy-mega\src\src\pluginstructs\p020_data_struct.h:4,
                 from src/src/PluginStructs/P020_data_struct.cpp:1:
C:/Users/fabio/.platformio/packages/framework-arduinoespressif32/libraries/WiFi/src/WiFiServer.h:47:16: note: declared here
   47 |     WiFiClient available() __attribute__((deprecated("Renamed to accept().")));
      |                ^~~~~~~~~
Compiling .pio\build\custom_ESP32_4M316k_LittleFS\src\src\PluginStructs\P024_data_struct.cpp.o

Archiving .pio\build\custom_ESP32_4M316k_LittleFS\libFrameworkArduino.a
Linking .pio\build\custom_ESP32_4M316k_LittleFS\ESP_Easy_mega_20231226_custom_ESP32_4M316k_LittleFS.elf
lto-wrapper.exe: warning: using serial compilation of 17 LTRANS jobs
lto-wrapper.exe: note: see the '-flto' option documentation for more information
Retrieving maximum program size .pio\build\custom_ESP32_4M316k_LittleFS\ESP_Easy_mega_20231226_custom_ESP32_4M316k_LittleFS.elf

TD-er · 2023-12-26T11:55:54Z

Yep, the deprecated warnings are something I still need to look into.
The deprecation status of available() is not even sure if it will remain deprecated and the pm_config still needs to be dealt with but for now it isn't an immediate issue.

I have no idea why the LTO option adds a warning as it is just info.
LTO does reduce the bin files (sadly not yet possible on ESP8266 builds) by checking which functions are really used and which aren't.
The unused functions will be removed from the bin file.

iz8mbw · 2023-12-26T15:07:48Z

I have successfully built both ESP_Easy_mega_20231226_custom_ESP32_4M316k and ESP_Easy_mega_20231226_custom_ESP32_4M316k_LittleFS (LittleFS with above warnings).

On one board with SPIFFS I uploaded the LittleFS firmware and as expected I lost my configuration.
I replaced (uploaded) all the flies (backupped before) config.dat, notification.dat, rules1.txt and security.dat with File Browser option and after reboot all my configurations was back.
I hope what I did is enough and it's a correct process to migrate an existing setup from SPIFFS to LittleFS.

tonhuisman · 2023-12-26T15:22:56Z

You did it just fine 👍

It could have been a little more easy if you had first uploaded the latest 'regular' build, with SPIFFS, on that unit, and let it run for > 5 minutes with a stable WiFi connection. Then the WiFi credentials should have been automatically saved in the NVS area of the flash, and after a subsequent OTA with the new LittleFS build it should have restored the WiFi credentials from NVS during first connection to the WiFi network.

NB: This intended feature (though it may seem like 'magic' 😃) is also available if you used one of the pre-releases of the IDF 5.1 PR on the unit; that PR has been merged yesterday.

iz8mbw · 2023-12-26T16:48:02Z

Hi Ton, I did exactly what you wrote:

Update all the ESP32 boards with latest "regular" (so SPIFFS) firmware
Wait 10 minutes
Download the files config.dat, notification.dat, rules1.txt and security.dat
Update ESP32 board with latest "new" (so LittleFS) firmware
Upload the files config.dat, notification.dat, rules1.txt and security.dat into the file system
Reboot

All seems OK.
Thanks!

TD-er · 2023-12-26T20:34:31Z

Only step missing is to check all checkboxes on the Factory Reset page (unless you have done it in a previous build of course)

iz8mbw · 2023-12-27T08:58:59Z

No, I didn't do the Factory Reset.
At what step it should be done?

TD-er · 2023-12-27T09:52:43Z

No you don't need to reset, just check the checkboxes on the Factory Reset page to make sure all the relevant info you want to keep is stored outside the file system as that gets erased when switching to LittleFS.

iz8mbw · 2023-12-27T09:55:05Z

Ok, yes all the checkboxes were/are selected.

iz8mbw · 2023-12-30T09:59:40Z

Hi @TD-er
Since some days I have finished to move from ESP32 Custom SPIFFS to ESP32 Custom LittleFS all my boards (16M and 4M).
All is OK here and I noticed a notable less CPU usage and a little less RAM usage.
Very good job, many thanks!

fly74 · 2023-12-30T11:52:55Z

What's difference between these two's?

iz8mbw · 2023-12-30T12:04:07Z

@fly74 read this #4826

tonhuisman · 2023-12-30T12:13:55Z

@fly74 read this #4826

That.
And:
SPIFFS is the flash-optimized file-system used on nearly all ESP8266 builds and all ESP32 builds with up to 1MB of file system, (except the few builds with LittleFS in the name).
All builds marked with LittleFS use the LittleFS file system, as is to be expected 🤓
SPIFFS has been deprecated for several years now, and has finally been removed from the IDF 5.x framework for ESP32. That's why TD-er has worked his ass off, the last couple of months, to make IDF 5.x work with/for ESPEasy, by squashing bugs and other stuff in IDF 5.x, Arduino-ESP32 3.0 and ESPEasy.
The technical layout of the file-systems are not compatible, so a migration plan has to be used, to be able to continue using your existing configuration. An exact, detailed, description will be published somewhere in 2024.

Introduced in the ESPEasy 20231225 release, once the WiFi connection is stable, and the Factory reset setting page has been configured accordingly, the WiFi parameters, and a few other settings, are stored in the NVS area/partition (non-volatile storage) of the flash, to be read & used by a subsequent update-install, as the settings files can't be read any longer... That's what iz8mbw has been testing, see the messages above 😃

NB: This only applies to ESP32 (all CPU families), as ESP8266 doesn't use a NVS partition.

TD-er added the Category: Build Related to building/IDE/releases label Dec 23, 2023

TD-er pinned this issue Dec 23, 2023

TD-er closed this as completed Dec 25, 2023

TD-er unpinned this issue Dec 25, 2023

tonhuisman mentioned this issue Dec 27, 2023

[BUG] [NeoPixel] [ESP8266] NeoPixel plugins are fixed on GPIO-2 since NeoPixelBus_Wrapper was introduced #4921

Closed

[Test Build] Upcoming X-mas build, testers wanted #4917

[Test Build] Upcoming X-mas build, testers wanted #4917

Comments

TD-er commented Dec 23, 2023 • edited Loading

Dickesplumpel commented Dec 23, 2023

fly74 commented Dec 23, 2023

TD-er commented Dec 23, 2023

TD-er commented Dec 23, 2023

TD-er commented Dec 23, 2023

Dickesplumpel commented Dec 23, 2023

TD-er commented Dec 23, 2023

fly74 commented Dec 23, 2023

fly74 commented Dec 23, 2023 • edited Loading

tonhuisman commented Dec 23, 2023

fly74 commented Dec 23, 2023

tonhuisman commented Dec 23, 2023

fly74 commented Dec 23, 2023 • edited Loading

tonhuisman commented Dec 23, 2023

fly74 commented Dec 23, 2023

tonhuisman commented Dec 23, 2023

tonhuisman commented Dec 23, 2023

TD-er commented Dec 23, 2023

fly74 commented Dec 23, 2023

fly74 commented Dec 23, 2023

TD-er commented Dec 23, 2023

fly74 commented Dec 23, 2023

tonhuisman commented Dec 23, 2023 • edited Loading

TD-er commented Dec 23, 2023 • edited Loading

fly74 commented Dec 23, 2023 • edited Loading

TD-er commented Dec 23, 2023 • edited Loading

TD-er commented Dec 23, 2023

fly74 commented Dec 23, 2023 • edited Loading

chemmex commented Dec 25, 2023

fly74 commented Dec 25, 2023 • edited Loading

chemmex commented Dec 25, 2023 • edited Loading

fly74 commented Dec 25, 2023

chemmex commented Dec 25, 2023

fly74 commented Dec 25, 2023

TD-er commented Dec 25, 2023

chemmex commented Dec 25, 2023

tonhuisman commented Dec 25, 2023

TD-er commented Dec 25, 2023

fly74 commented Dec 25, 2023

iz8mbw commented Dec 26, 2023

tonhuisman commented Dec 26, 2023

iz8mbw commented Dec 26, 2023

iz8mbw commented Dec 26, 2023

TD-er commented Dec 26, 2023

iz8mbw commented Dec 26, 2023

tonhuisman commented Dec 26, 2023

iz8mbw commented Dec 26, 2023

TD-er commented Dec 26, 2023

iz8mbw commented Dec 27, 2023

TD-er commented Dec 27, 2023

iz8mbw commented Dec 27, 2023

iz8mbw commented Dec 30, 2023

fly74 commented Dec 30, 2023

iz8mbw commented Dec 30, 2023

tonhuisman commented Dec 30, 2023

TD-er commented Dec 23, 2023 •

edited

Loading

fly74 commented Dec 23, 2023 •

edited

Loading

fly74 commented Dec 23, 2023 •

edited

Loading

tonhuisman commented Dec 23, 2023 •

edited

Loading

TD-er commented Dec 23, 2023 •

edited

Loading

fly74 commented Dec 23, 2023 •

edited

Loading

TD-er commented Dec 23, 2023 •

edited

Loading

fly74 commented Dec 23, 2023 •

edited

Loading

fly74 commented Dec 25, 2023 •

edited

Loading

chemmex commented Dec 25, 2023 •

edited

Loading