Developer Guide - ESP-IDF Installation

Hi dexter, thank you for your efforts. I'm using ESP IDF #e97f72e now but esp-idf-tools-setup-2.0.exe installation fails at follows:

019-07-27 10:48:21.329 Setting PATH for this process: C:\Users\Matthias\AppData\Local\Programs\Python\Python37\;C:\Program Files\Git\cmd\;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\GNU\GnuPG\pub;C:\Program Files\Calibre2\;C:\Program Files\PuTTY\;C:\Program Files\TortoiseGit\bin;C:\Program Files\Git\cmd;C:\Users\Matthias\AppData\Local\Programs\Python\Python37\Scripts\;C:\Users\Matthias\AppData\Local\Programs\Python\Python37\
2019-07-27 10:48:21.329 PYTHONPATH=
2019-07-27 10:48:21.329 idf_tools.py does not exist in IDF directory, using a fallback version
2019-07-27 10:48:21.329 idf_tools.py command: C:\Users\Matthias\AppData\Local\Programs\Python\Python37\python.exe C:\Users\Matthias\.espressif\idf_tools_fallback.py --idf-path C:\Users\Matthias\Documents\GIT\esp-idf --tools C:\Users\Matthias\.espressif\tools_fallback.json
2019-07-27 10:48:21.329 Installing tools:C:\Users\Matthias\AppData\Local\Programs\Python\Python37\python.exe C:\Users\Matthias\.espressif\idf_tools_fallback.py --idf-path C:\Users\Matthias\Documents\GIT\esp-idf --tools C:\Users\Matthias\.espressif\tools_fallback.json install
2019-07-27 10:51:20.153 Done, exit code=1
2019-07-27 10:51:20.153 --------
2019-07-27 10:51:20.153 Running command: C:\Users\Matthias\AppData\Local\Programs\Python\Python37\python.exe C:\Users\Matthias\.espressif\idf_tools_fallback.py --idf-path C:\Users\Matthias\Documents\GIT\esp-idf --tools C:\Users\Matthias\.espressif\tools_fallback.json install
Installing tools: xtensa-esp32-elf, esp32ulp-elf, cmake, openocd-esp32, mconf, ninja, idf-exe, ccache
Installing xtensa-esp32-elf@1.22.0-80-g6c4433a5-5.2.0
Downloading xtensa-esp32-elf-win32-1.22.0-80-g6c4433a-5.2.0.zip to C:\Users\Matthias\Documents\ESPRESSIF\.espressif\dist\xtensa-esp32-elf-win32-1.22.0-80-g6c4433a-5.2.0.zip.tmp
Done
Extracting C:\Users\Matthias\Documents\ESPRESSIF\.espressif\dist\xtensa-esp32-elf-win32-1.22.0-80-g6c4433a-5.2.0.zip to C:\Users\Matthias\Documents\ESPRESSIF\.espressif\tools\xtensa-esp32-elf\1.22.0-80-g6c4433a5-5.2.0
Installing esp32ulp-elf@2.28.51.20170517
Downloading binutils-esp32ulp-win32-2.28.51-esp32ulp-20180809.zip to C:\Users\Matthias\Documents\ESPRESSIF\.espressif\dist\binutils-esp32ulp-win32-2.28.51-esp32ulp-20180809.zip.tmp
Done
Extracting C:\Users\Matthias\Documents\ESPRESSIF\.espressif\dist\binutils-esp32ulp-win32-2.28.51-esp32ulp-20180809.zip to C:\Users\Matthias\Documents\ESPRESSIF\.espressif\tools\esp32ulp-elf\2.28.51.20170517
Installing cmake@3.13.4
Downloading cmake-3.13.4-win64-x64.zip to C:\Users\Matthias\Documents\ESPRESSIF\.espressif\dist\cmake-3.13.4-win64-x64.zip.tmp
Done
Extracting C:\Users\Matthias\Documents\ESPRESSIF\.espressif\dist\cmake-3.13.4-win64-x64.zip to C:\Users\Matthias\Documents\ESPRESSIF\.espressif\tools\cmake\3.13.4
Traceback (most recent call last):
File "C:\Users\Matthias\.espressif\idf_tools_fallback.py", line 1249, in <module>
main(sys.argv[1:])
File "C:\Users\Matthias\.espressif\idf_tools_fallback.py", line 1245, in main
action_func(args)
File "C:\Users\Matthias\.espressif\idf_tools_fallback.py", line 1048, in action_install
tool_obj.install(tool_version)
File "C:\Users\Matthias\.espressif\idf_tools_fallback.py", line 554, in install
strip_container_dirs(dest_dir, self._current_options.strip_container_dirs)
File "C:\Users\Matthias\.espressif\idf_tools_fallback.py", line 261, in strip_container_dirs
os.rename(path, tmp_path)
PermissionError: [WinError 5] Zugriff verweigert: 'C:\\Users\\Matthias\\Documents\\ESPRESSIF\\.espressif\\tools\\cmake\\3.13.4' -> 'C:\\Users\\Matthias\\Documents\\ESPRESSIF\\.espressif\\tools\\cmake\\3.13.4.tmp'

Not sure what is about idf_tools.py

Thank you for your help. I run git submodule update --init --recursive after cloning OVMS ESP IDF #e97f72e.

It seems during the first try to install esp-idf-tools-setup-2.0.exe I made the 'mistake' to select some other installation directory that was used for IDF_TOOLS_PATH. This was not removed on uninstall and the installation directory was also not prompted for during the next installations tries.

So now, after the installation has finished, a cmd window is opened where the following issues are reported:

Using Python in C:\Users\Matthias\AppData\Local\Programs\Python\Python37\
Python 3.7.3
Using Git in C:\Program Files\Git\cmd\
git version 2.22.0.windows.1
IDF version does not include export.bat. Using the fallback version.
IDF version does not include tools\tools.json. Using the fallback version.
IDF version does not include tools\idf_tools.py. Using the fallback version.

Setting IDF_PATH: C:\Users\Matthias\Documents\GIT\esp-idf

Adding ESP-IDF tools to PATH...
WARNING: tool idf-exe version 1.0 is installed, but the tool failed to run

<<<POPUP WINDOW "idf.py.exe funktioniert nicht mehr">>>

ERROR: tool idf-exe has no installed versions. Please run 'C:\Users\Matthias\AppDat
a\Local\Programs\Python\Python37\python.exe C:\Users\Matthias\.espressif\idf_tools_
fallback.py --tools-json C:\Users\Matthias\.espressif\tools_fallback.json install'
to install it.

<<<DOING SO>>>

C:\Users\Matthias\AppData\Local\Programs\Python\Python37>python.exe C:\Users\Matthias\
.espressif\idf_tools_fallback.py --tools-json C:\Users\Matthias\.espressif\tools_fallback.json install
Installing tools: xtensa-esp32-elf, esp32ulp-elf, cmake, openocd-esp32, mconf, ninja, idf-exe, ccache
Skipping xtensa-esp32-elf@1.22.0-80-g6c4433a5-5.2.0 (already installed)
Skipping esp32ulp-elf@2.28.51.20170517 (already installed)
Skipping cmake@3.13.4 (already installed)
Skipping openocd-esp32@v0.10.0-esp32-20190313 (already installed)
Skipping mconf@v4.6.0.0-idf-20190628 (already installed)
Skipping ninja@1.9.0 (already installed)
WARNING: tool idf-exe version 1.0 is installed, but the tool failed to run
Installing idf-exe@1.0
file idf-exe-v1.0.zip is already downloaded
WARNING: destination path already exists, removing
Traceback (most recent call last):
File "C:\Users\Matthias\.espressif\idf_tools_fallback.py", line 1249, in <module>

main(sys.argv[1:])
File "C:\Users\Matthias\.espressif\idf_tools_fallback.py", line 1245, in main
action_func(args)
File "C:\Users\Matthias\.espressif\idf_tools_fallback.py", line 1048, in action_i
nstall
tool_obj.install(tool_version)
File "C:\Users\Matthias\.espressif\idf_tools_fallback.py", line 550, in install
shutil.rmtree(dest_dir)
File "C:\Users\Matthias\AppData\Local\Programs\Python\Python37\lib\shutil.py", li
ne 513, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\Matthias\AppData\Local\Programs\Python\Python37\lib\shutil.py", li
ne 397, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\Matthias\AppData\Local\Programs\Python\Python37\lib\shutil.py", li
ne 395, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 5] Zugriff verweigert: 'C:\\Users\\Matthias\\.espressif\
\tools\\idf-exe\\1.0\\idf.py.exe'

Not sure what is going on exactly.

Maybe someone could give me a hint, if the messages above (they also show when 'ESP-IDF command prompt (cmd.exe)' is opened) are normal or not:

"IDF version does not include export.bat. Using the fallback version.
IDF version does not include tools\tools.json. Using the fallback version.
IDF version does not include tools\idf_tools.py. Using the fallback version."

I don't see these files in OVMS ESP IDF #e97f72e.

Ok tanks, I was just following the instructions in the Developer's Guide. I was using the ESP-IDF Tools Installer from https://dl.espressif.com/dl/esp-idf-tools-setup-2.0.exe as this was what I found at "Step 1. Install prerequisites - Windows" when the link provided in Developer's Guide p.6 (https://esp-idf.readthedocs.io/en/latest/get-started/index.html) was automatically redirected to https://docs.espressif.com/projects/esp-idf/en/latest/get-started/index.....

Maybe someone could fix the Developer's Guide with a realy suitable link for docs.espressif.com?

I the meantime I got a hint to use an even older toolchain: https://dl.espressif.com/dl/esp32_win32_msys2_environment_and_toolchain-...

Trying this I actually can compile and link the OVMS3 project, but unfortunately my generated ovms3.bin differs from the reference:

$ ls -l build/ovms3.bin edge/ovms3.bin
-rw-r--r-- 1 Matthias None 3036096 2. Aug 08:26 build/ovms3.bin
-rw-r--r-- 1 Matthias None 2594192 2. Aug 07:59 edge/ovms3.bin

I have downloaded the second ovms3.bin (3.2.002-151-g56116fe
Thu Aug 1 16:04:20 UTC 2019 Automated build (markhk8)), the first one I build myself in the mingw32 console using

$ make clean
$ make all
Toolchain path: /opt/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc
Toolchain version: crosstool-ng-1.22.0-80-g6c4433a5
Compiler version: 5.2.0
Python requirements from C:/msys32/home/.../esp/esp-idf/requirements.txt are satisfied.

App "ovms3" version: 3.2.002-151-g56116fe1
CC build/app_update/esp_app_desc.o
AR build/app_update/libapp_update.a
Build version is 3.2.002-151-g56116fe1
CXX build/main/ovms_version.o
AR build/main/libmain.a
Generating libapp_update.a.sections_info
Generating libmain.a.sections_info
Generating esp32.project.ld
LD build/ovms3.elf

$ md5sum.exe build/ovms3.*
ca1ef125eda09cabe3c38ed5518e1b6c *build/ovms3.bin

I'm not sure why the binaries differ so much. I'm using Python 3.7.3 as this come with my first try of toolchain installation.

I would like to have a clean toolchain reference, so that I can reproducing the same binary (for the same GIT #tag of course) from where I can start and later share own development.

Now I found something about "IoT Development Framework Configuration". The Developer's Guide says (p.6):

"make menuconfig - The config tool may prompt for some new config opt"ions, use the default values for all"

I'v seen that in my environment, the default for the Compiler Optimization was set to "Debug -Og" which was the reason why ma ovms3.bin was so much larger. Now, using "Release -Os", the size is similar to edge/ovms3.bin, but not the same.

Maybe someone could tell about other important settings in "IoT Development Framework Configuration"? As instructed, I had copied sdkconfig before from support/sdkconfig.default.hw31 (GIT #fd9dc50edfb, 2019-07-21, Michael Balzer) where CONFIG_OPTIMIZATION_LEVEL_DEBUG=y is set, obviously not matching markhk8's setting for 'Automated build'