From 45f449bfc1089ad21b3974b518478d5e2d9f12eb Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 13:18:26 -0600 Subject: binman: Add coverage tools info for Python 3 Test coverage with Python 3 requires a new package. Add details about this. Signed-off-by: Simon Glass --- tools/binman/README | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index ac193f16cf..decca47bbf 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -699,7 +699,7 @@ implementations target 100% test coverage. Run 'binman -T' to check this. To enable Python test coverage on Debian-type distributions (e.g. Ubuntu): - $ sudo apt-get install python-coverage python-pytest + $ sudo apt-get install python-coverage python3-coverage python-pytest Concurrent tests -- cgit From c7d8035ec185c789f8182c7d3374db4dcdbd250b Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 13:18:28 -0600 Subject: binman: Add a --toolpath option to set the tool search path Sometimes tools used by binman may not be in the normal PATH search path, such as when the tool is built by the U-Boot build itself (e.g. mkimage). Provide a way to specify an additional search path for tools. The flag can be used multiple times. Update the help to describe this option. Signed-off-by: Simon Glass --- tools/binman/README | 10 ++++++++++ 1 file changed, 10 insertions(+) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index decca47bbf..28624fadb3 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -691,6 +691,16 @@ Not all properties can be provided this way. Only some entries support it, typically for filenames. +External tools +-------------- + +Binman can make use of external command-line tools to handle processing of +entry contents or to generate entry contents. These tools are executed using +the 'tools' module's Run() method. The tools generally must exist on the PATH, +but the --toolpath option can be used to specify additional search paths to +use. This option can be specified multiple times to add more than one path. + + Code coverage ------------- -- cgit From d8d40748dd6f202948a8173073cf34f3b6468b6d Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 13:18:35 -0600 Subject: binman: Add a function to decode an ELF file Add a function which decodes an ELF file, working out where in memory each part of the data should be written. Signed-off-by: Simon Glass --- tools/binman/README | 4 ++++ 1 file changed, 4 insertions(+) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index 28624fadb3..0ff30ef6fd 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -181,6 +181,10 @@ the configuration of the Intel-format descriptor. Running binman -------------- +First install prerequisites, e.g. + + sudo apt-get install python-pyelftools python3-pyelftools + Type: binman -b -- cgit From 07d9e70bf9a65f3c94573b9e11f12025b472cff1 Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 13:18:41 -0600 Subject: patman: Add functions to compress and decompress data Add utility functions to compress and decompress using lz4 and lzma algorithms. In the latter case these use the legacy lzma support favoured by coreboot's CBFS. No tests are provided as these functions will be tested by the CBFS tests in a separate patch. Signed-off-by: Simon Glass --- tools/binman/README | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index 0ff30ef6fd..7eda244bbe 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -183,7 +183,8 @@ Running binman First install prerequisites, e.g. - sudo apt-get install python-pyelftools python3-pyelftools + sudo apt-get install python-pyelftools python3-pyelftools lzma-alone \ + liblz4-tool Type: -- cgit From d5164a79703df76254d8c0ac67037d629d113518 Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 13:18:49 -0600 Subject: binman: Allow preserving test directories Sometimes when debugging tests it is useful to keep the input and output directories so they can be examined later. Add an option for this and update the binman tests to support it. This affects both the test class and the tearDown() function called after each test. Signed-off-by: Simon Glass --- tools/binman/README | 8 ++++++++ 1 file changed, 8 insertions(+) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index 7eda244bbe..2f4c7fec21 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -731,6 +731,14 @@ Use '-P 1' to disable this. It is automatically disabled when code coverage is being used (-T) since they are incompatible. +Debugging tests +--------------- + +Sometimes when debugging tests it is useful to keep the input and output +directories so they can be examined later. Use -X or --test-preserve-dirs for +this. + + Advanced Features / Technical docs ---------------------------------- -- cgit From 488b2251fecd9eeb5241e75947923b114ffbb154 Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 14:25:24 -0600 Subject: binman: Update future features A few features have been completed and a few items are added. Signed-off-by: Simon Glass --- tools/binman/README | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index 2f4c7fec21..1851a4e536 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -811,13 +811,17 @@ Some ideas: - Use of-platdata to make the information available to code that is unable to use device tree (such as a very small SPL image) - Allow easy building of images by specifying just the board name -- Produce a full Python binding for libfdt (for upstream). This is nearing - completion but some work remains - Add an option to decode an image into the constituent binaries - Support building an image for a board (-b) more completely, with a configurable build directory -- Consider making binman work with buildman, although if it is used in the - Makefile, this will be automatic +- Support putting the FDT in an image with a suitable magic number +- Support adding a pointer to the FDT map +- Support listing files in images +- Support logging of binman's operations, with different levels of verbosity +- Support updating binaries in an image (with no size change / repacking) +- Support updating binaries in an image (with repacking) +- Support adding FITs to an image +- Support for ARM Trusted Firmware (ATF) -- Simon Glass -- cgit From 57460184686f83386712c366cd7722d1aa35f09d Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 14:25:25 -0600 Subject: binman: Update help for new features A few new features have been added. This has rendered part of the README obsolete. Update it. Signed-off-by: Simon Glass --- tools/binman/README | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index 1851a4e536..b067d3b78c 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -36,10 +36,9 @@ suitable padding and alignment. It provides a way to process binaries before they are included, by adding a Python plug-in. The device tree is available to U-Boot at run-time so that the images can be interpreted. -Binman does not yet update the device tree with the final location of -everything when it is done. A simple C structure could be generated for -constrained environments like SPL (using dtoc) but this is also not -implemented. +Binman can update the device tree with the final location of everything when it +is done. Entry positions can be provided to U-Boot SPL as run-time symbols, +avoiding device-tree code overhead. Binman can also support incorporating filesystems in the image if required. For example x86 platforms may use CBFS in some cases. -- cgit From 086cec9f980efd6f25e184b84f626d4a667e6645 Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 14:25:27 -0600 Subject: binman: Add an FDT map An FDT map is an entry which holds a full description of the image entries, in FDT format. It can be discovered using the magic string at its start. Tools can locate and read this entry to find out what entries are in the image and where each entry is located. Signed-off-by: Simon Glass --- tools/binman/README | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index b067d3b78c..8a5f3320dc 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -638,6 +638,10 @@ the image definition, binman calculates the final values and writes these to the device tree. These can be used by U-Boot at run-time to find the location of each entry. +Alternatively, an FDT map entry can be used to add a special FDT containing +just the information about the image. This is preceded by a magic string so can +be located anywhere in the image. + Compression ----------- @@ -814,7 +818,6 @@ Some ideas: - Support building an image for a board (-b) more completely, with a configurable build directory - Support putting the FDT in an image with a suitable magic number -- Support adding a pointer to the FDT map - Support listing files in images - Support logging of binman's operations, with different levels of verbosity - Support updating binaries in an image (with no size change / repacking) -- cgit From cf2289435c7e1134c75d9217ef32238aec0ecdbf Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 14:25:28 -0600 Subject: binman: Add an image header It is useful to be able to quickly locate the FDT map in the image. An easy way to do this is with a pointer at the start or end of the image. Add an 'image header' entry, which places a magic number followed by a pointer to the FDT map. This can be located at the start or end of the image, or at a chosen location. As part of this, update GetSiblingImagePos() to detect missing siblings. Signed-off-by: Simon Glass --- tools/binman/README | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index 8a5f3320dc..ef62d1f1ec 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -640,7 +640,9 @@ of each entry. Alternatively, an FDT map entry can be used to add a special FDT containing just the information about the image. This is preceded by a magic string so can -be located anywhere in the image. +be located anywhere in the image. An image header (typically at the start or end +of the image) can be used to point to the FDT map. See fdtmap and image-header +entries for more information. Compression @@ -817,7 +819,6 @@ Some ideas: - Add an option to decode an image into the constituent binaries - Support building an image for a board (-b) more completely, with a configurable build directory -- Support putting the FDT in an image with a suitable magic number - Support listing files in images - Support logging of binman's operations, with different levels of verbosity - Support updating binaries in an image (with no size change / repacking) -- cgit From 53cd5d921dd76d4651f2c99681a3c050743b6ba1 Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 14:25:29 -0600 Subject: binman: Convert to use ArgumentParser This class is the new way to handle arguments in Python. Convert binman over to use it. At the same time, introduce commands so that we can separate out the different parts of binman functionality. Signed-off-by: Simon Glass --- tools/binman/README | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index ef62d1f1ec..fe734c1e5f 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -187,7 +187,7 @@ First install prerequisites, e.g. Type: - binman -b + binman build -b to build an image for a board. The board name is the same name used when configuring U-Boot (e.g. for sandbox_defconfig the board name is 'sandbox'). @@ -195,7 +195,7 @@ Binman assumes that the input files for the build are in ../b/. Or you can specify this explicitly: - binman -I + binman build -I where is the build directory containing the output of the U-Boot build. @@ -483,7 +483,7 @@ Entry Documentation For details on the various entry types supported by binman and how to use them, see README.entries. This is generated from the source code using: - binman -E >tools/binman/README.entries + binman entry-docs >tools/binman/README.entries Hashing Entries @@ -715,7 +715,7 @@ Code coverage ------------- Binman is a critical tool and is designed to be very testable. Entry -implementations target 100% test coverage. Run 'binman -T' to check this. +implementations target 100% test coverage. Run 'binman test -T' to check this. To enable Python test coverage on Debian-type distributions (e.g. Ubuntu): -- cgit From 8287ee852d23b81bda364f9a4ed11c2fcc17da43 Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 14:25:30 -0600 Subject: binman: Move compression into the Entry base class Compression is currently available only with blobs. However we want to report the compression algorithm and uncompressed size for all entries, so that other entry types can support compression. This will help with the forthcoming 'list' feature which lists entries in the image. Move the compression properties into the base class. Also fix up the docs which had the wrong property name. Signed-off-by: Simon Glass --- tools/binman/README | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index fe734c1e5f..abbf809b82 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -339,6 +339,10 @@ expand-size: limited by the size of the image/section and the position of the next entry. +compress: + Sets the compression algortihm to use (for blobs only). See the entry + documentation for details. + The attributes supported for images and sections are described below. Several are similar to those for entries. @@ -649,15 +653,16 @@ Compression ----------- Binman support compression for 'blob' entries (those of type 'blob' and -derivatives). To enable this for an entry, add a 'compression' property: +derivatives). To enable this for an entry, add a 'compress' property: blob { filename = "datafile"; - compression = "lz4"; + compress = "lz4"; }; The entry will then contain the compressed data, using the 'lz4' compression -algorithm. Currently this is the only one that is supported. +algorithm. Currently this is the only one that is supported. The uncompressed +size is written to the node in an 'uncomp-size' property, if -u is used. -- cgit From c52c9e7da809e36001d125891e594c4740235055 Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 14:25:37 -0600 Subject: binman: Allow entries to expand after packing Add support for detecting entries that change size after they have already been packed, and re-running packing when it happens. This removes the limitation that entry size cannot change after PackEntries() is called. Signed-off-by: Simon Glass --- tools/binman/README | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index abbf809b82..9860633792 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -566,7 +566,8 @@ tree. This sets the correct 'offset' and 'size' vaues, for example. The default implementatoin does nothing. This can be overriden to adjust the contents of an entry in some way. For example, it would be possible to create an entry containing a hash of the contents of some other entries. At this -stage the offset and size of entries should not be adjusted. +stage the offset and size of entries should not be adjusted unless absolutely +necessary, since it requires a repack (going back to PackEntries()). 10. WriteSymbols() - write the value of symbols into the U-Boot SPL binary. See 'Access to binman entry offsets at run time' below for a description of -- cgit From 61f564d15f35e5f5600ed639201b257efa09d1f1 Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 14:25:48 -0600 Subject: binman: Support listing an image Add support for listing the entries in an image. This relies on the image having an FDT map. Signed-off-by: Simon Glass --- tools/binman/README | 44 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index 9860633792..146e0fd470 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -490,6 +490,49 @@ see README.entries. This is generated from the source code using: binman entry-docs >tools/binman/README.entries +Listing images +-------------- + +It is possible to list the entries in an existing firmware image created by +binman, provided that there is an 'fdtmap' entry in the image. For example: + + $ binman ls -i image.bin + Name Image-pos Size Entry-type Offset Uncomp-size + ---------------------------------------------------------------------- + main-section c00 section 0 + u-boot 0 4 u-boot 0 + section 5fc section 4 + cbfs 100 400 cbfs 0 + u-boot 138 4 u-boot 38 + u-boot-dtb 180 108 u-boot-dtb 80 3b5 + u-boot-dtb 500 1ff u-boot-dtb 400 3b5 + fdtmap 6fc 381 fdtmap 6fc + image-header bf8 8 image-header bf8 + +This shows the hierarchy of the image, the position, size and type of each +entry, the offset of each entry within its parent and the uncompressed size if +the entry is compressed. + +It is also possible to list just some files in an image, e.g. + + $ binman ls -i image.bin section/cbfs + Name Image-pos Size Entry-type Offset Uncomp-size + -------------------------------------------------------------------- + cbfs 100 400 cbfs 0 + u-boot 138 4 u-boot 38 + u-boot-dtb 180 108 u-boot-dtb 80 3b5 + +or with wildcards: + + $ binman ls -i image.bin "*cb*" "*head*" + Name Image-pos Size Entry-type Offset Uncomp-size + ---------------------------------------------------------------------- + cbfs 100 400 cbfs 0 + u-boot 138 4 u-boot 38 + u-boot-dtb 180 108 u-boot-dtb 80 3b5 + image-header bf8 8 image-header bf8 + + Hashing Entries --------------- @@ -825,7 +868,6 @@ Some ideas: - Add an option to decode an image into the constituent binaries - Support building an image for a board (-b) more completely, with a configurable build directory -- Support listing files in images - Support logging of binman's operations, with different levels of verbosity - Support updating binaries in an image (with no size change / repacking) - Support updating binaries in an image (with repacking) -- cgit From eea264ead3ca198ed66f62a78dc4940075621ae7 Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 14:25:49 -0600 Subject: binman: Allow for logging information to be displayed Binman generally operates silently but in some cases it is useful to see what Binman is actually doing at each step. Enable some logging output with different logging levels selectable via the -v flag. Signed-off-by: Simon Glass --- tools/binman/README | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index 146e0fd470..1655a9d50d 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -533,6 +533,24 @@ or with wildcards: image-header bf8 8 image-header bf8 +Logging +------- + +Binman normally operates silently unless there is an error, in which case it +just displays the error. The -D/--debug option can be used to create a full +backtrace when errors occur. + +Internally binman logs some output while it is running. This can be displayed +by increasing the -v/--verbosity from the default of 1: + + 0: silent + 1: warnings only + 2: notices (important messages) + 3: info about major operations + 4: detailed information about each operation + 5: debug (all output) + + Hashing Entries --------------- @@ -868,7 +886,6 @@ Some ideas: - Add an option to decode an image into the constituent binaries - Support building an image for a board (-b) more completely, with a configurable build directory -- Support logging of binman's operations, with different levels of verbosity - Support updating binaries in an image (with no size change / repacking) - Support updating binaries in an image (with repacking) - Support adding FITs to an image -- cgit From 71ce0ba284aeb388ddcefb4f6f0056c759a758cc Mon Sep 17 00:00:00 2001 From: Simon Glass Date: Mon, 8 Jul 2019 14:25:52 -0600 Subject: binman: Add an 'extract' command It is useful to be able to extract all binaries from the image, or a subset of them. Add a new 'extract' command to handle this. Signed-off-by: Simon Glass --- tools/binman/README | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) (limited to 'tools/binman/README') diff --git a/tools/binman/README b/tools/binman/README index 1655a9d50d..756c6a0010 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -533,6 +533,30 @@ or with wildcards: image-header bf8 8 image-header bf8 +Extracting files from images +---------------------------- + +You can extract files from an existing firmware image created by binman, +provided that there is an 'fdtmap' entry in the image. For example: + + $ binman extract -i image.bin section/cbfs/u-boot + +which will write the uncompressed contents of that entry to the file 'u-boot' in +the current directory. You can also extract to a particular file, in this case +u-boot.bin: + + $ binman extract -i image.bin section/cbfs/u-boot -f u-boot.bin + +It is possible to extract all files into a destination directory, which will +put files in subdirectories matching the entry hierarchy: + + $ binman extract -i image.bin -O outdir + +or just a selection: + + $ binman extract -i image.bin "*u-boot*" -O outdir + + Logging ------- @@ -883,7 +907,6 @@ Some ideas: - Use of-platdata to make the information available to code that is unable to use device tree (such as a very small SPL image) - Allow easy building of images by specifying just the board name -- Add an option to decode an image into the constituent binaries - Support building an image for a board (-b) more completely, with a configurable build directory - Support updating binaries in an image (with no size change / repacking) -- cgit