U boot integration [WIP] #1411
Open
U boot integration [WIP] #1411
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When switching an SDWire to 'host' mode, it may take time for the partition to appear. During that time the device (e.g. '/dev/sdo1') may not exist, so trying to cat e.g. '/sys/call/block/sdo1' fails. Handle this by returning a zero size in that case, as is done when the returned size is empty. Even when the block device is present, udisks2 may take a short time to make the device available. Handle this by retrying a few times, until things settle. Add the call to _wait_for_medium() in write_files() while we are here, to match what is done in write_image(). Also fix the parameter type for write_files() and mention the exception it may raise. Fixes: labgrid-project#1357 Signed-off-by: Simon Glass <[email protected]>
When running a process it is sometimes necessary to pass environment variables and to run the process in a particular directory. Add support for these features. Signed-off-by: Simon Glass <[email protected]>
It isn't generally useful to print out every command before executing it. Now that the client shows failed commands fully, it seems better to change this to show only when debugging. Signed-off-by: Simon Glass <[email protected]>
The current subprocess helper assumes it is the only thing running. In some cases (such as the Dediprog em100 tool) we want to start a process and have it run for a while in the background. Rather than invent another helper, modify the existing helper to support this. For existing use cases, no functional change is intended. Signed-off-by: Simon Glass <[email protected]>
The EM100-pro is a SPI-flash emulator which supports a variety of chips. It can be used to provide an image to a booting system. Add a driver which allows images to be loaded ready for use. Provide a trace facility for use when the board does not boot. Signed-off-by: Simon Glass <[email protected]>
Some boards have a way to put them into a 'recovery' mode provided by the SoC. Recovery is a signal asserted at power-on (or when the board is reset). It causes the SoC to listen on USB. Add a protocol for this feature. Signed-off-by: Simon Glass <[email protected]>
The reset protocol only supports doing a reset. Sometimes we want to assert reset while something is done (e.g. a power cycle or a recovery signal), then de-assert it afterwards. Add a new set_reset_enable() method for this. The name is a little strange: one might expect set_enable() instead. But with things like the PowerResetMixin we want to avoid ambiguity over whether this refers to powering the device or resetting it. Signed-off-by: Simon Glass <[email protected]>
Some drivers need to be told what to do from the command line. This is particularly true of complex strategy drivers, such as the one to write U-Boot to board media, or send it over USB. Provide a way to specify driver-specific variables with a string value, so that they can control the strategy. Examples include whether to bootstrap the device or just boot the existing image, whether to build a new image or use an existing one and whether to write the image to boot media or send it over USB. With these variables a useful selection of shell scripts can provide a highly productive development and test environment, without a huge prolification of very similar drivers. This approach also avoids the complex interactions that would be necessary between different drivers aimed towards the same goal. Signed-off-by: Simon Glass <[email protected]>
When integrating Labgrid into other projects it is useful to be able to query information known by a particular driver. For example, U-Boot wants to know the board name of a particular target, so that it can pass this to pytest. Add a new driver method to provide this feature. Signed-off-by: Simon Glass <[email protected]>
The role (or is it target?) is used in the environment and allows multiple roles to use the same place. It is therefore often better to specify the role to use rather than the place, since the place can be determined from the role. Add a new -r option to support this. Signed-off-by: Simon Glass <[email protected]>
It is common to use a digital output (such as a relay) to assert the recovery signal. Add a driver to handle this. Signed-off-by: Simon Glass <[email protected]>
Servo is a range of boards used by ChromiumOS devices to provide reset, recovery and power control, as well as multiple serial consoles. Add a resource and driver which provides these basic facilities. Signed-off-by: Simon Glass <[email protected]>
Building firmware images is fairly complex these days, due to the wide variety of binaries that need to be assembled together. U-Boot provides two useful tools here: - buildman can build U-Boot itself, selecting a suitable toolchain - binman can assemble / sign images, pulling in required binaries It is convenient to build these images within Labgrid, since each board has a selection of necessary binaries, which can be provided as part of the lab configuration. This allows a fully functional firmware image to be produced for any board as part of setting up a Labgrid configuration for that board. A particularly useful case is for interactive use, where the current source tree is built and booted on a selected board, resulting in a console for that board (if everything goes well). The driver includes some features which make this easy to implement. Note that Labgrid itself does not build U-Boot. It just calls buildman to make it happen. Signed-off-by: Simon Glass <[email protected]>
Each SoC has its own way of booting and this means that the firmware layout for each SoC is typically different. On top of that, some SoCs allow writing an image over USB. which uses a different format, e.g. separate binaries for each stage. Trying to make all this work is extremely challenging. Labgrid provides a nice unification point, where the method appropriate for each SoC can be included, with a small amount of configuration to control things. Binary blobs can be provided to make the writer work correctly. This driver is capable of writing SD-card images for several popular devices as well as handling USB booting in some cases. It can be easily extended as necessary. Signed-off-by: Simon Glass <[email protected]>
Some devices do not have power control. Add a driver which models this situation, so that strategies don't need to have it as a special case. Signed-off-by: Simon Glass <[email protected]>
Add quotes around the driver name, as expected. Signed-off-by: Simon Glass <[email protected]>
Some SoCs need multiple USB-loading steps to function. For example, Samsung devices require BL1, then SPL then U-Boot. Add a new parameter to the protocol to support this. Signed-off-by: Simon Glass <[email protected]>
Add a USB loader for sunxi, supporting SPL and U-Boot. Signed-off-by: Simon Glass <[email protected]>
Add a USB loader for tegra, supporting BCT and U-Boot. Signed-off-by: Simon Glass <[email protected]>
Add a USB loader for samsung, supporting BL1, SPL and U-Boot. Signed-off-by: Simon Glass <[email protected]>
Sometimes it is useful to specify the block size when writing to media, since some SoCs require different block sizes. Add a new block_size argument which allows the value to be set. Support the existing defaults based on other arguments. IMO the behaviour of the automatic block-size-guessing logic is a little odd. Perhaps we should just have a default of 512 in all cases? Signed-off-by: Simon Glass <[email protected]>
Sometimes it is useful to specify the block count when writing to media, e.g. when a limited area is available or an area must be filled. Add a new count argument which allows the value to be set. Signed-off-by: Simon Glass <[email protected]>
The current approach is slow, since each block must be individually written to the disk before the next can start. The more common approach is to do a manual 'sync' after writing. Adjust write_image() to use this approach. It is several times faster on non-trivial files, even with the need for the sync. Signed-off-by: Simon Glass <[email protected]>
It isn't very useful to show the dd stats on every transfer. Make this silent. It may be useful to tie this to the -v option but I am not sure how best to do that. Signed-off-by: Simon Glass <[email protected]>
Provide an overall time for the dd operation. Signed-off-by: Simon Glass <[email protected]>
For generic labs with lots of boards it is helpful to use the same test for all boards. Add a command-line option to specify which target to use, so that we don't need to add a separate test class for every single target. Signed-off-by: Simon Glass <[email protected]>
Allow setting variables when running a test so that it is possible to control complex strategy-drivers. Signed-off-by: Simon Glass <[email protected]>
On some boards a relay is used to assert the recovery signal, which can be used to force the DUT into SoC-recovery mode (boot ROM). Add support for this. Signed-off-by: Simon Glass <[email protected]>
This tool prints lots of information to the terminal so it is not easy to see if anything went wrong. Use the logging subsystem to handle most of this output. Signed-off-by: Simon Glass <[email protected]>
This tool prints out the configuration information that it reads. This is not generally useful, so put it behind a -v flag. Signed-off-by: Simon Glass <[email protected]>
It isn't obvious when the exporter has finished setting things up and the labe is ready for use. This seems to take quite a while - e.g. 15 seconds for 125 resources. Add a message to provide this signal, including how many resources are exported. Signed-off-by: Simon Glass <[email protected]>
It is common to acquire a place and then use it to access the console. Add a -a option to support this, to avoid running labgrid-client twice, which takes longer and makes error handling harder. If the place is acquired, it is released on exit. Signed-off-by: Simon Glass <[email protected]>
Some messages are printed regardless of the logging level. Convert them to logging statements so they can be suppressed. Signed-off-by: Simon Glass <[email protected]>
When a command fails in labgrid-client it just shows the return code. This makes it very hard to see what actually went wrong. Handle this exception explicitly, showing the output of the tool which failed. This allows easy debugging of the problem. This is particularly important with things like USB loaders. Signed-off-by: Simon Glass <[email protected]>
When a target may or may not be acquired, it is useful to be able to release it without checking first. Add an option for this, so that it is not necessary to check first. Signed-off-by: Simon Glass <[email protected]>
Fix a small typo in a message. Signed-off-by: Simon Glass <[email protected]>
Use logging rather than print so that this output can be controlled. Signed-off-by: Simon Glass <[email protected]>
At present if a console device is used by a strategy driver it is then removed by the client immediately afterwards. If the 'console' command is used, this means that any console data is lost. This is done so that the console device is release for microcom, so move the logic in with the microcom logic. With the forthcoming internal terminal, it will not be needed. Signed-off-by: Simon Glass <[email protected]>
Put this code into a function to reduce the size of the _get_target() function. Signed-off-by: Simon Glass <[email protected]>
It is possible to set the initial state of a strategy, but not the final state. In many cases we want to power the board off and otherwise clean things up at the end. Provide a -e option to accomplish this. Signed-off-by: Simon Glass <[email protected]>
At present Labgrid uses microcom as its console. This has some limitations: - console output is lost between when the board is reset and microcom connects - txdelay cannot be handled in microcom, meaning that boards may fail to receive expected output - the console may echo a few characters back to the caller in the time between when 'labgrid-client console' is executed and when microcom starts (which causes failures with U-Boot test system) For many use cases, microcom is more than is needed, so provide a simple internal terminal which resolved the above problems. It is enabled by a '-i' option to the 'console' command, as well as an environment variable, so that it can be adjustly without updating a lot of scripts. To exit, press Ctrl-] twice, quickly. Signed-off-by: Simon Glass <[email protected]>
Transferring a file around is an operation which happens internally within Labgrid. It is confusing to show these transfers as they happen. Use quiet mode with rsync to avoid unwanted output. Signed-off-by: Simon Glass <[email protected]>
This class has a lot of members whose type and purpose is hard to figure out. Add some documentation at the top. Signed-off-by: Simon Glass <[email protected]>
This function lacks documentation. Tidy it up. While we are here, reverse the order of the 'active' and 'activate' arguments, to separate the arguments into those at the start which provide the filter and the one at the end which requests an action. Signed-off-by: Simon Glass <[email protected]>
Some strategies may want to use a driver if it exists, without failing if it does not. Add an argument to get_driver() to support this. Signed-off-by: Simon Glass <[email protected]>
U-Boot prints a banner when it starts up, which contains the version string. Collect this so it can be used by the strategy driver as needed. Signed-off-by: Simon Glass <[email protected]>
When something goes wrong it is useful to see what output was processed as a clue to what happened. Collect everything up to the autoboot prompt, since once it hits that things are normally working OK. Signed-off-by: Simon Glass <[email protected]>
At present the 'start' state of U-Boot consumes all the initial U-Boot output, including the SPL and proper headers and any version information. This makes it impossible to run the U-Boot pytests, since this output is expected from the board. There are two ways to fix this: - Show the output - Print a message indicating that U-Boot has booted, so tests don't wait for the headers Do both, the first to make tests work existing U-Boot work with tests, the second to resolve the problem for future U-Boot versions, once the U-Boot pytest patch is accepted: test: Allow signaling that U-Boot is ready Signed-off-by: Simon Glass <[email protected]>
The U-Boot strategy only allows bringing up a U-Boot prompt. If for some reason this fails (e.g. due to a bug), nothing is shown on the console. For interactive use, it is helpful to see any output, even if it does not ultimately result in reach the U-Boot prompt. Add a new 'start' state, to start U-Boot. Use this from the 'uboot' state. Signed-off-by: Simon Glass <[email protected]>
It is useful to write a new U-Boot version to the board before starting it up. This is particularly valuable when using labgrid interactively, to test out / debug U-Boot on various boards. We end up with several different paths through the strategy: - just power on/reset the board - bootstrap: write an existing U-Boot to the board, then power it on - build and bootstrap: build a new U-Boot, write it to the board, then power it on All of these are useful in an interactive context. Implement this by calling the U-Boot provider to build U-Boot, then the U-Boot writer to write it to the board. Finally, power on/reset the board as normal. This is controlled by two variables: do-bootstrap: 1 to bootstrap the board, 0 to leave it along do-build: 1 to build U-Boot, 0 to use the existing build Signed-off-by: Simon Glass <[email protected]>
Some boards do not have boot media, or it cannot be accessed by the test system. In this case it is useful to send U-Boot to the board using the USB-recovery mechanism. Add the logic for this, controlled by a new 'do-send' variable. Even for boards which do have boot media, it can still be useful to use the USB mechanism. Add the logic for this, using a new 'do-send' variable to control it. If 'do-send' is 1, U-Boot is written over USB. If it is 0, then U-Boot is written to the boot media. Signed-off-by: Simon Glass <[email protected]>
When Labgrid fails to get to the U-Boot command line, show the output that was received, so that the user can see what is going on. Signed-off-by: Simon Glass <[email protected]>
U-Boot's pytest framework has lots of logic to detect the U-Boot banner and deal with failures to do so. Labgrid supports this in its U-Boot driver. We don't need both and Labgrid's implementation is better. It has more knowledge of what is going on, since it is in charge of bringing the board to life. It can be configured to know how long the board takes to start up. Add a 'lab mode' where Labgrid does all the work and just announces when things are ready. This is enabled by presence of the U_BOOT_SOURCE_DIR environment variable, which is set by U-Boot's pytest system. This greatly simplifies operation of lab testing and ensures that pytest is not trying to guess what happened. Signed-off-by: Simon Glass <[email protected]>
Provide some scripts which facilitate U-Boot testing and development using Labgrid. Signed-off-by: Simon Glass <[email protected]>
sjg20
force-pushed
the
u-boot-integration
branch
from
68aeabc
to
4cdfb54
Compare
Cover the drivers, scripts, pytest and the Gitlab information. Note: This implementation is the result of working through what is needed in Labgrid to support U-Boot labs. I did initially file quite a few issues[1] but there has not been a lot of traction and I got feedback that I have overwhelmed people with too many. So I have stopped filing issues on the things I hit along the way. I have carried on with the implementation in the hope that this can be a better basis for discussion. Note that all of these patches are work-in-progress. Feedback on any or all may change the implementation and documentation substantially. Some changes since last time: - Rationalise the flags for the U-Boot scripts - Support tracing with em100 - Support an internal terminal instead of microcom - Add a -D flag for debugging - Support send-only boards - Add a way to build the U-Boot config - Add a control for buildman's process-limit - allow the build-dir to be specified in a variable - add documentation about U-Boot-pytest integration - add source_dir and config_file to UBootProviderDriver - add an internal terminal - expand the U-Boot scripts - significantly improve the U-Boot-pytest integration The approximate diffstat is: contrib/u-boot/.gitignore | 1 + contrib/u-boot/_ub-bisect-try | 47 +++++ contrib/u-boot/conftest.py | 21 ++ contrib/u-boot/get_args.sh | 128 ++++++++++++ contrib/u-boot/index.rst | 232 +++++++++++++++++++++ contrib/u-boot/lg-client | 11 + contrib/u-boot/lg-env | 10 + contrib/u-boot/test_smoke.py | 3 + contrib/u-boot/ub-bisect | 44 ++++ contrib/u-boot/ub-cli | 39 ++++ contrib/u-boot/ub-int | 41 ++++ contrib/u-boot/ub-pyt | 65 ++++++ contrib/u-boot/ub-smoke | 48 +++++ doc/configuration.rst | 611 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- doc/usage.rst | 288 ++++++++++++++++++++++++++ labgrid/driver/__init__.py | 10 +- labgrid/driver/common.py | 11 + labgrid/driver/consoleexpectmixin.py | 7 + labgrid/driver/powerdriver.py | 29 +++ labgrid/driver/recoverydriver.py | 22 ++ labgrid/driver/resetdriver.py | 7 + labgrid/driver/servodriver.py | 157 ++++++++++++++ labgrid/driver/sfemulatordriver.py | 102 ++++++++++ labgrid/driver/ubootdriver.py | 27 ++- labgrid/driver/ubootproviderdriver.py | 264 ++++++++++++++++++++++++ labgrid/driver/ubootwriterdriver.py | 139 +++++++++++++ labgrid/driver/usbhidrelay.py | 11 +- labgrid/driver/usbloader.py | 170 +++++++++++++++- labgrid/driver/usbstoragedriver.py | 35 +++- labgrid/factory.py | 4 +- labgrid/protocol/__init__.py | 1 + labgrid/protocol/bootstrapprotocol.py | 8 +- labgrid/protocol/recoveryprotocol.py | 14 ++ labgrid/protocol/resetprotocol.py | 12 ++ labgrid/pytestplugin/fixtures.py | 21 +- labgrid/pytestplugin/hooks.py | 8 + labgrid/remote/client.py | 220 +++++++++++++------- labgrid/remote/config.py | 7 +- labgrid/remote/exporter.py | 124 ++++++++++- labgrid/resource/__init__.py | 6 + labgrid/resource/remote.py | 24 +++ labgrid/resource/servo.py | 485 +++++++++++++++++++++++++++++++++++++++++++ labgrid/resource/sfemulator.py | 33 +++ labgrid/resource/suggest.py | 6 + labgrid/resource/udev.py | 33 +++ labgrid/strategy/ubootstrategy.py | 138 +++++++++++-- labgrid/target.py | 120 +++++++++-- labgrid/util/agents/udisks2.py | 49 ++++- labgrid/util/helper.py | 220 +++++++++++--------- labgrid/util/ssh.py | 3 +- labgrid/util/term.py | 124 +++++++++++ labgrid/var_dict.py | 8 + man/labgrid-client.1 | 6 + man/labgrid-client.rst | 4 + man/labgrid-device-config.5 | 4 + 55 files changed, 4017 insertions(+), 245 deletions(-) [1] https://github.com/labgrid-project/labgrid/issues/created_by/sjg20 Signed-off-by: Simon Glass <[email protected]>
sjg20
force-pushed
the
u-boot-integration
branch
from
4cdfb54
to
c4b13af
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: This implementation is the result of working through what is
needed in Labgrid to support U-Boot labs. I did initially file quite
a few issues[1] but there has not been a lot of traction and I got
feedback that I have overwhelmed people with too many. So I have
stopped filing issues on the things I hit along the way. I have carried
on with the implementation in the hope that this can be a better basis
for discussion.
Note that all of these patches are work-in-progress. Feedback on any or
all may change the implementation and documentation substantially.
The approximate diffstat is:
contrib/u-boot/.gitignore | 1 +
contrib/u-boot/_ub-bisect-try | 47 ++
contrib/u-boot/conftest.py | 21 +
contrib/u-boot/get_args.sh | 50 +++
contrib/u-boot/index.rst | 203 +++++++++
contrib/u-boot/lg-client | 11 +
contrib/u-boot/lg-env | 9 +
contrib/u-boot/test_smoke.py | 3 +
contrib/u-boot/ub-bisect | 37 ++
contrib/u-boot/ub-cli | 30 ++
contrib/u-boot/ub-int | 30 ++
contrib/u-boot/ub-pyt | 40 ++
contrib/u-boot/ub-smoke | 38 ++
doc/configuration.rst | 606 +++++++++++++++++++++++++-
doc/usage.rst | 287 ++++++++++++
labgrid/driver/init.py | 10 +-
labgrid/driver/common.py | 11 +
labgrid/driver/consoleexpectmixin.py | 7 +
labgrid/driver/powerdriver.py | 29 ++
labgrid/driver/recoverydriver.py | 22 +
labgrid/driver/resetdriver.py | 7 +
labgrid/driver/servodriver.py | 153 +++++++
labgrid/driver/sfemulatordriver.py | 50 +++
labgrid/driver/ubootproviderdriver.py | 214 +++++++++
labgrid/driver/ubootwriterdriver.py | 139 ++++++
labgrid/driver/usbhidrelay.py | 11 +-
labgrid/driver/usbloader.py | 171 +++++++-
labgrid/driver/usbstoragedriver.py | 36 +-
labgrid/factory.py | 4 +-
labgrid/protocol/init.py | 1 +
labgrid/protocol/bootstrapprotocol.py | 8 +-
labgrid/protocol/recoveryprotocol.py | 14 +
labgrid/protocol/resetprotocol.py | 4 +
labgrid/pytestplugin/fixtures.py | 21 +-
labgrid/pytestplugin/hooks.py | 8 +
labgrid/remote/client.py | 96 +++-
labgrid/remote/config.py | 7 +-
labgrid/remote/exporter.py | 124 +++++-
labgrid/resource/init.py | 6 +
labgrid/resource/remote.py | 24 +
labgrid/resource/servo.py | 485 +++++++++++++++++++++
labgrid/resource/sfemulator.py | 33 ++
labgrid/resource/suggest.py | 6 +
labgrid/resource/udev.py | 33 ++
labgrid/strategy/ubootstrategy.py | 95 +++-
labgrid/target.py | 120 ++++-
labgrid/util/agents/udisks2.py | 49 ++-
labgrid/util/helper.py | 9 +-
labgrid/util/ssh.py | 3 +-
labgrid/var_dict.py | 8 +
man/labgrid-client.1 | 6 +
man/labgrid-client.rst | 4 +
man/labgrid-device-config.5 | 4 +
53 files changed, 3352 insertions(+), 93 deletions(-)
[1] https://github.com/labgrid-project/labgrid/issues/created_by/sjg20