.../82_auto-generation_process.md | 20 ++++++++++++++++++++ README.md | 1 + appendix_d_buildexe_command/d4_usage.md | 19 +++++++++++++++---- 3 files changed, 36 insertions(+), 4 deletions(-)
V2:
change the option name to --binary-destination and --binary-source.
fixes:https://bugzilla.tianocore.org/show_bug.cgi?id=689
Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael Kinney <michael.d.kinney@intel.com>
Cc: Kevin W Shaw <kevin.w.shaw@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
---
.../82_auto-generation_process.md | 20 ++++++++++++++++++++
README.md | 1 +
appendix_d_buildexe_command/d4_usage.md | 19 +++++++++++++++----
3 files changed, 36 insertions(+), 4 deletions(-)
diff --git a/8_pre-build_autogen_stage/82_auto-generation_process.md b/8_pre-build_autogen_stage/82_auto-generation_process.md
index 671a7d5..f2ddf32 100644
--- a/8_pre-build_autogen_stage/82_auto-generation_process.md
+++ b/8_pre-build_autogen_stage/82_auto-generation_process.md
@@ -1031,10 +1031,30 @@ maximum command line length. The default value is 4096.
**Note:** The following `FLAGS` options are included in the response file:
`PP_FLAGS`, `CC_FLAGS`, `VFRPP_FLAGS`, `APP_FLAGS`, `ASLPP_FLAGS`, `ASLCC_FLAGS`,
and `ASM_FLAGS`.
**********
+#### 8.2.4.15 Build with Binary Cache
+
+**build** tool provides three new options for binary cache feature.
+--hash enables hash-based caching during build process. when --hash is enabled,
+build tool will base on the module hash value to do the incremental build, without
+--hash, build tool will base on the timestamp to do the incremental build. --hash
+option use md5 method to get every hash value, DSC/FDF, tools_def.txt, build_rule.txt
+and build command are calculated as global hash value, Package DEC and its include
+header files are calculated as package hash value, Module source files and its INF
+file are calculated as module hash value. Library hash value will combine the global
+hash value and its dependent package hash value. Driver hash value will combine the
+global hash value, its dependent package hash value and its linked library hash value.
+When --hash and --binary-destination are specified, build tool will copy the generated
+binary files for each module into the directory specified by binary-destination at the
+build phase. Binary-destination directory caches all the generated binary files.
+When --hash and --binary-source are specified, build tool will try to get the binary
+files from the binary source directory at the build phase. If the cached binary has
+the same hash value, it will be directly used. Otherwise, build tool will compile the
+source files and generate the binary files.
+
### 8.2.5 Post processing
Once all files are parsed, the build tools will do following work for each EDK
II module:
diff --git a/README.md b/README.md
index 52abb6a..ca59a35 100644
--- a/README.md
+++ b/README.md
@@ -215,5 +215,6 @@ Copyright (c) 2008-2017, Intel Corporation. All rights reserved.
| | [#523](https://bugzilla.tianocore.org/show_bug.cgi?id=523) Build spec: add EBNF for the --pcd syntax in the Section D.4 | |
| | [#517](https://bugzilla.tianocore.org/show_bug.cgi?id=517) Build spec: chapter 5.2.2 Guided Tools add description for Pkcs7Sign tool and BrotliCompress tool | |
| | [#481](https://bugzilla.tianocore.org/show_bug.cgi?id=481) Build Spec: add clarification for not used Pcd that build tool will not do additional checks on its value | |
| | [#518](https://bugzilla.tianocore.org/show_bug.cgi?id=518) Build Spec: Update Precedence of PCD Values | |
| | [#669](https://bugzilla.tianocore.org/show_bug.cgi?id=669) Build Spec: Add multi-arg support to PREBUILD/POSTBUILD | |
+| | [#689](https://bugzilla.tianocore.org/show_bug.cgi?id=689) Build spec: add description for build with binary cache | |
diff --git a/appendix_d_buildexe_command/d4_usage.md b/appendix_d_buildexe_command/d4_usage.md
index b71f2d0..c901266 100644
--- a/appendix_d_buildexe_command/d4_usage.md
+++ b/appendix_d_buildexe_command/d4_usage.md
@@ -32,19 +32,20 @@
## D.4 Usage
```ini
Usage: build.exe [options]
[all|fds|genc|genmake|clean|cleanall|cleanlib|modules|libraries|run]
-Copyright (c) 2007 - 2014, Intel Corporation All rights reserved.
+Copyright (c) 2007 - 2017, Intel Corporation All rights reserved.
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-a TARGETARCH, --arch=TARGETARCH
- ARCHS is one of list: IA32, X64, IPF, ARM, or EBC,
- which overrides target.txt's TARGET_ARCH definition.
- To specify more archs, please repeat this option.
+ ARCHS is one of list: IA32, X64, IPF, ARM, AARCH64 or
+ EBC, which overrides target.txt's TARGET_ARCH
+ definition. To specify more archs, please repeat this
+ option.
-p PLATFORMFILE, --platform=PLATFORMFILE
Build the platform specified by the DSC file name
argument, overriding target.txt's ACTIVE_PLATFORM
definition.
-m MODULEFILE, --module=MODULEFILE
@@ -112,10 +113,20 @@ Options:
-N, --no-cache Disable build cache mechanism
--conf=CONFDIRECTORY Specify the customized Conf directory.
--check-usage Check usage content of entries listed in INF file.
--ignore-sources Focus to a binary build and ignore all source files
--pcd=OPTIONPCD Set PCD value by command line. Format: "PcdName=Value"
+ -l COMMANDLENGTH, --cmd-len=COMMANDLENGTH
+ Specify the maximum line length of build command.
+ Default is 4096.
+ --hash Enable hash-based caching during build process.
+ --binary-destination=BINCACHEDEST
+ Generate a cache of binary files in the specified
+ directory.
+ --binary-source=BINCACHESOURCE
+ Consume a cache of binary files from the specified
+ directory.
```
### D.4.1 Debug Levels
The numeric debug levels are defined as integer values 0-9.
--
2.6.1.windows.1
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel
Reviewed-by: Liming Gao <liming.gao@intel.com> >-----Original Message----- >From: Zhu, Yonghong >Sent: Tuesday, September 19, 2017 2:48 PM >To: edk2-devel@lists.01.org >Cc: Gao, Liming <liming.gao@intel.com>; Kinney, Michael D ><michael.d.kinney@intel.com>; Shaw, Kevin W <kevin.w.shaw@intel.com> >Subject: [Patch V2] Build spec: add description for build with binary cache > >V2: >change the option name to --binary-destination and --binary-source. > >fixes:https://bugzilla.tianocore.org/show_bug.cgi?id=689 >Cc: Liming Gao <liming.gao@intel.com> >Cc: Michael Kinney <michael.d.kinney@intel.com> >Cc: Kevin W Shaw <kevin.w.shaw@intel.com> >Contributed-under: TianoCore Contribution Agreement 1.1 >Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com> >--- > .../82_auto-generation_process.md | 20 ++++++++++++++++++++ > README.md | 1 + > appendix_d_buildexe_command/d4_usage.md | 19 >+++++++++++++++---- > 3 files changed, 36 insertions(+), 4 deletions(-) > >diff --git a/8_pre-build_autogen_stage/82_auto-generation_process.md >b/8_pre-build_autogen_stage/82_auto-generation_process.md >index 671a7d5..f2ddf32 100644 >--- a/8_pre-build_autogen_stage/82_auto-generation_process.md >+++ b/8_pre-build_autogen_stage/82_auto-generation_process.md >@@ -1031,10 +1031,30 @@ maximum command line length. The default >value is 4096. > **Note:** The following `FLAGS` options are included in the response file: > `PP_FLAGS`, `CC_FLAGS`, `VFRPP_FLAGS`, `APP_FLAGS`, `ASLPP_FLAGS`, >`ASLCC_FLAGS`, > and `ASM_FLAGS`. > ********** > >+#### 8.2.4.15 Build with Binary Cache >+ >+**build** tool provides three new options for binary cache feature. >+--hash enables hash-based caching during build process. when --hash is >enabled, >+build tool will base on the module hash value to do the incremental build, >without >+--hash, build tool will base on the timestamp to do the incremental build. -- >hash >+option use md5 method to get every hash value, DSC/FDF, tools_def.txt, >build_rule.txt >+and build command are calculated as global hash value, Package DEC and its >include >+header files are calculated as package hash value, Module source files and its >INF >+file are calculated as module hash value. Library hash value will combine the >global >+hash value and its dependent package hash value. Driver hash value will >combine the >+global hash value, its dependent package hash value and its linked library >hash value. >+When --hash and --binary-destination are specified, build tool will copy the >generated >+binary files for each module into the directory specified by binary- >destination at the >+build phase. Binary-destination directory caches all the generated binary files. >+When --hash and --binary-source are specified, build tool will try to get the >binary >+files from the binary source directory at the build phase. If the cached binary >has >+the same hash value, it will be directly used. Otherwise, build tool will >compile the >+source files and generate the binary files. >+ > ### 8.2.5 Post processing > > Once all files are parsed, the build tools will do following work for each EDK > II module: > >diff --git a/README.md b/README.md >index 52abb6a..ca59a35 100644 >--- a/README.md >+++ b/README.md >@@ -215,5 +215,6 @@ Copyright (c) 2008-2017, Intel Corporation. All rights >reserved. > | | [#523](https://bugzilla.tianocore.org/show_bug.cgi?id=523) Build >spec: add EBNF for the --pcd syntax in the Section D.4 >| | > | | [#517](https://bugzilla.tianocore.org/show_bug.cgi?id=517) Build >spec: chapter 5.2.2 Guided Tools add description for Pkcs7Sign tool and >BrotliCompress tool >| | > | | [#481](https://bugzilla.tianocore.org/show_bug.cgi?id=481) Build >Spec: add clarification for not used Pcd that build tool will not do additional >checks on its value >| | > | | [#518](https://bugzilla.tianocore.org/show_bug.cgi?id=518) Build >Spec: Update Precedence of PCD Values >| | > | | [#669](https://bugzilla.tianocore.org/show_bug.cgi?id=669) Build >Spec: Add multi-arg support to PREBUILD/POSTBUILD >| | >+| | [#689](https://bugzilla.tianocore.org/show_bug.cgi?id=689) Build >spec: add description for build with binary cache >| | >diff --git a/appendix_d_buildexe_command/d4_usage.md >b/appendix_d_buildexe_command/d4_usage.md >index b71f2d0..c901266 100644 >--- a/appendix_d_buildexe_command/d4_usage.md >+++ b/appendix_d_buildexe_command/d4_usage.md >@@ -32,19 +32,20 @@ > ## D.4 Usage > > ```ini > Usage: build.exe [options] > [all|fds|genc|genmake|clean|cleanall|cleanlib|modules|libraries|run] >-Copyright (c) 2007 - 2014, Intel Corporation All rights reserved. >+Copyright (c) 2007 - 2017, Intel Corporation All rights reserved. > > Options: > --version show program's version number and exit > -h, --help show this help message and exit > -a TARGETARCH, --arch=TARGETARCH >- ARCHS is one of list: IA32, X64, IPF, ARM, or EBC, >- which overrides target.txt's TARGET_ARCH definition. >- To specify more archs, please repeat this option. >+ ARCHS is one of list: IA32, X64, IPF, ARM, AARCH64 or >+ EBC, which overrides target.txt's TARGET_ARCH >+ definition. To specify more archs, please repeat this >+ option. > -p PLATFORMFILE, --platform=PLATFORMFILE > Build the platform specified by the DSC file name > argument, overriding target.txt's ACTIVE_PLATFORM > definition. > -m MODULEFILE, --module=MODULEFILE >@@ -112,10 +113,20 @@ Options: > -N, --no-cache Disable build cache mechanism > --conf=CONFDIRECTORY Specify the customized Conf directory. > --check-usage Check usage content of entries listed in INF file. > --ignore-sources Focus to a binary build and ignore all source files > --pcd=OPTIONPCD Set PCD value by command line. Format: >"PcdName=Value" >+ -l COMMANDLENGTH, --cmd-len=COMMANDLENGTH >+ Specify the maximum line length of build command. >+ Default is 4096. >+ --hash Enable hash-based caching during build process. >+ --binary-destination=BINCACHEDEST >+ Generate a cache of binary files in the specified >+ directory. >+ --binary-source=BINCACHESOURCE >+ Consume a cache of binary files from the specified >+ directory. > ``` > > ### D.4.1 Debug Levels > > The numeric debug levels are defined as integer values 0-9. >-- >2.6.1.windows.1 _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
On 09/19/17 08:48, Yonghong Zhu wrote: > V2: > change the option name to --binary-destination and --binary-source. > > fixes:https://bugzilla.tianocore.org/show_bug.cgi?id=689 What are the benefits of a hash-based incremental build over a timestamp-based incremental build? Thank you, Laszlo > Cc: Liming Gao <liming.gao@intel.com> > Cc: Michael Kinney <michael.d.kinney@intel.com> > Cc: Kevin W Shaw <kevin.w.shaw@intel.com> > Contributed-under: TianoCore Contribution Agreement 1.1 > Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com> > --- > .../82_auto-generation_process.md | 20 ++++++++++++++++++++ > README.md | 1 + > appendix_d_buildexe_command/d4_usage.md | 19 +++++++++++++++---- > 3 files changed, 36 insertions(+), 4 deletions(-) > > diff --git a/8_pre-build_autogen_stage/82_auto-generation_process.md b/8_pre-build_autogen_stage/82_auto-generation_process.md > index 671a7d5..f2ddf32 100644 > --- a/8_pre-build_autogen_stage/82_auto-generation_process.md > +++ b/8_pre-build_autogen_stage/82_auto-generation_process.md > @@ -1031,10 +1031,30 @@ maximum command line length. The default value is 4096. > **Note:** The following `FLAGS` options are included in the response file: > `PP_FLAGS`, `CC_FLAGS`, `VFRPP_FLAGS`, `APP_FLAGS`, `ASLPP_FLAGS`, `ASLCC_FLAGS`, > and `ASM_FLAGS`. > ********** > > +#### 8.2.4.15 Build with Binary Cache > + > +**build** tool provides three new options for binary cache feature. > +--hash enables hash-based caching during build process. when --hash is enabled, > +build tool will base on the module hash value to do the incremental build, without > +--hash, build tool will base on the timestamp to do the incremental build. --hash > +option use md5 method to get every hash value, DSC/FDF, tools_def.txt, build_rule.txt > +and build command are calculated as global hash value, Package DEC and its include > +header files are calculated as package hash value, Module source files and its INF > +file are calculated as module hash value. Library hash value will combine the global > +hash value and its dependent package hash value. Driver hash value will combine the > +global hash value, its dependent package hash value and its linked library hash value. > +When --hash and --binary-destination are specified, build tool will copy the generated > +binary files for each module into the directory specified by binary-destination at the > +build phase. Binary-destination directory caches all the generated binary files. > +When --hash and --binary-source are specified, build tool will try to get the binary > +files from the binary source directory at the build phase. If the cached binary has > +the same hash value, it will be directly used. Otherwise, build tool will compile the > +source files and generate the binary files. > + > ### 8.2.5 Post processing > > Once all files are parsed, the build tools will do following work for each EDK > II module: > > diff --git a/README.md b/README.md > index 52abb6a..ca59a35 100644 > --- a/README.md > +++ b/README.md > @@ -215,5 +215,6 @@ Copyright (c) 2008-2017, Intel Corporation. All rights reserved. > | | [#523](https://bugzilla.tianocore.org/show_bug.cgi?id=523) Build spec: add EBNF for the --pcd syntax in the Section D.4 | | > | | [#517](https://bugzilla.tianocore.org/show_bug.cgi?id=517) Build spec: chapter 5.2.2 Guided Tools add description for Pkcs7Sign tool and BrotliCompress tool | | > | | [#481](https://bugzilla.tianocore.org/show_bug.cgi?id=481) Build Spec: add clarification for not used Pcd that build tool will not do additional checks on its value | | > | | [#518](https://bugzilla.tianocore.org/show_bug.cgi?id=518) Build Spec: Update Precedence of PCD Values | | > | | [#669](https://bugzilla.tianocore.org/show_bug.cgi?id=669) Build Spec: Add multi-arg support to PREBUILD/POSTBUILD | | > +| | [#689](https://bugzilla.tianocore.org/show_bug.cgi?id=689) Build spec: add description for build with binary cache | | > diff --git a/appendix_d_buildexe_command/d4_usage.md b/appendix_d_buildexe_command/d4_usage.md > index b71f2d0..c901266 100644 > --- a/appendix_d_buildexe_command/d4_usage.md > +++ b/appendix_d_buildexe_command/d4_usage.md > @@ -32,19 +32,20 @@ > ## D.4 Usage > > ```ini > Usage: build.exe [options] > [all|fds|genc|genmake|clean|cleanall|cleanlib|modules|libraries|run] > -Copyright (c) 2007 - 2014, Intel Corporation All rights reserved. > +Copyright (c) 2007 - 2017, Intel Corporation All rights reserved. > > Options: > --version show program's version number and exit > -h, --help show this help message and exit > -a TARGETARCH, --arch=TARGETARCH > - ARCHS is one of list: IA32, X64, IPF, ARM, or EBC, > - which overrides target.txt's TARGET_ARCH definition. > - To specify more archs, please repeat this option. > + ARCHS is one of list: IA32, X64, IPF, ARM, AARCH64 or > + EBC, which overrides target.txt's TARGET_ARCH > + definition. To specify more archs, please repeat this > + option. > -p PLATFORMFILE, --platform=PLATFORMFILE > Build the platform specified by the DSC file name > argument, overriding target.txt's ACTIVE_PLATFORM > definition. > -m MODULEFILE, --module=MODULEFILE > @@ -112,10 +113,20 @@ Options: > -N, --no-cache Disable build cache mechanism > --conf=CONFDIRECTORY Specify the customized Conf directory. > --check-usage Check usage content of entries listed in INF file. > --ignore-sources Focus to a binary build and ignore all source files > --pcd=OPTIONPCD Set PCD value by command line. Format: "PcdName=Value" > + -l COMMANDLENGTH, --cmd-len=COMMANDLENGTH > + Specify the maximum line length of build command. > + Default is 4096. > + --hash Enable hash-based caching during build process. > + --binary-destination=BINCACHEDEST > + Generate a cache of binary files in the specified > + directory. > + --binary-source=BINCACHESOURCE > + Consume a cache of binary files from the specified > + directory. > ``` > > ### D.4.1 Debug Levels > > The numeric debug levels are defined as integer values 0-9. > _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
Laszlo: Hash way may improve the incremental build performance. If hash value is not changed, module AutoGen and Make will be skipped. Time stamp way bases on Makefile. This way will run AutoGen every time. If no change, Makefile will not be updated. Thanks Liming >-----Original Message----- >From: Laszlo Ersek [mailto:lersek@redhat.com] >Sent: Thursday, September 28, 2017 4:19 PM >To: Zhu, Yonghong <yonghong.zhu@intel.com>; edk2-devel@lists.01.org >Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Shaw, Kevin W ><kevin.w.shaw@intel.com>; Gao, Liming <liming.gao@intel.com> >Subject: Re: [edk2] [Patch V2] Build spec: add description for build with binary >cache > >On 09/19/17 08:48, Yonghong Zhu wrote: >> V2: >> change the option name to --binary-destination and --binary-source. >> >> fixes:https://bugzilla.tianocore.org/show_bug.cgi?id=689 > >What are the benefits of a hash-based incremental build over a >timestamp-based incremental build? > >Thank you, >Laszlo > >> Cc: Liming Gao <liming.gao@intel.com> >> Cc: Michael Kinney <michael.d.kinney@intel.com> >> Cc: Kevin W Shaw <kevin.w.shaw@intel.com> >> Contributed-under: TianoCore Contribution Agreement 1.1 >> Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com> >> --- >> .../82_auto-generation_process.md | 20 >++++++++++++++++++++ >> README.md | 1 + >> appendix_d_buildexe_command/d4_usage.md | 19 >+++++++++++++++---- >> 3 files changed, 36 insertions(+), 4 deletions(-) >> >> diff --git a/8_pre-build_autogen_stage/82_auto-generation_process.md >b/8_pre-build_autogen_stage/82_auto-generation_process.md >> index 671a7d5..f2ddf32 100644 >> --- a/8_pre-build_autogen_stage/82_auto-generation_process.md >> +++ b/8_pre-build_autogen_stage/82_auto-generation_process.md >> @@ -1031,10 +1031,30 @@ maximum command line length. The default >value is 4096. >> **Note:** The following `FLAGS` options are included in the response file: >> `PP_FLAGS`, `CC_FLAGS`, `VFRPP_FLAGS`, `APP_FLAGS`, `ASLPP_FLAGS`, >`ASLCC_FLAGS`, >> and `ASM_FLAGS`. >> ********** >> >> +#### 8.2.4.15 Build with Binary Cache >> + >> +**build** tool provides three new options for binary cache feature. >> +--hash enables hash-based caching during build process. when --hash is >enabled, >> +build tool will base on the module hash value to do the incremental build, >without >> +--hash, build tool will base on the timestamp to do the incremental build. -- >hash >> +option use md5 method to get every hash value, DSC/FDF, tools_def.txt, >build_rule.txt >> +and build command are calculated as global hash value, Package DEC and its >include >> +header files are calculated as package hash value, Module source files and >its INF >> +file are calculated as module hash value. Library hash value will combine >the global >> +hash value and its dependent package hash value. Driver hash value will >combine the >> +global hash value, its dependent package hash value and its linked library >hash value. >> +When --hash and --binary-destination are specified, build tool will copy the >generated >> +binary files for each module into the directory specified by binary- >destination at the >> +build phase. Binary-destination directory caches all the generated binary >files. >> +When --hash and --binary-source are specified, build tool will try to get the >binary >> +files from the binary source directory at the build phase. If the cached >binary has >> +the same hash value, it will be directly used. Otherwise, build tool will >compile the >> +source files and generate the binary files. >> + >> ### 8.2.5 Post processing >> >> Once all files are parsed, the build tools will do following work for each EDK >> II module: >> >> diff --git a/README.md b/README.md >> index 52abb6a..ca59a35 100644 >> --- a/README.md >> +++ b/README.md >> @@ -215,5 +215,6 @@ Copyright (c) 2008-2017, Intel Corporation. All rights >reserved. >> | | [#523](https://bugzilla.tianocore.org/show_bug.cgi?id=523) Build >spec: add EBNF for the --pcd syntax in the Section D.4 >| | >> | | [#517](https://bugzilla.tianocore.org/show_bug.cgi?id=517) Build >spec: chapter 5.2.2 Guided Tools add description for Pkcs7Sign tool and >BrotliCompress tool >| | >> | | [#481](https://bugzilla.tianocore.org/show_bug.cgi?id=481) Build >Spec: add clarification for not used Pcd that build tool will not do additional >checks on its value >| | >> | | [#518](https://bugzilla.tianocore.org/show_bug.cgi?id=518) Build >Spec: Update Precedence of PCD Values >| | >> | | [#669](https://bugzilla.tianocore.org/show_bug.cgi?id=669) Build >Spec: Add multi-arg support to PREBUILD/POSTBUILD >| | >> +| | [#689](https://bugzilla.tianocore.org/show_bug.cgi?id=689) Build >spec: add description for build with binary cache >| | >> diff --git a/appendix_d_buildexe_command/d4_usage.md >b/appendix_d_buildexe_command/d4_usage.md >> index b71f2d0..c901266 100644 >> --- a/appendix_d_buildexe_command/d4_usage.md >> +++ b/appendix_d_buildexe_command/d4_usage.md >> @@ -32,19 +32,20 @@ >> ## D.4 Usage >> >> ```ini >> Usage: build.exe [options] >> [all|fds|genc|genmake|clean|cleanall|cleanlib|modules|libraries|run] >> -Copyright (c) 2007 - 2014, Intel Corporation All rights reserved. >> +Copyright (c) 2007 - 2017, Intel Corporation All rights reserved. >> >> Options: >> --version show program's version number and exit >> -h, --help show this help message and exit >> -a TARGETARCH, --arch=TARGETARCH >> - ARCHS is one of list: IA32, X64, IPF, ARM, or EBC, >> - which overrides target.txt's TARGET_ARCH definition. >> - To specify more archs, please repeat this option. >> + ARCHS is one of list: IA32, X64, IPF, ARM, AARCH64 or >> + EBC, which overrides target.txt's TARGET_ARCH >> + definition. To specify more archs, please repeat this >> + option. >> -p PLATFORMFILE, --platform=PLATFORMFILE >> Build the platform specified by the DSC file name >> argument, overriding target.txt's ACTIVE_PLATFORM >> definition. >> -m MODULEFILE, --module=MODULEFILE >> @@ -112,10 +113,20 @@ Options: >> -N, --no-cache Disable build cache mechanism >> --conf=CONFDIRECTORY Specify the customized Conf directory. >> --check-usage Check usage content of entries listed in INF file. >> --ignore-sources Focus to a binary build and ignore all source files >> --pcd=OPTIONPCD Set PCD value by command line. Format: >"PcdName=Value" >> + -l COMMANDLENGTH, --cmd-len=COMMANDLENGTH >> + Specify the maximum line length of build command. >> + Default is 4096. >> + --hash Enable hash-based caching during build process. >> + --binary-destination=BINCACHEDEST >> + Generate a cache of binary files in the specified >> + directory. >> + --binary-source=BINCACHESOURCE >> + Consume a cache of binary files from the specified >> + directory. >> ``` >> >> ### D.4.1 Debug Levels >> >> The numeric debug levels are defined as integer values 0-9. >> _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
On 09/28/17 11:02, Gao, Liming wrote: > Laszlo: > Hash way may improve the incremental build performance. If hash value is not changed, module AutoGen and Make will be skipped. > Time stamp way bases on Makefile. This way will run AutoGen every time. If no change, Makefile will not be updated. I did some tests, with ArmVirtQemu (AARCH64) and OVMF (Ia32 / Ia32X64 / X64). First I built all these platforms twice in a row, *without* --hash: - build ArmVirtQemu (AARCH64) - build OVMF (Ia32) - build OVMF (Ia32X64) - build OVMF (X64) - repeat all four builds in the same order, and now capture times in the report Then I built them twice in a row, *with* --hash: - build ArmVirtQemu (AARCH64) - build OVMF (Ia32) - build OVMF (Ia32X64) - build OVMF (X64) - repeat all four builds in the same order, and now capture times in the report The idea was that in the "repeat" builds, there was nothing to actually rebuild. Then we can compare how much time "--hash" saved, between the "repeat" builds: * without --hash: build.aa64virt.report:Build Duration: 00:00:15 build.aa64virt.report:AutoGen Duration: 00:00:04 build.aa64virt.report:Make Duration: 00:00:04 build.aa64virt.report:GenFds Duration: 00:00:06 build.ovmf.32.report:Build Duration: 00:00:16 build.ovmf.32.report:AutoGen Duration: 00:00:04 build.ovmf.32.report:Make Duration: 00:00:06 build.ovmf.32.report:GenFds Duration: 00:00:06 build.ovmf.3264.report:Build Duration: 00:00:20 build.ovmf.3264.report:AutoGen Duration: 00:00:06 build.ovmf.3264.report:Make Duration: 00:00:07 build.ovmf.3264.report:GenFds Duration: 00:00:07 build.ovmf.report:Build Duration: 00:00:18 build.ovmf.report:AutoGen Duration: 00:00:04 build.ovmf.report:Make Duration: 00:00:06 build.ovmf.report:GenFds Duration: 00:00:06 * With --hash: build.aa64virt.report:Build Duration: 00:00:11 build.aa64virt.report:AutoGen Duration: 00:00:04 build.aa64virt.report:GenFds Duration: 00:00:06 build.ovmf.32.report:Build Duration: 00:00:11 build.ovmf.32.report:AutoGen Duration: 00:00:04 build.ovmf.32.report:GenFds Duration: 00:00:06 build.ovmf.3264.report:Build Duration: 00:00:14 build.ovmf.3264.report:AutoGen Duration: 00:00:06 build.ovmf.3264.report:GenFds Duration: 00:00:07 build.ovmf.report:Build Duration: 00:00:12 build.ovmf.report:AutoGen Duration: 00:00:05 build.ovmf.report:GenFds Duration: 00:00:06 With "--hash", the "Make" step disappeared entirely (which is very welcome, and I'll start using --hash today). However, the "AutoGen" step took just as long. Can you think of a reason why? (Above you mention that AutoGen should be skipped too.) Thanks! Laszlo >> -----Original Message----- >> From: Laszlo Ersek [mailto:lersek@redhat.com] >> Sent: Thursday, September 28, 2017 4:19 PM >> To: Zhu, Yonghong <yonghong.zhu@intel.com>; edk2-devel@lists.01.org >> Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Shaw, Kevin W >> <kevin.w.shaw@intel.com>; Gao, Liming <liming.gao@intel.com> >> Subject: Re: [edk2] [Patch V2] Build spec: add description for build with binary >> cache >> >> On 09/19/17 08:48, Yonghong Zhu wrote: >>> V2: >>> change the option name to --binary-destination and --binary-source. >>> >>> fixes:https://bugzilla.tianocore.org/show_bug.cgi?id=689 >> >> What are the benefits of a hash-based incremental build over a >> timestamp-based incremental build? >> >> Thank you, >> Laszlo >> >>> Cc: Liming Gao <liming.gao@intel.com> >>> Cc: Michael Kinney <michael.d.kinney@intel.com> >>> Cc: Kevin W Shaw <kevin.w.shaw@intel.com> >>> Contributed-under: TianoCore Contribution Agreement 1.1 >>> Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com> >>> --- >>> .../82_auto-generation_process.md | 20 >> ++++++++++++++++++++ >>> README.md | 1 + >>> appendix_d_buildexe_command/d4_usage.md | 19 >> +++++++++++++++---- >>> 3 files changed, 36 insertions(+), 4 deletions(-) >>> >>> diff --git a/8_pre-build_autogen_stage/82_auto-generation_process.md >> b/8_pre-build_autogen_stage/82_auto-generation_process.md >>> index 671a7d5..f2ddf32 100644 >>> --- a/8_pre-build_autogen_stage/82_auto-generation_process.md >>> +++ b/8_pre-build_autogen_stage/82_auto-generation_process.md >>> @@ -1031,10 +1031,30 @@ maximum command line length. The default >> value is 4096. >>> **Note:** The following `FLAGS` options are included in the response file: >>> `PP_FLAGS`, `CC_FLAGS`, `VFRPP_FLAGS`, `APP_FLAGS`, `ASLPP_FLAGS`, >> `ASLCC_FLAGS`, >>> and `ASM_FLAGS`. >>> ********** >>> >>> +#### 8.2.4.15 Build with Binary Cache >>> + >>> +**build** tool provides three new options for binary cache feature. >>> +--hash enables hash-based caching during build process. when --hash is >> enabled, >>> +build tool will base on the module hash value to do the incremental build, >> without >>> +--hash, build tool will base on the timestamp to do the incremental build. -- >> hash >>> +option use md5 method to get every hash value, DSC/FDF, tools_def.txt, >> build_rule.txt >>> +and build command are calculated as global hash value, Package DEC and its >> include >>> +header files are calculated as package hash value, Module source files and >> its INF >>> +file are calculated as module hash value. Library hash value will combine >> the global >>> +hash value and its dependent package hash value. Driver hash value will >> combine the >>> +global hash value, its dependent package hash value and its linked library >> hash value. >>> +When --hash and --binary-destination are specified, build tool will copy the >> generated >>> +binary files for each module into the directory specified by binary- >> destination at the >>> +build phase. Binary-destination directory caches all the generated binary >> files. >>> +When --hash and --binary-source are specified, build tool will try to get the >> binary >>> +files from the binary source directory at the build phase. If the cached >> binary has >>> +the same hash value, it will be directly used. Otherwise, build tool will >> compile the >>> +source files and generate the binary files. >>> + >>> ### 8.2.5 Post processing >>> >>> Once all files are parsed, the build tools will do following work for each EDK >>> II module: >>> >>> diff --git a/README.md b/README.md >>> index 52abb6a..ca59a35 100644 >>> --- a/README.md >>> +++ b/README.md >>> @@ -215,5 +215,6 @@ Copyright (c) 2008-2017, Intel Corporation. All rights >> reserved. >>> | | [#523](https://bugzilla.tianocore.org/show_bug.cgi?id=523) Build >> spec: add EBNF for the --pcd syntax in the Section D.4 >> | | >>> | | [#517](https://bugzilla.tianocore.org/show_bug.cgi?id=517) Build >> spec: chapter 5.2.2 Guided Tools add description for Pkcs7Sign tool and >> BrotliCompress tool >> | | >>> | | [#481](https://bugzilla.tianocore.org/show_bug.cgi?id=481) Build >> Spec: add clarification for not used Pcd that build tool will not do additional >> checks on its value >> | | >>> | | [#518](https://bugzilla.tianocore.org/show_bug.cgi?id=518) Build >> Spec: Update Precedence of PCD Values >> | | >>> | | [#669](https://bugzilla.tianocore.org/show_bug.cgi?id=669) Build >> Spec: Add multi-arg support to PREBUILD/POSTBUILD >> | | >>> +| | [#689](https://bugzilla.tianocore.org/show_bug.cgi?id=689) Build >> spec: add description for build with binary cache >> | | >>> diff --git a/appendix_d_buildexe_command/d4_usage.md >> b/appendix_d_buildexe_command/d4_usage.md >>> index b71f2d0..c901266 100644 >>> --- a/appendix_d_buildexe_command/d4_usage.md >>> +++ b/appendix_d_buildexe_command/d4_usage.md >>> @@ -32,19 +32,20 @@ >>> ## D.4 Usage >>> >>> ```ini >>> Usage: build.exe [options] >>> [all|fds|genc|genmake|clean|cleanall|cleanlib|modules|libraries|run] >>> -Copyright (c) 2007 - 2014, Intel Corporation All rights reserved. >>> +Copyright (c) 2007 - 2017, Intel Corporation All rights reserved. >>> >>> Options: >>> --version show program's version number and exit >>> -h, --help show this help message and exit >>> -a TARGETARCH, --arch=TARGETARCH >>> - ARCHS is one of list: IA32, X64, IPF, ARM, or EBC, >>> - which overrides target.txt's TARGET_ARCH definition. >>> - To specify more archs, please repeat this option. >>> + ARCHS is one of list: IA32, X64, IPF, ARM, AARCH64 or >>> + EBC, which overrides target.txt's TARGET_ARCH >>> + definition. To specify more archs, please repeat this >>> + option. >>> -p PLATFORMFILE, --platform=PLATFORMFILE >>> Build the platform specified by the DSC file name >>> argument, overriding target.txt's ACTIVE_PLATFORM >>> definition. >>> -m MODULEFILE, --module=MODULEFILE >>> @@ -112,10 +113,20 @@ Options: >>> -N, --no-cache Disable build cache mechanism >>> --conf=CONFDIRECTORY Specify the customized Conf directory. >>> --check-usage Check usage content of entries listed in INF file. >>> --ignore-sources Focus to a binary build and ignore all source files >>> --pcd=OPTIONPCD Set PCD value by command line. Format: >> "PcdName=Value" >>> + -l COMMANDLENGTH, --cmd-len=COMMANDLENGTH >>> + Specify the maximum line length of build command. >>> + Default is 4096. >>> + --hash Enable hash-based caching during build process. >>> + --binary-destination=BINCACHEDEST >>> + Generate a cache of binary files in the specified >>> + directory. >>> + --binary-source=BINCACHESOURCE >>> + Consume a cache of binary files from the specified >>> + directory. >>> ``` >>> >>> ### D.4.1 Debug Levels >>> >>> The numeric debug levels are defined as integer values 0-9. >>> > > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel > _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
Laszlo: I would like more people use it. Thank you! If you find any issue, please let me know. When hash is enabled, AutoGen phase calculates hash value of every source files, INF, DSC and FDF, but has no AutoGen header and source file generation. So, AutoGen still takes some time. Thanks Liming >-----Original Message----- >From: Laszlo Ersek [mailto:lersek@redhat.com] >Sent: Thursday, September 28, 2017 7:15 PM >To: Gao, Liming <liming.gao@intel.com>; Zhu, Yonghong ><yonghong.zhu@intel.com>; edk2-devel@lists.01.org >Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Shaw, Kevin W ><kevin.w.shaw@intel.com> >Subject: Re: [edk2] [Patch V2] Build spec: add description for build with binary >cache > >On 09/28/17 11:02, Gao, Liming wrote: >> Laszlo: >> Hash way may improve the incremental build performance. If hash value is >not changed, module AutoGen and Make will be skipped. >> Time stamp way bases on Makefile. This way will run AutoGen every time. >If no change, Makefile will not be updated. > >I did some tests, with ArmVirtQemu (AARCH64) and OVMF (Ia32 / Ia32X64 / >X64). > >First I built all these platforms twice in a row, *without* --hash: >- build ArmVirtQemu (AARCH64) >- build OVMF (Ia32) >- build OVMF (Ia32X64) >- build OVMF (X64) >- repeat all four builds in the same order, and now capture times in the >report > >Then I built them twice in a row, *with* --hash: >- build ArmVirtQemu (AARCH64) >- build OVMF (Ia32) >- build OVMF (Ia32X64) >- build OVMF (X64) >- repeat all four builds in the same order, and now capture times in the >report > >The idea was that in the "repeat" builds, there was nothing to actually >rebuild. > >Then we can compare how much time "--hash" saved, between the "repeat" >builds: > >* without --hash: > >build.aa64virt.report:Build Duration: 00:00:15 >build.aa64virt.report:AutoGen Duration: 00:00:04 >build.aa64virt.report:Make Duration: 00:00:04 >build.aa64virt.report:GenFds Duration: 00:00:06 > >build.ovmf.32.report:Build Duration: 00:00:16 >build.ovmf.32.report:AutoGen Duration: 00:00:04 >build.ovmf.32.report:Make Duration: 00:00:06 >build.ovmf.32.report:GenFds Duration: 00:00:06 > >build.ovmf.3264.report:Build Duration: 00:00:20 >build.ovmf.3264.report:AutoGen Duration: 00:00:06 >build.ovmf.3264.report:Make Duration: 00:00:07 >build.ovmf.3264.report:GenFds Duration: 00:00:07 > >build.ovmf.report:Build Duration: 00:00:18 >build.ovmf.report:AutoGen Duration: 00:00:04 >build.ovmf.report:Make Duration: 00:00:06 >build.ovmf.report:GenFds Duration: 00:00:06 > >* With --hash: > >build.aa64virt.report:Build Duration: 00:00:11 >build.aa64virt.report:AutoGen Duration: 00:00:04 >build.aa64virt.report:GenFds Duration: 00:00:06 > >build.ovmf.32.report:Build Duration: 00:00:11 >build.ovmf.32.report:AutoGen Duration: 00:00:04 >build.ovmf.32.report:GenFds Duration: 00:00:06 > >build.ovmf.3264.report:Build Duration: 00:00:14 >build.ovmf.3264.report:AutoGen Duration: 00:00:06 >build.ovmf.3264.report:GenFds Duration: 00:00:07 > >build.ovmf.report:Build Duration: 00:00:12 >build.ovmf.report:AutoGen Duration: 00:00:05 >build.ovmf.report:GenFds Duration: 00:00:06 > >With "--hash", the "Make" step disappeared entirely (which is very >welcome, and I'll start using --hash today). > >However, the "AutoGen" step took just as long. Can you think of a reason >why? (Above you mention that AutoGen should be skipped too.) > >Thanks! >Laszlo > > >>> -----Original Message----- >>> From: Laszlo Ersek [mailto:lersek@redhat.com] >>> Sent: Thursday, September 28, 2017 4:19 PM >>> To: Zhu, Yonghong <yonghong.zhu@intel.com>; edk2-devel@lists.01.org >>> Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Shaw, Kevin W >>> <kevin.w.shaw@intel.com>; Gao, Liming <liming.gao@intel.com> >>> Subject: Re: [edk2] [Patch V2] Build spec: add description for build with >binary >>> cache >>> >>> On 09/19/17 08:48, Yonghong Zhu wrote: >>>> V2: >>>> change the option name to --binary-destination and --binary-source. >>>> >>>> fixes:https://bugzilla.tianocore.org/show_bug.cgi?id=689 >>> >>> What are the benefits of a hash-based incremental build over a >>> timestamp-based incremental build? >>> >>> Thank you, >>> Laszlo >>> >>>> Cc: Liming Gao <liming.gao@intel.com> >>>> Cc: Michael Kinney <michael.d.kinney@intel.com> >>>> Cc: Kevin W Shaw <kevin.w.shaw@intel.com> >>>> Contributed-under: TianoCore Contribution Agreement 1.1 >>>> Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com> >>>> --- >>>> .../82_auto-generation_process.md | 20 >>> ++++++++++++++++++++ >>>> README.md | 1 + >>>> appendix_d_buildexe_command/d4_usage.md | 19 >>> +++++++++++++++---- >>>> 3 files changed, 36 insertions(+), 4 deletions(-) >>>> >>>> diff --git a/8_pre-build_autogen_stage/82_auto-generation_process.md >>> b/8_pre-build_autogen_stage/82_auto-generation_process.md >>>> index 671a7d5..f2ddf32 100644 >>>> --- a/8_pre-build_autogen_stage/82_auto-generation_process.md >>>> +++ b/8_pre-build_autogen_stage/82_auto-generation_process.md >>>> @@ -1031,10 +1031,30 @@ maximum command line length. The default >>> value is 4096. >>>> **Note:** The following `FLAGS` options are included in the response >file: >>>> `PP_FLAGS`, `CC_FLAGS`, `VFRPP_FLAGS`, `APP_FLAGS`, `ASLPP_FLAGS`, >>> `ASLCC_FLAGS`, >>>> and `ASM_FLAGS`. >>>> ********** >>>> >>>> +#### 8.2.4.15 Build with Binary Cache >>>> + >>>> +**build** tool provides three new options for binary cache feature. >>>> +--hash enables hash-based caching during build process. when --hash is >>> enabled, >>>> +build tool will base on the module hash value to do the incremental build, >>> without >>>> +--hash, build tool will base on the timestamp to do the incremental build. >-- >>> hash >>>> +option use md5 method to get every hash value, DSC/FDF, tools_def.txt, >>> build_rule.txt >>>> +and build command are calculated as global hash value, Package DEC and >its >>> include >>>> +header files are calculated as package hash value, Module source files >and >>> its INF >>>> +file are calculated as module hash value. Library hash value will combine >>> the global >>>> +hash value and its dependent package hash value. Driver hash value will >>> combine the >>>> +global hash value, its dependent package hash value and its linked >library >>> hash value. >>>> +When --hash and --binary-destination are specified, build tool will copy >the >>> generated >>>> +binary files for each module into the directory specified by binary- >>> destination at the >>>> +build phase. Binary-destination directory caches all the generated binary >>> files. >>>> +When --hash and --binary-source are specified, build tool will try to get >the >>> binary >>>> +files from the binary source directory at the build phase. If the cached >>> binary has >>>> +the same hash value, it will be directly used. Otherwise, build tool will >>> compile the >>>> +source files and generate the binary files. >>>> + >>>> ### 8.2.5 Post processing >>>> >>>> Once all files are parsed, the build tools will do following work for each >EDK >>>> II module: >>>> >>>> diff --git a/README.md b/README.md >>>> index 52abb6a..ca59a35 100644 >>>> --- a/README.md >>>> +++ b/README.md >>>> @@ -215,5 +215,6 @@ Copyright (c) 2008-2017, Intel Corporation. All >rights >>> reserved. >>>> | | [#523](https://bugzilla.tianocore.org/show_bug.cgi?id=523) >Build >>> spec: add EBNF for the --pcd syntax in the Section D.4 >>> | | >>>> | | [#517](https://bugzilla.tianocore.org/show_bug.cgi?id=517) >Build >>> spec: chapter 5.2.2 Guided Tools add description for Pkcs7Sign tool and >>> BrotliCompress tool >>> | | >>>> | | [#481](https://bugzilla.tianocore.org/show_bug.cgi?id=481) >Build >>> Spec: add clarification for not used Pcd that build tool will not do additional >>> checks on its value >>> | | >>>> | | [#518](https://bugzilla.tianocore.org/show_bug.cgi?id=518) >Build >>> Spec: Update Precedence of PCD Values >>> | | >>>> | | [#669](https://bugzilla.tianocore.org/show_bug.cgi?id=669) >Build >>> Spec: Add multi-arg support to PREBUILD/POSTBUILD >>> | | >>>> +| | [#689](https://bugzilla.tianocore.org/show_bug.cgi?id=689) >Build >>> spec: add description for build with binary cache >>> | | >>>> diff --git a/appendix_d_buildexe_command/d4_usage.md >>> b/appendix_d_buildexe_command/d4_usage.md >>>> index b71f2d0..c901266 100644 >>>> --- a/appendix_d_buildexe_command/d4_usage.md >>>> +++ b/appendix_d_buildexe_command/d4_usage.md >>>> @@ -32,19 +32,20 @@ >>>> ## D.4 Usage >>>> >>>> ```ini >>>> Usage: build.exe [options] >>>> [all|fds|genc|genmake|clean|cleanall|cleanlib|modules|libraries|run] >>>> -Copyright (c) 2007 - 2014, Intel Corporation All rights reserved. >>>> +Copyright (c) 2007 - 2017, Intel Corporation All rights reserved. >>>> >>>> Options: >>>> --version show program's version number and exit >>>> -h, --help show this help message and exit >>>> -a TARGETARCH, --arch=TARGETARCH >>>> - ARCHS is one of list: IA32, X64, IPF, ARM, or EBC, >>>> - which overrides target.txt's TARGET_ARCH definition. >>>> - To specify more archs, please repeat this option. >>>> + ARCHS is one of list: IA32, X64, IPF, ARM, AARCH64 or >>>> + EBC, which overrides target.txt's TARGET_ARCH >>>> + definition. To specify more archs, please repeat this >>>> + option. >>>> -p PLATFORMFILE, --platform=PLATFORMFILE >>>> Build the platform specified by the DSC file name >>>> argument, overriding target.txt's ACTIVE_PLATFORM >>>> definition. >>>> -m MODULEFILE, --module=MODULEFILE >>>> @@ -112,10 +113,20 @@ Options: >>>> -N, --no-cache Disable build cache mechanism >>>> --conf=CONFDIRECTORY Specify the customized Conf directory. >>>> --check-usage Check usage content of entries listed in INF file. >>>> --ignore-sources Focus to a binary build and ignore all source files >>>> --pcd=OPTIONPCD Set PCD value by command line. Format: >>> "PcdName=Value" >>>> + -l COMMANDLENGTH, --cmd-len=COMMANDLENGTH >>>> + Specify the maximum line length of build command. >>>> + Default is 4096. >>>> + --hash Enable hash-based caching during build process. >>>> + --binary-destination=BINCACHEDEST >>>> + Generate a cache of binary files in the specified >>>> + directory. >>>> + --binary-source=BINCACHESOURCE >>>> + Consume a cache of binary files from the specified >>>> + directory. >>>> ``` >>>> >>>> ### D.4.1 Debug Levels >>>> >>>> The numeric debug levels are defined as integer values 0-9. >>>> >> >> _______________________________________________ >> edk2-devel mailing list >> edk2-devel@lists.01.org >> https://lists.01.org/mailman/listinfo/edk2-devel >> _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
On 09/19/17 08:48, Yonghong Zhu wrote: > V2: > change the option name to --binary-destination and --binary-source. > > fixes:https://bugzilla.tianocore.org/show_bug.cgi?id=689 > Cc: Liming Gao <liming.gao@intel.com> > Cc: Michael Kinney <michael.d.kinney@intel.com> > Cc: Kevin W Shaw <kevin.w.shaw@intel.com> > Contributed-under: TianoCore Contribution Agreement 1.1 > Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com> > --- > .../82_auto-generation_process.md | 20 ++++++++++++++++++++ > README.md | 1 + > appendix_d_buildexe_command/d4_usage.md | 19 +++++++++++++++---- > 3 files changed, 36 insertions(+), 4 deletions(-) > > diff --git a/8_pre-build_autogen_stage/82_auto-generation_process.md b/8_pre-build_autogen_stage/82_auto-generation_process.md > index 671a7d5..f2ddf32 100644 > --- a/8_pre-build_autogen_stage/82_auto-generation_process.md > +++ b/8_pre-build_autogen_stage/82_auto-generation_process.md > @@ -1031,10 +1031,30 @@ maximum command line length. The default value is 4096. > **Note:** The following `FLAGS` options are included in the response file: > `PP_FLAGS`, `CC_FLAGS`, `VFRPP_FLAGS`, `APP_FLAGS`, `ASLPP_FLAGS`, `ASLCC_FLAGS`, > and `ASM_FLAGS`. > ********** > > +#### 8.2.4.15 Build with Binary Cache > + > +**build** tool provides three new options for binary cache feature. > +--hash enables hash-based caching during build process. when --hash is enabled, > +build tool will base on the module hash value to do the incremental build, without > +--hash, build tool will base on the timestamp to do the incremental build. --hash > +option use md5 method to get every hash value, DSC/FDF, tools_def.txt, build_rule.txt > +and build command are calculated as global hash value, Package DEC and its include > +header files are calculated as package hash value, Module source files and its INF > +file are calculated as module hash value. Library hash value will combine the global > +hash value and its dependent package hash value. Driver hash value will combine the > +global hash value, its dependent package hash value and its linked library hash value. > +When --hash and --binary-destination are specified, build tool will copy the generated > +binary files for each module into the directory specified by binary-destination at the > +build phase. Binary-destination directory caches all the generated binary files. > +When --hash and --binary-source are specified, build tool will try to get the binary > +files from the binary source directory at the build phase. If the cached binary has > +the same hash value, it will be directly used. Otherwise, build tool will compile the > +source files and generate the binary files. > + I have another question about this feature: how can I invalidate the binary cache, so that the next invocation with "--hash" fall back to the timestamp-based incremental build? Is it enough if I remove all "*.hash" files from the Build tree? This question is relevant for bisection. Assume that we are bisecting a commit range that straddles the introduction of "--hash" to BaseTools, and that we use a build script that uses "--hash" whenever it is available. Consider the following build order: (1) Build the tree with "--hash" (because "--hash" is available at the commit that we're testing). Assume this build displays the symptom we are bisecting, so we enter "git bisect bad", and git checks out an earlier commit. (2) Assume BaseTools lacks "--hash" at this earlier commit, so we build the tree without "--hash". Using the timestamp checks, the modules changed or affected by the checkout done by git-bisect in step (1) will be rebuilt correctly. We test this build and find that it works okay, so we enter "git bisect good". Git checks out a later commit (such that it is earlier than the one tested in (1)). (3) Assume that at this commit, "--hash" is again available, so we build the tree with "--hash". This is incorrect however, because the cached values come from step (1), not step (2). This means that every time the build script notices that "--hash" is unavailable, it must invalidate (delete) all the cached values, so that *next time* it returns to using "--hash", all the hash values are recalculated from scratch, and only the timestamp based checks are used for that build. So, how can I invalidate all the cached values? Is it enough to delete the *.hash files? Thanks, Laszlo _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
On 09/28/17 14:06, Laszlo Ersek wrote: > So, how can I invalidate all the cached values? Is it enough to delete > the *.hash files? ... I'm asking because I tried the --binary-destination option as well, and in the bin cache directory, *.depex and *.inf files were stored as well, not just *.hash values. So I figure, whatever gets stored in the binary cache directory, should be deleted when I'd like to invalidate the cache. Now, removing this separate cache directory altogether would solve my question; but that would require me to use "--binary-destination" and "--binary-source" together. The idea is that "--hash" would fetch the hash values from that separate directory, and it would also write the new hash values back to that directory. In addition this directory would be very easy to remove, so "--hash" would know whenever the hash was empty. However, it looks like "--binary-destination" and "--binary-source" cannot be used together. Why is that? Thanks, Laszlo _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
Laszlo: --binary-destination is to cache the generated binary in this directory, --binary-source is to fetch the cached binary from this directory. If they are used together, the behavior will be same that --hash option only and Build output directory. We don't think it is the necessary usage model. So, we don't add this support. If you have the real case that depends on their combination, we can plan to add it. Thanks Liming >-----Original Message----- >From: Laszlo Ersek [mailto:lersek@redhat.com] >Sent: Thursday, September 28, 2017 8:14 PM >To: Zhu, Yonghong <yonghong.zhu@intel.com>; edk2-devel@lists.01.org >Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Shaw, Kevin W ><kevin.w.shaw@intel.com>; Gao, Liming <liming.gao@intel.com> >Subject: Re: [edk2] [Patch V2] Build spec: add description for build with binary >cache > >On 09/28/17 14:06, Laszlo Ersek wrote: > >> So, how can I invalidate all the cached values? Is it enough to delete >> the *.hash files? > >... I'm asking because I tried the --binary-destination option as well, >and in the bin cache directory, *.depex and *.inf files were stored as >well, not just *.hash values. > >So I figure, whatever gets stored in the binary cache directory, should >be deleted when I'd like to invalidate the cache. > >Now, removing this separate cache directory altogether would solve my >question; but that would require me to use "--binary-destination" and >"--binary-source" together. The idea is that "--hash" would fetch the >hash values from that separate directory, and it would also write the >new hash values back to that directory. In addition this directory would >be very easy to remove, so "--hash" would know whenever the hash was >empty. > >However, it looks like "--binary-destination" and "--binary-source" >cannot be used together. Why is that? > >Thanks, >Laszlo _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
If use both flags is same as --hash. Can we just get rid of --hash and just use the existing two flags? In this way, we can hide the internal implementation of how to check modification. Thanks/Ray > -----Original Message----- > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Gao, > Liming > Sent: Saturday, September 30, 2017 1:03 PM > To: Laszlo Ersek <lersek@redhat.com>; Zhu, Yonghong > <yonghong.zhu@intel.com>; edk2-devel@lists.01.org > Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Shaw, Kevin W > <kevin.w.shaw@intel.com> > Subject: Re: [edk2] [Patch V2] Build spec: add description for build with binary > cache > > Laszlo: > --binary-destination is to cache the generated binary in this directory, --binary- > source is to fetch the cached binary from this directory. If they are used > together, the behavior will be same that --hash option only and Build output > directory. We don't think it is the necessary usage model. So, we don't add this > support. If you have the real case that depends on their combination, we can > plan to add it. > > Thanks > Liming > >-----Original Message----- > >From: Laszlo Ersek [mailto:lersek@redhat.com] > >Sent: Thursday, September 28, 2017 8:14 PM > >To: Zhu, Yonghong <yonghong.zhu@intel.com>; edk2-devel@lists.01.org > >Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Shaw, Kevin W > ><kevin.w.shaw@intel.com>; Gao, Liming <liming.gao@intel.com> > >Subject: Re: [edk2] [Patch V2] Build spec: add description for build > >with binary cache > > > >On 09/28/17 14:06, Laszlo Ersek wrote: > > > >> So, how can I invalidate all the cached values? Is it enough to > >> delete the *.hash files? > > > >... I'm asking because I tried the --binary-destination option as well, > >and in the bin cache directory, *.depex and *.inf files were stored as > >well, not just *.hash values. > > > >So I figure, whatever gets stored in the binary cache directory, should > >be deleted when I'd like to invalidate the cache. > > > >Now, removing this separate cache directory altogether would solve my > >question; but that would require me to use "--binary-destination" and > >"--binary-source" together. The idea is that "--hash" would fetch the > >hash values from that separate directory, and it would also write the > >new hash values back to that directory. In addition this directory > >would be very easy to remove, so "--hash" would know whenever the hash > >was empty. > > > >However, it looks like "--binary-destination" and "--binary-source" > >cannot be used together. Why is that? > > > >Thanks, > >Laszlo > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
Laszlo: Yes. Delete .hash can invalidate hash way. But, --hash option is to try hash first. If hash has no change, there is no AutoGen and Make. If hash is change, AutoGen and Make will still be trigged. So, for your case, if any file is changed, it will cause hash value be changed, then trig time-stamp Make build. I don't think you need to invalidate hash. Thanks Liming >-----Original Message----- >From: Laszlo Ersek [mailto:lersek@redhat.com] >Sent: Thursday, September 28, 2017 8:07 PM >To: Zhu, Yonghong <yonghong.zhu@intel.com>; edk2-devel@lists.01.org >Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Shaw, Kevin W ><kevin.w.shaw@intel.com>; Gao, Liming <liming.gao@intel.com> >Subject: Re: [edk2] [Patch V2] Build spec: add description for build with binary >cache > >On 09/19/17 08:48, Yonghong Zhu wrote: >> V2: >> change the option name to --binary-destination and --binary-source. >> >> fixes:https://bugzilla.tianocore.org/show_bug.cgi?id=689 >> Cc: Liming Gao <liming.gao@intel.com> >> Cc: Michael Kinney <michael.d.kinney@intel.com> >> Cc: Kevin W Shaw <kevin.w.shaw@intel.com> >> Contributed-under: TianoCore Contribution Agreement 1.1 >> Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com> >> --- >> .../82_auto-generation_process.md | 20 >++++++++++++++++++++ >> README.md | 1 + >> appendix_d_buildexe_command/d4_usage.md | 19 >+++++++++++++++---- >> 3 files changed, 36 insertions(+), 4 deletions(-) >> >> diff --git a/8_pre-build_autogen_stage/82_auto-generation_process.md >b/8_pre-build_autogen_stage/82_auto-generation_process.md >> index 671a7d5..f2ddf32 100644 >> --- a/8_pre-build_autogen_stage/82_auto-generation_process.md >> +++ b/8_pre-build_autogen_stage/82_auto-generation_process.md >> @@ -1031,10 +1031,30 @@ maximum command line length. The default >value is 4096. >> **Note:** The following `FLAGS` options are included in the response file: >> `PP_FLAGS`, `CC_FLAGS`, `VFRPP_FLAGS`, `APP_FLAGS`, `ASLPP_FLAGS`, >`ASLCC_FLAGS`, >> and `ASM_FLAGS`. >> ********** >> >> +#### 8.2.4.15 Build with Binary Cache >> + >> +**build** tool provides three new options for binary cache feature. >> +--hash enables hash-based caching during build process. when --hash is >enabled, >> +build tool will base on the module hash value to do the incremental build, >without >> +--hash, build tool will base on the timestamp to do the incremental build. -- >hash >> +option use md5 method to get every hash value, DSC/FDF, tools_def.txt, >build_rule.txt >> +and build command are calculated as global hash value, Package DEC and its >include >> +header files are calculated as package hash value, Module source files and >its INF >> +file are calculated as module hash value. Library hash value will combine >the global >> +hash value and its dependent package hash value. Driver hash value will >combine the >> +global hash value, its dependent package hash value and its linked library >hash value. >> +When --hash and --binary-destination are specified, build tool will copy the >generated >> +binary files for each module into the directory specified by binary- >destination at the >> +build phase. Binary-destination directory caches all the generated binary >files. >> +When --hash and --binary-source are specified, build tool will try to get the >binary >> +files from the binary source directory at the build phase. If the cached >binary has >> +the same hash value, it will be directly used. Otherwise, build tool will >compile the >> +source files and generate the binary files. >> + > >I have another question about this feature: how can I invalidate the >binary cache, so that the next invocation with "--hash" fall back to the >timestamp-based incremental build? > >Is it enough if I remove all "*.hash" files from the Build tree? > >This question is relevant for bisection. Assume that we are bisecting a >commit range that straddles the introduction of "--hash" to BaseTools, >and that we use a build script that uses "--hash" whenever it is available. > >Consider the following build order: > >(1) Build the tree with "--hash" (because "--hash" is available at the >commit that we're testing). Assume this build displays the symptom we >are bisecting, so we enter "git bisect bad", and git checks out an >earlier commit. > >(2) Assume BaseTools lacks "--hash" at this earlier commit, so we build >the tree without "--hash". Using the timestamp checks, the modules >changed or affected by the checkout done by git-bisect in step (1) will >be rebuilt correctly. We test this build and find that it works okay, so >we enter "git bisect good". Git checks out a later commit (such that it >is earlier than the one tested in (1)). > >(3) Assume that at this commit, "--hash" is again available, so we build >the tree with "--hash". This is incorrect however, because the cached >values come from step (1), not step (2). > > >This means that every time the build script notices that "--hash" is >unavailable, it must invalidate (delete) all the cached values, so that >*next time* it returns to using "--hash", all the hash values are >recalculated from scratch, and only the timestamp based checks are used >for that build. > >So, how can I invalidate all the cached values? Is it enough to delete >the *.hash files? > >Thanks, >Laszlo _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel
On 09/30/17 04:49, Gao, Liming wrote:
> Laszlo:
> Yes. Delete .hash can invalidate hash way. But, --hash option is to
> try hash first. If hash has no change, there is no AutoGen and Make.
> If hash is change, AutoGen and Make will still be trigged. So, for
> your case, if any file is changed, it will cause hash value be
> changed, then trig time-stamp Make build. I don't think you need to
> invalidate hash.
I've done the following test:
number hash subject commit timestamp
------ ------------ -------------------------------------------------------------------- --------------------------
1 8932679df5be MdeModulePkg/DxeCore: Add check to ensure no possible NULL ptr deref 2017-09-26 09:38:46 +08:00
2 1b8eca8b1aff BaseTools: report build time measured by module of EDKII Build 2017-09-26 13:39:39 +08:00
3 36d083ef0018 BaseTools: add support for BIOS build with binary cache 2017-09-27 17:14:54 +08:00
4 119d8c42ab8b BaseTools: Fix the regression bug to build single module 2017-09-28 10:10:39 +08:00
5 62634215f320 ShellPkg/UefiHandleParsingLib.c: Map SmmPciRootBridgeIo correctly 2017-09-29 01:08:07 +08:00
6 770f3f6144b4 ShellPkg/dh: Add the 'dh' dump support for Partition Info protocol 2017-09-29 09:39:35 +08:00
I built the tree at commits #6, #1 and #5. The idea is that commit #3
introduced "--hash", so it separates #1 from #5 and #6.
Commits #2 and #4 are also used for separation, because #2 introduced a
build regression, and #4 fixed it. This means that all commits before
commit #2 lack "--hash" *and* do not have the regression, and all
commits after #4 have "--hash" and also do not have the regression.
Additionally, both commits #5 and #6 modify the file
"ShellPkg/Library/UefiHandleParsingLib/UefiHandleParsingLib.c", so
moving from #6, to #1, to #5 (as if we were bisecting) requires
recompiling "UefiHandleParsingLib.c" at every step.
(1) First build: at commit #6 (770f3f6144b4). This build was done from
zero.
The build uses "--hash". "UefiHandleParsingLib.c" is compiled.
(2) Second build: at commit #1 (8932679df5be).
"--hash" is not available, so it is not used. "UefiHandleParsingLib.c"
is compiled correctly. (I checked in the build log.)
(3) Third build: at commit #5 (62634215f320). "UefiHandleParsingLib.c"
is compiled correctly. (I checked in the build log.)
So it indeed looks like I don't need to invalidate the hash manually.
Thanks!
Laszlo
>> -----Original Message-----
>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>> Sent: Thursday, September 28, 2017 8:07 PM
>> To: Zhu, Yonghong <yonghong.zhu@intel.com>; edk2-devel@lists.01.org
>> Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Shaw, Kevin W
>> <kevin.w.shaw@intel.com>; Gao, Liming <liming.gao@intel.com>
>> Subject: Re: [edk2] [Patch V2] Build spec: add description for build with binary
>> cache
>>
>> On 09/19/17 08:48, Yonghong Zhu wrote:
>>> V2:
>>> change the option name to --binary-destination and --binary-source.
>>>
>>> fixes:https://bugzilla.tianocore.org/show_bug.cgi?id=689
>>> Cc: Liming Gao <liming.gao@intel.com>
>>> Cc: Michael Kinney <michael.d.kinney@intel.com>
>>> Cc: Kevin W Shaw <kevin.w.shaw@intel.com>
>>> Contributed-under: TianoCore Contribution Agreement 1.1
>>> Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
>>> ---
>>> .../82_auto-generation_process.md | 20
>> ++++++++++++++++++++
>>> README.md | 1 +
>>> appendix_d_buildexe_command/d4_usage.md | 19
>> +++++++++++++++----
>>> 3 files changed, 36 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/8_pre-build_autogen_stage/82_auto-generation_process.md
>> b/8_pre-build_autogen_stage/82_auto-generation_process.md
>>> index 671a7d5..f2ddf32 100644
>>> --- a/8_pre-build_autogen_stage/82_auto-generation_process.md
>>> +++ b/8_pre-build_autogen_stage/82_auto-generation_process.md
>>> @@ -1031,10 +1031,30 @@ maximum command line length. The default
>> value is 4096.
>>> **Note:** The following `FLAGS` options are included in the response file:
>>> `PP_FLAGS`, `CC_FLAGS`, `VFRPP_FLAGS`, `APP_FLAGS`, `ASLPP_FLAGS`,
>> `ASLCC_FLAGS`,
>>> and `ASM_FLAGS`.
>>> **********
>>>
>>> +#### 8.2.4.15 Build with Binary Cache
>>> +
>>> +**build** tool provides three new options for binary cache feature.
>>> +--hash enables hash-based caching during build process. when --hash is
>> enabled,
>>> +build tool will base on the module hash value to do the incremental build,
>> without
>>> +--hash, build tool will base on the timestamp to do the incremental build. --
>> hash
>>> +option use md5 method to get every hash value, DSC/FDF, tools_def.txt,
>> build_rule.txt
>>> +and build command are calculated as global hash value, Package DEC and its
>> include
>>> +header files are calculated as package hash value, Module source files and
>> its INF
>>> +file are calculated as module hash value. Library hash value will combine
>> the global
>>> +hash value and its dependent package hash value. Driver hash value will
>> combine the
>>> +global hash value, its dependent package hash value and its linked library
>> hash value.
>>> +When --hash and --binary-destination are specified, build tool will copy the
>> generated
>>> +binary files for each module into the directory specified by binary-
>> destination at the
>>> +build phase. Binary-destination directory caches all the generated binary
>> files.
>>> +When --hash and --binary-source are specified, build tool will try to get the
>> binary
>>> +files from the binary source directory at the build phase. If the cached
>> binary has
>>> +the same hash value, it will be directly used. Otherwise, build tool will
>> compile the
>>> +source files and generate the binary files.
>>> +
>>
>> I have another question about this feature: how can I invalidate the
>> binary cache, so that the next invocation with "--hash" fall back to the
>> timestamp-based incremental build?
>>
>> Is it enough if I remove all "*.hash" files from the Build tree?
>>
>> This question is relevant for bisection. Assume that we are bisecting a
>> commit range that straddles the introduction of "--hash" to BaseTools,
>> and that we use a build script that uses "--hash" whenever it is available.
>>
>> Consider the following build order:
>>
>> (1) Build the tree with "--hash" (because "--hash" is available at the
>> commit that we're testing). Assume this build displays the symptom we
>> are bisecting, so we enter "git bisect bad", and git checks out an
>> earlier commit.
>>
>> (2) Assume BaseTools lacks "--hash" at this earlier commit, so we build
>> the tree without "--hash". Using the timestamp checks, the modules
>> changed or affected by the checkout done by git-bisect in step (1) will
>> be rebuilt correctly. We test this build and find that it works okay, so
>> we enter "git bisect good". Git checks out a later commit (such that it
>> is earlier than the one tested in (1)).
>>
>> (3) Assume that at this commit, "--hash" is again available, so we build
>> the tree with "--hash". This is incorrect however, because the cached
>> values come from step (1), not step (2).
>>
>>
>> This means that every time the build script notices that "--hash" is
>> unavailable, it must invalidate (delete) all the cached values, so that
>> *next time* it returns to using "--hash", all the hash values are
>> recalculated from scratch, and only the timestamp based checks are used
>> for that build.
>>
>> So, how can I invalidate all the cached values? Is it enough to delete
>> the *.hash files?
>>
>> Thanks,
>> Laszlo
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel
>
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel
© 2016 - 2024 Red Hat, Inc.