Commit graph

533 commits

Author SHA1 Message Date
Dmitry Muhomor
0691a32fa6 generate-all: pass extra commands to lunch/choosecombo in envsetup.sh
Requires the corresponding patch in build/make repo.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
c8f93555c0 filter out several unused overlays without parsing them
Vendor module trees remain the same.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
5f80831d88 update vendor module specs after resource overlay changes
Notable changes:
- multiple previously-unparseable overlays are added for felix and tangorpro
- fingerprint setup strings are no longer cut-off at first line
- udfps_lottie_translate_y dimension is now parsed correctly
- "Face unlock"-related strings are removed ("Face unlock" is not shipped)
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
29cd63fdda update overlay filters for felix and tangorpro
Previously-removed overlays no longer break the resource parser.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
c036c79a5e filter out HbmSVManager overlay on all devices
HbmSVManager ("high brightness mode manager") itself is not shipped, even though this overlay
no longer breaks resource parser.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
55c6f0db6b rewrite aapt2-based overlay parser
Previous parser relied on text output of `aapt2/ dump xmltree` and `aapt2 dump resources` commands,
which is not meant for parsing.

This led to multiple parsing bugs: cutoff multiline strings, floating point parsing issues,
quoting issues, non-English locale issues etc.

Rewritten parser instead relies on protobuf aapt2 output, which allows to resolve most of these
issues.

A simple patch to aapt2 in frameworks/base is required to slightly expand its protobuf output.

Other notable changes:
- support for the "fraction" resource type
- parsing is now parallelized at APK level
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
2543191d23 add ts-proto output for Resources.proto from aapt2 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
386f1710fe add ts-proto dependency
It allows to use aapt2 resource protobuf types.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
adec586797 add protobufjs dependency
It's needed for parsing protobuf aapt2 output.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
e46c929e60 remove @types/hasha from devDependencies
It's not neeeded, hasha includes types already.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
456fddae1b remove product/etc/CarrierSettings/ from dep_files
Inline carrier settings extraction is no longer used.

This directory is not filtered out elsewhere, it will still be included in full (non-prep) vendor
modules. However, collected system state files that were generated prior to this commit need to be
regenerated, since they list these files as included in AOSP due to the way dep_files directive
works.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
86c7472066 do not use BUILD_BROKEN_ELF_PREBUILT_PRODUCT_COPY_FILES 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
7403e95179 add vendor-{specs,skels} for all supported devices 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
cb4a7e34b9 update gitignore
android-info.txt files are about to be added to vendor-skels/.

/dl is useful to have as a symlink.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
3416e4a554 generate-all: verify the result against a reference FileTreeSpec
This commit adds an extra step to generate-all:
- if --updateSpec flag is specified, FileTreeSpec of the generated vendor module is written out
to vendor-specs/$VENDOR/$DEVICE.yaml file ("reference FileTreeSpec").
- if --updateSpec is not specified, FileTreeSpec of the generated vendor module is compared against
the reference FileTreeSpec. If there are any differences, generated vendor module is deleted and an
exception is thrown.

Also, when --updateSpec flag is used, text files inside vendor module are written out to
vendor-skels/$VENDOR/$DEVICE directory.

Contents of vendor-skels/ are useful for keeping track of vendor module changes via git and for
showing diffs in case text files in the generated module don't match the reference ones.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
73c1f665f3 add code for generating and comparing FileTreeSpecs 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
c84d027c23 update links to adevtool GitHub 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
bcc612c622 sort items in generated overlay resource files
This makes them easier to read.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
5c9e695863 reorder PRODUCT_{PACKAGES,COPY_FILES} in device makefile
These directives are usually very large and are less interesting than PRODUCT_<partition>_PROPERTIES,
PRODUCT_OVERRIDE_FINGERPRINT variables. Move them to the bottom to improve readability.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
2aaa970886 add show-status command
It shows BUILD_ID, AOSP tag, platform security patch level and stock OS image availability for
each of specified devices.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
3bf5977e81 add command for making "BUILD_ID to AOSP tag" map 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
dcc8339fd7 update generate-all command
- pass device config with the standard -d flag instead of through argument
- default to using factory and OTA images for current build ID when --stockSrc is not specified
- add extract_android_ota_payload.py as the final step, with an option to skip it
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
12e586385f update generate-prep command
- pass device config with the standard -d flag instead of through argument. This allows to specify
multiple devices without using DeviceList config file
- default to using factory image for current build ID when --stockSrc is not specified
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
97b724790b improve handling of device factory images
- support unpacking factory images without root by using debugfs rdump (ext4) and
fsck.erofs --extract (erofs)
- support reusing unpacked factory images between adevtool runs to reduce latency
- unpack factory images for multiple devices in parallel. Extraction of files from partition images
is parallelized too
- extract partition images from factory image inner zip directly, without extracting it, since this
inner zip is uncompressed
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
9750841f3d collect-state: support specifying device names with DeviceList(s)
For example, `collect-state -d pixel-gen{6,7} barbet`
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
40354ec422 collect-state: set default output path to vendor/state 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
2a3fba5d2e generate-all: set default of customSrc flag to vendor/state 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
22b2d700db add helper functions for async spawning of processes 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
9be2a36aa5 add hasha dependency
This is a simple wrapper for built-in node.js hashing APIs.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
40abb08329 add yauzl-promise dependency
This is a zip unpacking library.

unzipit library that is already present doesn't support extracting from inner factory image zips
without unpacking them and extracts files to memory first, which consumes huge amount of memory
when unpacking factory images, especially in parallel.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
8d8604dbe7 check PLATFORM_SECURITY_PATCH value in product Makefile
On devices that no longer receive vendor updates (e.g. Pixel 4 / 4 XL), PLATFORM_SECURITY_PATCH
should be overriden to correctly indicate that.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
70129836eb check in product Makefile that BUILD_ID is set correctly 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
0b0b76ef70 do not use bright green color for action spinner text
It has bad readability when using dark-on-light terminal color schemes.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
d7594443cd make file system type of OS partitions part of device config 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
694afd844b improve device image downloading
- define a default download directory
- obtain image URL and checksum from the Git-tracked local build index instead of fetching it from
remote server
- default to downloading images for the current build ID
- download image to a temporary file first, mark it as complete only after checksum verification
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
e29dbb7aea add update-build-index command
See its description.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
a4d615c057 add index of Pixel Android 14 Beta 5.2 builds 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
061f631abc add index of Pixel Android 14 Beta 5.1 builds 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
6f2c9525e0 add index of Pixel Android 14 Beta 5 builds 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
1be5571e1e add index of Pixel Android 14 Beta 4.1 builds 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
47db1ba94c add index of stock OS builds for supported Pixel devices 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
1d3c8c59c8 add support for loading build index from YAML
Default path is config/build-index/build-index.yml, 'includes' directive is supported.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
e1d69888dd add fetch-build-index command 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
83ac2dba38 update node-fetch version to 2.6.13
2.6.6 is vulnerable to CVE-2022-0235
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
822c196d02 add parser of Pixel factory, OTA and vendor image pages
Extracts the following image properties:
- device name
- build ID
- build desctiption
- image download link
- image file SHA-256

Images are grouped by '$DEVICE $BUILD_ID' key.

Beta factory/OTA image pages are supported too.

JSDOM dependency is added for simple and correct parsing of these pages.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
31d493f58c make current and previous build ID part of device config
Previous build ID is useful when doing comparisons.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
81adbb0c42 improve device config loading
- support specifying device{,-list} config with config file name instead of config path
- support loading mutliple device{,-list} configs at once. Previously, loading multiple device
configs was supported only by passing a device-list config
- parallelize config loading with promises
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
4503081dbf require device config name to match device name
This is already the case for all device configs.
2023-08-31 16:03:43 -04:00
Dmitry Muhomor
65e3e84583 add path constants for OS checkout, adevtool, (device) config dirs 2023-08-31 16:03:43 -04:00
Dmitry Muhomor
409c89cbdc require adevtool to always be started from the root of OS tree
Several commands (e.g. generate-all) break when this is not the case. Several other commands have
non-obvious dependency on this invariant.
2023-08-31 16:03:43 -04:00