tobias/entropice
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Make aggregations work

Browse source
This commit is contained in:
Tobias Hölzer 2025-11-28 00:03:51 +01:00
parent 98314fe8b3
commit 7b09dda6a3
6 changed files with 328 additions and 97 deletions
Download patch file Download diff file Expand all files Collapse all files

40
Processing Documentation.md
View file

@ -38,18 +38,7 @@ All spatial aggregations relied heavily on CPU compute, since Cupy lacking suppo
and for higher resolution grids the amount of pixels to reduce where too small to overcome the data movement overhead of using a GPU.
The aggregations scale through the number of concurrent processes (specified by `--concurrent_partitions`) accumulating linearly more memory with higher parallel computation.
| grid | time | memory | processes |
| ----- | ------ | ------ | --------- |
| Hex3 | | | |
| Hex4 | | | |
| Hex5 | | | |
| Hex6 | | | |
| Hpx6 | 37 min | ~300GB | 40 |
| Hpx7 | | | |
| Hpx8 | | | |
| Hpx9 | 25m | ~300GB | 40 |
| Hpx10 | 34 min | ~300GB | 40 |
All spatial aggregations into the different grids done took around 30 min each, with a total memory peak of ~300 GB partitioned over 40 processes.
## Alpha Earth
@ -71,4 +60,31 @@ Each scale was choosen so that each grid cell had around 10000px do estimate the
## Era5
### Spatial aggregations into grids
All spatial aggregations relied heavily on CPU compute, since Cupy lacking support for nanquantile
and for higher resolution grids the amount of pixels to reduce where too small to overcome the data movement overhead of using a GPU.
The aggregations scale through the number of concurrent processes (specified by `--concurrent_partitions`) accumulating linearly more memory with higher parallel computation.
Since the resolution of the ERA5 dataset is spatially smaller than the resolution of the higher-resolution, different aggregations methods where used for different grid-levels:
- Common aggregations: mean, min, max, std, median, p01, p05, p25, p75, p95, p99 for low resolution grids
- Only mean aggregations for medium resolution grids
- Linar interpolation for high resolution grids
For geometries crossing the antimeridian, geometries are corrected.
| grid | method |
| ----- | ----------- |
| Hex3 | Common |
| Hex4 | Common |
| Hex5 | Mean |
| Hex6 | Interpolate |
| Hpx6 | Common |
| Hpx7 | Common |
| Hpx8 | Common |
| Hpx9 | Mean |
| Hpx10 | Interpolate |
???