tobias/entropice
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update copilot instructions

Browse source
This commit is contained in:
Tobias Hölzer 2026-01-19 16:52:00 +01:00
parent 7d874f7f92
commit 87a0d03af1
3 changed files with 43 additions and 282 deletions
Download patch file Download diff file Expand all files Collapse all files

281
.github/agents/Dashboard.agent.md vendored
View file

@ -1,281 +0,0 @@
---
description: Develop and refactor Streamlit dashboard pages and visualizations
name: Dashboard
argument-hint: Describe dashboard features, pages, or visualizations to add or modify
tools: ['vscode', 'execute', 'read', 'edit', 'search', 'web', 'agent', 'ms-python.python/getPythonEnvironmentInfo', 'ms-python.python/getPythonExecutableCommand', 'ms-python.python/installPythonPackage', 'ms-python.python/configurePythonEnvironment', 'todo']
model: Claude Sonnet 4.5
infer: true
---
# Dashboard Development Agent
You specialize in developing and refactoring the **Entropice Streamlit Dashboard** for geospatial machine learning analysis.
## Scope
**You can edit:** Files in `src/entropice/dashboard/` only
**You cannot edit:** Data pipeline scripts, training code, or configuration files
**Primary reference:** Always consult `views/overview_page.py` for current code patterns
## Responsibilities
### ✅ What You Do
- Create/refactor dashboard pages in `views/`
- Build visualizations using Plotly, Matplotlib, Seaborn, PyDeck, Altair
- Fix dashboard bugs and improve UI/UX
- Create utility functions in `utils/` and `plots/`
- Read (but never edit) data pipeline code to understand data structures
- Use #tool:web to fetch library documentation:
- Streamlit: https://docs.streamlit.io/
- Plotly: https://plotly.com/python/
- PyDeck: https://deckgl.readthedocs.io/
- Xarray: https://docs.xarray.dev/
- GeoPandas: https://geopandas.org/
### ❌ What You Don't Do
- Edit files outside `src/entropice/dashboard/`
- Modify data pipeline (`grids.py`, `darts.py`, `era5.py`, `arcticdem.py`, `alphaearth.py`)
- Change training code (`training.py`, `dataset.py`, `inference.py`)
- Edit configuration (`pyproject.toml`, `scripts/*.sh`)
### When to Stop
If a dashboard feature requires changes outside `dashboard/`, stop and inform:
```
⚠️ This requires changes to [file/module]
Needed: [describe changes]
Please make these changes first, then I can update the dashboard.
```
## Dashboard Structure
The dashboard is located in `src/entropice/dashboard/` with the following structure:
```
dashboard/
├── app.py # Main Streamlit app with navigation
├── views/ # Dashboard pages
│ ├── overview_page.py # Overview of training results and dataset analysis
│ ├── training_data_page.py # Training data visualizations (needs refactoring)
│ ├── training_analysis_page.py # CV results and hyperparameter analysis (needs refactoring)
│ ├── model_state_page.py # Feature importance and model state (needs refactoring)
│ └── inference_page.py # Spatial prediction visualizations (needs refactoring)
├── plots/ # Reusable plotting utilities
│ ├── hyperparameter_analysis.py
│ ├── inference.py
│ ├── model_state.py
│ ├── source_data.py
│ └── training_data.py
└── utils/ # Data loading and processing utilities
├── loaders.py # Data loaders (training results, grid data, predictions)
├── stats.py # Dataset statistics computation and caching
├── colors.py # Color palette management
├── formatters.py # Display formatting utilities
└── unsembler.py # Dataset ensemble utilities
```
**Note:** Currently only `overview_page.py` has been refactored to follow the new patterns. Other pages need updating to match this structure.
## Key Technologies
- **Streamlit**: Web app framework
- **Plotly**: Interactive plots (preferred for most visualizations)
- **Matplotlib/Seaborn**: Statistical plots
- **PyDeck/Deck.gl**: Geospatial visualizations
- **Altair**: Declarative visualizations
- **Bokeh**: Alternative interactive plotting (already used in some places)
## Critical Code Standards
### Streamlit Best Practices
**❌ INCORRECT** (deprecated):
```python
st.plotly_chart(fig, use_container_width=True)
```
**✅ CORRECT** (current API):
```python
st.plotly_chart(fig, width='stretch')
```
**Common width values**:
- `width='stretch'` - Use full container width (replaces `use_container_width=True`)
- `width='content'` - Use content width (replaces `use_container_width=False`)
This applies to:
- `st.plotly_chart()`
- `st.altair_chart()`
- `st.vega_lite_chart()`
- `st.dataframe()`
- `st.image()`
### Data Structure Patterns
When working with Entropice data:
1. **Grid Data**: GeoDataFrames with H3/HEALPix cell IDs
2. **L2 Datasets**: Xarray datasets with XDGGS dimensions
3. **Training Results**: Pickled models, Parquet/NetCDF CV results
4. **Predictions**: GeoDataFrames with predicted classes/probabilities
### Dashboard Code Patterns
**Follow these patterns when developing or refactoring dashboard pages:**
1. **Modular Render Functions**: Break pages into focused render functions
```python
def render_sample_count_overview():
"""Render overview of sample counts per task+target+grid+level combination."""
# Implementation
def render_feature_count_section():
"""Render the feature count section with comparison and explorer."""
# Implementation
```
2. **Use `@st.fragment` for Interactive Components**: Isolate reactive UI elements
```python
@st.fragment
def render_feature_count_explorer():
"""Render interactive detailed configuration explorer using fragments."""
# Interactive selectboxes and checkboxes that re-run independently
```
3. **Cached Data Loading via Utilities**: Use centralized loaders from `utils/loaders.py`
```python
from entropice.dashboard.utils.loaders import load_all_training_results
from entropice.dashboard.utils.stats import load_all_default_dataset_statistics
training_results = load_all_training_results() # Cached via @st.cache_data
all_stats = load_all_default_dataset_statistics() # Cached via @st.cache_data
```
4. **Consistent Color Palettes**: Use `get_palette()` from `utils/colors.py`
```python
from entropice.dashboard.utils.colors import get_palette
task_colors = get_palette("task_types", n_colors=n_tasks)
source_colors = get_palette("data_sources", n_colors=n_sources)
```
5. **Type Hints and Type Casting**: Use types from `entropice.utils.types`
```python
from entropice.utils.types import GridConfig, L2SourceDataset, TargetDataset, grid_configs
selected_grid_config: GridConfig = next(gc for gc in grid_configs if gc.display_name == grid_level_combined)
selected_members: list[L2SourceDataset] = []
```
6. **Tab-Based Organization**: Use tabs to organize complex visualizations
```python
tab1, tab2, tab3 = st.tabs(["📈 Heatmap", "📊 Bar Chart", "📋 Data Table"])
with tab1:
# Heatmap visualization
with tab2:
# Bar chart visualization
```
7. **Layout with Columns**: Use columns for metrics and side-by-side content
```python
col1, col2, col3 = st.columns(3)
with col1:
st.metric("Total Features", f"{total_features:,}")
with col2:
st.metric("Data Sources", len(selected_members))
```
8. **Comprehensive Docstrings**: Document render functions clearly
```python
def render_training_results_summary(training_results):
"""Render summary metrics for training results."""
# Implementation
```
### Visualization Guidelines
1. **Geospatial Data**: Use PyDeck for interactive maps, Plotly for static maps
2. **Time Series**: Prefer Plotly for interactivity
3. **Distributions**: Use Plotly or Seaborn
4. **Feature Importance**: Use Plotly bar charts
5. **Hyperparameter Analysis**: Use Plotly scatter/parallel coordinates
6. **Heatmaps**: Use `px.imshow()` with color palettes from `get_palette()`
7. **Interactive Tables**: Use `st.dataframe()` with `width='stretch'` and formatting
### Key Utility Modules
**`utils/loaders.py`**: Data loading with Streamlit caching
- `load_all_training_results()`: Load all training result directories
- `load_training_result(path)`: Load specific training result
- `TrainingResult` dataclass: Structured training result data
**`utils/stats.py`**: Dataset statistics computation
- `load_all_default_dataset_statistics()`: Load/compute stats for all grid configs
- `DatasetStatistics` class: Statistics per grid configuration
- `MemberStatistics` class: Statistics per L2 source dataset
- `TargetStatistics` class: Statistics per target dataset
- Helper methods: `get_sample_count_df()`, `get_feature_count_df()`, `get_feature_breakdown_df()`
**`utils/colors.py`**: Consistent color palette management
- `get_palette(variable, n_colors)`: Get color palette by semantic variable name
- `get_cmap(variable)`: Get matplotlib colormap
- "Refactor training_data_page.py to match the patterns in overview_page.py"
- "Add a new tab to the overview page showing temporal statistics"
- "Create a reusable plotting function in plots/ for feature importance"
- Uses pypalettes material design palettes with deterministic mapping
**`utils/formatters.py`**: Display formatting utilities
- `ModelDisplayInfo`: Model name formatting
- `TaskDisplayInfo`: Task name formatting
- `TrainingResultDisplayInfo`: Training result display names
## Workflow
1. Check `views/overview_page.py` for current patterns
2. Use #tool:search to find relevant code and data structures
3. Read data pipeline code if needed (read-only)
4. Leverage existing utilities from `utils/`
5. Use #tool:web to fetch documentation when needed
6. Implement changes following overview_page.py patterns
7. Use #tool:todo for multi-step tasks
## Refactoring Checklist
When updating pages to match new patterns:
1. Move to `views/` subdirectory
2. Use cached loaders from `utils/loaders.py` and `utils/stats.py`
3. Split into focused `render_*()` functions
4. Wrap interactive UI with `@st.fragment`
5. Replace hardcoded colors with `get_palette()`
6. Add type hints from `entropice.utils.types`
7. Organize with tabs for complex views
8. Use `width='stretch'` for charts/tables
9. Add comprehensive docstrings
10. Reference `overview_page.py` patterns
## Example Tasks
**✅ In Scope:**
- "Add feature correlation heatmap to overview page"
- "Create PyDeck map for RTS predictions"
- "Refactor training_data_page.py to match overview_page.py patterns"
- "Fix use_container_width deprecation warnings"
- "Add temporal statistics tab"
**⚠️ Out of Scope:**
- "Add new climate variable" → Requires changes to `era5.py`
- "Change training metrics" → Requires changes to `training.py`
- "Modify grid generation" → Requires changes to `grids.py`
## Key Reminders
- Only edit files in `dashboard/`
- Use `width='stretch'` not `use_container_width=True`
- Always reference `overview_page.py` for patterns
- Use #tool:web for documentation
- Use #tool:todo for complex multi-step work

31
.github/copilot-instructions.md vendored
View file

@ -63,6 +63,37 @@ DATA_DIR/
- `entropice.dashboard`: Streamlit visualization app
- `entropice.utils`: Paths, codecs, types
## Organisation of the Dashboard
- `entropice.dashboard.app`: Main Streamlit app, entry point, imports the pages
- `entropice.dashboard.pages`: Individual dashboard pages - functions which handle data loading and building each page, always called `XXX_page()`
- `entropice.dashboard.sections`: Reusable Streamlit sections - functions which build parts of pages, always called `render_XXX()`
- `entropice.dashboard.plots`: Plotting functions for the dashboard - functions which create plots, always called `create_XXX()` and return plotly figures
- `entropice.dashboard.utils`: Dashboard-specific utilities, also contain data loading functions
### Note on deprecated use_container_width parameter of streamlit functions
**❌ INCORRECT** (deprecated):
```python
st.plotly_chart(fig, use_container_width=True)
```
**✅ CORRECT** (current API):
```python
st.plotly_chart(fig, width='stretch')
```
**Common width values**:
- `width='stretch'` - Use full container width (replaces `use_container_width=True`)
- `width='content'` - Use content width (replaces `use_container_width=False`)
This applies to:
- `st.plotly_chart()`
- `st.altair_chart()`
- `st.vega_lite_chart()`
- `st.dataframe()`
- `st.image()`
## Testing & Notebooks
- Production code belongs in `src/entropice/`, not notebooks

13
src/entropice/dashboard/utils/stats.py
View file

@ -9,6 +9,7 @@ from dataclasses import asdict, dataclass
from typing import Literal
import pandas as pd
import streamlit as st
import xarray as xr
from stopuhr import stopwatch
@ -265,7 +266,7 @@ class DatasetStatistics:
)
# @st.cache_data # ty:ignore[invalid-argument-type]
@st.cache_data # ty:ignore[invalid-argument-type]
def load_all_default_dataset_statistics() -> dict[GridLevel, dict[TemporalMode, DatasetStatistics]]:
"""Precompute dataset statistics for all grid-level combinations and temporal modes."""
cache_file = entropice.utils.paths.get_dataset_stats_cache()
@ -309,6 +310,8 @@ def load_all_default_dataset_statistics() -> dict[GridLevel, dict[TemporalMode,
@dataclass(frozen=True)
class TrainingDatasetStatistics:
"""Statistics about the training dataset used for a specific training run."""
n_samples: int # Total number of samples in the dataset
n_features: int # Number of features
feature_names: list[str] # Names of all features
@ -341,6 +344,7 @@ class TrainingDatasetStatistics:
task: Task,
target: TargetDataset,
) -> "TrainingDatasetStatistics":
"""Compute training dataset statistics from a DatasetEnsemble and training settings."""
training_dataset = ensemble.create_training_set(task=task, target=target)
# Sample counts
@ -423,6 +427,8 @@ class TrainingDatasetStatistics:
@dataclass(frozen=True)
class CVMetricStatistics:
"""Cross-validation statistics for a specific metric."""
best_score: float
mean_score: float
std_score: float
@ -461,6 +467,8 @@ class CVMetricStatistics:
@dataclass(frozen=True)
class ParameterSpaceSummary:
"""Summary statistics for a hyperparameter in the search space."""
parameter: str
type: Literal["Numeric", "Categorical"]
min: float | None
@ -470,6 +478,7 @@ class ParameterSpaceSummary:
@classmethod
def compute(cls, result: TrainingResult, param_col: str) -> "ParameterSpaceSummary":
"""Get cross-validation statistics for a metric."""
param_name = param_col.replace("param_", "")
param_values = result.results[param_col].dropna()
@ -496,6 +505,8 @@ class ParameterSpaceSummary:
@dataclass(frozen=True)
class CVResultsStatistics:
"""Cross-validation results statistics for all metrics and parameters."""
metrics: dict[str, CVMetricStatistics]
parameter_summary: list[ParameterSpaceSummary]