wxylib/design
Design notes, surveys and staging plans for wxylib: Rust tooling that takes the modular concepts GDAL already contains, gives each one its own home, and builds an ecosystem of crates, bindings and documentation around them.
New to wxylib? Start at https://wxylib.org/docs/: what works, how-tos, explainers and how to help. These documents are rendered there too, under Design.
Nothing here is code. Each document ends in a gate: a decision to accept, revise or reject before the next document starts.
Principles
- Slow incremental steps that are productive, consistent, modular and extensible.
- Separate planning, reading and computing. A plan can be inspected, serialised and costed before any pixel is touched (the rwarp model).
- Leverage existing crates (object_store, async-tiff, zarrs, proj, proj4rs, vaster) where they fit. Build only what has no good home.
- GDAL is the inspiration, the test oracle and, for now, a workaround we may call, never the basis. If we reach for functionality before we have the foundation for it, we stop and build the foundation. Nothing we publish carries “gdal” in its name.
- Virtualisation is the default pathway for formats. A format we need gets a scanner that emits chunk references (byte ranges plus codecs) readable as virtual Zarr, rather than a reader of its own. This bridges to future format standardisation and defers legacy complexity; GDAL and other tools may serve as scanners where nothing else exists.
- Every module, design choice and stage carries a rationale: what we get, what it costs, and what happens if we don’t do it.
Plan
| Step | Document | Gate |
|---|---|---|
| 1 | Module survey | Agree the module map (passed 2026-10-16) |
| 2 | One data model, from raster to vector | Accept core types and traits (provisional) |
| 3 | Architecture and staged roadmap | Pick the first implementation stage (S1 chosen, provisional) |
| 4 | The landscape: related work, what to leverage, what to track | Upstream-first, Icechunk refs, browser CRS, conventions work (passed 2026-10-16) |
| 5 | Pain points in GDAL, R and Python | S4b mdim warp, curvilinear after S5, an extraction stage, no global state (passed 2026-10-16) |
Whether wxylib earns a long-term commitment is tested by four commit criteria, with their live status.
Which stages are built is kept in one place, doc 03’s stages section; what they add up to for a user is on the capability map.
Musings
Side notes, not plans and not gated. A musing becomes work only by being picked up into a numbered document. Some end in questions; where those have been answered, the status says so and the musing records the decisions.
| Musing | About | Status |
|---|---|---|
| m01: Downstream UX, blue sky | What front ends (a modular QGIS among them) could build from the core, and which doors the core keeps open for them | Open, no questions |
| m02: Pilot, a byte-reference table as a wxy-virtual source | The mdim-refs/0.2 table (OISST) through wxy-virtual into Icechunk, checked against GDAL |
Done: pilot built in core; walk-through on the docs site |
| m03: Presentation on raw data, and the map as a document | Image scaling that travels with untouched values, and a map that references its sources (doc 05, X1 and X2) | Open, no questions |
| m04: The loftier goals, reassessed | Doc 03 section 5 and m01 to m03 checked against what had landed by 2026-10-17 | Assessment; its gate 4 closure is superseded by m06 |
| m05: The chunk cache as a growing implicit store | Keeping the encoded bytes of every chunk a run reads beside the byte-reference rows, so a store accumulates without dictating a format | Decided 2026-10-17; pilot C1 built in core (wxy-cache); C2 replaced by S3c |
| m06: Response to the foundation assessment | An outside review of wxylib checked point by point; proposes the flat refs store with the cache inside, and a Bluelink-shaped fixture | Section 6 decided 2026-10-19; became stage S3c in doc 03 |
| m07: Index space and coordinate space | The playproj grids placed on the ladder, the gaps they show, and five small Zarr fixtures to pin them | Decided 2026-10-17; fixtures I1 and I3 built |
| m08: Point clouds, lidar and PDAL on the ladder | Point clouds as R6 on an R1 lattice with an R4 octree chunk scheme (COPC), the Rust crates, and a stance: ladder moves, PDAL as oracle, no processing toolkit | Decided 2026-10-18 (all six as recommended) |
| m09: The modules that don’t exist yet | The CLI as an operation registry, VSI through object_store, byte views and a chain grammar as the real gaps, a crate check before each stage, and GDAL’s other infrastructure | Open: five questions with recommended answers |
| m10: Overlapping COG mosaics, measured on REMA v2 and COP30 | Overlap, variable chunk sizes and padded tiles as three problems; tile-edge and origin cuts as duals; the predictor codec as the first blocker | Open, measured; requirements in section 8 |
| m11: Quiet operations | What composite raster readers do without saying so, a ladder from hidden to separated, and REMA v2, COP30 and Sentinel-2 as the fixtures that expose it | Open; COP30 source table in measurements/cop30 |
Reference specs
Specs from adjacent work that the design documents refer to. Not wxylib formats.
| Spec | About | Status |
|---|---|---|
| mdim-refs/0.2 | A byte-reference table built from a gdal mdim mosaic VRT (SQLite; Parquet and kerchunk exports) |
Draft; open questions on per-axis chunk edges and the Parquet export layout |
License
MIT OR Apache-2.0, matching rwarp and the crates we build on.