The Complete Modern GIS Roadmap

This is a post I have been meaning to write for a while now and it is probably going to be the most important thing I write this year.

For the past few years I have published a number of different versions of this graphic that showcases the Modern GIS Landscape, at least as I see it, and it always gets a lot of interest and responses. 

If you read nothing else from this email please read this: for over 40ish years the center of GIS was the GIS. You brought data to a GIS server or system and then it lived there, never needing to go outside of it.

That is over. I generally dislike highly declarative posts with headlines like “The end of…” but I actually think that is no longer the case and more use cases are pivoting towards the new modern GIS paradigm.

That being the data infrastructure is the center of geospatial data, and all systems including GIS revolve around that. 

Why: Esri writing Parquet, Iceberg’s native geometry type, spatial SQL in every major warehouse.

To be clear, if you are a traditional GIS user this isn’t a warning or a red flag. Your use cases are still valid and important, but the data you use is coming from somewhere else. 

It is creating new roles, jobs, and processes.

Problem is there is no roadmap for this at all. Not one. No guide. So this is my first attempt to build that.

But I realize that I haven’t ever written about this in great detail to actually explain how you can use this, how you can build with it, and most importantly what to do if you want to learn it. Now again, this represents the way I see the space but I am pulling this information from my day to day working with individuals, large organizations, teams using modern tools and those using traditional GIS, and the great people I get to learn and collaborate with daily.

So that is what this post is. It will be a long one so be prepared for that but I hope it provides a clear set of steps in terms of what to do.

If you are interested we are actually learning this over the course of 15 lessons in the Spatial Lab. You can learn more and become a member here: https://forrest.nyc/go/spatial-lab/

The structure here is broken into three different areas:

• The Data Foundations
• The Analytical Core
• The Delivery

Each layer has different parts that play a critical role in the modern GIS ecosystem. I also want to deliver this information in a few ways:

• For people learning on their laptops or computers, building locally
• For individuals and organizations doing this with larger, cloud resources
• For those using a traditional GIS ecosystem, how to leverage this in a hybrid way (and draw parallels to what you use today)

Last thing, no AI used here for ideas or for writing. I did use it to do a grammar check but made changes myself where needed. 

Okay no more delay let’s jump in. 

Part 1: The Data Foundations

The first step, and sometimes the one that can feel the most difficult is actually getting all the data you need in one place. Now that “place” might vary based on your set up but the problem is always the same. Everyone has data in different places. That could be from local files on your computer, a shared internal server with more files, an internal GIS server, public datasets online, APIs, cloud native files, public datasets behind a log in, or any other source.

Long story is that there is no single easy method to get your data in house, in one usable location, but that is the job to be done at the end of the day. Get all the data you need, into one place, in one unified format, updating at an interval you deem fit (one time, daily, hourly, etc).

That format can vary quite a bit and for a long time that meant your central GIS server that stored data, layers, and tiles. This isn’t that to be clear.

The way you need to think about this is that you are building your data foundations from the ground up. This isn’t the data you will use for your analysis. This is the landing zone. The carbon copy of the original data source in a format that works for you and your system.

We never really needed to think about this step much in the past. For the solo builder you just started by downloading files. For the enterprise a lot of this was stored in your GIS server. 

Problem is that neither of these approaches accounts for getting live data in, timing, dependency graphs (step B depends on step A). And making sure the system can scale is a whole other problem. So this stage focuses directly on that.

ETL the modern way

This first step is very simple. 

• Map every data source you need including source location and update frequency
• Create a process to do that
• Schedule and run it

In short:

• Data Source A is an API -> Script calls API and pulls results -> Saves it in a format I can use as Data Layer A
• Data Source B is from an open GIS portal -> Script calls the portal and downloads the file -> Saves it in a format I can use as a timestamped copy as Data Layer B
• Data Source C is from Source Cooperative (or other cloud native data sources) -> Script calls the portal and reads the data you need-> Saves it in a format I can use as Data Layer C

Repeat.

We will discuss what format it should be in when it lands in your data system next but for now we just need to move it.

The complicated part is the sheer number of choices you have to do this. It’s a lot and the landscape is complicated.

In a traditional GIS set up this means FME or ArcGIS Data Pipelines. These are great all in one tools and can handle the job of all three required steps (extract, transform, and load and even scheduling). They are all in one solutions but in terms of building it yourself with a modern stack we want to split it into three considerations:

1. Orchestrate: decide when things run and in what order
2. Move (Extract + Load): Get data from sources systems into your storage
3. Transform: Turn raw data into modeled tables

Transform steps are the actual engine that you run underneath all this so this will be discussed later. For now we are focusing on the Move pieces.

Those tools include but are not limited to:

• GDAL/ogr2ogr: This is the universal translator of geospatial files and it is critical. It is why I teach it first in the Modern GIS Accelerator. Most commonly written in shell/terminal scripts although there are Python bindings and it is inside every library (GeoPandas, DuckDB, etc.)
• dlt: Code first E+L all written in Python. Simple and easy when you need something more than just a shell script.
• Airbyte: Open source (with a cloud hosted option) GUI with hundreds of prebuilt source→destination connectors. Has dbt integration (more later) and sync scheduling 
• Fivetran: Managed connector service similar to Airbyte billed on rows

If you are starting out all you need is GDAL and script either in shell or Python to make this work. Use dlt when you need some extra support (comes with AI skills to build these faster) and go for it.

Here is the blueprint to build or prompt an AI coding agent:

• Describe the location of the data and how you get access to it
• Pick the tools (if not GDAL) and build out a script (Shell or Python + Shell for GDAL) 
• Give it credentials if it needs access
• Designate an output for the file type (spoiler alert that should be GeoParquet, Cloud-Optimized GeoTIFF or Zarr)
• Decide how this landing zone layer is organized (aka file system or cloud storage, create bucket naming rules, data retention – i.e. are you keeping the raw data forever, for 30 days, dumping it)
• Run the scripts

Now for an enterprise use case you probably want this running on the cloud and plenty of teams just run this same set up, some even on smaller processes using built in cloud tools like Cloud/Lambda functions and schedulers or they host dlt. 

You can go up to something like Airbyte or Fivetran as it brings a ton of other connectors along with it but that would be the main reason I would go there along with their scheduling and UIs.

• https://doc.arcgis.com/en/data-pipelines/latest/get-started/about-data-pipelines.htm
• https://dlthub.com/
• https://fme.safe.com/
• https://docs.airbyte.com/

Formats & Storage

Okay so this shouldn’t be much of a surprise but the formats you want to unify your data in should be cloud native formats. That means GeoParquet, Cloud-Optimized GeoTIFF, and Zarr.

A few things to note though:

• That doesn’t mean these are the formats that the data will end up in for their final state. We are still in pipeline mode and they are the most efficient to work with the tools as they move through: these are still raw ingredients.
• Our job from here is to make the final formats as efficient as possible for the upstream systems. If something is trying to get you to move it into their system faster, there is likely a reason they want you to do that.
• We have a few critical choices to make at this stage in terms of data unification and those will be outlined below.

What we are building now is the bronze layer of the medallion architecture or the multi-hop architecture. It is basically a carbon copy of your data. The next step you can either do inside the bronze layer or set up a silver layer to handle this (you can have more than one silver layer too). 

You can see my diagram of this here but also please watch this video as it does an excellent job of explaining this. 

Our first task is to unify into one CRS. You can handle transformations to correct CRS systems for final data products later so pick one that works or default to 4326 for ease but note you will need to transform for any distance calculations in a future silver layer although those operations are starting to drop in terms of computational complexity.

Second is that we need to organize and sort our data. This makes everything upstream more efficient. To be 100% clear here there are no hard written rules about this and it all depends on your data but the choices you need to make are here:

Geoparquet

• Sorting of data likely using Hilbert curve or some other method
• Ordering of data (Parquet uses empty space in between repeated values so this helps cut down on total space on the file)
• Partitioning of the data: usually spatial is the most optimal across the entire dataset but you can do it by date or some other field too
• Compression: ZSTD is good for most everything, Snappy is good for pure read-based analytics
• BBox column: yes, include it.
• Row groups: I think start with 100,000 to begin but adjust as needed depending on data complexity (i.e. points v. polygons)

https://sedona.apache.org/latest/api/sql/Optimizer/#push-spatial-predicates-to-geoparquet

https://wherobots.com/blog/spatial-data-geoparquet-and-apache-sedona/

https://dewey.dunnington.ca/post/2024/partitioning-strategies-for-bigger-than-memory-spatial-data

https://dewey.dunnington.ca/post/2024/wrangling-and-joining-130m-points-with-duckdb–the-open-source-spatial-stack

https://geomermaids.com/cookbook/geoparquet

Vector and Geoparquet are the data types I work with most so I happen to know the most about these

Cloud Optimized GeoTIFF (COG)

• Single or split COGs: either store everything in one giant GeoTIFF or split it by tile 
• Pick a tile/internal block size (usually 256×256 or 512×512) 
• Do you need to use overviews for visualization?
• Compression? Yes or no? Deflate/LZW, JPEG, or LERC
• Pick your NODATA values and projection
• Does the COG need to be web-optimized to view on a standard web map?

To be clear here are some things that we still don’t know yet (direct from the CNG Guide)

• The optimum size of data at which splitting across files improves performance as a multi-file dataset instead of a single file.
• When to recommend particular internal tile sizes
• Compression impacts on http transfer rates.
• Support for COG creation in all common geospatial tools varies.

Zarr

Zarr is basically, in my simple terms, a stack of arrays. Each array represents a date or range of data. The stack itself is the Zarr. They are most commonly used for weather and climate data. Much of this information is from the Cloud Native Geospatial (CNG) Guide as this is the best source for this right now.

• Chunking: the decision that matters most. Size for ~1MB+ uncompressed per chunk, and shape chunks (i.e. the spatial windows across all of time for zonal stats or small area for time series – a few cells across many time periods). Misaligned chunks mean every query fetches everything.
• Sharding (available in v3): use it. Packs many chunks into one object so big stores don’t become millions of tiny files.
• Compression: default ZSTD is solid. Blosc with lz4 + shuffle can be dramatically smaller on smooth numeric data but you need to test.
• Consolidated metadata: Generally a good idea because one request reads the whole tree instead of one per array, which is the difference on object storage.
• Version: Generally use V3

https://github.com/earth-mover/workshop-cng-2025-zarr

https://www.earthmover.io/blog/what-is-zarr

Right now these are all ways to make your data more efficient. By optimizing the foundation everything you do upstream will be more efficient. This is the core task here and shouldn’t be overlooked as the format alone solves a lot, but this is where you make it actually scalable.

The other thing to be clear is that these recommendations are still being developed and tested and discussed. And they may change in the future and you most importantly, need to test and figure out what works for your data.

Simply by adopting cloud native formats you will see a speed and storage size improvement in this layer, but these extra steps make it better.

Catalogs (or not)

While names and tools like Iceberg, Delta Lake, Duck Lake, Polaris, Icechunk, Unity, Glue, S3 Tables, STAC, stac-geoparquet, and more can seem extremely overwhelming, but it is actually something we have had for a long time.

The closest comparison in GIS is either an Esri Server or an open data portal. It is an organizing layer on top of raw data. That’s it!

Now the tricky part is catalogs come with two parts: the table system and the catalog itself. The two work together to do specific things.

A table format (Iceberg, Delta) is a metadata spec that turns a pile of Parquet files into a real table with schema, snapshots, and time travel. 

A catalog is the service that tracks those tables by name, holds the pointer to each table’s current state, and controls access. 

Files hold data, the table format makes them a table, the catalog tells everyone where the tables are. Here is a breakdown of these different tools:

Tool	What it is	How it relates
Parquet / GeoParquet	File format	The actual data on disk. Everything below organizes these files.
Apache Iceberg	Table format	Spec that makes files a table: schema evolution, snapshots, native geometry type. Needs a catalog to be findable.
Delta Lake	Table format	Iceberg’s Databricks-world equivalent.
DuckLake	Table format + catalog in one	Metadata lives in a SQL database (SQLite/Postgres), data in Parquet. Both jobs, one afternoon. The below-Spark-scale answer.
Icechunk	Table format, for arrays.	What Iceberg does for tables, Icechunk does for Zarr: git-like versioning, safe concurrent commits, and virtual references to existing NetCDF/HDF5 chunks. The raster/cube side of the same idea.
Lakekeeper / Polaris	Catalog (open, Iceberg REST)	Serve Iceberg tables to many engines safely. Self-hosted or managed.
Unity / Glue / S3 Tables / Horizon	Catalog (vendor)	Same job, attached to your cloud or warehouse. Use the one you already pay for.
STAC / stac-geoparquet	Discovery catalog	Different job entirely: helps humans find assets by space and time. No transactions, no schema management. Pairs with, never replaces, the above.

You will see that STAC is a little different. It helps you discover data more than anything else rather than keep track of the data history or help make them data formats. This acts much like a public data catalog that you would find from ArcGIS Data Hubs. 

If you are doing a one off project you likely don’t need a full blown catalog although some of them come with additional optimizations on the tables such as Iceberg. But it can be overkill if you are doing a one off project.

For local you can use DuckLake for an easy set up or Iceberg via PyIceberg and Icechunk if you need Zarr support. 

The majority of the use cases for this will be with enterprise support where you will likely use a managed service at this layer that also, likely bundles in compute too. Think Wherobots, Databricks, Snowflake, Earthmover, etc. You can set this up yourself but having it in a managed form removes a lot of headaches.

https://wherobots.com/blog/apache-iceberg-spatial-data/

Orchestration

Now that we have our data foundations we need some way to make it all work together and that means orchestration. In short that means deciding when things run, in what order, and remember what happened.

So it has nothing to do with extract and load and nothing to do with transformation. These are simply tools that help you manage the order, timing, and dependency of tasks.

Our GIS reference for this is Model Builder. It handles order of operations and you can schedule tasks in it. But the entire thing is still inside the GIS system at the end of the day so you can choose your compute or other steps.

The tools below do allow you to do that and use multiple steps. You can swap out SQL engines, use cloud and local, switch to a Python library or shell script and more. They also generally include alerts for failures, retry logic, and features that allow you to look at sources and see if there are updates and run if there is an update before a scheduled run.

They all, again generally, use what is known as a directed acyclic graph or DAG which looks like this:

These are steps that flow in one direction that are connected in various ways. Fancy term for a simple concept. Again there are a lot of choices here and I have a full chart of them below but you can go simple to complex but let’s start with what they are.

Tool	Where it sits / how it works	Strengths	Weaknesses	Geospatial
cron	Rung one. A time trigger on a machine you own runs a command. No state, no retries, no UI.	Zero setup, zero cost, universally available. Correct for one script on one server.	No state, no alerting, no dependencies, dies with the machine, UTC-vs-local bites everyone once.	None, and doesn’t need any. It just runs your spatial script.
GitHub Actions	Rung two. A workflow YAML next to your code: cron trigger + hosted runner + run history + secrets + failure emails.	Versioned with the repo, free at small scale, logs and alerting built in, zero infrastructure. A legitimate permanent home for nightly pipelines.	6-hour job limit, modest runners (RAM caps big spatial jobs), no backfills or data-aware dependencies, clunky cross-workflow coordination.	None natively. Runners can install GDAL/DuckDB/SedonaDB fine; large rasters can exceed runner disk/RAM.
Apache Airflow	Rung three, and the incumbent. Python DAGs of tasks; a scheduler service + workers + metadata DB you run (or buy managed: MWAA, Composer, Astronomer). Task-oriented: schedules verbs.	Massive ecosystem of providers, battle-tested at enterprise scale, every data engineer knows it, Airflow 3 modernized the UI and versioning.	Heavyweight to run and upgrade, task-centric model knows nothing about your data, local dev is painful, teams routinely deploy it years before they need it.	None. Spatial work happens inside tasks you write; Airflow neither knows nor cares about geometry.
Prefect	Rung three, the lighter Pythonic challenger. Decorate Python functions into flows; run anywhere; state and UI via server or Prefect Cloud.	Much lower friction than Airflow, great local dev story, dynamic runtime DAGs, clean retries/caching.	Smaller ecosystem and community, fewer prebuilt integrations.	None. Same as Airflow: your code does the spatial work.
Dagster	Rung three, the asset-oriented one. You declare data assets (each medallion table = an asset) and how each is computed. Dagster derives the DAG, schedules, partitions, backfills. Schedules nouns, not verbs.	The asset model maps 1:1 to bronze/silver/gold, lineage graph for free, first-class backfills and partitions, tight dbt integration, strong local dev, generous OSS tier.	Steeper conceptual learning curve, asset thinking is overkill for one or two scripts, smaller talent pool than Airflow.	None built in, but the asset model fits spatial pipelines unusually well (each GeoParquet layer/table is a natural asset).
Kestra	Rung three, declarative flavor. YAML-defined flows, event and time triggers, plugin ecosystem, runs as a JVM service.	Language-agnostic (good for mixed shell/SQL/Python teams), clean UI, growing fast.	Younger ecosystem, YAML gets unwieldy for complex logic, smaller community than any of the big three.	None.
Temporal	Different category, often confused in. Durable execution for application workflows: long-running, stateful processes that must survive crashes (payments, provisioning).	Bulletproof state machines, exactly-once semantics.	Wrong tool for batch analytics; you’d be building a data platform from primitives.	None, and not its job. Mentioned only to keep it off your shortlist.

Okay so what should you pick and how. Basically there are three rungs to focus on here:

1. Start with chron. It’s on every computer and your computer just needs to be open and running (or a cloud compute service) and you build the logic into your pipeline script. Simple and effective especially for small things.
2. Move to GitHub Actions. For small simple jobs it’s free and it works plus you get a lot of functionality out of the box. Not for production use cases but for personal small things it does the trick and you learn a solid skill.
3. If you get past this you need to pick a system. For starters I recommend Prefect as it is pure Python and the set up headache is near nothing. Dagster is also a good option but has some specifics about how assets are defined which can be good depending on your mental model. However, Airflow is still the best. It can be overkill for one off or small projects but it is the gold standard and good for any big team, works up to enterprise scale as well.

These help you keep everything in order. Last bit in this layer coming up.

Data contracts and governance

Data contracts and governance cover two specific things in my mind: the technical validity of the data as well as the promise you make to the end user of the data.

Let’s start with the technical. Data governance tools serve a single purpose which is to check if you data is valid as you designed it to be, up to date, and correct. They do this by checking non spatial things like the data schema (i.e. are the columns and data types for the columns as expected) and the data freshness (i.e. if you promise that the data is updated daily that it is less than 24 hours out of date).

The spatial components of the data contracts are more specific and need to be customized for any of these. What would you want to check? 

• CRS validity and expected CRS
• Geometry validity
• Extent of data bounds
• Band profiles

There are, again, lots of tools to handle this but I really recommend two for most cases. For the raw data to bronze step I recommend pandera. It is pure Python and has a Geopandas engine to check everything we listed below. The others are datacontract-cli which uses YAML and requires more customization. Soda and Great Expectations are cloud tools that have more integrations with things like Slack and Email if you need, but again you need to customize for spatial.

Once it is past that stage there is dbt (which we will use again later) and checks can be written in SQL. Now there are enterprise options which are cloud hosted but dbt has a cloud as well and pandora is just Python but these are there for you to see regardless.

The other side of the coin is the promise you make of the data. All those things listed above: CLI, valid geoms, refresh rate, data extent, all should be documented and written down so people know what to expect from the data at each stage. It is this human element that is critical to share and distribute to both your internal stakeholders and external users.

https://pandera.readthedocs.io/en/stable

https://docs.getdbt.com

Part 2: The Analytical Core (processing, analytics, scalable data)

Now to be clear, we still haven’t done any processing or moved past landing raw data in our bronze layer. At this point we have just designed the system and only the early stages (phew, still a lot of work). Now I know it might not seem worth all the work. 

Why do all this work when you can download a bunch of files, dump them into your GIS, and start analyzing in a desktop GIS, Python, or other system?

Here we are not designing an analysis, we are building an autonomous system that can grow and scale over time. The end outcome is this entire system will run on its own and you will be alerted only when something goes wrong. When designed well it can scale, run, and get updated data every single hour of the data at the lowest possible cost without any additional effort from you and your stakeholders will all be served as they are needed.

That is the promise from modern data engineering systems. You will get this data, in this format, at this frequency. Column names and data types won’t change and if they do you will know about it. And each layer is optimized for speed and consistency. Each data user (or mart as we will call them) is served with the exact right data they need, nothing more or less. 

This is what this section is all about. Designing the system then letting it run.

Designing the data layer

So we first need to talk about the data layers that we have: bronze, silver, and gold. I will also call them raw, staging, warehouse. Literally this is the best video on the internet about this so please do watch: https://www.youtube.com/watch?v=8p77fOWp5F4

Okay so what happens in each layer and what do you need to do.

Bronze (Raw)

As we stated earlier this layer just takes the data that you have built ETL pipelines for and turns it into an optimized CNG format. The ONLY other things you should do here is organize and name your data. We won’t even transform it here (although you can if you so choose) but we aren’t optimizing for engines at all so in most cases we just want to convert and land the data.

What you are starting to see take shape is the separation of concerns here. Each step in the process has exactly one specific concern or thing to focus on. That allows us to scale up and down as we need and select the right engine for the job as we move forward.

Silver (Staging)

Now conventional wisdom would tell use to have a single layer here for all the spatial processes but I am going to challenge that a bit and have you consider adding multiple layers here, or at least two in total.

The silver layer is where your heavy processing will live. These are the big spatial tasks: joins, zonal stats, distance measurements and more. But you also need to have your spatial optimization here too. That is your transformation unification and ordering/optimizing your data to improve those processes.

So in your first silver warehouse you should take each layer and convert to your desired CRS, drop the old geometry, and then organize and optimize the files as you need based on the parameters about. For GeoParquet that means ordering, partitioning, etc. COG that means deciding on the things like overviews, splitting or not, etc. 

Then you can have one (or more) silver warehouses to do the spatial processing. If you need to combine vector data with raster or climate data you will probably need at least 2 more since those engines will require something outside of a pure vector engine.

Then in the final step you want to prepare your data outputs for their final form. This means one final query structure and one final format.

Gold (Warehouse)

In most warehouses the gold layer literally is just a query to structure the data for the final consumer (that could be a person or a technology) with the right time window, filters, columns, and data types to make it most efficient for that system.

That is part 1. But for spatial data that also means outputting different formats for the correct system. For some that might mean leaving it in GeoParqet or raster formats. But for something like a GIS that might mean connecting it to a server, or turning it into a Shapefile (yes, people still use those). For a web app that might mean making tiles. You get the picture.

Literally that is it in this layer. A warehouse or mart for each consumer with the right data structure and format so that, as much as possible, that layer doesn’t need to requery or process data. The data itself should serve the use case and reduce compute to as close to zero as possible. Or put another way a GIS web app that has to requery data constantly, call to a server, regenerate tiles, and then render the result is a big no-no in this paradigm.

Thinking like a data engineer

Now there are a million ways to think like a data engineer and endless tutorials you could watch and use to get this. But the simple rules I try to keep in mind are these:

• Keep operations separate: don’t compress operations into different tools and engines where they don’t belong
• Design for the user: design for the person using the data or the tool consuming it not just the data that needs to come out
• Remember you are making a promise: if that data breaks decisions break
• Preserve compute: anything done with your data should be fast at the final layer

But there are many other rules as well. Here are a few I have collected and tried to make as easy to consume as possible:

• If you run any step twice, get the same state. The test is literal: run it twice, compare row counts (this is called idempotency)
• Incremental models: process only new or changed rows, keyed on something like snapshot date
• Backfills: when logic changes, history is stale by definition. This only works if you respect the above rules.
• Dev/prod separation: a dev target you can break and a prod target you can’t. 
• The code is the system. Not a software.
• Master data: one record per real-world thing (one parcel, one building), with written (yes written) rules for which source wins each field.
• Use data types smartly: aka use arrays for time series: 30 years of monthly climate = 1 row with a 360-element array, not 360 rows. 

Now the tools here are usually dbt or SQLMesh which is emerging. This is because all your data should be in a SQL system at this point and this allows you to chain together SQL logic and test steps as you process them through. 

I wanted to point out one thing on those data arrays. If your upstream data consumer doesn’t need data in a raster or Zarr format but does need data from those in a vector format this can be a powerful option. Imagine you have a table of points or polygons and you did some zonal stats on 30 days of climate data. Just compress that into an ordered array. The upstream query needs to be structured correctly but as long as that is well documented you should be good to go.

There are a ton of other concepts you can apply here but if you try and embed these rules into what you are doing and think strategically here you should be in good shape. Also watch these videos 

Spatial processing at scale

Now we get into a topic that is one of the most important one I think. There are a ton of engines you can choose to process your data. Between SQL and Python tools, cloud and local, there are too many choices and the BIGGEST mistake I see anyone make is choosing ONE AND ONLY ONE of these. 

It is like trying to find one hammer to build an entire house. You wouldn’t do it, you would take multiple tools to a job site. Let’s start with the fact that we have a plethora of choices each with their strengths and weaknesses and we get to choose the best ones. That is a good problem to have.

Now generally these fall into these categories: at least in my mind:

• Databases: generally called OLTP or transactional databases these help store and keep records for fast look up and retrieval. This is PostGIS at the core and it is amazing at that. It has been a stand in for processing for a long time just because it is so feature dense. It has a logical limit of how much data you want to push through it because the data has to live inside the database so keep that in mind but if there is a feature or function there that you can’t find anywhere else, then you might need it. And if your workflow requires referencing and looking up specific records, run don’t walk to use PostGIS.
• Processing: these tools are for data pipelines pure and simple. They scale up to the level you need to run big batch jobs at any scale. They aren’t for record lookup or on demand analytics (because they have a start time associated with them) but if you have heavy data to move and process they work. This is SedonaDB (locally) or Apache Spark + Apache Sedona or Wherobots in the cloud.
  • Now a note about raster and array or climate data. These things generally are done with similar underlying compute infrastructure in the past splitting a job up and running them in the cloud or in a high performance compute environment. They generally require optimizing CPU or sometimes GPUs as well. Increasingly there is a movement to move these formats to easier to use things faster, namely Zarr (a time series of raster data is way more efficiently stored as a Zarr) and also integrating into the wider distributed computing environment but we just aren’t there yet, at least in an easy way. It is here you will see systems like Dask or Ray used to scale up and distribute basic Python workflows which is still what many of these things use (i.e xarray and Rasterio)
• Analytics: these are known as OLAP systems as they process analytical workloads which means a ton of computing power to answer ad-hoc questions on big data. Now they get confused with processing because they are so powerful but they work on a single question and their cost, both on your computer and on the cloud is billed by query (or question) not scale and time which you can control for in the processing scenario. They are perfect at asking and querying data but multi step processing shouldn’t be done here. Prepare the data for these systems first (more below) and then use them to reduce cost upstream.

Now what tool you use for each isn’t particularly relevant but here are my recommendations:

1. Use PostGIS for your transactional database. Doesn’t matter if that is local, cloud, or what tool just use it.
2. For processing, specifically spatial processing use SedonaDB locally (it is getting better at multimodal) or Apache Sedona and then Spark/Sedona or Wherobots in the cloud. You can predict what you will spend and manage scale easily.
3. For analytics use DuckDB on well structured data and simple spatial queries and then you can use a cloud service like MotherDuck or any of the other data warehouses should you want to use them.

I have strong opinions here so please do share if you think otherwise but I can share some evidence to back those up if needed.

Raster and climate data

Okay I alluded to this before but this is, in my opinion, the single biggest issue and blind spot in geospatial right now. 

Vector, raster, and climate/array based formats all kind of live in their own worlds. What I mean by this is that they have systems that they work very well inside of when just analyzing that data type but when you need to combine them, say join polygons to raster or climate data, that is where things get dicey. 

Vector to raster zonal stats is getting better. In Python you can use GeoPandas plus Rasterio or a package called RasterStats to do this. But I think SedonaDB and Apache Sedona or even PostGIS are much better at doing this over many different polygons. I even scaled it up to calculate the mean elevation for all 2.1 billion (at the time) buildings in the Overture Maps dataset if you want to check that out. 

Climate data in NetCDF or Zarr basically has a one way path which is either to vectorize your data and do a vector to vector join or move the vector data into a Zarr or NextCDF compatible tool like xarray or Xvec which is outlined in this post by EarthMover. 

Apache Sedona has a compromise approach which I think makes sense which is to have NetCDF be a reader and read that file into a format that treats it like a raster which we already know works well. You don’t change your format and with smart data engineering principles you can run a daily job to keep updating the data and move it into usable upstream gold layers. 

Not to mention I am not covering point clouds, LiDAR, SAR, or any other types here.

True spatial multimodal analysis is a blind spot and one that has a ton of opportunity in it. We still only have a small number of teams running true spatial processing in any meaningful way but this is something to fix.

To say it clearly now this is a problem and over my career I have seen a lot of band-aid solutions to solving this. My recommendation?

Use Apache Sedona or SedonaDB, or you will need to have a format switch at some point. But as this is a changing area I am open to ideas here too.

https://www.earthmover.io/blog/vector-datacube-pt1

One last note on silver and gold

I said this earlier but I think it bears repeating. 

Silver is for spatial. Have multiple steps with different engines to handle the scale as you need. Don’t try and pack it all in. Get it ready to go into gold.

Gold should be one final step away from ready to use data for your final consumer. You may be doing some extra advanced work here even like running ML models or predictive analysis in TorchGeo too, definitely spatial analytics like Getis Ord GI* or spatial data science techniques as needed, and feature engineering for pure ML upstream too. 

But the big one is formats, data types, structure, and queries that run super fast. Minimal heavy lifting here the data should be in a one query away step from the final consumer and if it isn’t you should consider adding more steps in silver to get it there.

Okay so now that we have that, what are these actual landing zones and how do I design data for them.

Part 3: Delivery (outcome design)

This is the final step and the key concept to keep in mind here is that one size does not fit all. We have the ability to design datasets and types for the different tools consuming them from one single system. And if you have worked long enough in GIS or any data field you know that the data used in a GIS system needs to be different than the data you deliver for a web application which is different from what you need to deliver for a data scientist. 

You can make exactly what they need! All from this one system. And you should. 

The concept name you want to remember here is the data mart. Yes like the store. Each store or mart has a specific purpose: to sell (aka provide) data to a specific customer. Now you need to know your customer in detail to design that for them and also how to get that data to them.

For someone like a GIS system, they likely want files, or better yet the data wired directly to their GIS server. That could be PostGIS or something else. But you don’t want to send it as files that push tons of data over the wire and there are choices of tools that can read your files and move them directly to tables.

The most important rules here is that your data should be:

• Fast for the system or customer to consume, read, render, and overall use – it is purpose built for the specific use case and system it’s going in to
• It is documented. It says exactly what the data is, when it is updated, what it can do, and how it is structured including the data lineage. Type this out and share it with the users.
• It doesn’t require processing! If it is going into a system that queries it that is fine like an analytics warehouse but it is built to run fast with the questions that user will be asking. 
• It is in the right format. GeoParquet isn’t right for everything (yet)
• It has the right data types and columns (only what is needed) for the final system it is going into.
• It has a direct and fast, reproducible path to connect it to the system that requires NO ADDITIONAL PROCESSING just connecting.

Okay so let’s show how to design this for some real systems.

Dashboards

I feel like dashboards got misused and badly positioned in GIS for may years now. The way we use them is to show data and let people ask questions of them. Explore, dive deep, and ultimately that leads them to want to reprocess or query the data deeper. 

While that is fine it creates a full loop problem where the dashboard has to call back to the source data, requery it, and then send it back to the dashboard to re-redner. This is time and compute consuming and a bad use of dashboards.

The original purpose of a BI dashboard is to show one set of data, one snapshot in time, one specific view, and allow people to filter that specific view to see what’s going on and see specific phenomenon, but not have dynamic queries that call back to the database. 

Look at any BI tool that’s out there, Tableau, Power BI, Looker, any of them. Their sole purpose is to query a set of data that shows a specific view of time. That time may be the past hour or it may be updating every day, but it’s that specific snapshot. And every filter allows you to filter that data set. It does not query or re-query the underlying data. 

And there’s a reason for this. It takes time and compute and resources to do this. It creates this dynamic loop and it’s very difficult to architect. In fact, it’s really only something that’s useful if you’re building a custom application. 

So what should we do with dashboards? Well they are powerful tools, but the first thing before building a dashboard is understanding the specific information and audience needs. And if there’s many different use cases with many different things that need to be seen, have many different dashboards. Don’t try to cram it all into one. 

One dashboard, one purpose, one message that’s being conveyed clearly and consistently. 

So the data you share should match that. It can give a specific snapshot. And as long as your underlying data system is updating that on a frequent basis, either one table that has the most frequent data, or you’re giving a time range or a window back for the specific hour or day that the data that you’re looking at, that is what the dashboard should do. 

So what do you need to do here?

• Define the specific thing that the user or user group needs to see
• How often does the data need to be updated?
• What is the time window they need to see?
• What ways do they want to filter the data?
• What are the best filters to use?
• What are the fewest filters you can use to convey that message?
• What is the single most important message the cartography needs to send?
• What are the fewest number of layers on the map you need to send that message (1 is the best case)?
• What are the fewest number of data columns you need to deliver that data and what data types are needed?
• What is the fastest way to render that data on the map without overwhelming the user?

Once you do this you can basically build a single query to connect to that final data source and build your gold table. You can connect directly to the file or use a connector that many of these tools have. You can load that into a database if it is small enough or use a data warehouse. The BI tools have nearly endless connectors so this should be easy.

The entire data query and final dataset should be fast to render on the map, have the lowest data overhead as possible, and the only the fields you need to actually complete and build out the dashboard. 

That’s the whole point of this layer. We shouldn’t get lost in trying to create complete queryable explorable solutions. Just many specific solutions that solve a very specific time and place problem. 

Applications

Web applications allow you to do more of the customization and even re-query data as needed. They allow you to build anything that you want and they allow for deeper exploration, but again they should have one clear focus and clear use case when they’re being built. 

The problem with web applications is that there’s a whole different set of problems that you have. Imagine that you’re serving that data set to people around the globe. How do you get that data moved over most efficiently without sending megabytes or, in a worse case, gigabytes of data over the wire to that application each time. 

The whole purpose of a web application is two things, customizations and fast data access. For the map data to quickly access that, you generally want to compress that into map tiles. Now in the GIS world we generally have done that through WMS layers where there are pre-prepared layers with styles that are put on a server and then that server is connected to the web application. 

In the 2010s, we updated that by using map tiles and compressing that data into tiles. That was originally done with raster tiles and then with vector tiles. Problem is you still need a server to serve up that data. 

But now in the cloud native world we have PM Tiles, which allows you to create a single file layer that’s compressed tiles and allow you to send that over the wire with HTTPS protocols. 

But that tile layer is a capture in time. You would have to regenerate those tiles anytime you need to show fresh or new data. Let’s say you need to show dynamic data where that data is being filtered or updated by a backend query system. You need to send that dynamically over the web, and retiling that each time is a painful process to build an architect. 

There are solutions for this. PostGIS has some built-in tools with tiling like pg_tileserv and ST_AsMVT that allows you to dynamically render tiles from a PostGIS query, which can be done, and I’ve seen people be successful with it. Generally if you need to do this, this is the approach I recommend. 

But your end goal should be to serve a tile layer that’s pre-prepared to an application that doesn’t need to be updated by a dynamic query. Tiles can be filtered and you can put a lot of data into the tiles to allow you to do this. But as much as you can, this is the best way to serve an application use case. 

Now you can also serve peer files, so you can return data as a GeoJSON or in the cloud native world that can be run as a FlatGeoBuff file. Those files themselves can be queried and processed and the Flat Geobuff can actually connect into a DuckDB database that’s running in WASM, aka in the browser. 

So there are multiple ways to do this, but again, the gold standard is saying that your data is so good that it can be put into tiles, and those tiles are updated maybe every day or every hour if you really need them to. But every use case in that application is served from that tile layer. It’s gonna render fast and efficiently every single time. 

So what sho should you strive to do here in order of most efficient? 

• Build tiles that can be served over HTTPS with PM tiles or any other tile service that you want. Mapbox has great options for this with applications, but you can use many different things. 
• If you need dynamic querying, PostGS provides a great option and you can serve this with the built-in tiling functions in PostGS and a tile service that will render that data. 
• If you want to do this on a little bit of a lower scale, you can use DuckDB WASM along with many different geospatial file types that will then re-render and re-query that data, all happening in the client side, no server needed. 
• Worst case, you’re sending big files over the web to the application. Not ideal, but it works and it can be done. 

The application itself is yours to create. It’s all built in a front-end service, however you want to construct that, and you can build filters on top of any of these things. You can allow it to submit data, push data back to a database, all these different things. It’s basically your playground to build exactly what you need and what you want to do. 

This works for web and mobile applications. Both the same setups work across each of these, although the programming languages to build them will look a little bit different and you will need to check some of the tiling protocols with mobile and things like that. 

So this layer is both about the query output to the actual dataset that’s pushing into the application. Make sure it’s fit for that use case. Again, it’s very similar to dashboards and shows exactly what it needs. 

But format is probably the most important part as you can see here. Saying how you send that data to the application and what format it needs to be in, there’s extra steps here and that’s pre-tiling step pushing those tiles to a specific bucket that can serve that web application is the data engineering piece that you need to consider here. A web app can’t just render a bunch of Parquet files in a snap. 

Feeding GIS

Now GIS systems as you know generally require data in specific formats and that is your primary job here. Now if you’re serving public systems, you need to make those datasets accessible in file downloads in multiple formats. 

Think of any ArcGIS hub page or any open data portal that you’ve encountered before. You have options to connect to that data and download it via API, but most people go in and grab it as a Shapefile or a GeoJSON or a CSV. That’s one consideration, and you may need to use a service to share that data. However, there are cloud native options that are coming out as well to basically allow a similar data access protocol for base files stored in Parquet. 

Hint hint: https://www.portolan-sdi.org/#demo

But for internal stakeholders it really boils down to three questions:

• Are they using data directly from a file service and accessing files themselves?
• Or are they accessing it from a catalog system like ArcGIS Server?
• Or are they accessing that data from a database like PostGIS? 

The good news is it’s easy to move parquet files into any of these systems. You want to structure the files in a very specific way so that the GIS layer answers that specific question or can be used in any upstream analysis. 

When you’re building boundary files, you don’t need to have multiple boundaries over time. You want to have one boundary layer that tells you one thing and allows a GS analyst to do their job upstream. 

The lesson here is don’t try to write analytical layers for that GIS system. Have that specific data that has a specific view and job to do upstream. GIS works in layers and it’s designed to work that way. Create layers, not data sets that are updating time over time and things like that. 

Then basically once you know the exact data that needs to be brought in, move it into that system, make it ready in the proper projection, and then build it into one of those three formats. 

That allows that person upstream to do exactly what they need to do in their sp system and move on with what they need to accomplish. These aren’t big massive layers, they’re basically GIS consumable layers that allow you to work inside a desktop GIS. Think of it that way. Don’t overcomplicate it. 

One last note is that partition data doesn’t work well in a GIS. Make one layer, please. 

Spatial data science and ML

Now spatial data science and machine learning workflows require data sets that have some additional data that can be processed, but effectively there will be some extra exploratory steps that people are doing upstream. 

What this means is that they may want access to a bigger time window of data. They may want to slice that in different ways, but effectively your job is to remove the headache from that person of creating features. Give them the features they actually need, and then allow Python, which is where they’ll be running either their machine learning workflow or their spatial data science workflow, to do its job. 

Python itself is not a scalable system. It is a scripting system that allows you to do different processes. For example, in spatial data science, that might be computing the Moran’s I spatial autocorrelation statistic. In a machine learning workflow that might be running XGBoost over pre-prepared features to do some mean machine learning predictive workflows. 

Or in more advanced use case, that might be using pre-prepared roster data to run through a deep learning Python learning pipeline in PyTorch. Or that might be sending prepared climatology data to a forecasting system. 

In any case here, the data formats matter the most to understand what that data is needed and the system that it’s running in. The good news is that most systems can run Parquet or even connect into catalogs. 

The most important thing is to make sure that the data is ready to consume in any pipeline that is needed.

For things like spatial autocorrelation or other spatial data science functions, consider what the data shape needs to be to work with things like PySAL. That may be computing things like an array of neighbor IDs or weights that are associated with those different things. Don’t make the upstream system compute that. 

For things like machine learning, you need to have the features engineered and ready to use in those different systems. They shouldn’t have to compute other features. Again, don’t make the upstream system compute that. 

Data format is less of a consideration here. You need to know what it is and what they require to see that in, that may be running it through a warehouse which connects into Python so they can query that dynamically in a specific area or something else or just access to the raw files. But most importantly is the computation of the features that are required by those systems. 

Through all of this, you can start to see a pattern. Understand exactly what the users need, build those features so that no one else has to again. 

AI integration

This is the final step that hasn’t been clearly defined yet. How do you connect your spatial data into an AI enabled system to help it query, analyze and use that data and whatever it’s doing with that? 

I think the first step is to find how you want to use AI, and that’s not really a data engineering problem, it’s more of a thought exercise. You need to decide if you want AI to be simply accessing and using the data and retrieving it for you, or if you wanted to retrieve it and do something additional with it, create some sort of thought around it, analyze it, some opinionated report, whatever that might be. 

As it stands today, my recommendation is to use it to retrieve your data, share it with you, and then do some sort of small scale spatial work with that data. That’s the best use of AI today is to actually connect it and allow it to retrieve that. 

I’ve said this before and I’ll say it again, these are language models, and they are not good at reading spatial distances. They are not compute systems on their own. So you need to connect into something that can do that. 

That either means a spatial SQL system or an MCP that has access to one. These language models are incredibly good at writing code and given the right structure and guidelines and skills that tell it exactly how you want it to write that and the documentation that you should be preparing for these final layers as well, it can do that really efficiently. 

What you shouldn’t do is hand off the spatial logic to that. You can write that in an agentec workflow to say do step A, then step B, then step C, but don’t have it make decisions for you. That’s not what it’s meant to do. It’s meant to retrieve data, present it, and do that through natural language 

Now the thing that is starting to enter into the discussion is leveraging additional context along with that. You can add in something known as a spatial knowledge graph. That graph itself contains nodes which understand relationships between things, like how related is a gas station, a car wash, and a repair center. 

Those nodes can also embed spatial relationships such as distances, proximity, and other basic things that can help it understand that from an AI perspective. That can help you answer those different questions that have spatial components embedded in a graph which the language model can understand and use. 

That is a completely different output. Or as we have been discussing it, a completely different format. You can build these and do that with any of the tools we’ve listed upstream. It’s just another output and another format that you can provide through this process. 

I recommend taking a look at how Portolan has done this with their cataloging system. They have specific skills that give very clear instructions on how to look at and understand those catalogs and access the data inside of them. 

https://github.com/portolan-sdi/portolan-skills

But this is evolving. I really think as it stands today, your best use is to do something similar to this. Build highly structured data sets. Give an AI skills, rules, definitions, data schemas, and ability to access and understand that data, have it retrieve it for you, but not do the analysis itself unless you’ve given it a very clear, hard-coded and structured analysis that it should be running. It can simply present the result. It should not interpret them for you. The AI shouldn’t do the thinking for you, but it can do the data retrieval and prep from something that you’ve structured well. 

And the good news is SQL can do this for you and build all that out. With this infrastructure you are already AI ready. 

Conclusion

This whole paradigm breaks a lot of what we’ve done traditionally in GIS. The entire process here is meant to focus on and structure the data in very clear ways. It helps solve problems of multimodality that we really haven’t addressed yet, how to blend these different datasets together, and create usable outputs. And it focuses on creating a clear set of deliverables for clear solutions. 

It doesn’t replace the work. I’m not saying that it’s gonna remove GIS Analysts completely from their job roles or it’s going to remove the need for a spatial data scientist. In fact, it will actually probably increase the ability to do those more. 

By focusing here, we can actually have frequently updated data that can move through pipelines efficiently and all these other roles can be better served because we know the data is going to be up to date. Has a very clear contract around it on when it should be updated and the frequency in that. 

The fact of the matter is that there is a very small circle and pool of people that can do this today. We need to create education paths and plans to help people do this and understand how they can do it and implement it in their organizations. 

More importantly, we shouldn’t be asking roles that do specific things like GIS analysis or spatial data science or AI development to do the data engineering work. Organizations in the tech industry have figured this out for a long time, yet GIS somehow has not come to this realization. We think we need to cram all the data and the entire process into one single tool, which just doesn’t make sense. 

So I’m proposing we actually increase and have more spatial roles. We have people do and specialize in geospatial data engineering to serve the upstream roles that are doing this type of work. And this is the basic process and the basic premise to do this. There’s the tools you need and the systems you need and all the different pieces here. I’ve tried to lay that out in as clear of different steps and the things you should focus in on as possible. 

And you can learn this locally, there’s nothing stopping you from spinning up any of these tools and testing this out in a small system on your local machine. 

In fact this is exactly what we’ve done and are doing in the Spatial Lab depending on when you’re reading this. Over a series of 15 lessons, we’re going through each of these steps and building this out for individuals’ problems and specifically showing them how to execute these pipelines. We’re going through and understanding the considerations you need to make, building the skills to agentically develop these different pieces together, especially if you don’t know all the different tools right off the bat. 

But the most important part is the architectural thinking around this. We are creating a clear plan and process on how to think through and do this. And once you’ve done that process once, you can repeat it over and over, and you’ll understand the systems you need to bring into place and the trade-offs between each of them. And that’s honestly the biggest part of the data engineering process. 

And more importantly, you’d be able to speak to the specific stakeholders that you need to to help them think through this process in great detail, understand their needs, and build them data sets that work every single time for an exact use case that they need. 

For me, I think this is the way that it goes and this is the future. I’d like to see more effort focused in on this layer, as much as possible to serve all the other upstream layers, because if this layer can win, everyone else wins as well and will have a need for more spatial analysis. 

Keeping things in silos has never worked, but opening them up and making them connect into multiple different systems helps that grow and expand. We’ve seen this with data engineering in the past. When more data has been present, everything else grows. We should be doing the same in geospatial. This is my way to get there. And I realize that there are multiple ways to start to think about this. But this is a discussion I want to have and I want to learn how you are doing this for using these tools as well. 

And if you’re interested in doing this, check out the full sprint in the Spatial Lab. There’s more information on the Lab here. If you have any questions, don’t hesitate to reach out.