In May, we published a story diving into the nuances of the USDA plant hardiness zone map, which was updated in 2023 for the first time in a decade. I am a gardener AND a map nerd, so this was the juiciest, data-rich gardening story I was ever going to see. So we built an immersive story explaining what changed across the country, and what it means for our readers’ gardens.

It honestly feels like a Stefon meme. This app has EVERYTHING:

• Walking azaleas
• Rainbow D3.js charts
• “Mad Libs”-style, dynamic text for 30k+ places
• Naked figs in a freezer
• A color ramp with 26 unique colors

And of particular interest to this blog post:

• Self-hosted slippy maps

Slippy maps, which I’m defining here as pan-and-zoomable vector tile maps, can be quite costly to host with a third party like Mapbox. Historically, self-hosting was possible, but required a lot of technical expertise. But over the past two years, Kevin Schaul (Washington Post), Chris Amico (MuckRock) and others have outlined new approaches that lower the technical bar to self-hosting maps — and cost significantly less.

Here’s what we learned:

Skip building your own OSM layer. Use Protomaps’ daily download instead.

The tools that Schaul outlined fit together something like this:

Workflow with OpenMapTiles
 

It starts by baking your own OpenStreetMap and Natural Earth-based vector tiles via OpenMapTiles’ command line interface. I dove into it, but got totally overwhelmed. Confession time: I have never used Docker before, and I avoid PostGIS like the plague. I only recently learned what a makefile is. And that’s like…the whole thing.

[If you do want to go this route, I suggest following this workshop (videos 1, 2, 3, 4) to understand the ecosystem. They also have an easier-to-use paid tier.]

Thankfully, Amico’s blog post and talk at NICAR 2024 pointed out a pathway to avoid this step: use the Protomaps weekly PMTiles world build.

Protomaps is a fully free and open-source web mapping ecosystem spearheaded by developer Brandon Liu. It includes (as described on their website):

• PMTiles, an open archive format for pyramids of tile data, accessible via HTTP range requests.
• An ecosystem of tools and libraries for creating, serving and manipulating PMTiles.
• A cartographic “basemap” showing features in the world like roads, water bodies and labels, based on the OpenStreetMap dataset, and delivered as one big PMTiles archive.

And the trick here is that Protomaps provides a copy of the whole world, downloadable for free. (See an example basemap using this data.) This contains all the data that would be available if I rolled my own tileset with OpenMapTiles, but without needing to wade into Docker and understand what a “schema” is.

Each build is about 120 GB. Assuming you have somewhere local to store that data, you can trim it to your desired area using Tippecanoe. You can also extract a specific area of your liking instead of downloading the whole world.

This simplified our workflow to the following:

Workflow with
OpenMapTiles
 

Workflow with
Protomaps weekly builds

Trimming the tiles by bounding box was fairly easy, and allowed us to store less than the whole world. But trimming data based on country was not easy to figure out. That’s why some Mexican and Canadian cities are in our basemap. This isn’t ideal, but it was expedient. If we really needed to trim these things out, I think we’d have to use something like OpenTileMaps, which would have given us more fine-tuned control over what data goes into the basemap. (But if you know a better way, please let me know!)

It really IS a lot cheaper.

Self-hosting maps, beyond being daunting, used to cost a lot more than it does now. Before the PMTiles filetype existed, hosting vector tiles required either a server-side component (a “tile server”) or pre-baking every tile at every zoom level — and uploading and hosting all those individual tiles.

But with PMTiles you do not need to run a server to host and deliver vector tiles. Instead, you only need to drop the big PMTiles file somewhere accessible (S3 for instance, and ideally behind a CDN like Cloudfront). And Protomaps relies on the magic of HTTP range requests, where users only request the small portion of the PMTiles file that they need at any given time. The result: You can self-host maps at a substantial savings.

Protomaps provides a handy cost calculator to estimate a project’s costs versus other hosted options. I adapted this for my own calculator in Google Sheets .

Here are some topline things we learned:

The real cost is transfer from S3 to the browser

(Caveat: All these estimates are based on prices and polices at the time of publishing.)

• About 90% of the costs are for bandwidth from Cloudfront (Amazon’s CDN) to the internet. This is the total data transferred to users, and it costs about a dime for every gigabyte transferred. (And remember, users aren’t downloading the whole giant PMTiles file — just a tiny range of it.) OSM and hillshade tiles each averaged about 100 KB per tile. Our custom tiles for garden zones were much smaller. Transferring the data for about 100,000 tiles would cost $1.
• About 8% of the costs are based on the total number of GET requests made by users. Each tile requested is one GET request. Each PMTiles file shows about 4 tiles per zoom level per layer displayed. 100,000 GET requests would cost about 9 cents.
• A hypothetical scenario: If your project requests 100 tiles in an average session, and it gets 1,000 pageviews, that totals 100,000 tile requests. If your project gets 1 million pageviews in a month, your estimated costs for that month would be around $1,100. (For most news projects, the highest traffic — and costs — would come during that first month, and then trail off in subsequent months.)

It’s hard to say precisely how well the estimates matched up with reality, but they seemed to be in the ballpark.

To compare, that’s about ⅓ the cost that similar traffic might cost with a hosted service.

Don’t fret about the large size of the world map.

Storage on S3 is extremely cheap, about 0.1% of all costs for this project. Storing the WHOLE WORLD (~120GB) only costs about $3 per month. Functionally, this means storage volume is not a concern. Data transfer into S3 is also free, so there are almost no costs associated with getting huge files onto S3.

Most requests to get data out of S3, including PUT requests, do cost a tiny amount ($0.005 per request) — this is why the older strategy of generating static tiles can be an expensive proposition.

How to save money (and speed up loading!)

The main two axes to save money are:

1. reduce the total number of tiles requested, and
2. reduce the size of each tile.

Here’s how we approached this:

• Hide layers at certain zooms. For instance, we opted to avoid showing the hillshade layer until the map was zoomed in. This reduced the number of large tiles requested.
• Lazy-load tile layers. Initially the show/hide logic was based on the opacity of layers. This requested 3-4 more tiles than needed to be shown at any given time, which was costly and inefficient. Changing this logic dramatically improved the browser’s paint speed and saved us money.
• Cap maximum zoom level in explore mode. At a certain point, the user doesn’t need to zoom any further, so limit it to save on the number of tile requests.
• Limit exploration if it’s not essential. We really wanted to have folks be able to find themselves in the data and explore it. Sometimes this is not necessary. Limiting exploration can confine the upper end of possible costs.

Use ChatGPT (or similar) to help you understand inscrutable documentation

The Mapbox/Maplibre GL JS style spec is inscrutable:

map.setPaintProperty('2023_zones','fill-opacity',['interpolate',['linear'],['zoom'],0, 1, 7, 1, 8, 0.78, 22, 0.78 ]);

ChatGPT at least pretended to understand it perfectly. When asked what this code is doing, it explained it line by line, ending with:

“In summary, the fill opacity for the 2023_zones layer starts fully opaque at lower zoom levels (0-7) and becomes slightly more transparent (0.78 opacity) at zoom level 8 and remains at that opacity for higher zoom levels.”

Helpful!

But beware: It can also lie! For instance, I asked “Is there a way in Maplibre GL JS style to adjust the opacity only on the fill-outline property?” The correct answer is no, as far as I can tell. But ever the people pleaser, ChatGPT confidently told me:

The only problem is that transparentize does not exist in either style spec. It exists in Sass and maybe elsewhere, but not here. I figured this out with some quick Googling, but similar stochastic parroting can lead to some weird wrong turns. Buyer beware. (It’s worth noting that this hallucination took place on GPT-3.5. When I recently asked the same question to GPT-4o, it provided a more reliable answer.)

Important notes for styling your basemap

Finally, I want to share a few tools that are helpful in making your wildest self-hosted map dreams come true.

Styling your own basemap in the Maplibre GL JS style without a graphical user interface (GUI) is next to impossible. Luckily, you can use the Maputnik GUI to edit styles. (I found it much easier to use online, but there is a version you can run locally.)

If you’re using tiles generated from the Protomaps weekly builds the easiest starting point is the Protomaps Light style, available in Maputnik. Open the editor, click “Open”, then scroll down and click on “Protomaps Light”. Edit the map styles to your liking in Maputnik, then export the style.json to your project. I recommend committing this file to your project so that you have a version history of your style as it changes.

(Note: If you start with a style like OSM Liberty, and try to connect it to a Protomaps tileset, you will have to tweak layer names in style.json to match Protomaps’ basemap schema. For instance, your source layer name for country boundaries will need to change from “boundary” to “boundaries”. These sorts of things are devilishly difficult to identify!)

To add your preferred fonts to the style.json, you will need to create the appropriate .pbf files for each zoom level and style. The MapLibre font maker makes this a cinch. Store the generated .pbf files on S3 and reference that location in your style.json.

Similarly, you may need to make a custom sprite sheet. A sprite sheet is a single png that contains all icons and symbols that you might see on a map. A json key tells the mapping software what section of the sprite sheet to slice and use for the desired symbol.

In our case, we didn’t really have any conventional icons that were needed. But in order to avoid including a large legend on every map page, and for accessibility reasons, we wanted to include a layer that would show the hardiness zone names repeated across the map.

Left, the map in full color. Right, the map as it may look to someone with deuteranopia color blindness. In this view, you can see how the repeated zone labels would help readers distinguish between zones 7b and 8b.

There are a few solutions out there to create custom sprite sheets; I found that Spreet worked like a charm.

Great strides, but there are other costs to consider

The recent advances in self-hosted web maps are really exciting, but it’s important to remember a few caveats:

Hosting maps with a commercial provider like Mapbox comes with a raft of support and a more polished toolset. While the community around self-hosted solutions is robust and often available to help, there is no dedicated support if you go this direction.

Learning new tech is time-consuming and daunting. If you feel overwhelmed, look me up on the internet or in the #maps channel of the News Nerdery Slack group and ask for help! Also, I recently presented on this topic at the NACIS conference in October (as did Protomaps’ Liu).

Good luck!

We talk about so many things when we talk about AI. The conversation can roam from self-driving cars to dynamic video generation, from conversational chatbots to satellite imagery object detection, and from better search engines to dreamlike imagery generation. You get the point.

It gets confusing! For laypeople, it’s hard to nail down what AI actually does (and doesn’t) do. For those in the field, we often have to break down and overspecify our terms before we can get to our desired conversations.

After plenty of discussions and tons of exploration, I think we can simplify the world of AI use cases into three simple, distinct buckets:

1. Gods: Super-intelligent, artificial entities that do things autonomously.
2. Interns: Supervised copilots that collaborate with experts, focusing on grunt work.
3. Cogs: Functions optimized to perform a single task extremely well, usually as part of a pipeline or interface.

Let’s break these down, one by one.

Gods

Gods are the human-replacement use cases. The much hyped artificial general intelligences (AGI) that are allegedly just around the corner…

(Or maybe they’re just imminent when you’re trying to close one of the largest funding rounds in history? After the round is finalized, you can go back to realism. But I digress…)

Gods require gigantic models. Their autonomous nature – their defining quality – means a low error tolerance and a broad, general model. But a giant LLM surely isn’t enough. We could throw every data center and every database at the problem and still be short. We’re going to need breakthroughs, oodles of researchers to find these breakthroughs, and billions of dollars (at least).

Few entities will work towards the AGI or god use case because the barriers are so high. But, so are the rewards. The massively funded private ventures and sovereign-backed labs will continue their work.

But for most of us the Gods use case will most impact us in the form of hype and hand-wringing. It will affect funding, regulation, and politics. (Though if a would-be god-builder visibly hits a wall, we’ll feel it in other ways.)

Interns

Interns are the copilots (and a term I’ve shamelessly borrowed from Simon Willison). Their defining quality is that they are used and supervised by experts. They have a high tolerance for errors because said expert is reviewing their output, and can prevent embarrassing mistakes from going further. Interns are focused on the grunt work: remembering documentation and navigating references, filling in the details after the broad strokes are defined, assisting with ideation by being a dynamic sounding board, and much more.

Github Copilot and Cursor are interns for programmers. Adobe Firefly and Visual Electric are interns for designers. Microsoft Copilot and Grammarly are interns for writers. NotebookLM is an intern for researchers. Monterey is an intern for product managers. There are many more examples.

Adobe’s Project Turntable – a utility for rotating 2D designs as if they were 3D objects – is a perfect example of the intern form. Jess Weatherbed, writing for the Verge, sums it up:

The tool allows users to click a button and then drag a slider along to automatically view and snap a vector image into a different viewing perspective — something that would typically require an artist to redraw the image entirely from scratch. The examples demonstrated at the event retained their original designs when rotated without warping into a new overall shape.

The work being done here – redrawing the vector drawing from many different angles – is the type of grunt work an intern at a design studio might perform. The look and feel of the design and its proportions are already set. The work required is tedious but easily defined. Further, the output of this tool is editable by the expert using it. If Turntable is sloppy with a line or two, the artist can drop in and tweak it.

Because they are tools for specific types of experts, Interns are limited to specific domains. They don’t have to be generalists. Their model sizes are large, but not massive. One could build new interns – starting from scratch – for millions. If you use an open model, the costs are dramatically lower.

Today, Interns are delivering the lion’s share of the realized value from AI. Engineering copilots alone are delivering massive returns, increasing the output and capabilities of expert programmers. I’ve heard of a few large companies who have slowed down hiring due to their effects and have witnessed many tiny teams ship products years beyond what they could have delivered even 3 years ago, thanks to Github Copilot and similar tools.

And while Interns are delivering tremendous value, they are secondary to the experts driving them. How do you improve the output of an AI copilot? Simple: find a better expert.

Interns Toys

“But Drew,” you might say, “what about the fun image generator or friendly chatbot I play with occasionally? It doesn’t really fit in the intern bucket.”

You’re right. I think these are Toys, a subcategory of Interns defined by their usage by non-experts. Toys have a high tolerance for errors because they’re not being relied on for much beyond entertainment.

To improve a toy you don’t usually need to improve the user or the model; rather, the opportunity is usually the interface. We, as a community, just tend to focus on the models because we’re nerds.

But moving on…

Cogs

Cogs is the last use case bucket. Cogs are comparable to functions. They’re designed to do one task, unsupervised, very well. Cogs have a low tolerance for errors because they run with little expert oversight.

Cogs may be built into data pipelines, performing some enrichment or transformation step, alongside bog-standard functions powered by regex or SQL. Cogs may be built into interfaces, transforming human output like speech or scribbles into machine-readable input.

Cogs exist in systems as reliable components. Their focus on one task and a low tolerance for errors mean they are usually built with fine-tuned or heavily prompted small, open models. Cogs are cheap to run and relatively cheap to ship. They are enabled and improved by data, used for fine-tuning and/or developing prompts.

We used a cog here recently! Ollama and an embedding model powered a step in our conflation pipeline, nestled among some SQL queries and string distance functions. This cog was cheap, quick, and reliable after some initial configuration.

Cogs are, by far, the dominant use case amongst enterprise teams building with AI. As we covered after the DataBricks summit, in most discussions among builders, “AI looks like just another pipeline function.”

Cloud platforms – DataBricks, Snowflake, Azure, AWS, and others – have recognized the dominance of the cog use case and have sprinted towards delivering tools and hardware for building, testing, and running them.

Sorting the ocean of AI use cases into the Gods, Interns, and Cogs (and fine, Toys) buckets has helped me tremendously. It’s easier to navigate the noise, ask questions about new products, and identify the bottlenecks holding back each category.

It’s challenging to work in emergent fields as the language has yet to settle. Communication challenges are a tax that must be dealt with before productive conversations can occur. By scaffolding the mess with appropriate structures we can ease exchange.