Scaling Up
As we're progressing with nationwide and force-specific hotspot determination we have had to reassess some of the tooling we've been using so far, particularly for presenting and disseminating results, and with a view to producing practical tools for practitioners and policymakers.
We've started to hit performance limitations with geopandas so are switching a lot of the CPU-intensive spatial code to use an ephemeral duckdb instance with the spatial extension. This is exceptionally fast, able to load 3 years of national crime data (~18m incidents, England & Wales) from over 1500 individual parquet files, and group the crimes into 200m hex cells (~1.5m) - all in well under 10 seconds.
We've previously used Streamlit as a front-end (see here) but it's also not scaling well, proving somewhat inflexible and has compatibility issues with duckdb, so we're looking at alternatives. Later in the project we'll post a more definitive article on recommendations for the best (python) tooling, but in the meantime we're excited about the possibilities duckdb enables and want to share some progress...
Embedded Interactive Hotspot Maps
Here are some illustrative results for a nationwide analysis of 3 crime types in the public data, covering the 3 year period up to September 2025.
A rolling window of crime counts (1, 3, and 12 months) are aggregated onto a hex grid (200m side, ~350m height) and ranked. The top 0.1% of cells are recorded for each window. Finally, spatial units are then ranked by the number of times each spatial unit features in the top 0.1% over the 3 year period.
In the interactive maps the hex cells are shaded according to the frequency they feature in the hotspots. Click on the tabs to switch crime type, and the links above each map to expand.
Increasing the size of the rolling window has the effect of making the hotspots more stable. There are clear differences in the characteristics of different crimes, with "Possession of weapons", with relatively low counts, showing a lesser degree of concentration.
(Note that Greater Manchester Police do not currently supply public crime data.)