Compare commits

..

7 Commits

8 changed files with 97 additions and 92 deletions
+1 -1
View File
@@ -4,6 +4,6 @@
- The generated result belongs in `tiles/`; do not hand-edit it. Run `docker compose up --build` or `./scripts/build-tiles.sh` to regenerate it.
- Preserve the public Arma endpoint format: `/tiles/{x}/{y}/{z}`. Internal files remain `tiles/{z}/{x}/{y}.png`.
- `CALLBACK_PROVIDER` is optional. When set, it must be an HTTPS template containing exactly `{z}`, `{x}`, and `{y}` in its path. Validate it before a tile build; no provider-specific host or branding belongs in application code or documentation.
- With a callback provider, configure MapLibre with `/base/{x}/{y}/{z}` as its raster layer below `/tiles/{x}/{y}/{z}`. This makes the provider visible through transparent edges of Arma overlays.
- With a callback provider, MapLibre uses only `/tiles/{x}/{y}/{z}`. The builder flattens Arma overlays over callback tiles, so transparent edges never expose black pixels.
- Keep source processing idempotent, do not modify input files, and preserve per-source error reporting in `tiles/metadata.json`.
- Before handing off changes, run `python3 tests/test_build_tiles.py`, `docker compose config --quiet`, and `docker compose build tiles` when Docker is available.
+1
View File
@@ -9,6 +9,7 @@ RUN apt-get update \
gdal-bin \
python3 \
python3-pil \
ca-certificates \
proj-data \
&& rm -rf /var/lib/apt/lists/*
+42 -51
View File
@@ -1,49 +1,43 @@
# Arma raster tiles
# Tilemap Server
Servidor estático de tiles raster XYZ para os mapas de Arma 3 fornecidos como KML/KMZ `GroundOverlay`. Todas as fontes compõem uma camada mundial única: não existe identificador de mapa na URL.
A static XYZ raster-tile server for maps supplied as KML/KMZ `GroundOverlay` files. All sources are composited into one global layer; the URL contains no map identifier.
## Arquitetura
## Architecture
`tile-builder` `source/` recursivamente, extrai KMZs somente em diretórios temporários, lê cada `GroundOverlay`, localiza a imagem local indicada por `Icon/href` e a reprojeta para Web Mercator (EPSG:3857). Para `LatLonBox` com `rotation`, os quatro cantos são girados no sentido anti-horário em torno do centro antes do warp GDAL; o raster final recebe alfa para conservar transparência e recortar as bordas rotacionadas.
`tile-builder` recursively reads `source/`, extracts KMZ files only into temporary directories, reads each `GroundOverlay`, finds the local image referenced by `Icon/href`, and reprojects it to Web Mercator (EPSG:3857). For a `LatLonBox` with `rotation`, its four corners are rotated counter-clockwise around the center before the GDAL warp. The final raster retains alpha to preserve transparency and clip rotated edges.
Cada overlay gera tiles temporários por `gdal2tiles --xyz`. O builder os compõe com alfa em uma única pirâmide. Uma área menor tem prioridade e é desenhada por cima; se as áreas forem iguais, vence primeiro o caminho relativo em ordem alfabética e depois o índice do `GroundOverlay`. A área é calculada a partir do polígono rotacionado em EPSG:3857.
Each overlay produces temporary tiles with `gdal2tiles --xyz`. The builder composites them with alpha into a single pyramid. A smaller area takes precedence and is rendered on top; where areas are equal, the relative path in alphabetical order wins, followed by the `GroundOverlay` index. Area is calculated from the rotated polygon in EPSG:3857.
O resultado físico é publicado diretamente na pasta local `tiles/`:
The generated files are published directly under the local `tiles/` directory:
```text
tiles/{z}/{x}/{y}.png
tiles/metadata.json
```
O Nginx aceita exclusivamente a rota pública abaixo e faz a tradução interna. Não publique nem use a ordem de armazenamento diretamente.
Nginx accepts only the public route below and translates it internally. Do not publish or use the storage order directly.
```text
https://arma_tiles.valmo.dev/tiles/{x}/{y}/{z}
https://arma_tiles.valmo.dev/tiles/18342/12417/15
https://tiles.example.com/tiles/{x}/{y}/{z}
https://tiles.example.com/tiles/18342/12417/15
```
Mesmo sem `.png` na URL, a resposta é `Content-Type: image/png` e inclui `Access-Control-Allow-Origin: *`, permitindo uso direto pelo MapLibre em outra origem. Sem provider configurado, coordenadas sem cobertura retornam `tiles/empty.png`; com provider configurado, são encaminhadas ao callback e armazenadas no cache local do Nginx.
Even without `.png` in the URL, the response uses `Content-Type: image/png` and includes `Access-Control-Allow-Origin: *`, so it can be used directly by MapLibre from another origin. Without a configured provider, uncovered coordinates return `tiles/empty.png`; with a provider, they are forwarded to the callback and cached locally by Nginx.
## Provider de callback e MapLibre
## Callback provider and MapLibre
Copie `.env.example` para `.env` e, se quiser uma camada base, defina um template HTTPS XYZ:
Copy `.env.example` to `.env` and, for a base layer, define an HTTPS XYZ template:
```dotenv
CALLBACK_PROVIDER=https://tiles.seu-provider.example/{z}/{x}/{y}.png
CALLBACK_PROVIDER=https://tiles.example-provider.com/{z}/{x}/{y}.png
```
O template é validado antes de qualquer leitura/processamento de fonte. Ele precisa usar `https`, não pode ter query, credenciais ou IP privado, e deve conter exatamente um `{z}`, `{x}` e `{y}` no caminho. Deixe a variável vazia ou ausente para usar o `empty.png` branco.
The template is validated before source processing begins. It must use `https`, cannot include a query, credentials, or a private IP address, and must contain exactly one `{z}`, `{x}`, and `{y}` in its path. Leave the variable empty or unset to use the white `empty.png`.
Para que o provider apareça tanto onde não há mapa Arma quanto nas bordas transparentes de uma tile parcialmente coberta, use a camada base local abaixo da camada Arma:
During generation, every tile with Arma coverage is flattened over the callback tile. Therefore MapLibre needs only one source:
```js
sources: {
providerBase: {
type: "raster",
tiles: ["http://localhost:9000/base/{x}/{y}/{z}"],
tileSize: 256,
scheme: "xyz"
},
arma: {
type: "raster",
tiles: ["http://localhost:9000/tiles/{x}/{y}/{z}"],
@@ -53,76 +47,73 @@ sources: {
maxzoom: 17
}
},
layers: [
{ id: "provider-base", type: "raster", source: "providerBase" },
{ id: "arma", type: "raster", source: "arma" }
]
layers: [{ id: "arma", type: "raster", source: "arma" }]
```
`/base/{x}/{y}/{z}` e o fallback de `/tiles/` consultam o callback sob demanda e compartilham um cache local persistente. Exiba a atribuição e cumpra os termos definidos pelo provider escolhido.
Only `/tiles/{x}/{y}/{z}` is public. Tiles without Arma coverage are fetched on demand from the callback and cached by Nginx. The builder also caches callback tiles while flattening covered areas. Display attribution and comply with the selected provider's terms.
## Entradas
## Inputs
Coloque `.kmz` e `.kml` em `source/`, inclusive em subpastas. Um KMZ deve conter um KML (o builder prefere `doc.kml`) e a imagem local referenciada pelo KML. KMLs soltos podem referenciar imagens dentro da mesma árvore de `source/`.
Place `.kmz` and `.kml` files in `source/`, including subdirectories. A KMZ must contain one KML file (the builder prefers `doc.kml`) and the local image referenced by the KML. Standalone KML files may reference images within the same `source/` tree.
Os arquivos de entrada nunca são modificados ou removidos. Erros por arquivo são registrados em `tiles/metadata.json` e não bloqueiam as demais fontes. Links HTTP(S), caminhos absolutos e `gx:LatLonQuad` não fazem parte desta primeira versão; use `LatLonBox` e imagens locais.
Input files are never modified or removed. Per-file errors are recorded in `tiles/metadata.json` and do not prevent other sources from being processed. HTTP(S) links, absolute paths, and `gx:LatLonQuad` are not supported in this first version; use `LatLonBox` and local images.
## Zoom e configuração
## Zoom and configuration
O máximo automático de cada overlay é calculado a partir da maior resolução reprojetada em metros por pixel. O builder escolhe o maior zoom cuja resolução de tile ainda não é mais detalhada que a fonte, evitando ampliação de pixels.
The automatic maximum zoom for each overlay is calculated from the highest reprojected resolution in metres per pixel. The builder selects the highest tile zoom whose resolution is no more detailed than the source, avoiding pixel upscaling.
O mínimo automático é `0`: os níveis amplos custam pouco e tornam mapas isolados encontráveis a partir da visão mundial. Cada fonte para no próprio máximo nativo, evitando ampliar pixels no servidor. Para permitir zoom visual adicional, configure o MapLibre com `maxZoom` alto e mantenha o `maxzoom` da source no máximo nativo; o MapLibre amplia a última tile disponível. Se nenhuma fonte cobrir a coordenada, o servidor usa o callback configurado ou o `empty.png` branco.
The automatic minimum zoom is `0`: broad levels are inexpensive and make isolated maps discoverable from a world view. This repository sets global `maxzoom` to `17`, deliberately allowing enlargement above the native resolution so a map does not disappear into the callback while zooming. Set it back to `null` to keep the automatic per-overlay maximum. If no source covers the coordinate, the server uses the configured callback or white `empty.png`.
`config/maps.json` permite overrides globais e por caminho relativo a `source/`:
`config/maps.json` supports global and source-relative overrides:
```json
{
"global": { "minzoom": null, "maxzoom": null },
"sources": {
"Altis.kmz": { "minzoom": 8, "maxzoom": 16 },
"subpasta/exemplo.kml": { "maxzoom": 14 }
"example.kmz": { "minzoom": 8, "maxzoom": 16 },
"subfolder/example.kml": { "maxzoom": 14 }
}
}
```
Use inteiro entre 0 e 22 ou `null`. A precedência é fonte, global, cálculo automático. Um `maxzoom` explícito acima do máximo automático é permitido e ficará marcado em `metadata.json`, pois solicita ampliação conscientemente.
Use an integer between 0 and 22 or `null`. Precedence is source, global, then automatic calculation. An explicit `maxzoom` above the automatic maximum is allowed and is marked in `metadata.json`, since it deliberately requests upscaling.
## Geração local
## Local build
Dependências locais: Python 3, Pillow, GDAL com `gdal_translate`, `gdalwarp`, `gdalinfo` e `gdal2tiles.py`.
Local dependencies: Python 3, Pillow, and GDAL with `gdal_translate`, `gdalwarp`, `gdalinfo`, and `gdal2tiles.py`.
```bash
./scripts/build-tiles.sh
./scripts/build-tiles.sh --source-filter Altis.kmz
./scripts/build-tiles.sh --source-filter example.kmz
```
O cache em `cache/` usa SHA-256 da fonte. Entradas inalteradas reutilizam os GeoTIFFs reprojetados. Para garantir a composição e a prioridade corretas, a v1 recompõe a pirâmide final inteira em staging a cada build; apenas a reprojeção das fontes inalteradas é reutilizada. Ao fim do build, a pasta local `tiles/` é substituída pelo resultado completo, após a geração de `metadata.json`.
The `cache/` directory uses the source SHA-256. Unchanged inputs reuse their reprojected GeoTIFFs. To ensure correct compositing and priority, version 1 rebuilds the entire final pyramid in staging for every build; only reprojection of unchanged sources is reused. At the end of a build, the local `tiles/` directory is replaced with the complete result after `metadata.json` is generated.
## Docker Compose e Coolify
## Docker Compose and Coolify
Para desenvolvimento local:
For local development:
```bash
docker compose up --build
```
O Nginx ficará disponível em `http://localhost:${HTTP_PORT:-9000}`. O Compose executa primeiro o builder e monta a pasta `tiles/` como somente leitura no Nginx.
Nginx is available at `http://localhost:${HTTP_PORT:-9000}`. Compose runs the builder first and mounts `tiles/` read-only in Nginx.
No Coolify, mantenha as entradas fora do Git e crie uma pasta/volume persistente no host. Defina `ARMA_TILES_SOURCE_DIR` para esse caminho; ele será montado como `/app/source` somente leitura. O volume nomeado `arma_tiles_data` preserva o cache entre deploys; a pasta `tiles/` do projeto é o volume de saída publicado tanto pelo builder quanto pelo Nginx. Configure o domínio `arma_tiles.valmo.dev` e TLS no proxy do Coolify, apontando para a porta 80 do serviço `tiles`.
In Coolify, keep inputs outside Git and create a persistent host directory or volume. Set the source-directory environment variable to that path; it is mounted read-only at `/app/source`. The named data volume preserves the cache between deployments, and the project's `tiles/` directory is the output volume published by both the builder and Nginx. Configure your domain and TLS in the Coolify proxy, targeting port 80 of the `tiles` service.
Para limitar paralelismo do GDAL, configure `GDAL2TILES_PROCESSES` (o padrão é `1`). As respostas de tile têm cache público de um dia e `stale-while-revalidate` de sete dias, além de ETag.
To limit GDAL parallelism, set `GDAL2TILES_PROCESSES` (default: `1`). Tile responses have one day of public caching plus seven days of `stale-while-revalidate`, as well as an ETag.
## Validação
## Validation
```bash
python3 tests/test_build_tiles.py
./scripts/smoke-test.sh
```
O teste de fumaça processa `Altis.kmz`, verifica `tiles/metadata.json` e um PNG, sobe o Nginx e consulta uma tile pela ordem pública `{x}/{y}/{z}` para confirmar `200`, `image/png` e CORS. O callback é opcional e não é exercitado pelo teste local.
The smoke test processes the bundled sample KMZ, checks `tiles/metadata.json` and a PNG, starts Nginx, and requests a tile using the public `{x}/{y}/{z}` order to confirm `200`, `image/png`, and CORS. The callback is optional and is not exercised by the local test.
## Limitações conhecidas
## Known limitations
- Não há watcher: após adicionar ou alterar uma entrada, execute o builder ou faça um novo deploy.
- A composição final ainda é uma reconstrução global; a estrutura de cache permite uma futura invalidação apenas dos tiles afetados.
- Esta versão não baixa imagens externas, não suporta `gx:LatLonQuad`, não trata overlays que cruzam o antimeridiano e não inclui DEM, hillshade, vetores, MBTiles, frontend, autenticação ou banco de dados.
- There is no watcher: after adding or changing an input, run the builder or deploy again.
- Final compositing is still a global rebuild; the cache structure allows future invalidation of only affected tiles.
- This version does not download external images, support `gx:LatLonQuad`, handle overlays crossing the antimeridian, or include DEM, hillshade, vectors, MBTiles, a frontend, authentication, or a database.
+1 -1
View File
@@ -1,7 +1,7 @@
{
"global": {
"minzoom": null,
"maxzoom": null
"maxzoom": 17
},
"sources": {
"Altis.kmz": {
+7 -13
View File
@@ -6,25 +6,19 @@ server {
root /var/lib/arma-tiles;
etag on;
# Keep these at server level. Named fallback locations inherit them, so each
# response has exactly one CORS value instead of the invalid "*, *".
add_header Cache-Control "public, max-age=86400, stale-while-revalidate=604800" always;
add_header Access-Control-Allow-Origin "*" always;
add_header Access-Control-Allow-Methods "GET, HEAD, OPTIONS" always;
add_header Access-Control-Allow-Headers "Origin, Range, Accept" always;
# Public API is x/y/z. try_files performs an internal lookup at z/x/y.png.
location ~ ^/tiles/(?<tile_x>[0-9]+)/(?<tile_y>[0-9]+)/(?<tile_z>[0-9]+)$ {
if ($request_method = OPTIONS) {
add_header Access-Control-Allow-Origin "*" always;
add_header Access-Control-Allow-Methods "GET, HEAD, OPTIONS" always;
add_header Access-Control-Allow-Headers "Origin, Range, Accept" always;
add_header Content-Length 0;
return 204;
}
try_files /tiles/$tile_z/$tile_x/$tile_y.png @callback_provider_tile;
add_header Cache-Control "public, max-age=86400, stale-while-revalidate=604800" always;
add_header Access-Control-Allow-Origin "*" always;
add_header Access-Control-Allow-Methods "GET, HEAD, OPTIONS" always;
add_header Access-Control-Allow-Headers "Origin, Range, Accept" always;
}
# A callback-provider base layer fills transparent edges when configured.
location ~ ^/base/(?<base_x>[0-9]+)/(?<base_y>[0-9]+)/(?<base_z>[0-9]+)$ {
try_files /__callback_provider_base__ @callback_provider_base;
}
include /etc/nginx/includes/callback-provider.conf;
+38 -13
View File
@@ -15,8 +15,10 @@ import tempfile
import uuid
import zipfile
from datetime import UTC, datetime
from io import BytesIO
from pathlib import Path
from urllib.parse import unquote, urlparse
from urllib.request import Request, urlopen
from xml.etree import ElementTree as ET
from PIL import Image
@@ -346,7 +348,34 @@ def alpha_is_empty(image: Image.Image) -> bool:
return image.getchannel("A").getbbox() is None
def compose_layer(source_tiles: Path, final_tiles: Path) -> None:
def callback_background(relative: Path, provider, cache_root: Path) -> Image.Image:
"""Return a cached callback tile, or an opaque white tile when disabled."""
if provider is None:
return Image.new("RGBA", (256, 256), (255, 255, 255, 255))
z, x, filename = relative.parts
y = filename.removesuffix(".png")
cached = cache_root / z / x / filename
if cached.exists():
with Image.open(cached) as opened:
image = opened.convert("RGBA")
image.load()
return image
url = provider.origin + provider.path_template.replace("{z}", z).replace("{x}", x).replace("{y}", y)
try:
request = Request(url, headers={"User-Agent": "arma-tiles-builder/1.0"})
with urlopen(request, timeout=30) as response:
payload = response.read()
with Image.open(BytesIO(payload)) as opened:
image = opened.convert("RGBA")
image.load()
except Exception as error:
raise SourceError(f"callback provider falhou para {z}/{x}/{y}: {error}") from error
cached.parent.mkdir(parents=True, exist_ok=True)
image.save(cached, format="PNG", optimize=False)
return image
def compose_layer(source_tiles: Path, final_tiles: Path, provider, callback_cache: Path) -> None:
for tile in sorted(source_tiles.rglob("*.png")):
relative = tile.relative_to(source_tiles)
destination = final_tiles / relative
@@ -360,21 +389,17 @@ def compose_layer(source_tiles: Path, final_tiles: Path) -> None:
with Image.open(destination) as opened:
base = opened.convert("RGBA")
base.load()
base.alpha_composite(layer)
result = base
else:
result = layer
if alpha_is_empty(result):
destination.unlink(missing_ok=True)
else:
result.save(destination, format="PNG", optimize=False)
base = callback_background(relative, provider, callback_cache)
base.alpha_composite(layer)
base.save(destination, format="PNG", optimize=False)
def write_empty_tile(tiles_dir: Path) -> None:
Image.new("RGBA", (256, 256), (255, 255, 255, 255)).save(tiles_dir / "empty.png", format="PNG", optimize=False)
def tile_overlay(overlay: dict, work_root: Path, final_tiles: Path, processes: int) -> None:
def tile_overlay(overlay: dict, work_root: Path, final_tiles: Path, processes: int, provider, callback_cache: Path) -> None:
layer_root = work_root / f"layer-{overlay['priority']:04d}-{overlay['index']}"
help_result = subprocess.run(["gdal2tiles.py", "--help"], text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
legacy = ["--legacy"] if "--legacy" in help_result.stdout else []
@@ -382,7 +407,7 @@ def tile_overlay(overlay: dict, work_root: Path, final_tiles: Path, processes: i
"gdal2tiles.py", *legacy, "-q", "--profile=mercator", "--xyz", "--exclude", "--webviewer=none", "--resampling=bilinear",
f"--zoom={overlay['minzoom']}-{overlay['maxzoom']}", f"--processes={processes}", overlay["warped"], str(layer_root),
])
compose_layer(layer_root, final_tiles)
compose_layer(layer_root, final_tiles, provider, callback_cache)
shutil.rmtree(layer_root, ignore_errors=True)
@@ -431,7 +456,7 @@ def source_record(descriptor: dict, config: dict) -> dict:
def build(arguments: argparse.Namespace) -> None:
# Validate before reading or warping any source, so a malformed provider never wastes a full build.
validate_callback_provider(os.environ.get("CALLBACK_PROVIDER"))
provider = validate_callback_provider(os.environ.get("CALLBACK_PROVIDER"))
source_dir = Path(arguments.source_dir).resolve()
data_dir = Path(arguments.data_dir).resolve()
tiles_dir = Path(arguments.tiles_dir).resolve()
@@ -476,7 +501,7 @@ def build(arguments: argparse.Namespace) -> None:
# alphabetical winner is painted last.
for overlay in sorted(overlays, key=lambda item: (item["area_m2"], item["source"], item["index"]), reverse=True):
try:
tile_overlay(overlay, work_root, stage, arguments.processes)
tile_overlay(overlay, work_root, stage, arguments.processes, provider, cache_root / "callback-provider")
except SourceError as error:
errors.append({"source": overlay["source"], "overlay": overlay["index"], "stage": "tile", "reason": str(error)})
global_min = min((item["minzoom"] for item in overlays), default=None)
@@ -486,7 +511,7 @@ def build(arguments: argparse.Namespace) -> None:
"generated_at": datetime.now(UTC).isoformat(),
"minzoom": global_min,
"maxzoom": global_max,
"tile_background": "transparent",
"tile_background": "callback_provider" if provider else "white",
"processed_sources": records,
"sources_with_error": errors,
"source_count": len(records),
+7 -13
View File
@@ -59,15 +59,10 @@ def validate_callback_provider(value: str | None) -> CallbackProvider | None:
return CallbackProvider(origin=f"https://{parsed.netloc}", path_template=parsed.path)
def render_location(name: str, provider: CallbackProvider | None, x: str, y: str, z: str, label: str) -> str:
headers = """ add_header Cache-Control \"public, max-age=86400, stale-while-revalidate=604800\" always;
add_header Access-Control-Allow-Origin \"*\" always;
add_header Access-Control-Allow-Methods \"GET, HEAD, OPTIONS\" always;
add_header Access-Control-Allow-Headers \"Origin, Range, Accept\" always;"""
def render_location(name: str, provider: CallbackProvider | None, x: str, y: str, z: str) -> str:
if provider is None:
return f"""location @{name} {{
try_files /tiles/empty.png =500;
{headers}
}}
"""
rendered_path = provider.path_template.replace("{x}", f"${x}").replace("{y}", f"${y}").replace("{z}", f"${z}")
@@ -79,18 +74,17 @@ def render_location(name: str, provider: CallbackProvider | None, x: str, y: str
proxy_cache callback_provider;
proxy_cache_valid 200 7d;
proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
add_header X-Map-Source \"Callback provider {label}\" always;
{headers}
proxy_hide_header Cache-Control;
proxy_hide_header Access-Control-Allow-Origin;
proxy_hide_header Access-Control-Allow-Methods;
proxy_hide_header Access-Control-Allow-Headers;
}}
"""
def render_nginx(provider: CallbackProvider | None) -> str:
return (
"# Generated at container startup. Do not edit.\n"
+ render_location("callback_provider_tile", provider, "tile_x", "tile_y", "tile_z", "fallback")
+ "\n"
+ render_location("callback_provider_base", provider, "base_x", "base_y", "base_z", "base")
return "# Generated at container startup. Do not edit.\n" + render_location(
"callback_provider_tile", provider, "tile_x", "tile_y", "tile_z"
)