What we adopt, what we avoid.
Our positions on technologies and practices, drawn from production engagements: rail, online betting, defence, retail… Four quadrants, four rings, and a data tag to follow everything data-related.
Every blip
Techniques
Adopt — Default choice, proven in production.
- 1. Extreme Programming
Pair and mob programming, retrospectives, paired reviews, small and frequent releases. Still the most effective frame for shipping software that lasts — and a multiplier once AI joins the pair.
In production · adtech, telecoms, retail
- 2. Test-first
Write the expectation before the implementation. With an AI assistant, tests become the contract that bounds what it generates: the faster code is produced, the more that contract matters.
- 3. Human-AI pair programming
The AI proposes, critiques and refactors; the engineer decides and stays accountable. A tireless pair, not a subcontractor: review, tests and judgement stay human.
- 4. Streaming-first, minimal batch data
Design data flows as events from day one and keep batch for recomputation. Data arrives fresh, and a single model serves both real time and reporting.
In production · adtech, rail
- 5. Ontologies for data catalogues data
Model the catalogue on established ontologies (the W3C's SOSA/SSN for sensors and observations) rather than home-grown conventions. Metadata becomes queryable, alignable and durable.
In production · defence
- 6. AI-agent-driven data quality data
Agents that read schemas and SQL, propose quality rules and tune them from observed anomalies. With typed, validated outputs, humans arbitrate instead of hand-writing every check.
In production · online betting
Trial — Worth using on a project that can handle the risk.
- 7. Schema inference from code data
Derive schemas (Protobuf, Parquet) straight from the C, C++ or Python code that produces the data, through analysis and specialisation, instead of maintaining them by hand. Powerful on heterogeneous sources, still demanding to tool.
In production · defence
Assess — Worth exploring to understand how it would affect you.
- 8. Mutation testing
Measure test quality by injecting mutations into the code. As more code gets generated, it's a good guard against complacent tests — it still needs to get fast enough for CI.
- 9. Reinforcement learning in production data
Closing the loop between decisions and measurements in production (pricing, allocation, prioritisation). The potential is real, but observability, policy safety and exploration cost must be under control first.
Hold — Avoid for new work.
- 10. AI-generated code without tests
Accepting generated code without tests or review just moves the debt to later, faster. Alternative: test-first and human-AI pairing, where tests bound what the AI produces.
Tools
Adopt — Default choice, proven in production.
- 11. Pulumi
Infrastructure as code in a real language (TypeScript, Python): types, functions, tests. Cloudflare, AWS and Koyeb declared alongside the rest of the code.
In production · short-term rentals, SaaS
- 12. Temporal
Durable workflows written as ordinary code: retries, timeouts and compensation without home-grown scaffolding. The right tool as soon as a process outlives a request.
In production · short-term rentals
- 13. Claude Code
An agentic coding assistant in the terminal: it reads the repository, runs the tests and proposes complete changes. Our everyday AI pair, always framed by tests and review.
- 14. Typst
Programmable typesetting, compiled in milliseconds and embeddable as a library. Replaces LaTeX for generating PDF, SVG and HTML from typed data.
- 15. Model Context Protocol (MCP)
The standard protocol for connecting AI agents to tools and data. One MCP server makes a system usable from Claude, Cursor and others without bespoke integrations.
Trial — Worth using on a project that can handle the risk.
- 16. DataHub data
Open-source, extensible, API-driven metadata catalogue. Solid once a real domain vocabulary is modelled in it; operating it still takes investment.
In production · defence
Hold — Avoid for new work.
- 17. Azkaban data
A Hadoop-era job scheduler, barely maintained, with no notion of data assets or fine-grained retries. Alternative: Temporal for application workflows, a modern orchestrator for data pipelines.
In production · retail
Platforms
Adopt — Default choice, proven in production.
- 18. Cloudflare Workers / R2
Edge compute with no servers to manage, and S3-compatible object storage with no egress fees. An API moved from Rails to Rust on Workers can run at near-zero hosting cost.
In production · environment
- 19. Redpanda data
The Kafka protocol without Kafka's complexity: a single C++ binary, no JVM or external coordination, lower latency. Kafka clients, connectors and tooling work unchanged.
- 20. BigQuery data
The reference serverless analytical warehouse: no infrastructure to size, standard SQL, and pay-per-use pricing that rewards good modelling.
In production · retail, luxury, adtech
- 21. Kubernetes
The common foundation for running heterogeneous services and workers, with Kustomize to keep manifests readable. Keep it for workloads that don't fit serverless.
In production · retail, SaaS, adtech
Trial — Worth using on a project that can handle the risk.
- 22. NATS data
Lightweight, fast, easy-to-operate messaging with persistence through JetStream. Excellent for IoT and service-to-service communication; weigh it against Redpanda once large histories must be replayed.
In production · rail
- 23. AWS Bedrock
Access to frontier models inside the client's AWS perimeter: data stays in the account, existing IAM and billing apply. The right entry point when compliance comes first.
In production · online betting, short-term rentals
Assess — Worth exploring to understand how it would affect you.
- 24. Cloudflare D1 / Vectorize
SQLite and a vector database at the edge, right next to Workers. Promising for low-cost AI applications; size and query limits still to be proven.
Hold — Avoid for new work.
- 25. Apache Kafka data
The Kafka protocol is still the streaming standard, but its reference implementation isn't: the JVM, coordination and running costs no longer pay off on new projects. Alternative: Redpanda, API-compatible.
In production · rail, online betting, retail
- 26. Databricks data
Powerful but expensive, and each building block (Lakeflow, Unity Catalog) ties data and processing a little more to the platform. Alternative: open formats (Parquet), an independent catalogue and portable processing with Polars and SqlGlot.
In production · rail
- 27. On-premise Hadoop data
Cloudera, Hive, Impala, Sqoop: a heavy stack to run, evolve and staff. Alternative: object storage, open formats and serverless or single-node engines.
In production · adtech
Languages & frameworks
Adopt — Default choice, proven in production.
- 28. Rust
Performance, memory safety and a type system that makes refactoring calm. Our default for services, the edge and demanding data tooling.
In production · environment, defence
- 29. Python
The lingua franca of data and AI. With strict typing and validated models it holds up in production; we switch to Rust where performance matters.
In production · online betting, defence
- 30. TypeScript
End-to-end strict typing for interfaces and the edge. This site is the demo: Astro, React islands and Workers, all in TypeScript.
In production · short-term rentals
- 31. Pydantic / Pydantic AI
Validated data models, and AI agents with typed inputs and outputs. The most reliable way to plug an LLM into production code.
In production · online betting
- 32. Polars data
Rust-powered DataFrames with lazy, parallel execution: most "big data" volumes fit on a single machine. A fraction of a cluster's cost, and testable like ordinary code.
- 33. SqlGlot data
SQL parsing, optimisation and transpilation across dialects. Essential for auditing queries, extracting lineage or moving between engines without manual rewrites.
In production · online betting
Hold — Avoid for new work.
- 34. Scala 3
An excellent language whose ecosystem no longer keeps up: a costly 2 → 3 migration, flagship frameworks losing momentum, a shrinking talent pool. Alternative: Rust for strong typing and performance, Python for data and AI; Scala remains relevant for existing systems and teaching.
In production · retail, education
- 35. Apache Spark data
Most real-world volumes no longer need a distributed cluster, nor its operational and testing overhead. Alternative: Polars and SqlGlot over Parquet; Spark remains justified at very large scale.
In production · online betting, adtech
- 36. Akka
Moved to a commercial licence (BSL) in 2022: a legal and budget risk for any new project. Alternative: Apache Pekko for existing systems, simpler models (Temporal, async Rust) for new work.
In production · adtech, SaaS