Sources keep changing
URLs move, files get replaced, API shapes drift and yesterday's spreadsheet gains three mystery columns.
Data collection · cleansing · integration · warehousing
Turn scattered data sources into one useful warehouse.
DataStitcher helps you bring together public datasets, paid sources you already have access to and your own data — then collect, inspect, clean and join it without rebuilding the same messy plumbing for every project.
BigQuery at the centre Source lineage retained Downstream databases stay purpose-built
incoming sources
GOV open government data portalversioned
API licensed providerscheduled
CSV internal exportprofiled
warehouseBigQueryraw → staging → canonical → marts
governedPostgresAPIBIexport
The boring bit is the expensive bit
Most data projects fail slowly in the plumbing.
Formats are all over the place
CSV is the easy case. Real public and commercial data arrives as archives, Excel, XML, GIS, APIs and stranger things.
Every consumer creates another copy
A dashboard, app and analyst each build their own pipeline, then quietly disagree about what the same field means.
Nobody can explain where a number came from
Without versions, profiles and lineage, a useful dataset eventually turns into folklore with SQL attached.
One repeatable data flow
Keep the messy source work upstream.
BigQuery becomes the durable analytical centre. Source-specific collection and cleaning happen once; downstream products consume documented, reusable data instead of learning every source's peculiarities.
PublicOpen dataCKAN · bulk files · portals
LicensedPaid sourcesAPIs · feeds · exports
YoursInternal dataFiles · systems · extracts
01Collectversion + retain source evidence
02Inspectunderstand format, schema + quality
03Cleannormalise without losing lineage
04Joinbuild reusable canonical models
Central warehouseBigQuerystaging · canonical data · marts
PostgresAPIsDashboardsSearchParquet / CSV
Warehouse first, not warehouse only
BigQuery holds the joined truth. Other databases serve the job.
DataStitcher is deliberately not a plan to make every application query the warehouse directly. Keep raw history, cleaned models and analytical joins in BigQuery, then publish the slice each consumer needs.
- 01Retain source evidenceversions, hashes, acquisition metadata and raw artefacts where policy allows.
- 02Build canonical warehouse modelsclean once, standardise identifiers and make joins reusable.
- 03Publish purpose-built outputsPostgres tables, APIs, search indexes, dashboards, Parquet or flat exports.
RAW / SOURCEWhat arrived, when, and from whereimmutable evidence · versions · hashes
STAGINGParsed and typed without pretending it is cleanprofiles · schema evidence · normalisation
CANONICALReusable entities, dimensions and joinsbusiness rules · tests · lineage
MARTS / OUTPUTSOnly what the consumer actually needsapps · analytics · exports · operational stores
Where it earns its keep
Useful when the answer lives across several datasets.
The sweet spot is not “I have one clean API”. It is “the useful answer needs six sources, half of them awkward, and we want to keep using the result next month”.
Regional and market intelligence
Combine government, business, demographic and commercial datasets so you can compare places, sectors and markets from one repeatable warehouse.
Data products that need many sources
Build a dependable data foundation behind customer-facing tools without making every product feature responsible for scraping, parsing and cleaning its own inputs.
Recurring research and reporting
Replace the monthly ritual of downloading spreadsheets, fixing columns and rebuilding joins with a versioned pipeline you can rerun and audit.
Licensed data plus public context
Get more value from paid sources by joining them with open datasets and your own records while keeping source boundaries and lineage visible.
Built around real ugly data
The source does not have to arrive nicely.
The current DataStitcher platform already has processing paths for common tabular, archive, XML, geospatial and transport formats, plus public-data discovery and BigQuery tooling. New source-specific collection can sit on the same acquisition, profiling and lineage model.
CSVJSONXLSXParquetXMLZIPGISGTFSArcGIS
DataStitcheris the layer underneath
Not another dashboard builder
Fix the data foundation first.
Dashboards and AI tools are easy to add once the data is dependable. DataStitcher focuses on the less glamorous work that makes those things trustworthy: acquisition, versions, parsing, profiling, cleaning, joins, lineage and a warehouse you can reuse.
Bring the awkward sources
Tell me what you are trying to combine.
Start with the problem, the sources you know about and the output you actually need. We can work out the smallest sensible pipeline before anyone builds a cathedral of YAML around it.