How do I set up a data pipeline from Adobe Commerce to BigQuery?

Create an API integration in Adobe Commerce Admin, set up a BigQuery dataset with denormalized schemas, build a Python extractor using Adobe's REST API with pagination, then deploy it as a Google Cloud Function with Cloud Scheduler for automated syncing every 6 hours.

What is the best way to run a bulk data migration from Adobe Commerce?

For initial historical loads, use BigQuery load jobs with a GCS staging bucket rather than streaming inserts — it's approximately 90% cheaper at high volumes. Extract data using paginated REST API calls with a page size of 100, and process orders, customers, and products as separate batches.

Should I use Fivetran or a custom connector for Adobe Commerce to BigQuery?

Fivetran makes sense for teams with fewer than two data engineers and under 500,000 monthly active rows. For larger volumes or multi-store setups needing custom EAV attributes, a custom pipeline is significantly cheaper — often under USD 100/month versus thousands for managed ETL.

How do I handle multiple currencies in my BigQuery commerce data?

Create a currency conversion reference table with daily exchange rates and build a normalized view that converts all transaction amounts to a base currency (typically USD). This is essential for APAC retailers operating across markets with HKD, SGD, TWD, and AUD.

Can I use Adobe Commerce data pipeline setup with AWS instead of GCP?

Yes, you can replicate this architecture on AWS using Redshift or Athena as the warehouse, Lambda functions for extraction, and EventBridge for scheduling. The extraction code using Adobe's REST API remains identical — only the loading and orchestration layers change.

How often should I sync Adobe Commerce data to BigQuery?

For most retail analytics, a 6-hour sync cycle balances freshness with cost. If you need near-real-time data for inventory or pricing decisions, implement Change Data Capture (CDC) using Debezium connected to Adobe Commerce's MySQL database for sub-minute latency.

Adobe Commerce to BigQuery Data Pipeline Setup Guide

Quick Answer: Set up an Adobe Commerce to BigQuery data pipeline by creating an API integration in Adobe Commerce, designing denormalized BigQuery schemas, building a Python extractor for the REST API, and deploying it on Google Cloud Functions with Cloud Scheduler for automated incremental syncing.

Most Adobe Commerce merchants I talk to across Asia-Pacific are sitting on a goldmine of transactional data they can barely query. Their order data lives in MySQL, their catalog sits in Elasticsearch, customer segments are scattered across Adobe's ecosystem — and when the CFO asks for a cohort retention report, someone exports a CSV. That's not analytics. That's archaeology.

An adobe commerce to bigquery data pipeline setup solves this by creating a continuous, automated flow of your commerce data into Google BigQuery, where you can run complex analytical queries in seconds rather than minutes. After building these pipelines for retailers including a 200-store jewellery chain and a multi-brand F&B group across Hong Kong and Singapore, I can tell you the architecture decisions you make in week one determine whether this becomes a competitive advantage or an expensive maintenance burden.

This guide covers three approaches — Fivetran (managed ETL), a custom Python connector using Adobe Commerce REST API, and a hybrid approach using Cloud Functions with Pub/Sub. I'll include the schema design we actually use for retail analytics and the exact configurations you can copy.

Prerequisites

Before you start, make sure you have the following in place:

Adobe Commerce Requirements

Adobe Commerce 2.4.4+ (Cloud or on-premise) with REST API enabled
An integration or admin user with API access to sales, catalog, and customer resources
Your store's base URL and OAuth credentials (consumer key, consumer secret, access token, access token secret)

To create an integration in Adobe Commerce Admin:

1Stores → Settings → Configuration → Services → OAuth → Consumer Settings
2→ Ensure "Allow OAuth Access Tokens to be used as standalone Bearer tokens" = Yes
3
4System → Extensions → Integrations → Add New Integration
5→ Name: bigquery-pipeline
6→ API → Resource Access: Custom
7→ Select: Sales, Catalog, Customers, Reports
8→ Save & Activate

Google Cloud Requirements

A GCP project with billing enabled
BigQuery API enabled
A service account with roles/bigquery.dataEditor and roles/bigquery.jobUser
The service account JSON key file downloaded locally

1# Create service account
2gcloud iam service-accounts create adobe-commerce-pipeline \
3  --display-name="Adobe Commerce Pipeline" \
4  --project=your-project-id
5
6# Grant BigQuery permissions
7gcloud projects add-iam-policy-binding your-project-id \
8  --member="serviceAccount:[email protected]" \
9  --role="roles/bigquery.dataEditor"
10
11gcloud projects add-iam-policy-binding your-project-id \
12  --member="serviceAccount:[email protected]" \
13  --role="roles/bigquery.jobUser"
14
15# Generate key file
16gcloud iam service-accounts keys create ./bq-service-account.json \
17  --iam-account=adobe-commerce-pipeline@your-project-id.iam.gserviceaccount.com

Local Development Requirements

Python 3.10+ with pip
google-cloud-bigquery library (v3.x)
requests and requests-oauthlib libraries
Docker (if deploying to Cloud Run)

1pip install google-cloud-bigquery==3.25.0 requests==2.32.3 requests-oauthlib==2.0.0

Step 1: Design Your BigQuery Schema for Retail Analytics

Schema design is where most teams go wrong. They replicate Adobe Commerce's EAV (Entity-Attribute-Value) model into BigQuery, which defeats the purpose. BigQuery is a columnar store — you want denormalized, analytics-ready tables, not a mirror of MySQL.

Here's the schema we use for most APAC retail clients. According to Google's own BigQuery best practices documentation, denormalized schemas with nested and repeated fields outperform normalized designs by 2-5x on typical analytical queries.

Create the BigQuery Dataset and Core Tables

1-- Create dataset with regional location for APAC compliance
2CREATE SCHEMA IF NOT EXISTS `your-project-id.commerce_analytics`
3OPTIONS (
4  location = 'asia-southeast1',  -- Singapore region
5  description = 'Adobe Commerce analytics pipeline'
6);
7
8-- Orders fact table (denormalized)
9CREATE TABLE IF NOT EXISTS `your-project-id.commerce_analytics.orders` (
10  order_id INT64 NOT NULL,
11  increment_id STRING NOT NULL,
12  store_id INT64,
13  store_name STRING,
14  customer_id INT64,
15  customer_email STRING,
16  customer_group STRING,
17  status STRING,
18  state STRING,
19  currency_code STRING,
20  grand_total NUMERIC,
21  subtotal NUMERIC,
22  tax_amount NUMERIC,
23  shipping_amount NUMERIC,
24  discount_amount NUMERIC,
25  total_qty_ordered NUMERIC,
26  coupon_code STRING,
27  payment_method STRING,
28  shipping_method STRING,
29  shipping_country STRING,
30  shipping_region STRING,
31  shipping_city STRING,
32  created_at TIMESTAMP,
33  updated_at TIMESTAMP,
34  items ARRAY<STRUCT<
35    item_id INT64,
36    sku STRING,
37    name STRING,
38    qty_ordered NUMERIC,
39    price NUMERIC,
40    row_total NUMERIC,
41    discount_amount NUMERIC,
42    product_type STRING
43  >>,
44  _extracted_at TIMESTAMP,
45  _batch_id STRING
46)
47PARTITION BY DATE(created_at)
48CLUSTER BY store_id, status;
49
50-- Customer dimension table
51CREATE TABLE IF NOT EXISTS `your-project-id.commerce_analytics.customers` (
52  customer_id INT64 NOT NULL,
53  email STRING,
54  first_name STRING,
55  last_name STRING,
56  group_id INT64,
57  group_name STRING,
58  store_id INT64,
59  created_at TIMESTAMP,
60  updated_at TIMESTAMP,
61  default_shipping_country STRING,
62  default_shipping_region STRING,
63  _extracted_at TIMESTAMP
64)
65CLUSTER BY store_id, group_id;
66
67-- Product catalog dimension
68CREATE TABLE IF NOT EXISTS `your-project-id.commerce_analytics.products` (
69  product_id INT64 NOT NULL,
70  sku STRING NOT NULL,
71  name STRING,
72  type_id STRING,
73  status INT64,
74  visibility INT64,
75  price NUMERIC,
76  special_price NUMERIC,
77  category_names ARRAY<STRING>,
78  created_at TIMESTAMP,
79  updated_at TIMESTAMP,
80  _extracted_at TIMESTAMP
81)
82CLUSTER BY type_id, status;

Note the PARTITION BY DATE(created_at) on the orders table. For a client running 15,000+ orders per month across four APAC markets, this partitioning reduced query costs by roughly 80% compared to scanning the full table — BigQuery charges per byte scanned, as documented in Google's BigQuery pricing page.

Ready to Transform Your Ecommerce Operations?

Branch8 specializes in ecommerce platform implementation and AI-powered automation solutions. Contact us today to discuss your ecommerce automation strategy.

Get Started

Step 2: Build the Extraction Layer (Adobe Commerce REST API)

Adobe Commerce's REST API supports pagination through searchCriteria. Here's a production-grade Python extractor:

1# extractor.py
2import requests
3import json
4from datetime import datetime, timedelta
5from typing import Generator
6
7class AdobeCommerceExtractor:
8    def __init__(self, base_url: str, access_token: str):
9        self.base_url = base_url.rstrip('/')
10        self.headers = {
11            'Authorization': f'Bearer {access_token}',
12            'Content-Type': 'application/json'
13        }
14        self.page_size = 100  # Max recommended by Adobe
15
16    def _paginated_get(self, endpoint: str, params: dict = None) -> Generator:
17        """Generic paginated API fetcher."""
18        current_page = 1
19        total_pages = 1
20
21        while current_page <= total_pages:
22            search_params = {
23                'searchCriteria[pageSize]': self.page_size,
24                'searchCriteria[currentPage]': current_page,
25                **(params or {})
26            }
27            response = requests.get(
28                f'{self.base_url}/rest/V1/{endpoint}',
29                headers=self.headers,
30                params=search_params,
31                timeout=30
32            )
33            response.raise_for_status()
34            data = response.json()
35
36            total_count = data.get('total_count', 0)
37            total_pages = (total_count + self.page_size - 1) // self.page_size
38
39            yield from data.get('items', [])
40            current_page += 1
41
42    def extract_orders(self, since: datetime = None) -> Generator:
43        """Extract orders, optionally since a given timestamp."""
44        params = {}
45        if since:
46            params.update({
47                'searchCriteria[filter_groups][0][filters][0][field]': 'updated_at',
48                'searchCriteria[filter_groups][0][filters][0][value]': since.strftime('%Y-%m-%d %H:%M:%S'),
49                'searchCriteria[filter_groups][0][filters][0][condition_type]': 'gteq'
50            })
51        return self._paginated_get('orders', params)
52
53    def extract_customers(self, since: datetime = None) -> Generator:
54        """Extract customer records."""
55        params = {}
56        if since:
57            params.update({
58                'searchCriteria[filter_groups][0][filters][0][field]': 'updated_at',
59                'searchCriteria[filter_groups][0][filters][0][value]': since.strftime('%Y-%m-%d %H:%M:%S'),
60                'searchCriteria[filter_groups][0][filters][0][condition_type]': 'gteq'
61            })
62        return self._paginated_get('customers/search', params)
63
64    def extract_products(self, since: datetime = None) -> Generator:
65        """Extract product catalog."""
66        params = {}
67        if since:
68            params.update({
69                'searchCriteria[filter_groups][0][filters][0][field]': 'updated_at',
70                'searchCriteria[filter_groups][0][filters][0][value]': since.strftime('%Y-%m-%d %H:%M:%S'),
71                'searchCriteria[filter_groups][0][filters][0][condition_type]': 'gteq'
72            })
73        return self._paginated_get('products', params)

The _paginated_get method handles Adobe's quirky search criteria syntax. One gotcha: Adobe Commerce Cloud's API rate limits vary by plan. According to Adobe's Developer Documentation for Commerce Cloud, the default is 50 requests per second for integration-type tokens. Add retry logic if you're pulling large catalogs.

Step 3: Transform and Load Into BigQuery

The transformation layer maps Adobe Commerce's nested JSON responses into the flat + nested STRUCT schema we defined. This is where you flatten the EAV mess.

1# loader.py
2from google.cloud import bigquery
3from datetime import datetime, timezone
4import uuid
5
6class BigQueryLoader:
7    def __init__(self, project_id: str, dataset_id: str, credentials_path: str):
8        self.client = bigquery.Client(
9            project=project_id,
10            credentials=self._load_credentials(credentials_path)
11        )
12        self.dataset_id = dataset_id
13        self.project_id = project_id
14
15    def _load_credentials(self, path):
16        from google.oauth2 import service_account
17        return service_account.Credentials.from_service_account_file(path)
18
19    def _table_ref(self, table_name: str) -> str:
20        return f'{self.project_id}.{self.dataset_id}.{table_name}'
21
22    def load_orders(self, raw_orders):
23        """Transform and load orders into BigQuery."""
24        batch_id = str(uuid.uuid4())
25        extracted_at = datetime.now(timezone.utc).isoformat()
26        rows = []
27
28        for order in raw_orders:
29            items = []
30            for item in order.get('items', []):
31                items.append({
32                    'item_id': item.get('item_id'),
33                    'sku': item.get('sku'),
34                    'name': item.get('name'),
35                    'qty_ordered': item.get('qty_ordered'),
36                    'price': item.get('price'),
37                    'row_total': item.get('row_total'),
38                    'discount_amount': item.get('discount_amount', 0),
39                    'product_type': item.get('product_type')
40                })
41
42            # Extract shipping address details
43            shipping = order.get('extension_attributes', {}).get(
44                'shipping_assignments', [{}]
45            )[0].get('shipping', {}).get('address', {})
46
47            rows.append({
48                'order_id': order['entity_id'],
49                'increment_id': order['increment_id'],
50                'store_id': order.get('store_id'),
51                'store_name': order.get('store_name'),
52                'customer_id': order.get('customer_id'),
53                'customer_email': order.get('customer_email'),
54                'customer_group': str(order.get('customer_group_id')),
55                'status': order.get('status'),
56                'state': order.get('state'),
57                'currency_code': order.get('order_currency_code'),
58                'grand_total': order.get('grand_total'),
59                'subtotal': order.get('subtotal'),
60                'tax_amount': order.get('tax_amount'),
61                'shipping_amount': order.get('shipping_amount'),
62                'discount_amount': order.get('discount_amount', 0),
63                'total_qty_ordered': order.get('total_qty_ordered'),
64                'coupon_code': order.get('coupon_code'),
65                'payment_method': order.get('payment', {}).get('method'),
66                'shipping_method': order.get('shipping_description'),
67                'shipping_country': shipping.get('country_id'),
68                'shipping_region': shipping.get('region'),
69                'shipping_city': shipping.get('city'),
70                'created_at': order.get('created_at'),
71                'updated_at': order.get('updated_at'),
72                'items': items,
73                '_extracted_at': extracted_at,
74                '_batch_id': batch_id
75            })
76
77        if rows:
78            errors = self.client.insert_rows_json(
79                self._table_ref('orders'), rows
80            )
81            if errors:
82                raise RuntimeError(f'BigQuery insert errors: {errors}')
83            print(f'Loaded {len(rows)} orders (batch: {batch_id})')
84        return len(rows)

A note on insert_rows_json vs. load jobs: for incremental syncs under 10,000 rows, streaming inserts work fine. For initial historical loads, use load_table_from_json with a GCS staging bucket — it's roughly 90% cheaper at scale according to Google Cloud's streaming insert pricing documentation.

Ready to Transform Your Ecommerce Operations?

Branch8 specializes in ecommerce platform implementation and AI-powered automation solutions. Contact us today to discuss your ecommerce automation strategy.

Get Started

Step 4: Orchestrate With Cloud Functions and Cloud Scheduler

For production, you need automated scheduling. Here's a Cloud Function that runs the full extraction:

1# main.py (Cloud Function entry point)
2import functions_framework
3from extractor import AdobeCommerceExtractor
4from loader import BigQueryLoader
5from datetime import datetime, timedelta, timezone
6import os
7import json
8
9@functions_framework.http
10def run_pipeline(request):
11    """HTTP Cloud Function for Adobe Commerce → BigQuery sync."""
12    config = {
13        'adobe_base_url': os.environ['ADOBE_COMMERCE_URL'],
14        'adobe_token': os.environ['ADOBE_COMMERCE_TOKEN'],
15        'gcp_project': os.environ['GCP_PROJECT_ID'],
16        'bq_dataset': os.environ['BQ_DATASET'],
17        'credentials_path': '/tmp/bq-credentials.json'
18    }
19
20    # Write credentials from Secret Manager
21    creds_json = os.environ.get('BQ_CREDENTIALS_JSON')
22    with open(config['credentials_path'], 'w') as f:
23        f.write(creds_json)
24
25    # Default: sync last 24 hours
26    since = datetime.now(timezone.utc) - timedelta(hours=24)
27
28    extractor = AdobeCommerceExtractor(
29        config['adobe_base_url'],
30        config['adobe_token']
31    )
32    loader = BigQueryLoader(
33        config['gcp_project'],
34        config['bq_dataset'],
35        config['credentials_path']
36    )
37
38    results = {}
39    results['orders'] = loader.load_orders(extractor.extract_orders(since))
40    # Add customers, products similarly
41
42    return json.dumps({
43        'status': 'success',
44        'records_loaded': results,
45        'sync_since': since.isoformat()
46    }), 200

Deploy and schedule it:

1# Deploy Cloud Function
2gcloud functions deploy adobe-commerce-bq-sync \
3  --gen2 \
4  --runtime=python312 \
5  --region=asia-southeast1 \
6  --source=. \
7  --entry-point=run_pipeline \
8  --trigger-http \
9  --memory=512MB \
10  --timeout=540s \
11  --set-env-vars="ADOBE_COMMERCE_URL=https://your-store.com,GCP_PROJECT_ID=your-project-id,BQ_DATASET=commerce_analytics" \
12  --set-secrets="ADOBE_COMMERCE_TOKEN=adobe-token:latest,BQ_CREDENTIALS_JSON=bq-creds:latest"
13
14# Create Cloud Scheduler job (runs every 6 hours)
15gcloud scheduler jobs create http adobe-commerce-sync-job \
16  --location=asia-southeast1 \
17  --schedule="0 */6 * * *" \
18  --uri="https://asia-southeast1-your-project-id.cloudfunctions.net/adobe-commerce-bq-sync" \
19  --http-method=POST \
20  --oidc-service-account-email=adobe-commerce-pipeline@your-project-id.iam.gserviceaccount.com

Step 5: Alternatively, Use Fivetran for Managed ETL

If you'd rather not maintain custom code, Fivetran offers a managed Adobe Commerce connector. Here's the honest trade-off:

When Fivetran makes sense

Teams with fewer than two data engineers
You need the pipeline running in hours, not weeks
Standard reporting needs (orders, products, customers)
Budget of approximately USD 1-2/mo per 1,000 monthly active rows (Fivetran's 2024 pricing model)

When custom makes sense

You need custom attributes from EAV tables
Multi-store setups with store-specific transformation logic
Real-time or near-real-time requirements (sub-hour)
Cost sensitivity above 500,000 MAR (monthly active rows)

When we built the adobe commerce to bigquery data pipeline setup for a Hong Kong-based jewellery retailer with 4 Adobe Commerce storefronts across HK, Macau, and Taiwan, we initially evaluated Fivetran. The estimated cost at their volume (2.3 million MAR) was approximately USD 3,200/month. We built the custom pipeline in three weeks using the architecture described above, and the ongoing Cloud Functions cost runs under USD 85/month. The trade-off is about 4 hours per month of maintenance — mostly handling Adobe Commerce API changes after upgrades.

Fivetran Configuration Steps

If you go the Fivetran route:

11. Fivetran Dashboard → Connectors → Add Connector
22. Search "Magento" (Fivetran still uses the legacy name)
33. Enter your Adobe Commerce base URL
44. Authentication: Bearer Token → paste your integration access token
55. Destination: Select your BigQuery warehouse
66. Schema prefix: adobe_commerce
77. Sync frequency: Set to 6 hours (or 1 hour for Pro plans)
88. Initial sync: Enable historical sync

Fivetran creates its own schema — normalized tables matching Adobe's database structure. You'll still need a transformation layer (dbt is the standard choice) to build the denormalized analytics tables.

Ready to Transform Your Ecommerce Operations?

Branch8 specializes in ecommerce platform implementation and AI-powered automation solutions. Contact us today to discuss your ecommerce automation strategy.

Get Started

Step 6: Validate Your Pipeline and Build Monitoring

A pipeline without monitoring is a pipeline that silently breaks. Here are the validation queries we run after every sync:

1-- Check for data freshness
2SELECT
3  MAX(_extracted_at) AS last_extraction,
4  TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), MAX(_extracted_at), HOUR) AS hours_since_sync,
5  COUNT(*) AS total_orders
6FROM `your-project-id.commerce_analytics.orders`;
7
8-- Detect missing orders (compare against Adobe's order count)
9-- Run this against your Adobe Commerce admin's reported total
10SELECT
11  DATE(created_at) AS order_date,
12  COUNT(*) AS bq_order_count
13FROM `your-project-id.commerce_analytics.orders`
14WHERE created_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
15GROUP BY order_date
16ORDER BY order_date DESC;
17
18-- Check for duplicate orders (idempotency validation)
19SELECT
20  order_id,
21  COUNT(*) AS duplicate_count
22FROM `your-project-id.commerce_analytics.orders`
23GROUP BY order_id
24HAVING COUNT(*) > 1;

For alerting, set up a BigQuery scheduled query that writes to a monitoring table, then connect it to Google Cloud Monitoring:

1# Create alert policy for stale data (no sync in 12+ hours)
2gcloud alpha monitoring policies create \
3  --display-name="Adobe Commerce Pipeline Stale" \
4  --condition-display-name="No fresh data in 12 hours" \
5  --condition-filter='resource.type="bigquery_dataset" AND metric.type="custom.googleapis.com/pipeline/hours_since_sync"' \
6  --condition-threshold-value=12 \
7  --condition-threshold-comparison=COMPARISON_GT \
8  --notification-channels=your-channel-id

Handling Multi-Store and Multi-Currency APAC Scenarios

Across Asia-Pacific, most enterprise Adobe Commerce deployments run multiple store views — often with different currencies (HKD, SGD, TWD, AUD). Your pipeline needs to handle this cleanly.

Add a currency normalization layer:

1-- Create a currency conversion reference table
2CREATE TABLE IF NOT EXISTS `your-project-id.commerce_analytics.currency_rates` (
3  from_currency STRING,
4  to_currency STRING,
5  rate NUMERIC,
6  rate_date DATE
7);
8
9-- Normalized revenue view across all markets
10CREATE OR REPLACE VIEW `your-project-id.commerce_analytics.orders_normalized` AS
11SELECT
12  o.*,
13  ROUND(o.grand_total * COALESCE(cr.rate, 1), 2) AS grand_total_usd,
14  ROUND(o.subtotal * COALESCE(cr.rate, 1), 2) AS subtotal_usd
15FROM `your-project-id.commerce_analytics.orders` o
16LEFT JOIN `your-project-id.commerce_analytics.currency_rates` cr
17  ON o.currency_code = cr.from_currency
18  AND cr.to_currency = 'USD'
19  AND cr.rate_date = DATE(o.created_at);

According to Statista's 2024 e-commerce market report, cross-border e-commerce in Asia-Pacific reached USD 2.1 trillion in GMV — meaning multi-currency support isn't optional for any serious analytics pipeline in the region.

Ready to Transform Your Ecommerce Operations?

Branch8 specializes in ecommerce platform implementation and AI-powered automation solutions. Contact us today to discuss your ecommerce automation strategy.

Get Started

What to Do Next

Once your adobe commerce to bigquery data pipeline setup is running and validated, here's where to focus next:

Immediate next steps (Week 1-2)

Connect Looker Studio or Metabase to your BigQuery dataset for dashboards
Build dbt models for derived tables: customer lifetime value, product affinity, cohort retention
Add inventory data to the pipeline for stock-level analytics

Medium-term improvements (Month 1-3)

Implement CDC (Change Data Capture) using Debezium if you need near-real-time sync from Adobe Commerce's MySQL database
Add Google Analytics 4 data into the same BigQuery project for unified marketing + commerce analytics
Set up row-level security in BigQuery if multiple teams access different market data

The direction this space is heading is clear: managed connectors will commoditize basic ETL, but the competitive edge sits in the transformation and modeling layer. Companies that build strong dbt models on top of their commerce data — especially those operating across multiple APAC markets with varying tax regimes, payment methods, and customer behaviors — will make faster, more accurate decisions than those still exporting CSVs from their Adobe Commerce admin panel.

If your team needs help architecting a commerce data pipeline that handles APAC's complexity — multiple currencies, cross-border tax, regional payment methods — reach out to Branch8. We've built these for retailers across Hong Kong, Singapore, Taiwan, and Australia, and we can typically get a production pipeline running within three weeks.

Sources

Google Cloud BigQuery Best Practices: https://cloud.google.com/bigquery/docs/best-practices-performance-overview
Adobe Commerce REST API Reference: https://developer.adobe.com/commerce/webapi/rest/
Google Cloud BigQuery Pricing: https://cloud.google.com/bigquery/pricing
Fivetran Magento Connector Documentation: https://fivetran.com/docs/connectors/applications/magento
Adobe Commerce Cloud API Rate Limits: https://experienceleague.adobe.com/docs/commerce-cloud-service/user-guide/develop/integrate/integrations.html
Statista Cross-Border E-commerce in APAC (2024): https://www.statista.com/topics/5765/cross-border-e-commerce-in-asia-pacific/
dbt Documentation — BigQuery Setup: https://docs.getdbt.com/docs/core/connect-data-platform/bigquery-setup

Adobe Commerce to BigQuery Data Pipeline Setup: A Step-by-Step Guide

Prerequisites

Adobe Commerce Requirements

Google Cloud Requirements

Local Development Requirements

Step 1: Design Your BigQuery Schema for Retail Analytics

Create the BigQuery Dataset and Core Tables

Step 2: Build the Extraction Layer (Adobe Commerce REST API)

Step 3: Transform and Load Into BigQuery

Step 4: Orchestrate With Cloud Functions and Cloud Scheduler

Step 5: Alternatively, Use Fivetran for Managed ETL

When Fivetran makes sense

When custom makes sense

Fivetran Configuration Steps

Step 6: Validate Your Pipeline and Build Monitoring

Handling Multi-Store and Multi-Currency APAC Scenarios

What to Do Next

Immediate next steps (Week 1-2)

Medium-term improvements (Month 1-3)

Sources

FAQ

Matt Li