​
0. Required pre work

Make sure the account you are using on the integration has the following permissions:

  • BigQuery Data Editor
  • BigQuery Job User

​
1. Add your BigQuery access

  1. In the Destinations tab, click on the “Add destination” button located on the top right of your screen. Then, select the BigQuery option from the list of connectors.

  2. Click Next and you’ll be prompted to add your access:

    • Project: The ID of the GCP project your BigQuery is linked to.
    • Dataset name: The target dataset where data will be sent to. For detailed instructions, click here.
    • Credentials file (JSON): The credentials file for the service account linked to your BigQuery project. Make sure the account has access to perform write operations on BigQuery. For detailed instructions, click here.

  3. Decide about:

    • Write method: The method to use for writing to BigQuery.
    • Denormalizing data: Determines whether to denormalize the data before writing to BigQuery. A false value will write data using a fixed JSON column based schema, while a true value will write data using a dynamic schema derived from the input table.
    • Upsert mode: A value of true will write to a temporary table and then merge into the target table (upsert) - this requires the target table to have unique key properties. A value of false will write to the target table directly (append).
    • Overwrite mode: Determines if the target table should be overwritten on load. This means replacing the actual data with the incoming one. If set to false, data will be appended. IMPORTANT: if Upsert is set to true, this setting will be ignored.

  4. Click Next.

​
2. Select your catalog data to send

  1. The next step is letting us know which data you want to send to BigQuery. You can select a whole layer or specific tables.

Tip: The tables can be easily found by typing its name.

  1. Click Next.

​
3. Map your data to send

  1. Give a name to be given to each table in the destination.

  2. For each table, select the column that will work as a primary key.

  3. By clicking on the fields mapping button, you will see a list of columns and their correspondent names in the destination. You can modify it if you want.

  4. Click Next.

​
4. Configure your BigQuery data destination

  1. Describe your destination for easy identification within your organization. You can inform things like what data it sends, to which team it belongs, etc.

  2. To define your Trigger, consider how often you want data to be sent to this destination. This decision usually depends on how frequently you need the new table data updated (every day, once a week, only at specific times, etc.).

  3. Click Done.

​
Check your new destination!

Once completed, you’ll receive confirmation that your new destination is set up!

You can view your new destination on the Destinations page. Now, for you to be able to see it on your BigQuery account, you have to wait for the pipeline to run. You can monitor it on the Destinations page to see its execution and completion. If needed, manually trigger the pipeline by clicking on the refresh icon.

Once executed, your data should be seen on BigQuery.

If you encounter any issues, reach out to us via Slack, and we’ll gladly assist you!

OverviewAmazon SQS