Making Datasets Publicly Available
Currently, only datasets and query results that are available in BigQuery and defined in the bigquery-etl repository can be made publicly available. See the bigquery-etl documentation for information on how to create and schedule datasets. Before data can be published, a data review is required.
To make query results publicly available, a metadata.yaml file
must be added alongside the query in bigquery-etl. For example:
friendly_name: SSL Ratios
description: >-
Percentages of page loads Firefox users have performed that were
conducted over SSL broken down by country.
owners:
- example@mozilla.com
labels:
application: firefox
incremental: true # incremental queries add data to existing tables
schedule: daily # scheduled in Airflow to run daily
public_json: true
public_bigquery: true
review_bugs: # Bugzilla bug IDs of data review
- 1414839
incremental_export: false # non-incremental JSON export writes all data to a single location
The following options define how data is published:
public_json: data is available through the public HTTP endpointpublic_bigquery: data is publicly available on BigQuery- tables will get published in the
mozilla-public-dataGCP project which is accessible by everyone, also external users
- tables will get published in the
incremental_export: determines how data gets split uptrue: data for eachsubmission_dategets exported into separate directories (e.g.files/2020-04-15,files/2020-04-16, ...)false: all data gets exported into onefiles/directory
- JSON output files are capped at 1GB each and split into sequentially numbered files (e.g.
000000000000.json,000000000001.json, ...), up to 10,000 files per table
incremental: indicates how data gets updated based on the query and Airflow configurationtrue: data gets incrementally updatedfalse: the entire table data gets updated
review_bugs: list of Bugzilla bug numbers of the data reviewlegacy: set totrueto indicate a dataset that selects only from legacy Telemetry
Data will get published when the query is executed in Airflow. Metadata of available public data on Cloud Storage is updated daily through a separate Airflow task.
More information about accessing public data can be found in Accessing Public Data.