VanderSat API User Guide
This page describes the VanderSat API and how to use it. For documentation on Planet products and APIs, please visit https://developers.planet.com.
For a comprehensive list of API endpoints and parameters, please visit https://maps.vandersat.com/api/v2. There you will also be able to try out API requests and inspect responses, directly in the browser.
Interactive API documentation used for trying out the product listing request
Note
Not all API endpoints are available for each user. In general, for new accounts only data subscriptions will be available. Contact Sales if you have a need for the other endpoints.
Field-based vs non-field-based products
The API has different support for products (or product versions) that are “field-based” and those that are not:
Most products are not field-based. The base processing for these products is independent of the requested geometry, and post-processing is performed to create the desired output. All API endpoints described below will work for those, but only if enabled for your account.
For field-based products, typically Biomass Proxy products, the base processing creates the output for the requested geometry directly. Many API endpoints described below make no sense for those products, as no post-processing is done at all.
To tell if a product (version) is field-based or not, see product availability.
For both types, the recommended way to get data is through data subscriptions.
API names
VanderSat API name |
Planet Subscriptions API Source ID |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Authentication
Username and password are supplied to you through separate channels. The password can easily be set or changed using the set password page.
We support HTTP basic authentication. There are two ways of providing your authentication:
Provide your credentials directly in the URL: https://username:password@maps.vandersat.com/api/v2/products
Provide your username and password in the HTTP header of the request. For example, use Python’s
requestslibrary:import requests import json r = requests.get('https://maps.vandersat.com/api/v2/products', auth=('username', 'password')) # change username and password r.raise_for_status() product_dict = json.loads(r.content)
to get a Python dictionary with all products available to this user account.
Error handling
In case of an error, the server often provides details in the response body. Like a 403 Forbidden
may include {"message": "You do not have access to product 'BIOMASS-PROXY_V2.0_10'"}, but not
all standard HTTP error handling may show that. For example, in Python use:
try:
r = requests.get(...)
r.raise_for_status()
...
except requests.HTTPError as e:
code = e.response.status_code
message = e.response.text
...
Python client
vds-api-client is a Python and command line interface to download data from the API. This
package simplifies and speeds up the process of making API requests and downloading data. It
does not currently support data subscriptions.
The project is on GitHub and the package can be
pip installed from PyPI:
$ pip install vds-api-client
More information on using the client can be found in the package documentation.
Product and data availability
Checking access to Planetary Variables products
Your specialized Planet planetary variables account gives access to a number of different products, depending on your contract with us. To get a complete list, make the following GET request (for example, by typing the URL in your browser address bar):
GET https://maps.vandersat.com/api/v2/products/
The most important fields for each product are api_name, which is needed for requesting data,
field_based to determine the type of product, and area_allowed to know for which regions
the product data can be request. If the latter is not given, then the geojson_area_allowed of
your account applies, which can be fetched using:
GET https://maps.vandersat.com/api/v2/users/me
Checking data availability for a specific product
To check the availability of a non-field-based product product, get the
api_name as described in the previous section and make the following request:
GET https://maps.vandersat.com/api/v2/products/{api_name}/availability
The returned JSON object contains a list of datetime values. The date component indicates days for which data is available. The time component can be ignored.
Data availability does not apply to field-based products, for which data subscriptions will start processing automatically as soon as applicable.
Data subscriptions
When one needs the same data every day, then data subscriptions allow for automatic creation as soon as the source data is available, and getting HTTP webhook notification when processing is complete. Each daily automatic creation is called a “fulfillment”.
Subscriptions allow for specifying the exact requested geometry. This works differently for products (or product versions) that are “field-based” and those that are not:
For non-field-based products a subscription schedules automatic data requests on behalf of the user as soon as data is available. This needs an
api_request_type(currently only"gridded-data") and accepts additional arguments such asformatandmin_coverageas described in the specific data requests endpoints. When done, the requested geometry is cut out of the generated tiles and available for download.For field-based products, the processing creates the output for the requested geometry directly, also using that user-defined geometry for model training and processing. For those products most additional arguments do not apply, as no post-processing is done at all.
To tell if a product (version) is field-based or not, see product availability.
Subscribing to data
A new subscription can be created using:
POST https://maps.vandersat.com/api/v2/subscriptions/
This expects Content-Type: application/json with a UTF-8 encoded JSON body. For non-field-based
products this is something like:
{
"name": "Soil Moisture V4 area around Metbrunnen, Germany",
"start_date": "2021-09-01",
"end_date": "2024-12-31",
"api_name": "SM-SMAP-L-DESC_V4.0_100",
"api_request_type": "gridded-data",
"arguments": {
"format": "gtiff",
"min_coverage": 80
},
"geojson": {
"type": "Polygon",
"coordinates": [[
[9.080522, 51.697751], [9.082560, 51.696953], [9.082689, 51.699626],
[9.080629, 51.699666], [9.080522, 51.697751]
]]
},
"http_notify": {
"url": "https://example.com/my/endpoint?my-auth=s3cr3t"
}
}
For field-based products, api_request_type should never be specified, and most
often arguments does not apply either:
{
"name": "Biomass Proxy V2 field around Metbrunnen, Germany",
"start_date": "2021-09-01",
"end_date": "2024-12-31",
"api_name": "BIOMASS-PROXY_V2.0_10",
"geojson": {
"type": "Polygon",
"coordinates": [[
[9.080522, 51.697751], [9.082560, 51.696953], [9.082689, 51.699626],
[9.080629, 51.699666], [9.080522, 51.697751]
]]
},
"http_notify": {
"url": "https://example.com/my/endpoint?my-auth=s3cr3t"
}
}
nameis any custom name for the subscription. It does not need to be unique.start_dateandend_date(optional) define the subscription period. The values are inclusive, and past dates (also known as “backfilling”) are supported. These refer to product dates, so the actual processed data for a given date may not be available until next day.api_nameis the product name.geojsondefines the areas for the data, eitherPolygonorMultiPolygon. These should fit into your account’s allowed area. The maximum allowed area size is 20,000,000 m2 being 2,000 hectare.api_request_typedefines the type of data. Currently, this only supports"gridded-data"for non-field-based products, and should not be specified for field-based products.argumentsholds a subset of those used for data requests. As a subscription always creates output for the current date and the given GeoJSON areas, this excludes settings such asstart_date,end_date,zippedor some bounding box. So, for"gridded-data"these are only the optional image format ("gtiff"or"netcdf4") and optionalmin_coverage(percentage). Subscriptions for field-based products, and future other subscriptions, may need different arguments or no arguments at all.http_notify(optional) defines an HTTP endpoint that will receive the push notifications.urlis the full URL of the endpoint. To secure the endpoint, consider adding some URL query parameter that is impossible to guess.
Multiple subscriptions for the same api_name can be created.
Note
For field-based processing, each subscription should define a single field, for a single crop. This can be a GeoJSON MultiPolygon if applicable, but different crops should not be combined into a single MultiPolygon in a shared subscription.
HTTP push notifications
A publicly accessible webhook endpoint URL can be configured to receive the following JSON payload:
{
"uuid": "83e9a8a6-8d5a-4b64-8900-b29fb4b78985",
"date": "2021-09-23",
"status": "Ready",
"status_message": null,
"files": [
"https://maps.vandersat.com/api/v2/subscriptions/98d32056-faf3-45be-933c-efc0bcbaf4fe/data/83e9a8a6-8d5a-4b64-8900-b29fb4b78985/2021-09-24.tif/download"
]
"subscription": {
"uuid": "98d32056-faf3-45be-933c-efc0bcbaf4fe",
"api_name": "BIOMASS-PROXY_V2.0_10",
"name": "Biomass Proxy V2 field around Metbrunnen",
"start_date": "2021-09-01",
"end_date": "2024-12-31",
"geojson": {
"type": "Polygon",
"coordinates": [[
[9.080522, 51.697751], [9.082560, 51.696953], [9.082689, 51.699626],
[9.080629, 51.699666], [9.080522, 51.697751]
]]
}
},
}
For gridded data, no result file may be generated if
min_coveragewas not satisfied.The UTF-8 encoded JSON payload is delivered using
POSTandContent-Type: application/json.Both HTTP 302 Found and HTTP 301 Moved Permanently will be followed, but HTTP 301 does not change the URL in the configuration.
HTTPS validation is done using commonly known root certificates. Self-signed certificates are not supported.
statusis"Ready"for success (and for test notifications), or a more specific code otherwise. The specific codes depend on the product that is requested, and may supply additional details instatus_message; please contact support if not. No notifications are sent for intermediate processing states. If the error is resolved, a new notification may be received.
In case of a connection failure or an HTTP error response, a retry mechanism with exponential
backoff will start, retrying after at least 900 * ({retry} - 1)^5 seconds, with a maximum of
7,200 seconds. This delays the retries at least 900, 901, 932, 1,143, 1,924, 4,025 and from then on
2 hours, repeating at most 7 days. To allow for easily fixing a wrong endpoint configuration, the
retry counter is reset when the subscription is changed while notifications are pending; after an
update any pending notifications are submitted as soon as possible.
Note
Notifications are only sent if a fulfillment was created, so when new data is available. If data is somehow delayed (due to acquisition or processing errors) then no notification will be sent. Also, delayed data and retry notifications may be delivered out of sequence: older notifications may be retried after notifications for more recent data. Notifications should be received at least once (unless the 7 days limit is reached), but in rare occasions may be delivered multiple times.
The files can be downloaded by making a request to the given URLs. For "gridded-data" this
is always a single file. If in files the name is just a date, then when downloading the server
may still suggest a more specific name in a so-called Content-Disposition header. Files needs to
be requested using authentication of the same user who created the data
subscription. For example:
GET https://maps.vandersat.com/api/v2/subscriptions/98d32056-faf3-45be-933c-efc0bcbaf4fe/data/83e9a8a6-8d5a-4b64-8900-b29fb4b78985/2021-09-24.tif/download
Note
Data is stored until 7 days after the subscription created the files.
Testing notifications
To test without one’s own server see for example webhook.site and pipedream.com/requestbin.
To test the HTTP notification:
POST https://maps.vandersat.com/api/v2/subscriptions/{uuid}/test-http-notification
This creates a dummy fulfillment and synchronously sends a HTTP push notification for the given
subscription, using its current configuration. The fulfillment date is set to 1900-01-01, hence
no files are actually generated. The response will include details about the first invocation:
{
"uuid": "7d8fcd95-36ae-4989-b760-b9499fe8b36f",
"date": "1900-01-01",
"http_notify_count": 1,
"http_notify_status": "Ready",
"last_http_notify_dt": "2021-09-20T11:00:55.690264",
"last_http_notify_result": "200",
"last_http_notify_message": "The <em>truncated</em> response from your server",
}
The dummy fulfillment has full support for retries, so if the HTTP notification fails then changing the subscription’s settings will retry the test notification as soon as possible. The request above also returns a fulfillment UUID that can be used to get the notification status for retries:
GET https://maps.vandersat.com/api/v2/subscriptions/{uuid}/fulfillment/{fulfillment_uuid}
Getting subscription details
A paginated list of a user’s subscriptions and their uuids can be fetched using:
GET https://maps.vandersat.com/api/v2/subscriptions?page=1&limit=10
To include cancelled subscriptions, add include_cancelled=true:
GET https://maps.vandersat.com/api/v2/subscriptions?include_cancelled=true
The details of a specific subscription can be retrieved using:
GET https://maps.vandersat.com/api/v2/subscriptions/{uuid}
Listing fulfillments and files created by a subscription
The preferred way to get to know about new results is to use the HTTP push notification described above. To get a paginated list of all results created by a subscription:
GET https://maps.vandersat.com/api/v2/subscriptions/{uuid}/fulfillments?page=1&limit=3
{
"fulfillments": [
{
"uuid": "0410ef88-bdeb-4c62-81cb-82551ab3aea1",
"created_dt": "2021-09-24T05:18:45.810523",
"date": "2021-09-23",
"http_notify_count": 1,
"http_notify_status": "Ready",
"last_http_notify_dt": "2021-09-24T05:19:13.000098",
"last_http_notify_result": "200",
"last_http_notify_message": "{ \"success\": true }",
"files": [
"https://maps.vandersat.com/api/v2/subscriptions/98d32056-faf3-45be-933c-efc0bcbaf4fe/data/83e9a8a6-8d5a-4b64-8900-b29fb4b78985/2021-09-24.tif/download"
]
"status": "Ready",
"status_message": null
},
{ },
{ }
],
"total_items": 51
}
To just get
total_items, simply use?page=1&limit=0.Unlike for notifications, when explicitly requesting fulfillment details then
statusmay include intermediate states such as"Scheduled"or"Processing". It is"Ready"for success or a more specific code in case the fulfillment failed. The specific codes depend on the product that is requested, and may supply additional details instatus_message; please contact support if not.This supports paginated results and filtering on date ranges, using query parameters such as
?page=1&limit=10&min_date=2021-09-20. See https://maps.vandersat.com/api/v2 for the full details.
The files can be downloaded by making a request to the given URLs. For "gridded-data" this
is always a single file. Files needs to be requested using authentication
of the same user who created the data subscription. For example:
GET https://maps.vandersat.com/api/v2/subscriptions/98d32056-faf3-45be-933c-efc0bcbaf4fe/data/83e9a8a6-8d5a-4b64-8900-b29fb4b78985/2021-09-24.tif/download
Note
Data is stored until 7 days after the subscription created the files.
Updating a subscription
To update a subscription, issue an HTTP PUT with a JSON payload for the attributes that need
updating. Only the following attributes can be updated:
namestart_date(not for field-based products)end_date(must benullor a future date for field-based products)http_notify
For the attributes in this list, it’s okay to also specify those that do not actually change. When changing the dates, processing and notifications for missing past dates will be triggered.
Like to change the webhook URL, but keep name, start_date and end_date as is:
PUT https://maps.vandersat.com/api/v2/subscriptions/98d32056-faf3-45be-933c-efc0bcbaf4fe
{
"http_notify": {
"url": "https://example.com/my/new/endpoint?my-auth=n3w-s3cr3t"
}
}
Cancelling a subscription
To cancel a subscription:
POST https://maps.vandersat.com/api/v2/subscriptions/{uuid}/cancel
When data is already being generated, one more notification will be received. Cancelled subscriptions cannot be reactivated.
Pausing a subscription
To pause a subscription, cancel it, and create a new subscription with the same settings to resume.
Non-field-based data requests
This section introduces the main endpoints for making data requests for non-field-based products.
Note
For field-based products, only data subscriptions are available.
Note
Not all API endpoints are available for each user. In general, for new accounts even for non-field-based products only data subscriptions will be available. Contact Sales if you have a need for the other endpoints.
Some endpoints are asynchronous, which means that the response contains a UUID with which you can monitor the processing status and download the data. How to download the data once it is ready, will be explained in the Data downloading section. Please be aware that the requested data is stored until 7 days after submitting the request.
Requesting images with xyztiles
The xyztiles endpoint can be used as an XYZ tile service. The returned PNG tiles of 256x256
pixels allow you to easily display the images in a web mapping application, similar to our Viewer.
The endpoint follows the slippy map tilenames specification as defined by OpenStreetMap. For
example, to add the images to a map built with the JavaScript library Leaflet, create the following tileLayer:
var VDS_map = L.tileLayer('https://{s}.maps.vandersat.com/api/v2/products/{api_name}/xyztiles?x={x}&y={y}&z={z}&date={date}', {
maxZoom: 15,
attribution: '© Planet Labs PBC'
});
You will need to specify the path parameters {api_name}, {date}, and {s}. Substitute
{s} with either a, b, or c, which allows for more concurrent requests. Leaflet will
automatically provide the {x}, {y}, and {z} values depending on map location and zoom
level.
Requesting images with gridded-data
The gridded-data endpoint lets you request image data. It is asynchronous and data can be requested in
either GeoTIFF or NetCDF format.
Example request - Get L band soil moisture for May 2018 in GeoTIFFs:
GET https://maps.vandersat.com/api/v2/products/SM-LN_V001_100/gridded-data?lon_min=3&lon_max=7&lat_min=51&lat_max=53&start_date=2018-05-01&end_date=2018-05-31&format=gtiff&zipped=false
Example request - Get land surface temperature for 2017 in zipped NetCDFs:
GET https://maps.vandersat.com/api/v2/products/TEFF_V001_100/gridded-data?lat_min=51&lat_max=53&lon_min=3&lon_max=7&start_date=2017-01-01&end_date=2017-12-31&format=NetCDF4&zipped=true
Requesting point time series with point-time-series
The endpoint point-time-series lets you request a time series for a point
location. It is asynchronous and data can be requested in either CSV or JSON format.
Example request - Get L band soil moisture time series from April 2015 to April 2019 at 51.386 degrees latitude and 4.647 degrees longitude:
GET https://maps.vandersat.com/api/v2/products/TEFF_V001_100/point-time-series?start_time=2015-04-01&end_time=2019-03-31&lat=51.386&lon=4.647&format=csv&avg_window_days=0&include_masked_data=false&climatology=false
Example request - Same request as above, but also include a 15-day moving average, masked data (marked as invalid), the climatology based on the average column, and the derived root zone calculation given T=10 days. Get the output data in JSON format:
GET https://maps.vandersat.com/api/v2/products/TEFF_V001_100/point-time-series?start_time=2015-04-01&end_time=2019-03-31&lat=51.386&lon=4.647&format=json&avg_window_days=15&include_masked_data=true&climatology=true&exp_filter_t=10
Requesting ROI time series with roi-time-series and roi-time-series-sync
The endpoints roi-time-series and roi-time-series-sync let you request time series for a
region of interest (ROI). They are asynchronous and synchronous, respectively, and data can be
requested in either CSV or JSON format.
The asynchronous roi-time-series request will calculate missing ROI time
series values, whereas the synchronous roi-time-series-sync request will
only provide the data that is already calculated. If automatic ROI calculation
is turned on for your account then the sync request is all you should need.
Please contact your Planet point of contact in case of doubt.
It is possible to define ROIs in the Viewer by drawing polygons on the map or
uploading a shapefile (see Managing regions of interest). It is also possible to manage ROIs through the rois
endpoint (see https://maps.vandersat.com/api/v2 for more details).
Example request - Get land surface temperature time series for 2017 as zipped NetCDFs for a previously uploaded ROI.
Using roi-time-series:
GET https://maps.vandersat.com/api/v2/products/TEFF_V001_100/roi-time-series?start_time=2015-04-01&end_time=2019-03-31&roi_id=4325
Using roi-time-series-sync:
GET https://maps.vandersat.com/api/v2/products/TEFF_V001_100/roi-time-series-sync?start_time=2015-04-01&end_time=2019-03-31&roi_id=4325
Note
The climatology parameter determines if a climatology should be supplied together with the requested time series data. It is important to realize that the climatology is determined for the period for which the time series is requested. Data available earlier or later is not taken into account.
Data downloading
Downloading data of data subscriptions
The fulfillment details include a files list that provides full URLs for downloading.
Downloading data of asynchronous requests
The response of an asynchronous request contains a UUID value:
{
"url": "api/v2/api_requests/83833aae-1376-4288-b7c7-3e99af57e29f/status",
"message": "Download request received",
"uuid": "83833aae-1376-4288-b7c7-3e99af57e29f"
}
with which the processing status can be queried:
GET https://maps.vandersat.com/api/v2/api_requests/83833aae-1376-4288-b7c7-3e99af57e29f/status
The response for Scheduled or Processing requests will look like:
{
"scheduled_dt": "2019-06-04T11:59:56.409228",
"percentage": 0,
"processing_status": "Scheduled"
}
while for requests that are Ready, the response will look like:
{
"scheduled_dt": "2019-01-24T17:18:45.810523",
"started_dt": "2019-01-24T17:19:07.409485",
"processing_status": "Ready",
"finished_dt": "2019-01-24T17:19:12.698098",
"data": [
"/api/v2/api-requests/83833aae-1376-4288-b7c7-3e99af57e29f/data/SM-LN_V001_100_2018-12-02T000000_6.481934_52.669720_8.602295_51.710012_83833.tif",
"/api/v2/api-requests/83833aae-1376-4288-b7c7-3e99af57e29f/data/SM-LN_V001_100_2018-12-03T000000_6.481934_52.669720_8.602295_51.710012_83833.tif",
"/api/v2/api-requests/83833aae-1376-4288-b7c7-3e99af57e29f/data/SM-LN_V001_100_2018-12-04T000000_6.481934_52.669720_8.602295_51.710012_83833.tif",
"/api/v2/api-requests/83833aae-1376-4288-b7c7-3e99af57e29f/data/SM-LN_V001_100_2018-12-05T000000_6.481934_52.669720_8.602295_51.710012_83833.tif",
"/api/v2/api-requests/83833aae-1376-4288-b7c7-3e99af57e29f/data/SM-LN_V001_100_2018-12-06T000000_6.481934_52.669720_8.602295_51.710012_83833.tif",
"/api/v2/api-requests/83833aae-1376-4288-b7c7-3e99af57e29f/data/SM-LN_V001_100_2018-12-07T000000_6.481934_52.669720_8.602295_51.710012_83833.tif"
],
"percentage": 100
}
Data of requests that are Ready can be downloaded by making a request to the download endpoint for each of the produced
files. For example:
GET https://maps.vandersat.com/api/v2/api-requests/83833aae-1376-4288-b7c7-3e99af57e29f/data/SM-LN_V001_100_2018-12-02T000000_6.481934_52.669720_8.602295_51.710012_83833.tif/download
Note
Data is stored until 7 days after submitting the request.
File naming
The server sets the Content-Disposition HTTP header to suggest a file name for each download.
This may not always match part of the download URL.
Data subscription files
For field-based products, regardless the download URL as given in the fulfillment details, the
server sets Content-Disposition as:
<api_name>_<subscription_uuid>_<date:YYYY-MM-DD>.<ext>
with:
<api_name>Product name, see API names.<subscription_uuid>Subscription UUID.<date:YYYY-MM-DD>Sensing date, e.g.2022-02-20.<ext>File extension depending on requested data format (tif).
For non-field-based products, see the next section.
gridded-data files
The file naming convention for a gridded-data file is:
<api_name>_<date:YYYY-MM-DDThhmmss>_<lon_min:.6f>_<lat_max:.6f>_<lon_max:.6f>_<lat_min:.6f>_<uuid4_short>.<ext>
or, for zipped files:
<api_name>_<start_date:YYYY-MM-DDThhmmss>_<end_date:YYYY-MM-DDThhmmss>_<lon_min:.6f>_<lat_max:.6f>_<lon_max:.6f>_<lat_min:.6f>_<uuid4_short>.<ext>
with:
<api_name>Product name, see API names.<date:YYYY-MM-DDThhmmss>Sensing date, e.g.2018-12-07T000000.<start_date:YYYY-MM-DDThhmmss>Sensing date of first image, e.g.2018-12-07T000000.<end_date:YYYY-MM-DDThhmmss>Sensing date of last image, e.g.2018-12-07T000000.<lon_min:.6f>Minimum longitude rounded to 6 decimals.<lat_max:.6f>Maximum latitude rounded to 6 decimals.<lon_max:.6f>Maximum longitude rounded to 6 decimals.<lat_min:.6f>Minimum latitude rounded to 6 decimals.<uuid4_short>First 5 characters of the API request UUID.<ext>File extension depending on requested data format (tif,nc, orzip)
Note
Although a datetime is specified, currently only the sensing date is reflected in date, start_date,
and end_date. The time is always set to T000000 and should be ignored.
time-series files
The file naming convention for a point-time-series file is:
ts_<api_name>_<start_date:YYYY-MM-DDThhmmss>_<end_date:YYYY-MM-DDThhmmss>_<lon:.6f>_<lat:.6f>_<uuid4_short>.<ext>
and for a roi-time-series file:
roi-ts_<api_name>_<start_date:YYYY-MM-DDThhmmss>_<end_date:YYYY-MM-DDThhmmss>_<roi_id:d>_<uuid4_short>.<ext>
with:
<api_name>Product name, see API names.<start_date:YYYY-MM-DDThhmmss>Sensing date of first time point, e.g.2018-12-07T000000.<end_date:YYYY-MM-DDThhmmss>Sensing date of last time point, e.g.2018-12-07T000000.<lon:.6f>Longitude rounded to 6 decimals.<lat:.6f>Latitude rounded to 6 decimals.<roi_id:d>Identifier of the region of interest (integer value).<uuid4_short>First 5 characters of the API request UUID.<ext>File extension depending on requested data format (csvorjson)
Note
Although a datetime is specified, currently only the sensing date is reflected in start_date
and end_date. The time is always set to T000000 and should be ignored.
Data sample
API endpoint |
File format |
Download |
|---|---|---|
|
GeoTIFF |
|
|
NetCDF |
|
|
CSV |
|
|
JSON |
|
|
CSV |
|
|
JSON |
Python script examples
Data subscriptions
The Python script below shows how to create data subscriptions and
download image data with the subscriptions endpoint. Rather than using the recommended webhook
for HTTP push notifications, this example relies on polling for new fulfillments. To easily test
with webhook notifications instead, see for example webhook.site and
pipedream.com/requestbin.
To run the script, create a Python 3.6+ environment with the requests
and requests-toolbelt libraries installed.
Customize the script by changing AUTH, OUTPUT_FOLDER and the data for new subscriptions. For
a field-based product such as "api_name": "BIOMASS-PROXY_V2.0_10", also remove attributes api_request_type
and arguments.
import glob
import logging
import math
import os
import time
from typing import Iterator, List
from requests import Response
from requests_toolbelt.downloadutils import stream
from requests_toolbelt.exceptions import StreamingError
from requests_toolbelt.sessions import BaseUrlSession
# Change these values
AUTH = ("username", "password")
OUTPUT_FOLDER = "downloads"
def create_subscription(session: BaseUrlSession) -> dict:
"""Create a subscription and return it."""
data = {
"name": "Soil Moisture V4 area around Metbrunnen, Germany",
"api_name": "SM-SMAP-L-DESC_V4.0_100",
# Do not define `api_request_type` for field-based products
"api_request_type": "gridded-data",
# Do not define `arguments` for field-based products
"arguments": {
"format": "gtiff",
"min_coverage": 30,
},
"start_date": "2022-01-01",
"end_date": "2022-01-15",
"geojson": {
"type": "Polygon",
"coordinates": [[
[9.080522, 51.697751], [9.082560, 51.696953], [9.082689, 51.699626],
[9.080629, 51.699666], [9.080522, 51.697751],
]],
},
# See also https://webhook.site and https://requestb.in
# "http_notify": {
# "url": "https://example.com/my/endpoint?my-auth=s3cr3t"
# }
}
result = session.post(url="subscriptions", json=data).json()
logging.info(f"Created subscription: {result}")
return result
def get_all_pages(
session: BaseUrlSession, url: str, page_size: int = 50
) -> Iterator[dict]:
"""Get a generator to fetch paginated API results page by page."""
params = {"page": 1, "limit": page_size}
first_page = session.get(url=url, params=params).json()
yield first_page
page_count = math.ceil(first_page["total_items"] / page_size)
for params["page"] in range(2, page_count + 1):
next_page = session.get(url=url, params=params).json()
yield next_page
def get_subscriptions(session: BaseUrlSession):
"""Fetch the details of all subscriptions."""
for page in get_all_pages(session, url="subscriptions"):
for subscription in page["subscriptions"]:
logging.info(f"Existing subscription: {subscription}")
def get_subscription(session: BaseUrlSession, subscription_uuid: str) -> dict:
"""Fetch the details of a single subscription."""
subscription = session.get(url=f"subscriptions/{subscription_uuid}").json()
logging.info(f"Fetched subscription: {subscription}")
return subscription
def download_files(session: BaseUrlSession, urls: List[str], output_folder: str):
"""Save URL(s) using the Content-Disposition header's file name."""
os.makedirs(output_folder, exist_ok=True)
for url in urls:
# Find existing files: assume the Content-Disposition header
# uses the same name as the URL or at most adds a prefix, so a
# wildcard search suffices. A real application should not rely
# on that: use the fulfillment date or UUID to track handling.
name = url.split("/")[-2]
existing = glob.glob(os.path.join(output_folder, f"*{name}"))
if existing:
logging.info(f"Skipped existing file: name={existing[0]}; url={url}")
continue
r = session.get(url=url, stream=True)
try:
filename = stream.stream_response_to_file(r, path=output_folder)
logging.info(f"Downloaded file: name={filename}")
except StreamingError as e:
logging.error(f"Failed to download file; error={str(e)}; url={url}")
def handle_fulfillment(
session: BaseUrlSession, subscription_uuid: str, fulfillment: dict
) -> bool:
"""Handle a single fulfillment, like from an HTTP notification."""
if fulfillment["status"] == "Ready":
output_folder = os.path.join(OUTPUT_FOLDER, subscription_uuid)
# Even if Ready, `files` may be empty, like if the requested
# min_coverage was not met for non-field-based gridded-data
download_files(session, urls=fulfillment["files"], output_folder=output_folder)
# When handling an HTTP push notification only statuses Ready and
# Error are expected, but the polling in this example may also
# yield intermediate statuses such as Scheduled and Processing
return fulfillment["status"] in ("Ready", "Error")
def get_subscription_fulfillments(
session: BaseUrlSession, subscription_uuid: str
) -> bool:
"""Get fulfillments; not needed when using HTTP notifications."""
url = f"subscriptions/{subscription_uuid}/fulfillments"
pending = None
for page in get_all_pages(session, url=url):
logging.info(f"Fetched page of fulfillments: result={page}")
for fulfillment in page["fulfillments"]:
pending = pending or not handle_fulfillment(
session, subscription_uuid=subscription_uuid, fulfillment=fulfillment
)
# True if fulfillment(s) found and all were handled, False otherwise
return not pending
def response_hook(response: Response, *_args, **_kwargs):
"""Hook to get detailed error details from the response body."""
if response.status_code >= 400:
logging.error(
f"Error invoking API: url={response.url}; code={response.status_code}; "
f"reason={response.reason}; message={response.text}"
)
exit(response.status_code)
if __name__ == "__main__":
logging.basicConfig(
level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s"
)
session = BaseUrlSession(base_url="https://maps.vandersat.com/api/v2/")
session.hooks["response"] = [response_hook]
session.auth = AUTH
try:
# List the current subscriptions
get_subscriptions(session)
# Create a new subscription and get its UUID
uuid = create_subscription(session)["uuid"]
# Fetch the subscription; not very useful in this example
get_subscription(session, subscription_uuid=uuid)
# NOTE: polling is not recommended, use `http_notify` instead
while True:
# Get the fulfillments and download the result files
if get_subscription_fulfillments(session, subscription_uuid=uuid):
break
logging.info("Not done yet; sleeping 10 minutes")
time.sleep(10 * 60)
finally:
session.close()
Data requests
The Python script below is a minimal example demonstrating how to request and download image data with the
gridded-data endpoint. Note that most accounts should use data subscriptions
instead, which has its own example above.
To run the script, please create a Python 3.6+ environment with the requests library installed. Also customize the script by changing the username and password, product name, and output folder.
import os
import time
import json
import requests
BASE_URL = 'https://maps.vandersat.com/api/v2'
def request_data(url, params, auth):
r = requests.get(url, params=params, auth=auth)
return json.loads(r.content)['uuid']
def get_status(uuid, auth):
url = f'{BASE_URL}/api-requests/{uuid}/status'
r = requests.get(url, auth=auth)
return json.loads(r.content)
def wait_until_ready(uuid, auth):
percentage = get_status(uuid, auth)['percentage']
while percentage < 100:
print(f"Processing status... {percentage} percent")
time.sleep(5)
percentage = get_status(uuid, auth)['percentage']
print('Finished')
def get_filenames(uuid, auth):
wait_until_ready(uuid, auth)
slugs = get_status(uuid, auth)['data']
filenames = []
for slug in slugs:
filenames.append(os.path.basename(slug))
return filenames
def download_data(filenames, output_folder, auth):
for filename in filenames:
url = f'{BASE_URL}/api-requests/{uuid}/data/{filename}/download'
r = requests.get(url, stream=True, auth=auth)
path = os.path.join(output_folder, filename)
with open(path, 'wb') as f:
for chunk in r.iter_content(1024):
f.write(chunk)
if __name__ == '__main__':
# Change these values
OUTPUT_FOLDER = ''
API_NAME = 'SM-AMSR2-C1N-DESC_V003_100'
AUTH = ('username', 'password')
# Define API request URL and parameters
url = f'{BASE_URL}/products/{API_NAME}/gridded-data'
params = {
'lat_min': 51.5,
'lat_max': 52.5,
'lon_min': 4,
'lon_max': 5,
'start_date': '2020-05-01',
'end_date': '2020-05-03',
'format': 'gtiff',
'zipped': 'false'
}
# Request and download data
uuid = request_data(url, params=params, auth=AUTH)
filenames = get_filenames(uuid, auth=AUTH) # wait for processing to finish
download_data(filenames, output_folder=OUTPUT_FOLDER, auth=AUTH)
This script can be easily modified to request data from other asynchronous endpoints. Below is an
abbreviated example for the point-time-series endpoint.
# ...
if __name__ == '__main__':
# Change these values
# ...
# Define API request URL and parameters
url = f'{BASE_URL}/products/{API_NAME}/point-time-series'
params = {
'lat': 52,
'lon': 4.5,
'start_time': '2020-01-01',
'end_time': '2020-05-01',
'avg_window_days': 10,
'avg_window_direction': 'center',
'climatology': 'true',
'format': 'csv'
}
# Request and download data
# ...