API guidance

Introduction

This page provides further details on the Data Portal’s application programming interface (API), for users who wish to write code to access information from the Data Portal site (rather than downloading through the user interface).

The intention of this guide is to provide an overview of the functionality and not to serve as a tutorial for using an API. To fully utilise the API functionality, general programming skills and knowledge of SQL are needed. 

The guidelines for using API 

In a continued effort to enhance customer experience, we are not only conducting maintenance to address your API-related queries we strongly suggest using the below guidelines to make your use of the data portal API efficient. If the data consumer is not following the best practise for using the API and therefore overloading the server which causes performance issues then we reserve the right to block that IP address.

The rate limits are as follows: 

• CKAN API:  It is recommended to limit requests to a maximum one request per second.
• Datastore API:  The Datastore API is designed to query and retrieve data records from datasets. While it's a powerful feature, querying large datasets can be resource-intensive. Therefore, to ensure that the server's resources are not excessively strained, we recommend limiting requests to a maximum of two requests per minute.

Here are some guidelines for efficient data retrieval:

1: Avoid frequent data fetching
If you are fetching data frequently to check whether it has changed, it is recommended to use the resource_show endpoint to obtain information about the resource modification date. 
 /resource_show?id=<resource_id>

By comparing the modification date of the resource, you can determine whether you need to fetch updated data and use Datastore API to fetch the resource if you need to.

2: Utilize the Datastore API 
If you need to consume data records from a CKAN resource, consider using the Datastore API for efficient retrieval and filtering of data.

e.g. using datastore_search  API
/datastore_search?resource_id=resource_id&filters={"column_name": "filter_value"}

e.g. using datastore_search_sql API
/datastore_search_sql?SELECT * FROM “<resource_id>” WHERE “column_name” = ‘filter_value’

This approach allows you to query and retrieve specific data records based on filters, minimizing the data transfer and processing overhead.
 

List of supported API calls  

The URL for the calls is: https://api.neso.energy/api/3/action/

It is worth noting that the naming conventions in the API differ from those in the Data Portal, the table below shows how the terms relate: 

The Data Portal has been built using the CKAN platform, the following subset of CKAN endpoints are supported on the Data Portal: 

Querying data via SQL 

The datastore_search_sql action allows a user to search data in a resource or connect multiple resources with join expressions. The underlying SQL engine is the PostgreSQL engine.  

Example:  

/datastore_search_sql? sql=SELECT”field id” FROM “resource id”

Field ID 

Field IDs must be in double quotes and can be found by previewing the table view on the explore section of any resource.  

For example: 

Image

Resource ID 

The resource ID can be obtained by accessing the “API” button from any resource page and will be in the “Querying” section under the “Query example (via SQL statement)” or from the bottom of the dataset page under “Integrate this dataset using cURL” section. The resource ID must also be represented in double quotes.  

Image

Examples: