{ "cells": [ { "cell_type": "markdown", "id": "cb836184-31c3-4113-a89d-d93548f8bded", "metadata": {}, "source": [ "# Building Intake-ESM datastores of ACCESS model output\n", "\n", "This tutorial will demonstrate how users can build Intake-ESM datastores for their ACCESS model runs using the `access-nri-intake` Python package.\n", "\n", "The ACCESS-NRI Intake Catalog consists of a set of intake-esm datastores, where a datastore has the result for one experiment. This tutorial focusses on building one datastore for one experiment. In general, it's assumed most user's wont build their own intake catalog, but could have several datastores for several experiments.\n", "\n", "Note: The ACCESS-NRI Intake Catalog is curated by ACCESS-NRI with further information on its documentation website and this ACCESS-Hive page. It contains a set of model output shared amongst the ACCESS-NRI working groups, including CMIP6 results and ACCESS-OM2 (as typically used in Cosima Recipes). \n", "\n", "We demonstrate two methods here - first, the more involved method, which was the only way of building a datastore prior to the `access-nri-intake v1.1.1` release, and secondly, the convenience method added with that release.\n", "\n", "In most cases, the convenience method will be preferable, but the more involved method can be useful as it affords a greater degree of control over the process.\n", "\n", "----\n", "# 1. The more involved method" ] }, { "cell_type": "markdown", "id": "4bb64c0d-bb8f-4989-9002-44d5518b7cae", "metadata": {}, "source": [ "A builder class is used for creating Intake-ESM datastores for each flavour of model output. Classes for ACCESS model outputs are available in the `builders` submodule of the `access_nri_intake` Python package. " ] }, { "cell_type": "markdown", "id": "d7af029b-68e1-4b43-993e-5dc13de4fdc7", "metadata": {}, "source": [ "We can list the available builders:" ] }, { "cell_type": "code", "execution_count": 1, "id": "a89c7407-8fa7-4021-81e0-487ff8fbbf82", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['AccessOm2Builder',\n", " 'AccessOm3Builder',\n", " 'Mom6Builder',\n", " 'AccessEsm15Builder',\n", " 'AccessCm2Builder',\n", " 'ROMSBuilder',\n", " 'WoaBuilder']" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from access_nri_intake.source import builders\n", "\n", "builders.__all__" ] }, { "cell_type": "markdown", "id": "a372ba48-2ac3-4c5e-86f7-564fd6c01b69", "metadata": {}, "source": [ "Because we're looking at ACCESS-OM2 output, we'll typically use the `AccessOm2Builder` class. For other models, use the correct Builder" ] }, { "cell_type": "markdown", "id": "bc200ca9-5cba-4413-a550-bd0a4f1b54bd", "metadata": {}, "source": [ "## Building the datastore" ] }, { "cell_type": "markdown", "id": "54ab0f35-68b2-40dc-a298-407bd7adc700", "metadata": {}, "source": [ "Building the Intake-ESM catalog should be as simple as passing the model run base output directory to the Builder and calling `.build()`. The build is parallelized (without dask) so will be faster if you throw more resources at it. The following was run using an Large `normal` ARE instance (12 cpus)." ] }, { "cell_type": "code", "execution_count": 2, "id": "db2a83f3-501d-48df-944d-67ba306bbfa5", "metadata": { "scrolled": true, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n", "/g/data/xp65/public/apps/med_conda/envs/analysis3-25.08/lib/python3.11/site-packages/access_nri_intake/source/utils.py:241: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.\n", " warnings.warn(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 5.91 s, sys: 957 ms, total: 6.87 s\n", "Wall time: 1min 11s\n" ] } ], "source": [ "%%time\n", "\n", "PATH = \"/g/data/ik11/outputs/access-om2/1deg_jra55_iaf_omip2_cycle6\" #Some example ACCESS-OM2 output\n", "\n", "builder = builders.AccessOm2Builder(\n", " path=PATH\n", ").build()" ] }, { "cell_type": "markdown", "id": "d2510acd-931a-4d0a-b91a-9426190f4e57", "metadata": { "scrolled": true, "tags": [] }, "source": [ "If using MOM6 standalone, just change the Builder used, e.g.:\n", "``` python\n", "PATH = \"/g/data/ol01/outputs/mom6-panan/panant-01-zstar-ACCESSyr2\" #Some example MOM6-SIS2 output\n", "\n", "builder = builders.Mom6Builder(\n", " path=PATH\n", ").build()\n", "```" ] }, { "cell_type": "markdown", "id": "e61775ac-a973-4cb0-a87f-b6a695b9e00d", "metadata": {}, "source": [ "The previous cell builds the Intake-ESM datastore in memory. We'll want to save it somewhere so we can reuse and share it. The following cell will create two new files (a `.json` and `.csv` file) in your current work directory. These files are how Intake-ESM datastores are stored on disk." ] }, { "cell_type": "code", "execution_count": 3, "id": "7f4ed770-df35-4467-ab5e-ce9f8b8ec075", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Successfully wrote ESM catalog json file to: file:///home/603/as2285/mydatastore.json\n" ] } ], "source": [ "builder.save(\n", " name=\"mydatastore\", \n", " description=\"An example datastore for ACCESS-OM2 ryf9091\",\n", ")" ] }, { "cell_type": "markdown", "id": "1acbbe16-04a3-4a7c-bc30-3f2a4ec81fc9", "metadata": { "tags": [] }, "source": [ "## Using your datastore" ] }, { "cell_type": "markdown", "id": "e01ccb09-1783-4856-9648-f8ddcd863a80", "metadata": {}, "source": [ "Now we can use our Intake-ESM datastore to query and load the model data - see also the Intake-ESM documentation [here](https://intake-esm.readthedocs.io/en/stable/index.html).\n", "\n", "We can load the datastore directly using `intake`." ] }, { "cell_type": "code", "execution_count": 4, "id": "029dd740-fa73-4403-b756-2dff8fd0a32f", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
mydatastore catalog with 11 dataset(s) from 2013 asset(s):
| \n", " | unique | \n", "
|---|---|
| filename | \n", "1473 | \n", "
| path | \n", "2013 | \n", "
| file_id | \n", "11 | \n", "
| frequency | \n", "4 | \n", "
| start_date | \n", "855 | \n", "
| end_date | \n", "854 | \n", "
| variable | \n", "338 | \n", "
| variable_long_name | \n", "306 | \n", "
| variable_standard_name | \n", "57 | \n", "
| variable_cell_methods | \n", "6 | \n", "
| variable_units | \n", "128 | \n", "
| realm | \n", "2 | \n", "
| derived_variable | \n", "0 | \n", "
demo_datastore catalog with 11 dataset(s) from 2013 asset(s):
| \n", " | unique | \n", "
|---|---|
| filename | \n", "1473 | \n", "
| path | \n", "2013 | \n", "
| file_id | \n", "11 | \n", "
| frequency | \n", "4 | \n", "
| start_date | \n", "855 | \n", "
| end_date | \n", "854 | \n", "
| variable | \n", "338 | \n", "
| variable_long_name | \n", "11 | \n", "
| variable_standard_name | \n", "11 | \n", "
| variable_cell_methods | \n", "11 | \n", "
| variable_units | \n", "131 | \n", "
| realm | \n", "2 | \n", "
| derived_variable | \n", "0 | \n", "
demo_datastore catalog with 11 dataset(s) from 2013 asset(s):
| \n", " | unique | \n", "
|---|---|
| filename | \n", "1473 | \n", "
| path | \n", "2013 | \n", "
| file_id | \n", "11 | \n", "
| frequency | \n", "4 | \n", "
| start_date | \n", "855 | \n", "
| end_date | \n", "854 | \n", "
| variable | \n", "338 | \n", "
| variable_long_name | \n", "11 | \n", "
| variable_standard_name | \n", "11 | \n", "
| variable_cell_methods | \n", "11 | \n", "
| variable_units | \n", "131 | \n", "
| realm | \n", "2 | \n", "
| derived_variable | \n", "0 | \n", "