CESM1.1 has new capability to run multiple component instances under one model executable. The only caveat to this usage is that if N multiple instances of any one active component is used, then N multiple instances of ALL active components are required. More details are discussed below. The primary motivation for this development was to be able to run an ensemble kalman filter for data assimilation and parameter estimation (e.g. UQ). However, it also provides you with the ability to run a set of experiments within a single CESM executable where each instance can have a different namelist, and have all the output go to one directory.

In the following an F compset will be used as an illustration. Utilizing the multiple instance code involves the following steps:

  1. create the case

    > create_newcase -case Fmulti -compset F  -res ne30_g16 -mach hopper
> cd Fmulti

  2. Lets assume the following out of the box pe-layout for hopper:

    <entry id="NTASKS_ATM"   value="128"  />
<entry id="NTHRDS_ATM"   value="1"  />
<entry id="ROOTPE_ATM"   value="0"  />
<entry id="NINST_ATM"   value="1"  />
<entry id="NINST_ATM_LAYOUT"   value="concurrent"  />

<entry id="NTASKS_LND"   value="128"  />
<entry id="NTHRDS_LND"   value="1"  />
<entry id="ROOTPE_LND"   value="0"  />
<entry id="NINST_LND"   value="1"  />
<entry id="NINST_LND_LAYOUT"   value="concurrent"  />

<entry id="NTASKS_ICE"   value="128"  />
<entry id="NTHRDS_ICE"   value="1"  />
<entry id="ROOTPE_ICE"   value="0"  />
<entry id="NINST_ICE"   value="1"  />
<entry id="NINST_ICE_LAYOUT"   value="concurrent"  />

<entry id="NTASKS_OCN"   value="128"  />
<entry id="NTHRDS_OCN"   value="1"  />
<entry id="ROOTPE_OCN"   value="0"  />
<entry id="NINST_OCN"   value="1"  />
<entry id="NINST_OCN_LAYOUT"   value="concurrent"  />

<entry id="NTASKS_GLC"   value="128"  />
<entry id="NTHRDS_GLC"   value="1"  />
<entry id="ROOTPE_GLC"   value="0"  />
<entry id="NINST_GLC"   value="1"  />
<entry id="NINST_GLC_LAYOUT"   value="concurrent"  />

<entry id="NTASKS_CPL"   value="128"  />
<entry id="NTHRDS_CPL"   value="1"  />
<entry id="ROOTPE_CPL"   value="0"  />

    For an F compset, only atm, lnd, rof and cice are prognostic components, whereas the ocn is a data components and glc is a stub component. Lets say we want to run 2 instances of CAM in this experiment. The current implementation of multi-instances will also require you to run 2 instances of CLM. However, you have the flexibility to run either 1 or 2 instances of DOCN (we can ignore glc since it does not do anything in this compset). To run 2 instances of CAM, CLM and CICE, all you need to do is to change NINST_ATM, NINST_LND and NINST_ICE above from 1 to 2. This will result in the following env_mach_pes.xml file:

    <entry id="NTASKS_ATM"   value="128"  />
<entry id="NTHRDS_ATM"   value="1"  />
<entry id="ROOTPE_ATM"   value="0"  />
<entry id="NINST_ATM"   value="2"  />
<entry id="NINST_ATM_LAYOUT"   value="concurrent"  />

<entry id="NTASKS_LND"   value="128"  />
<entry id="NTHRDS_LND"   value="1"  />
<entry id="ROOTPE_LND"   value="0"  />
<entry id="NINST_LND"   value="2"  />
<entry id="NINST_LND_LAYOUT"   value="concurrent"  />

<entry id="NTASKS_ICE"   value="128"  />
<entry id="NTHRDS_ICE"   value="1"  />
<entry id="ROOTPE_ICE"   value="0"  />
<entry id="NINST_ICE"   value="2"  />
<entry id="NINST_ICE_LAYOUT"   value="concurrent"  />

<entry id="NTASKS_OCN"   value="128"  />
<entry id="NTHRDS_OCN"   value="1"  />
<entry id="ROOTPE_OCN"   value="0"  />
<entry id="NINST_OCN"   value="2"  />
<entry id="NINST_OCN_LAYOUT"   value="concurrent"  />

<entry id="NTASKS_GLC"   value="128"  />
<entry id="NTHRDS_GLC"   value="1"  />
<entry id="ROOTPE_GLC"   value="0"  />
<entry id="NINST_GLC"   value="1"  />
<entry id="NINST_GLC_LAYOUT"   value="concurrent"  />

<entry id="NTASKS_CPL"   value="128"  />
<entry id="NTHRDS_CPL"   value="1"  />
<entry id="ROOTPE_CPL"   value="0"  />

    As a result of this, you will have 2 instances of CAM, CLM and CICE (prescribed) and DOCN, each running concurrently on 64 MPI tasks.

  3. A separate user_nl_xxx_NNNN file (where NNNN is the number of the component instances) will be generated when cesm_setup is called. In particular, calling cesm_setup with the above env_mach_pes.xml file will result in the following user_nl_* files in $CASEROOT

    user_nl_cam_0001
user_nl_cam_0002
user_nl_cice_0001
user_nl_cice_0002
user_nl_clm_0001
user_nl_clm_0002
user_nl_cpl
user_nl_docn_0001
user_nl_docn_0002

    and the following *_in_* files and *txt* files in $CASEROOT/CaseDocs:

    atm_in_0001
atm_in_0002
docn.streams.txt.prescribed_0001
docn.streams.txt.prescribed_0002
docn_in_0001
docn_in_0002
docn_ocn_in_0001
docn_ocn_in_0002
drv_flds_in
drv_in
ice_in_0001
ice_in_0002
lnd_in_0001
lnd_in_0002

    The namelist for each component instance can be modified by changing the corresponding user_nl_xxx_NNNN file for that component instance. Modifying the user_nl_cam_0002 will result in the namelist changes you put in to be active ONLY for instance 2 of CAM. To change the DOCN stream txt file instance 0002, you should place a copy of docn.streams.txt.prescribed_0002 in $CASEROOT with the name user_docn.streams.txt.prescribed_0002 and modify it accordlingly.

It is also important to stress the following points:

  1. Different component instances can ONLY differ by differences in namelist settings - they are ALL using the same model executable.

  2. Only 1 coupler component is supported in the CESM1.1 multiple instance implementation.

  3. user_nl_* files once they are created by cesm_setup ARE NOT removed by calling cesm_setup -clean. The same is true for Macros files.

  4. In general, you should run multiple instances concurrently (the default setting in env_mach_pes.xml). The serial setting is only for EXPERT USERS in upcoming development code implementations.