Chapter 3. CPL7 Implementation
Time Management
Driver Clocks
The CCSM4 driver manages the main clock in the system. That clock advances at the shortest coupling period and uses alarms to trigger component coupling and other events. In addition, the driver maintains a clock that is associated with each component. The driver's component clocks have a timestep associated with the coupling period of that component. The main driver clock and the component clocks in the driver advance in a coordinated manor and are always synchronized. The advancement of time is managed as follows in the main run loop. First, the main driver clock advances one timestep and the component clocks are advanced in a synchronous fashion. The clock time represents the time at the end of the next model timestep. Alarms may be triggered at that timestep to call the the atmosphere, land, sea ice, land ice, or ocean run methods. If a component run alarm is triggered, the run method is called and the driver passes that component's clock to that component. The component clock contains information about the length of the next component integration and the expected time of the component at the end of the integration period.
Generally, the component models have indepedent time management software. When a component run method is called, the component must advance the proper period and also check that their internal clock is consistent with the coupling clock before returning to the driver. The clock passed to the component by the driver contains this information. Component models are also responsible for making sure the coupling period is consistent with their internal timestep. History files are managed independently by each component, but restart files are coordinated by the driver (see the Section called History and Restart Files).
The clocks in CCSM4 are based on the ESMF clock datatype are are supported in software by either an official ESMF library or by software included in CCSM called esmf_wrf_timemgr. The esmf_wrf_timemgr software is a much simplified Fortran implementation of a subset of the ESMF time manager interfaces.
The Driver Time Loop
The driver time loop is hardwired to sequence the component models in a specific way to meet scientific requirements and to provide the maximum possible concurrency of work. The results of the model integration are not dependent on the processor layout of the components.
The driver is configured to couple the atmosphere, land, and sea ice models using the same coupling frequency, while the ocean model may be coupled at the same or a lower frequency. To support this, the driver performs temporal averaging of coupling inputs to the ocean and computes the surface ocean albedo at the higher frequency. No averaging is done for other component interactions; land and sea ice surface albedos are calculated within their respective components. Additional functionality could be added to support alternative coupling schemes, but this would require careful handling of the interaction between albedo calculations and atmospheric radiation.
The coupler processors (PEs) manage data exchange between components. Tasks include deriving fields, transferring data to/from components, and executing component models. The time loop proceeds as follows:
The driver clock is advanced first. Initial data is computed on the coupler PEs. Ocean data is rearranged to ocean PEs. Land data is rearranged to land PEs. Ice data is rearranged to ice PEs. The ocean model is run. The ice model is run. The land model is run. Ocean inputs are accumulated; atmosphere/ocean fluxes are computed. Land data is sent back to the coupler. Ice data is sent back to the coupler. Atmospheric forcing data is computed. Atmospheric data is rearranged to atmosphere PEs. The atmosphere model is run. Atmospheric data is returned to the coupler. Ocean data is returned to the coupler. The loop returns.
Within this loop, operations like mapping, merging, diagnostics, area corrections, and flux computations are overlapped with component execution when possible. The land ice model is not included in the above sequence as its coupling design is still under development, and it will likely operate on a much longer timescale.
The sequencing strategy in CCSM4 has evolved over two decades to ensure conservation of mass and heat, minimize lags, and maintain system stability. This design aligns with concurrency limitations discussed here. Land and sea ice models always run before the atmospheric model, while the coupler and ocean models can run concurrently with other components. The atmosphere, land, sea ice, and atmosphere/ocean flux steps are not lagged; however, the ocean state is lagged by one ocean coupling period. See more details here.
You can reduce the ocean lag by setting the namelist variable ocean_tight_coupling to true. This changes the step where ocean data is sent back to the coupler so it occurs earlier in the loop — before the flux computation — reducing the lag by one atmosphere coupling period. However, it also limits the ability for the ocean and atmosphere models to run concurrently. This setting is most useful when the ocean coupling frequency matches the other components.
Coupling Frequency
In the current implementation, the coupling period must be identical for the atmosphere, sea ice, and land components. The ocean coupling period can be the same or greater. All coupling periods must be multiple integers of the smallest coupling period and should evenly divide a day of time.
The coupling periods are set in the driver namelist for each component via variables called something like atm_cpl_dt and atm_cpl_offset. The units of these inputs are seconds. The coupler template file derives these values from CCSM4 script variable names like ATM_NCPL which is the coupling frequency per day. The *_cpl_dt input specifies the coupling period in seconds and the *_cpl_offset input specifies the temporal offset of the coupling time relative to initial time. An example of an offset might be a component that couples every six hours. That would normally be on the 6th, 12th, 18th, and 24th hour of every day. An offset of 3600 seconds would change the coupling to the 1st, 7th, 13th, and 19th hour of every day. The offsets cannot be larger than the coupling period and the sign of the offsets is such that a positive offset shifts the alarm time forward by that number of seconds. The offsets are of limited use right now because of the limitations of the relative coupling frequencies.
Offsets play an important role in supporting concurrency. There is an offset of the smallest coupling period automatically introduced in every coupling run alarm for each component clock. This is only mentioned because it is an important but subtle point of the implementation and changing the coupling offset could have an impact on concurrency performance. Without this explicit automatic offset, the component run alarms would trigger at the end of the coupling period. This is fine for components that are running at the shortest coupling period, but will limit the ability of models to run concurrently for models that couple at longer periods. What is really required for concurrency is that the run alarm be triggered as early as possible and that the data not be copied from that component to the coupler pes until the coupling period has ended. The detailed implementation of this feature is documented in the seq_timemgr_mod.F90 file and the impact of it for the ocean coupling is implemented in the ccsm_driver.F90 code via use of the ocnrun_alarm and ocnnext_alarm variables.