The development of an efficient algorithm for teleseismic wave field modeling is valuable for calculating the gradients of the misfit function (termed “misfit gradients”) or Fréchet derivatives when the teleseismic waveform is used for adjoint tomography. Here, we introduce an element-by-element parallel spectral-element method (EBE-SEM) for the efficient modeling of teleseismic wave field propagation in a reduced geology model. Under the plane-wave assumption, the frequency–wavenumber (FK) technique is implemented to compute the boundary wave field used to construct the boundary condition of the teleseismic wave incidence. To reduce the memory required for the storage of the boundary wave field for the incidence boundary condition, a strategy is introduced to efficiently store the boundary wave field on the model boundary. The perfectly matched layers absorbing boundary condition (PML ABC) is formulated using the EBE-SEM to absorb the scattered wave field from the model interior. The misfit gradient can easily be constructed in each time step during the calculation of the adjoint wave field. Three synthetic examples demonstrate the validity of the EBE-SEM for use in teleseismic wave field modeling and the misfit gradient calculation.

The increasing demand for high-resolution imaging of deep lithospheric structures requires the utilization of teleseismic datasets for waveform inversion. Teleseismic waves provide tremendous amounts of information for the detection of crustal and upper mantle structures (Rondenay, 2009). Over the past forty years, many techniques have been developed to analyze teleseismic wave datasets, including receiver function analysis (Langston, 1977; Kind et al., 2012), teleseismic wave travel-time tomography based on ray theory (Zhang et al., 2011), teleseismic migration (Shragge et al., 2006), and teleseismic scattering tomography (Roecker et al., 2010; Tong et al., 2014a). To achieve the high-resolution imaging of lithospheric structures, the adjoint-state method has become the state-of-the-art technique for teleseismic wave imaging (Tong et al., 2014a; Monteiller et al., 2015).

The adjoint tomography technique constructs the Fréchet derivatives of the objective function with respect to the model parameters by numerically solving the full seismic wave equation twice if the forward wave fields are stored on a computer disk at every given time interval (Tromp et al., 2005; Liu and Tromp, 2006; Fichtner, 2011). Adjoint tomography has been successfully implemented to investigate crustal (Tape et al., 2009) and continental subsurface heterogeneity (Chen et al., 2015). Seismic adjoint waveform tomography, which has a greater resolution than the ray-based travel-time tomography for the same dense seismic ray coverage (Liu and Gu, 2012), is able to image small-scale (half of the minimum wavelength) heterogeneity (Virieux and Operto, 2009). The main drawback of the adjoint tomography is its large computational burden. The computational requirement is linearly related to the number of earthquakes used for the tomographic inversion and the iterations required by the optimization technique. For a typical local-scale model, such as the southern California region, several thousand 3-D full-wave field simulations are required to perform an adjoint tomography inversion (Tape et al., 2007, 2009).

Because most earthquakes occur in the crust, and seismic rays cannot easily
illuminate the deep lithosphere in local seismic tomography, imaging the
deep lithospheric structures can be difficult (Tong et al., 2014b, c).
Increasing the model size enables more deep reflections and refractions to
be included in the inversion dataset; as a result, deep structures can be
inverted by fitting these reflected and refracted waveforms (Chen et al.,
2015). However, for continental-scale models, it is difficult to invert
short-period seismic data on a standard computing cluster, such as 1–2

To reduce the amount of computation involved in solving the full seismic wave equation, many hybrid methods have been developed to localize 2-D/3-D numerical solvers. The boundary conditions for the reduced simulation model are provided by rapid 1-D analytical solutions for the 1-D background Earth model (Capdeville et al., 2003; Monteiller et al., 2013, 2015; Tong et al., 2014a, 2015). The 2-D/3-D responses to the heterogeneity inside the reduced model contribute to the coda waves of the teleseismic phases, and the 2-D/3-D effects outside the model are neglected. This assumption of a 1-D background layered Earth model is similar to that in receiver function analysis and is often effective for a station that is sufficiently far from the source (Langston, 1977; Rondenay, 2009).

Although the computational requirements can be efficiently reduced by these hybrid methods, the computational costs are still excessive for a small workstation when we are faced with the several thousand forward simulations required in 3-D teleseismic adjoint tomography. To further reduce the computational costs, Roecker et al. (2010) constructed a frequency domain 2.5-D hybrid method for teleseismic wave simulations. To simplify the teleseismic wave field invariant along a particular axis, the 2.5-D formulation can significantly reduce the computational demands. However, the 2.5-D formulation may restrict the application of the method in arbitrarily heterogeneous media (Tong et al., 2015).

In addition to the large computational demand (CPU time) associated with the 3-D hybrid methods, satisfying the memory requirements for storing the boundary wave fields to construct teleseismic incident boundary conditions is another important issue that should be carefully considered. Tong et al. (2015) adapted the Clayton and Engquist-type (CE-type) boundary condition (Clayton and Engquist, 1977) to interface the 1-D background analytical solution with a numerical solver on the boundary of a reduced model. This treatment can both assign the teleseismic incident condition for the computational domain and absorb the scattered wave field from the interior of the heterogeneous model. Implementing the CE-type boundary condition is extremely simple and does not substantially increase the required CPU time. However, the CE-type boundary condition can efficiently absorb waves only at approximately normal incident angles, and incident waves at grazing angles may be reflected back to the model (Yang et al., 2003), which may reduce the accuracy of the forward and adjoint wave fields in teleseismic adjoint tomography and thus decrease the accuracy of the constructed Fréchet derivatives. Note that the CE-type boundary condition requires nine wave field components (three displacement components and six stress components) to be stored on the model boundary; this requirement may be a significant burden on the computer memory for a relatively large-scale model decomposed by a dense numerical mesh.

Here, we introduce the element-by-element parallel spectral-element method (EBE-SEM) for the efficient modeling of teleseismic plane-wave propagation in reduced models. A significant advantage of the EBE-SEM is the easy parallelization of the spectral-element algorithm, which does not require the assembly of the global stiffness matrix. The spectral elements are equally allocated to every CPU core, and the product of the stiffness matrix and the solution vector is calculated element by element; these aspects are quite useful for ensuring load balance among the CPU cores. The element stiffness matrix can be written as the tensor product of the submatrices, which greatly reduce the computer memory. In addition, the misfit gradient can be efficiently constructed because the element matrices for calculating the misfit gradient can also be formulated from the tensor products of the element submatrices. The perfectly matched layers (PMLs; Collino and Tsogka, 2001; Komatitsch and Tromp, 2003; Liu et al., 2014) are formulated by the EBE-SEM to effectively absorb scattered waves. A detailed discussion is presented to incorporate the teleseismic incident boundary condition in the EBE-SEM, and only six components on the interface between the computational domain and the PML domain must be stored in the computer memory. The high efficiency of the EBE-SEM for teleseismic wave modeling and misfit gradient construction is demonstrated by using three numerical examples.

A schematic of a teleseismic plane wave entering a localized model is
depicted in Fig. 1, where the localized model is delineated by the blue
lines. We first introduce the EBE-SEM for isotropic elastic wave propagation
in an infinite half space, which includes the localized model. We denote the
total wave field

Schematic of a teleseismic plane wave entering a local two-layered
crust–upper-mantle model. The study area is delineated by the blue lines.
The origin of the Cartesian system is located on the surface at corner A of
the model. The positive

We assume that the total wave field obeys the following elastic wave equation
(Tong et al., 2014a):

To further increase the computational efficiency of SEM, the GLL quadrature
rule is fully considered in the product of the stiffness matrix and the
solution vector. To simplify the discussion, the product of

In our previous work, a second-order PML absorbing boundary condition (PML
ABC) was formulated using the mixed-grid finite-element method (Liu et al.,
2014). Here, we use this type of PML ABC to absorb the scattered wave field

Although the EBE-SEM is specially designed for teleseismic wave modeling (i.e., Eq. 9 is for teleseismic total wave field propagation if the proper teleseismic incident boundary condition is added and Eq. 16 is for absorbing the scatted wave field), EBE-SEM can be directly used for wave field simulation of an earthquake that occurred in the interior of the model (computational domain) if a source term is added to Eq. (9).

Seriani (1997) is the seminal work that introduced the EBE-SEM. In the 2-D
case, the element-by-element scheme is combined with the Chebyshev
orthogonal polynomial-based SEM. Because the Chebyshev orthogonal polynomial
is orthogonal and associated with the weight

The recent improvements in the SPECFEM3D software also incorporate the tensor products of the element stiffness matrix (Peter et al., 2011). Geology models can be decomposed by fully unstructured hexahedral meshes. Great load balancing is achieved based on graph partitioning. This software does not explicitly assemble the global stiffness matrix. The main contribution of this paper is the detailed introduction of EBE-SEM for high-efficient teleseismic wave modeling.

If appropriate boundary conditions are added for the computational domain and the PML domain, then the computation will be isolated in the counterpart domain, i.e., Eq. (9) for the computational domain and Eq. (16) for the PML domain. The boundary conditions are discussed presented below.

For the completeness of this paper, we first simply introduce the FK method for determining the 3-D elastic-wave-equation-based plane-wave propagation in layered media. For more details, the reader is referred to Haskell (1953) and Tong et al. (2014a). Our focus is to construct the teleseismic wave incident boundary conditions and develop a highly efficient method for the storing the boundary wave fields.

Transforming the elastic wave equation into the

1-D layered background model. The thickness, P wave velocity, S wave velocity, and density of each layer are shown.

We assume that the infinite space is composed of the following hexahedron set:

To simplify the discussion, the boundary condition of the first equation of
Eq. (16) is discussed. From Eq. (16), we have

Memory requirements of EBE-SEM and the conventional SEM for teleseismic wave modeling. All the values are stored in memory with single float precision. GB denotes gigabyte.

We use the model in Fig. 1 to quantitatively discuss the computational cost of EBE-SEM for teleseismic wave modeling. Because the thickness of the PML domain in our numerical examples is only three elements wide, the number of floating point operations is trivial compared with the computational domain, and the main computational cost of the PML domain is from the storage requirement of the boundary condition (Eq. 41).

The model is decomposed into 75 000 cubic elements with a size of

The floating point operations in the computational domain are mainly from
the element matrix–vector product. A total of

The memory requirements of EBE-SEM and the conventional SEM for teleseismic
wave modeling are presented in Table 1. A total of 13.12

Plane-wave incident on a 1-D crust–upper-mantle model. The
snapshots from the top to the bottom are taken at

Comparison of the waveforms generated by EBE-SEM (dotted line) and
FK (solid line). The waveform computed by FK is considered the reference
solution. Panels

Parallel efficiency of the EBE-SEM. The red line denotes the theoretical CPU time, and the blue line with circles represents the actual CPU times. The red arrows designate the abnormal CPU times compared with the neighboring CPU times.

Three numerical examples are provided to validate the efficiency of EBE-SEM
for teleseismic wave modeling. All three examples use the Gaussian
source–time function with a cutoff frequency of 2

Material parameters of the crust–upper-mantle model.

Except for the model parameters presented in Sect. 4, the material
parameters are shown in Table 2; the incidence angle is 15

Figure 3 shows snapshots at four instants. When

To qualitatively evaluate the accuracy of EBE-SEM for teleseismic wave
modeling, the synthetic seismology at the station is compared with the
reference solution, which is generated by the FK technique. The results are
shown in Fig. 4. As depicted in Fig. 4a, the synthetic and reference
waveforms show excellent agreement. The direct wave with the largest
amplitude is followed by the converted S wave and the crust multiples with
relatively small amplitudes. Although the amplitudes of the crust multiples
are small, their phases, amplitudes, and travel times are correctly modeled.
The error curve is shown in the right panel of Fig. 4. The maximum
relative difference between the numerical solution and the reference
solution is less than 5

The computation was performed on a workstation with 2 Intel Xeon CPUs
(E5-2680 v3) and 128

To demonstrate that EBE-SEM works well in 3-D heterogeneous models, an
abnormal structure with cube shape of an additional 15

Snapshots in a heterogeneous model. The snapshots in the upper and
lower panels are taken at

The black arrows in the upper plots of Fig. 6 indicate the distortion of the wave front because of the velocity anomaly in the media. Because of the positive velocity anomaly, the distorted wave front travels faster than the undistorted plane wave. The lower plots in Fig. 6 show strong scattered waves. As the yellow arrows indicates, the strong scattered waves do not reflect back to the interior of the model because of the efficiency of the PML ABC used in this paper.

Construction of the adjoint source. In the left panels, the
synthetic waveforms from the second example are considered the observed data
(blue line), and the synthetic waveforms from the first example are treated
as the theoretical data (red line). The rectangular boxes indicate the time
window [20, 24] to isolate the waveforms to construct the adjoint source.
The top, middle, and bottom panels correspond to the

Misfit gradients in the plane at

One key advantage of the EBE-SEM is its convenience for constructing the
misfit gradient because the element stiffness matrix can easily be assembled
based on the tensor product of the submatrices. To illustrate this
advantage, we first define the misfit function:

We consider the model in the second numerical example to be the real model and the 1-D layered model in Fig. 1 as the initial model. The observed and synthetic waveforms at the station (red triangle in Fig. 1) are shown in Fig. 7a. As Fig. 7b shows, the time-reversed waveform differences between the observed data and the theoretical data act as the source term in Eq. (43).

Figure 8 shows the constructed misfit gradients in the plane at

Teleseismic wave adjoint tomography has the ability to image the deep structure of the lithosphere. Thus, a highly efficient method for teleseismic wave forward-modeling and misfit calculation is important. In this work, the EBE-SEM was specially tailored to teleseismic wave modeling and misfit gradient calculation. In this approach, the PML ABC is discretized by EBE-SEM, and the method can efficiently absorb scattered teleseismic waves. Teleseismic wave incident conditions are constructed for the computational and PML domains. An economic technique for boundary wave field storage is introduced that can greatly reduce the required amount of computer memory.

The numerical results from the first and second numerical examples demonstrate not only the efficiency of EBE-SEM in modeling teleseismic waves but also the validity of the constructed teleseismic wave incident boundary condition. As shown in the third numerical example, hardly any extra effort is required to construct the misfit gradient. The EBE-SEM has advantages over the traditional SEM in three respects: the reduction in the required computer memory requirement, easy calculation of the misfit gradient, and significant parallelization efficiency.

The original source codes for the
numerical examples are written in C. To obtain these source codes, please
contact the first author (email: slliu@math.tsinghua.edu.cn) or the
corresponding author by email. The MPI library was downloaded from

The element matrices for the computational domain are listed below:

The element matrices for the PML domain are listed below:

In Eqs. (44) and (45),

The authors declare that they have no conflict of interest.

We greatly appreciate the detailed suggestions from Michal Afanasiev and the anonymous reviewer. Their valuable suggestions greatly improved the quality of the paper. This study was supported by the National Natural Science Foundation of China (grant nos. 41230210 and 41604034). Shaolin Liu was financially supported by the China Postdoctoral Science Foundation (grant no. 2015M580085). Edited by: Michal Malinowski Reviewed by: Michael Afanasiev and one anonymous referee