This article explains:
ZEMAX Development Corporation thanks Mr. Robert E. Fischer of Optics 1 for the permission to use the graphics from his book Optical System Design, and also for providing us the original graphic files as well.
Many surfaces in ZEMAX can have diffractive power in addition to refractive power. Diffractive power is independent of the substrate index and the surface sag, and introduces phase change to rays. .
All diffractive surfaces in ZEMAX bends rays according to the following equation.
where m is the diffraction order, l is the wavelength and T is the grating period (inverse of the line spacing d). The equation above is Snell's law for refraction, plus an additional ray bending term representing diffraction. The diagram below shows the diffraction for a ray incident normally (sinq1 =0) for a diffractive surface with no refractive power.
A surface such as the Diffraction Grating surface has constant period of grating lines along one axis, and is commonly used in spectrometers. The real power of computer-generated diffractive surfaces is that the grating period can be made to vary spatially across the surface, so that diffractive power can be added exactly where it is needed. For example:
In all these cases, rays are bend by the gradient of the phase profile introduced by the diffractive. The design process is therefore to compute the required phase profile, and then to compute the grating structure required to produce the phase profile.
A diffraction order has to be specified to each diffractive surfaces in the Lens Data Editor. Multiple diffraction orders can be modeled simultaneously via multiple configurations, as shown in the diagram below.

According to the equation above, the diffraction angle depends only on the period (T) of the repetitive structure where the incident light hits, and not on the shape of the structure within that particular period. The surface structure does affect the diffraction efficiency, which is not modeled by geometrical rays. The efficiency to the specified diffraction order is assumed to be 100%, meaning all rays incident on the diffractive surface will exit at the diffraction angle of the specified order.
The sign of the diffractive order determines the sign of the diffraction angle with respect to the optical axis. The sign convention for the diffraction order is purely arbitrary. The convention used by ZEMAX is positive diffraction angles (with respect to the optical axis) for positive diffraction orders.
Diffractive surfaces in ZEMAX can have refractive as well as diffractive powers. The diffractive power introduces a continuous phase across the surface, according to their formula described in the manual. Since the phase is continuous, they represent ideal diffractive optical elements (DOE), where the period of the diffractive structure is infinitesimally small or at least very small compared to the wavelength.
To maximize the diffraction efficiency in a DOE, the sag of the surface within the diffraction zones can be made such that the phase of the wavefront is parallel to the diffracted waves (of the desired diffraction order) everywhere. Figure 13.3 (b) shows a "blazed" transmission grating in which the blaze angle is optimized to maximize efficiency to a particular order.

Figure 13.3 from Optical System Design
A DOE with continuous surface profile shown in figure 13.3 (b) is often referred as kinoform. If the sag is approximated by discrete steps, as it is often the case when photolithography is used, it is commonly referred as a Binary Optic (see diagram below). Diffractive surfaces in ZEMAX are closer approximation to kinoforms than true binary optics, since the phase is continuous everywhere. It is up to the user to decide what surface structure to use to approximate the phase modeled by a diffractive surface.

Figure 13.6 from Optical System Design
The following diagram shows the theoretical efficiency of binary surface as a function of number of steps. 
Figure 13.10 from Optical System Design
This article has discussed diffractive surface modeling in ZEMAX . In summary:
References
1. Optical System Design by Robert E Fischer / Biljana Tadic-Galeb