Two sample user DLLs and C code files can be found in \ZEMAX\Object\Dll\UserObjects: Half cylinder and Elliptical volume.  These DLLs create the solid bodies shown below.

Sample DLL objects in NSC mode

Each object has flat front and rear surfaces, and a user-defined side surface.  For each user- defined object, we must (1) describe its visual appearance and (2) organize it for ray trace.  Each surface can be represented by a set of flat triangular facets.  This approximation is used for ZEMAX object drawing layouts and gives the starting point for finding the ray-surface intersection point, as will be described further.

Description of Zemax ray-tracing algorithm

For raytracing to non-flat surfaces, you must find the ray with surface intersection point and the normal vector to the surface at this point.  This is done for each object surface. The algorithms for those calculations depend on the surface representation: parametric form, implicit form, or flat surface.  If the surface is flat however, raytracing is only needed using facets.  In ZEMAX sample DLLs code, information regarding C language routines and details, as well as comments about ZEMAX and DLL exchange data, is available.