For the Shaded Model layout in sequential mode, your options include:
1) Reduce the “Number of Rays” in the Shaded Model settings:

Shaded Model layout settings showing "Number of Rays" setting 
2) Reduce the number of surfaces that you are viewing in the Shaded Model settings:

 Shaded Model layout settings showing surfaces settings
3) Reduce the number of “Angular Segments” and/or “Radial segments” in the Shaded Model settings:

 Shaded Model layout settings showing segments settings
(Note:  Reducing the number of segments will effectively reduce the amount of memory required for rendering but the quality of the rendered image will be reduced as well)

4) Increase the amount of memory on your video card or install a video card with more memory.

Methods for alleviating the memory error for non-sequential OpenGL features are listed on the next page.