These instructions are for Maya 8.0 and above.

1. Make sure you have the Mental Ray plugin loaded.

If not, go to Window -> Settings/Preferences->Plug-in Manager
Then enable Mayatomr.dll

2. Create/Select your objects. I’m just using a polygon sphere.

3. Optional: Create a new render layer and add the objects here.

2. Apply a background shader to the object.

Use these settings:

Specular Color: black
Reflectivity: 0.000
Reflection Limit: 0
Shadow Mask: 0
Matte Opacity Mode: solid matte
Matte Opacity: 1

3. In the attribute editor view of the background shader above, select the right arrow at the top to go into the SHADING GROUP.

4. Now, in the attribute editor view of your Shading Group, expand the mental ray section. Then expand the Contours section.

Check ‘Enable Contour Rendering‘

5. Set the values for Color, Alpha, and Width. Color is the color of the wireframe lines. Alpha controls transparency of the lines. Width is the thickness of the lines.

I used these values:

Color: white
Alpha: 1
Absolute Width
Width: 0.5

6. Open the render globals window and go to the mental ray tab.

Expand the Contours section.
Expand the Contours->General section.
Check ‘Enable Contour Rendering’.

Expand the Contours->Quality section.
Set Over-sample to 3 (Higher values create smoother lines)

Expand the Contours->Draw Contours section.

Check ‘Around All Poly Faces’

Here is a wireframe screenshot using the Mental Ray contour line method above:

Here is an alternate method (toon/Vector shader) for generating wireframes in Maya:

Click here to view the embedded video.