Edit_Normals modifier issues

By uhfath · 2012-09-25

Description

Hi! I'm new hear and forgive me if this was already asked but after searching I haven't found any info on this.
For past few days this modifier gives me a real headache. It screws my geometry exporting pretty much.
The steps:
1. create a sphere
2. add Edit_Normals modifier
3. modify some normals, just to make the changes visible
4. collapse the modifier stack
5. now I'm trying to export them in this way:
a. if there are smooth groups, then use meshOp.getFaceRNormals
b. if there are no smooth groups present, then use getFaceNormal

After exporting I clearly see that modifications to normals are lost. But Max sees them intact.
The only way around this that I found is to manually (actually from script during exporting) add Edit_Normals to the mesh, export normals right from it (not from mesh itself) and then delete it.
This way I see the modifications made after exporting. But I'm not sure that this is the only "workaround". Perhaps there is another way of extracting normals? It's just that if I always use this method (creating Edit_Normals modifier manually) then exporting normals with different smooth groups becomes messy. Don't know why.
Here is the code that I tested with Edit_Normals:

bufFaceNormalList = #{}
bufEditNormals.ConvertFaceSelection #{faceIndex} &bufFaceNormalList
bufNormalIndex = bufFaceNormalList as array
normal1 = bufEditNormals.GetNormal bufNormalIndex[1]
normal2 = bufEditNormals.GetNormal bufNormalIndex[2]
normal3 = bufEditNormals.GetNormal bufNormalIndex[3]

What am I missing here? How to properly export normals per vertex in different situations?

Comments (2)

uhfath · 2012-09-26

Sorry for vague explanation. I'll try to clarify a bit.
What I'm doing is the exporting "plugin" for my engine. The process looks like this (I'll try to strip the code):

bufMesh = bufObject.mesh
faceCount = meshOp.getNumFaces bufMesh
faceData = #()
positionData = #()
normalData = #()
for i = 1 to faceCount do
(
  bufNormal = #()
  if ((getFaceSmoothGroup bufMesh i) != 0) then
    bufNormal = meshOp.getFaceRNormals bufMesh i
  else
  (
    bufFaceNormal = getFaceNormal bufMesh i
    append bufNormal bufFaceNormal
    append bufNormal bufFaceNormal
    append bufNormal bufFaceNormal
  )

  bufFace = getFace bufMesh i
  for j = 1 to 3 do
  (
    append faceData ((i - 1) * 3 + (j - 1))

    bufPosition = meshOp.getVert bufMesh bufFace[j]
    bufPosition = bufPosition * offsetMatrix
    bufPosition = bufPosition * sceneScale
    append positionData bufPosition.x
    append positionData bufPosition.z
    append positionData -bufPosition.y

    append normalData bufNormal[j].x
    append normalData bufNormal[j].z
    append normalData -bufNormal[j].y
  )
)

Basically what is done here, is that we take a face, take the vertices from face indexes and put them into an array, which is then goes along the pipe to the file. Same thing with normals. Almost same. We check if smooth groups are present and if there are, then we take 3 normals with getFaceRNormals(). If there are not smooth groups, we take a face normal with getFaceNormal and output this single normal as 3 normals for 3 vertices of the triangle. There is no optimization here, which is currently not needed.
Everything seems to work perfectly OK, until I stumbled upon a model without smooth groups, but which is smooth by itself in 3DS Max viewport and render. Found it somewhere in the internet, so perhaps it is buggy by itself, but there is no guarantee of this. The model looks like this:

As you can see it's smooth, but there are no smooth groups in it and the exporter behaves actually correct, producing faceted normals (one normal per face). If I set one single smooth group to all triangles everything works correctly after that.
But that's one part of the "normals issue". Perhaps if you advice me how to deal with this case, the other part will be also solved (the one with "Edit_Normals" modifier).

MKlejnowski · 2012-09-26

What do you mean when you say "After exporting I clearly see that modifications to normals are lost. But Max sees them intact." How are you exporting them? Got more code of the entire process? I am having trouble figuring out what you are having issues with.