Tertium Organum CollectoREX: user's manual

Install

Unpack archive. In Max select MAXScript->Run script and choose mzp-file. Then select Custimize->Custoimize User Interface..., in window "Custoimize User Interface" go to Toolbars tab and choose "Tertium Organum" from Category combo box. Draq button "Tertium Organum CollectoREX" to toolbar. In order to view CollectoREX icon you must restart Max.

Quick start

Main utilite purpose - collect all files linked with project to choosen by user folders. It's for order in mind and on hard disk. It's for transferring scenes to another computers. For collector quick start:
1. Load your scene (for beginning - not containing Xrefs).
2. Launch utility (utility button on toolbar is trigger button - click it once more to close utility).
3. To realtime log view open MAXScript Listener (F11 or MAXScript -> MAXScript Listener)
4. Choose collect folders, options from category Affect will become available



5. Click Collect Resources button.



6. Utitlity will collect all project resources, update resource links in scene and save scene. In scene folder will appear two files: backup-file and log-file.

Terms in this manual

XRef-tree – current scene plus all scenes linked to it by XRef links (XRef Scene, XRef Object, XRef Material, XRef Atmospheric). XRef-links may be looped
IES-file – simplified name of photometric data files, used by photometric lights.
Links (to files/resources) – properties of scene entities (materials, objects, modifiers, etc.) of "FileName/String" and "Bitmap" types, pointing to files of any type (bitmaps, shaders, IES-files, etc.), except Xref links.
Collect folders – three choosing by user folders in which utility will copy: Link's tree in scene – list of objects connected by relation "the whole - part", showing link's place in scene. For example link-property of subtexture of texture of submaterial of multimaterial, applied to one of scene objects, have such link's tree: object name -> multimaterial name -> submaterial name -> texture name -> subtexture name -> property name.
Real file path – path of existing file - not always path stored in max scene. When Max can not find file by stored path it tries to search in scene folder and all it's subfoilders, then in system and user defined paths. That's why stred path may differ from real file path. When you merge objects in your scene - things become more confusing.
Project – in this manual context project - current scene's XRef-tree. If scene contains no Xref links, project is scene itself.

Technical requirements and usage hints

Tertium Organum CollectoREX created to work with scenes, which render by Default Scanline Renderer or Chaous Group VRay Renderer. Fry Renderer not supported. Maxwell renderer's materials: utility can extract only current layer and current coating info.
Utility has tested with following 3dsmax versions: 9sp1-sp2, 2008, 2009. Work in 9 w/o sp1 and previous versions - impossible.
Utility supports network paths. If you want to collect to network folder you must have read and write access to it.
It's desirable - but not indispensable - to restart Max before using utility. Utility operates with big data arrays - so MAXScript garbage collector errors may appear. In some cases instability and amassing gc errors may cause Max crash.
It's recommended to increase size of Initial Heap Allocation on MAXScript tab of Preference Settings window (Customize->Preferences…) at least to 100Mb. It will help utility to work more stable.

Why choose Tertium Organum CollectoREX

Because it:

Utility functions

1. Collect resources – collection of all resources (images, HDR images, Vray meshes, IES-files, shader files) pointed by current scene or Xref-tree to specified folders and update scene links
2. Unused – search in collect folders files that no longer used by any scene (or whole Xref-tree) link. You can delete founded files or move them to "local recycle bin" - subfolder "$trash$" created in folder where unused file had founded
3. Relink to resources roots – attempt to rewrite all scene (or whole Xref-tree) links so that they poin to files with same names as before but within specified collect folders. If some file doesn't exist in corresponding collect folder, link pointing to it remains the same. This function is needed when you move the whole project or its resources to another location or even computer
4. Resolve links to existing – this function is for ascertainment of real file paths (instead of stored in scene) of resources used by scene (or whole Xref-tree) and for rewriting all scene links so that they point to really existing files
5. Missing files Identity – output full object trees of scene (or whole Xref-tree) entities, that point to missing files
6. Convert to JPEG/TGA – conversion ".tif", ".tiff", ".psd", ".png", ".tga" and ".bmp" files to JPEG or TGA (w/alpha, RLE-compressed) files; if converting file has alpha channel, user can choose operation variant

Interface

Interface consists of three rollouts.

 

Rollout "TO CollectoREX"

Files with same names
Operations with files, having the same names as existing in collect folders
Ask even if bmps are equal – show coparison dialog with operation variants "Replace", "Use existing" and "Rename" even if images are equal.
Non-img full compare (slow!) – compare non-image files byte per byte, if their CRCs are equal. It's very slow if comparing files are big. If you are collecting only images, HDRI and IES-files - you can easily use this option. But still - it's not necessary.
Ask even if binary are equal – show comparison dialog with operation variants for non-image files even if their are equal

Security
Data security
Make back-ups – make backups before executing utility function changing scene
Incremental back-ups – if backup file already exists create new file with name, which consists of: scene name, ordinal number and dual extention "max.backup": myscene.max.backup, myscene1.max.backup, myscene2.max.backup, etc. If this option is not checked, backup file (<scene name>.max.backup) will rewrite each time
Ask user when file op fails – show full abort of current function request if file system error appears. Such errors appear if some files opened by another programs (e.g. by Photoshop), due some network or hard disk problems. If user agrees to abort function, function execution is aborted and user can locate and fix problem manually. If user answers "No", error will be ignored and copying of this file will be skipped. If user aborts function execution, he can undo all scene changes simply by reloading file: when aborting function utility does not save changes. What files (and how much) was copied during aborted function user can find out from function log.
Reload while fix (slow!) – reload images while changing links. It takes much time and memory (sometimes extremally much!). But reload is not necessary. Now this option is almost useless. But still it's possible situation, when: while collecting, you meet file with same name as file in collect folder, and choose "Use existing"; but that files are different; you do not see changed texture after collection. Most Max entity properties update when needed, so described situation did not appear so far while testing, but it's still possible. So, if you want you can use this option, but be careful - it can even force system show Windows Virtual Memory Increase warning!

Content Scene content
Process medit-only content – also collect materials and textures that exist only in Material Editor and not applied to any scene object.
Process XRef structure – process not only current scene but the whole Xref tree.
Clear medit before proceed – clear Material Editor palette before function execution. If this option is checked, option Process medit-only content disabled.
Ignore shaders – ignore ".cgfx", ".mi", ".psh", ".vsh", ".fx", ".fxh", ".axml", ".msl" files. Uncheck this option only if you use your own shaders not including or loading any files, and it is possible and nedded to collect them.
Scan all tex instances also – also scan all TextureMap objects in scene. This option is added just in case standard algorithm miss something. Now it seems to be useless due no textures skipped by standard algorithm was found while testing. If you think that not all files was collected by utility, you can try to check this option. Unfortunately, this option does not resolve problems with Fry and Maxwell renderers.

Collect Roots
Collect folders. Button with elipsis launches folder select dialog
Images & ph.web root – images (except HDRI) and IES-files collect folder. Shaders also go here if unchecked option Ignore shaders from category Content
HDRI root – HDR-images collect folder
VRay meshes root – Vray-meshes collect folder

Affect
What to process
Images & ph.web – shaders will be processed also if Ignore shaders is checked
HDR images
VRay meshes

Resource roots related
Utilite functions relaited to collect folders. Only selected types of resources will be affected.
Collect resources – execute "Collect" function of utility
Unused - execute "Unused" function of utility; Summary only - if processing Xref-tree show only summary (don't show for every file)
Relink to resources roots – execute "Relink to resources roots" function of utility

Resolve links to existing – execute "Resolve links to existing" function of utility
Missing files identity – execute "Missing files identity" function of utility
Continue op after crash – continue last function, launched for Xref-tree but interrupted by Max crash. (see below – Continue op after crash)
 

Rollout "Convert to JPEG/TGA"

Convert ".tif", ".tiff", ".psd", ".png", ".tga" and ".bmp" images to JPEG. If it has alpha-channel utility offers operation variants.
If image has alpha (ask/skip/JPG/TGA) – if image has alpha-channel: ask user / skip / convert to JPEG anyway (alpha will be lost) / convert to TGA with RLE-compression (saving alpha)
Premultiplied alpha in TGA – mix semitransparent pixels color with background color (TGA only)
JPEG quality (JPEG only)
Use NConvertif exists – if NConvertฉ exists in utility folder ("maxroot\scripts\tocollectorex" or "maxroot\scripts\tocollectorex Demo" for demo version), windows root folder or in Windows\System32 folder, it will be used (as it is more accurate) for JPEG conversion instead of standard Max compressor
Convert – execute Convert to JPEG/TGA function of utility; Affect options will be ignored
 

Rollout "Help & config"

The documentation and configuration
Configuration automatically saved together with the scene and loads in scene load time. Besides it can be globally saved in ini-file. Such configuration called "global user defaults" or default settings. Default settings load, if the current scene does not contain configuration. Also it is possible to load them compulsorily or to reset them to initial values.
Config
Save config as defaults - save current configuration as default values
Load config from defaults - load configuration from default values
Clear defaults - reset default settings to initial state

About - start a browser with a help file
License search/request (in a demo - How to buy) - search the suitable license in: utility folder ("maxroot\scripts\tocollectorex"), 3dsmax root folder, Windows root folder and Windows\System32 folder. In a demo this button is substituted by button How to buy which outputs short information on reception of the complete version.

The detailed description of utility functions

The functions working with collect folders


All functions output the detailed information about all the operations - in a log-file and in console MAXScript Listener. To see it, press F11 or click MAXScript-> MAXScript Listener.
Progress of each stage of function (and its name) is displayed by the progress indicator at the 3dsmax window bottom. Each function can be interrupted at any stage by pressing of button Cancel near the indicator of progress or key ESC.

Collect resources


This function includes:
- A choice by the user of collect folders
- The scene backup saving
- Collection of links information
- Output of a map of real paths of resources used in scene
- Search and output of not found files and a complete tree of each object referring to this file
- Copying of files to collect folders
- Fixing links
- Saving scene file

Collect folders
All collect folders of one project can be the same. The same collect folders can be specified for several projects (for example, for storage big VRay meshes). In this case Unused function of utility should be used with care - files which are not used only by the current scene or its XRef-tree will be shown. For example, if every project uses its own collect folders for images, but all use the same folder for collection VRay meshes, function Unused on any of these projects with checked option VRay meshes will show not used only by the current project.
The backup saving
The utility allows carrying out the backup saving. Two variants are thus possible: incremental (by default) and simple. Incremental backup: file name will consist of a name of an original file, a sequence number (if it is not the first backup saving for this file) and double "max.backup" extension. Simple backup: the name does not contain a sequence number, and if the file with such name already is, it is rewritten atop.

Collection of links information
The utility searches links in following 3dsmax entities:
- Geometry
- Lights
- Space warps
- Helpers
- Splines
- The textures, assigned to a renderer (for example, VRay environment map)
- 3dsmax environment map
- Effects
- Atmospherics
- All objects of parent type for all textures in system (if option Scan all tex instances also from category Content of the main rollout is checked)
- All materials and the maps which containing in Material Editor and have been not applied to objects of the scene (if option Process medit-only content also categories Content of the main rollout is checked)
In entities utility searches for maps, IES-files, HDRI and VRay mesh’่ - depending on what options are checked in category Affect of the main rollout

Map of real layout of files
Terms see: Real path of a file. The utility outputs to a log-file paths in a format "the path saved in the scene"-> "really found path". If the file is not found, instead of really found path there will be a syntactic word "missing". While output repetitions are not considered - any file will be shown only once.

Collection: Copying of files to collect folders. Files with the same names
If while copying of files into collect folders utility finds out, that the file with such name already is, it tries to compare files. It compares maps completely (with alpha and without), for other files calculates the check sum algorithm HMACSHA512. If user wishes it is possible to use a complete byte per byte matching for such files, but it occupies too much time, and matching of check sums in most cases guarantees, that files are identical. When files with identical names are equal the links will be rewritten so, that they point to existing in collect folders files (this behavior can be changed by means of options Ask even if bmps are equal and Ask even if binary are equal).
If names of copied and existing files are equal, and files not equal (or it is checked one or both options Ask even if bmps are equal and Ask even if binary are equal) user sees the comparison window to make more correctly decision what to do. He can: replace existing file with a copied file, use existing, or rename it (accordingly, buttons Replace, Use Existing, Rename to …). The name of a file for renaming, unique for collect folders, is substituted in a comparison window. If user wishes it can be changed, but the utility will not allow setting a name, non-unique for collect folders. In a matching window between slide miniatures equal-signs (==) or inequalities (<>) are put as results of comparison of maps without an alpha-channel (rgb) and with it (rgba). Under maps the information on files is shown: if it is maps - the dimensions, the size, presence of additional channels (for RLA) and an alpha-channel. For the files which are not maps, the size is shown only. The initial file path before collection user can see below.
If user wishes he can cancel collection function at this stage, having pressed button ABORT COLLECT.



Fixing links in the scene
After end of copying the utility walk through all objects in which references to files have been found and writes their new values. If option Reload while fix (slow!) is checked (category Security of the main rollout), the utility can reload maps. Usually it is not required. Now this option is almost useless. But still it's possible situation, when: while collecting, you meet file with same name as file in collect folder, and choose "Use existing"; but that files are different; you do not see changed texture after collection. Most Max entity properties update when needed, so described situation did not appear so far while testing, but it's still possible. So, if you want you can use this option, but be careful - it can even force system show Windows Virtual Memory Increase warning!

Unused


This function should be used after the scene has passed collection process. Unused allows to find out, what files in collect folders are not used by the current project. Function includes:
- A choice by the user of collect folders
- Collection of links information
- Show of the table of not used files

The table of not used files




In this table files not used by the scene are displayed. For convenience the information on the size and date of the last change of each file is shown. Opposite to each file at the left there is a flag. Operations Kill checked and Checked to trash concern those files for which this flag is set.
The user can delete files or move to a local recycle bin. The local recycle bin is a folder $trash$, creating in folder in which unused files are found.
Window buttons:
Kill checked - delete the files which flag is set
Kill all - delete all files
Checked to trash - move to a local trash can the files which flag is set
All to trash - move all files to a local recycle bin
Checkers: Invert - invert flags
Checkers: None - reset all flags
Checkers: All - set all flags
Checkers: Selected on - set flags for the selected strings (some strings it is possible to select, holding CTRL or SHIFT)
Checkers: Selected off - reset flags for the selected strings
The mouse double click on string opens the map in VFB (virtual frame buffer). Each time new VFB window is opened, it is necessary not to forget to close these windows manually not to spend memory in vain.

Relink to resources roots


This function is applied in the following situation: the project has passed collection process, after that collect folders (or collect folders and the project) have been transferred to another place: disk or computer. Attention: if project files only (the scene or the several linked scenes) is moved, this function is not required. The matter is that after change of location of collect folders all resource links in project will point to nonexistent files.
The situation when these files will not get to a category of missing files is possible: if collect folders are transferred somewhere in a project folder (i.e. in a folder or in any its subfolder with any degree of an enclosure) or in a folder containing in paths which "sees" 3dsmax (for example, user paths). However even in such situation if user left things "as is" he may get missing files issue when attempting to merge objects from this scene to another.
The given function changes all links so, that they specified in files with the same names, but being in the specified collect folders (if earlier specified folders do not exist appropriate fields of config will be cleared).

Operation includes:
- A choice by the user of collect folders
- The scene backup saving
- Collection the information of links
- Output of a map of real paths of resources used in scene
- Search and output of not found files and a complete tree of each object referring to this file
- Fixing links
- Saving scene file

Other functions


Following three functions are not dealing with collect folders and can be applied on the scene, not passed collection process.

Resolve links to existing


This function fixes links so, that they point to real files. It can work bypassing a technique of collection and is parallel from it.
It includes:
- The scene backup saving
- Collection the information of links
- Output of a map of real paths of the files used in the scene
- Search and output of not found files and a complete tree of each object referring to this file
- Fixing links
- Saving scene file

Missing files Identity


Function simply shows what objects point to missing files, with output of a complete tree of links.
It includes:
- Collection the information of links
- Output of a map of real paths of resources used in scene
- Search and output of not found files and a complete tree of each object referring to this file

Convert to JPEG/TGA


This function converts ".tif", ".tiff", ".psd", ".png", ".tga" and ".bmp" to JPEG. ATTENTION: options of category Affect have no effect to this function.
If the map contains an alpha-channel it cannot be saved without alpha loss in JPEG, therefore variants are offered to the user: convert in JPEG, save as TGA (if it is not a TGA-file) or leave without changes. Operation by default can be chosen in category If image has alpha.
When the utility meets the bitmap with alpha-channel, it shows a window, where there is a slide miniature of this bitmap, the information about its dimensions, the size, and additional channels.
Two variants of a window (on left the map already has format TGA) are shown below.




Window buttons:
Convert to JPEG - convert to JPEG with loss of data of an alpha-channel
Don't convert - leave without changes
Convert to TGA - convert the map to TGA with a RLE-compression; if option PreMultAlpha is checked, color of semitransparent pixels will be mixed with a background color; whether this option is set at each opening of dialogue, depends on a status of option Premultiplied alpha in TGA in rollout Convert to JPEG/TGA.

Working with a XRef-tree

If the scene contains XRef-links (XRef Scenes, XRef Objects, XRef Materials and XRef Atmospherics) and option Process XRef structure from category Content of the main rollout is checked, the utility can handle all scenes linked with flowing by means of this links.
With an XRef-tree all functions of utility can work. Then at function start the XRef-tree window is shown. By indents user can see depth of an enclosure.

Search of the scenes relating to an XRef-tree, is performed:
- In XRef Scenes and the scenes switched on in them etc.; the graph can be looped, in the list constructed by the utility there will be no duplicates
- In objects of the scene which have type XRefObject; if the object containing in this object has type XRefObject (through inclusion) search goes to such degree of an enclosure where the base object - not XRefObject
- In a proxy of XRef-objects of the scene
- In materials of the scene and, if it is necessary (option Process medit-only content from category Content of the main rollout), in materials of Material Editor which has been not applied to objects of the scene - also to necessary depth of an enclosure
- In the effects of atmosphere imported from other scenes by means of the XRef-link - also to necessary depth of an enclosure




In the lower left corner the name of current function is shown.
Window buttons:
Abort operation - cancel function execution
Continue without XRef tree - continue only with the current scene
Continue - continue with complete processing of all XRef-tree
At processing of all XRef-tree, 3dsmax loads, processes and save, if it is necessary, all files from a tree. Processing of each file occurs, as is described above in utility functions description. Feature is only that to all scenes in the course of processing the same utility config (including paths of collect folders) are applied.

Continue op after crash


As critical failures 3dsmax are frequent enough, in particular - at loading-unloading of big files, the situation when there is a crash during processing of the long list of files is quite possible. To continue the started function execution after 3dsmax reboot (it is necessary to be sure, that after window closing 3dsmax its process also was closed), Continue op after crash function is intended.
After 3dsmax restart if the previous function execution has been interrupted by critical error or aborted by user, button Continue op after crash in the main rollout is in the enabled state. After pressing it user sees window Continue After Crash. In a window are displayed: settings with which function execution has been started, function name (in the lower left corner) and the list of files which are not handled yet (Files scheduled).




Window buttons:
Cancel - simply close a window: in case the user has simply wanted to look, that contains in a file of the information for Continue op after crash
Abort operation - cancel the interrupted function execution and delete all data about it
Continue - continue the interrupted function execution

During XRef-tree processing in a batch-mode after each file current status of function execution is saved to make continuation in case of possible failure. At successful end of function execution this file is erased. The file in which the information for Continue op after crash is stored, has name "TOCollectoREX_cac.ini" and is in a folder of 3dsmax plug-ins config: if you enter in MAXScript Listener string symbolicPaths.getPathValue "$plugcfg" and press ENTER, MAXScript will show, where there is this folder, for example:
"C:\Documents and Settings\Administrator\Local Settings\Application Data\Autodesk\3dsmax\2008 - 64bit\enu\plugcfg".
It is not recommended to edit it manually; the utility itself will delete it when it became useless.

Utility configuration

For user convenience utility configuration is saved separately in each scene, for user not to set it up again next time, when he starts any utility function. If utility configuration is not saved in the scene yet, default settings (they are shown in the table of interface description above) are load.
Changing of these defaults provided to customize the utility for user completely. "Defaults of the user" are saved in a file with a name "TOCollectoREX_defaults.ini", in a folder of 3dsmax plug-ins configuration (see above). By means of buttons from category Config of rollout Help & Config the user can save configuration from the current scene as defaults, load configuration from defaults, reset defaults in an initial status.

The typical utility workflow

The user creates the scene, links to it various resources and edits it. After end of each stage (if his work style means division into stages) or simply periodically user uses Collect resources function of utility. If required (for example, upon end of development of complex textures in Adobe Photoshop) user uses Convert to JPEG/TGA and from time to time - Unused to find out, what resources are not necessary any more, and so can be deleted. If in current scene objects from other scenes are merged, the user can use Resolve links to existing to find out, what actual resources loaded in the merged or imported piece of the scene. If any files are output in list Missing Files at scene loading, it is possible to use Missing Files Identity to know, which objects point to missing files and to fix a problem.
The same workflow if the scene uses XRef-links, but usage Continue op after crash is possible at 3dsmax failures.
For convenience the user can save a utility configuration as global defaults (for example if he prefers simple backup than incremental or does not want clear Material Editor every time utility function starts).

Refusal from using the utility

Using the utility or refusal from it does not impose any limitations on your technological process. If you refuse its usage and delete it from the system, no data will be lost, your scenes processed by the utility, will continue to open and be ok, as before. The same concerns and the resource files collected or converted by means of the utility.

Known problems and solutions

1. If you use displace space warp, and in it use the bitmap field, and are going to make Relink To Resources Roots (and now all files of the scene have status Missing), there is probability, that this picture will not relink to new path. Therefore we advise to use parameter Map in displace space warp instead of Bitmap (which is, actually, obsolete)
2. If during execution of Find Unused Files function on X-Ref tree there was a 3dsmax crash, after restoring (Continue op after crash) the total list of the files which are not used by all XRef-tree will not shown - only for each file separately. Try to restart 3dsmax and repeat operation
3. The utility uses possibilities of Microsoft.NET. Connection MAXScript - .NET is imperfect, therefore periodically can not redraw some controls. It is enough to click on these controls or to switch to other task, and then return in 3dsmax, to redraw them.
4. After durable work of utility strange things (errors) in 3dsmax's work are possible. Usually it is preceded by output of one or several dialogues from 3dsmax, informing that in work of garbage collector of MAXScript there was an error. In this case restart 3dsmax and ensure that its process has been removed after closing of its window. It is better to start the utility in "new-started" 3dsmax.
5. IFL - if names of files not relative then as a result of collection IFL will point to that place where specify paths, only if files there are present. It is recommended to create IFL-files, not using absolute path - in each string there should be only a filename (variant by default)
6. The utility copies shader files, however it cannot copy the files specified in them. The option "do not copy shaders" allows not collecting shader files. It is checked by default.


Tertium Organum ฎ 2008-2019