Hello There, Guest! Login Register


Thread Rating:
  • 1 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Consolidating .tre files
#1
Ok, before I start adding guides on how to add this or that I need to explain something that's simplifies all my modding and development. That being the way I handle my .tre files themselves...

First we know that SWGEMU as it is today uses some 53 different .tre files to load data into the Core3 server and well... work properly. Thing is, it doesn't HAVE to be 53 different .tre's. I think anyone who's done any server work at all gets tired of bouncing around from .tre to .tre looking for the latest version of a particular object or file. Since the .tre files are loaded in a specific order, it can be assumed that newer assets overwrite older assets. knowing this you can consolidate the .tre files into easier to work with files. All my development is based upon consolidated .tre files using this method:

first, pull up your swgemu_live.cfg file AND your core3 conf file. Both will show the proper order in which the client and core load the 53 different .tre files. I used the Core3 list.

1) in treexplorer open the first .tre file (should be bottom.tre). Select all the folders and choose export. For me I exported to a folder call alltres.

2) once the export is done, open up the next .tre file in the list.
a. select all the files / nodes, export to the same alltres directory, and if prompted overwrite.
b. rinse / repeat until you've added ALL the .tre files to your alltres directory, always overwriting.

3) so now you'll want to repack all this data in to one single .tre file right? well kind of. I'm not sure if it's a size limitation or what but the trepacker app won't pack more than about 2gb of data or 127K data items (those are the maxes I've run into). So you'll need to split your alltres directory in two, I made my split at the object folder, but use your best judgement to get the files as close to equal as possible both in size and item counts. Could take some trial and error.

4) Once you get your folders split, repack them into .tre files (will take awhile) and you're done. you should now have two .tre files that contain all the needed data using all the latest assets and info. This works perfectly with the core and client, although load times are increased on slower machines so be aware. Make backups of your swgemu_live.cfg and core3 conf, change each config file to use the two new .tre files and you're good to go.

5) swgemu_live example:
maxSearchPriority is always total number of files to load. searchTree_00_x= always begins with zero and counts up to the newest file.
[SharedFile]
maxSearchPriority=3
searchTree_00_2=mytre.tre
searchTree_00_1=alltre02.tre
searchTree_00_0=alltre01.tre

Now that you've gotten this far it's time for the next step which is optional if you're using the Core strait up with 14.1 with no new expansion content. However, if you want to use the Expansion content this is by far the easiest way to implement it:

1) First yo need to obtain the last sku0_data.TOC (clarify file name) that SOE released. I don't have the file handy ATM so I'll edit to clarify date/time stamps and such.

2) using the same procedure outlined above for my alltre's, Open the .toc file using treexplorer and export the entire contents to a folder, I used fulltoc.

3) once the export is complete you'll need to organize your fulltoc directory into useable chunks of data. So for instance the appearance folder alone contains more files than trepacker will work with so you need to divide it up. Same with some of the other data groups as well. I ended up needing to make 6 sub folders (fulltoc01-fulltoc06) in order to keep the data groups small enough for trepacker to work without stack overflows.

4) pack each of the sub folders into their own .tre file.

5) modify your swgemu_live.cfg & core3 conf files to load the new .tre files in addition to the 14.1 consolidated .tre files we made above, here is the order I use. BE AWARE THAT THE 14.1 ASSETS ARE WHAT THE CORE NEEDS, so you must set your order such that the 14.1 assets overwrite the TOC assets. Otherwise it's buggy central.

Here's my order:
[SharedFile]
maxSearchPriority=9
searchTree_00_8=mytre.tre
searchTree_00_7=alltre02.tre
searchTree_00_6=alltre01.tre
searchTree_00_5=fulltoc06.tre
searchTree_00_4=fulltoc05.tre
searchTree_00_3=fulltoc04.tre
searchTree_00_2=fulltoc03.tre
searchTree_00_1=fulltoc02.tre
searchTree_00_0=fulltoc01.tre

Use this load order for the core too.

Now, if you've managed to consolidate the assets as I have you can add objects at will, only adding what you need to change / modify to your own personal .tre file (mytre.tre in this example). If you don't need to modify an asset for it work properly, then there's no reason to include it in it's own .tre file. let it load it from the consolidated asset data.

And there you have it. Hopefully this helps some to understand how my other guides and mods work.

-Duff
 
Reply
#2
I need to qualify here that while consolidating the original 14.1 .tre files (all ~53 of them) works just fine with zero issues in all my testing, Consolidating the .TOC's into the 6 additional .tre files CAN cause some issues, especially in non-test, full load production environment.

Example: The TOC data does something funky in and around Restuss, thus you'll get a Crash To Desktop if you happen to wander NE of Restus. I'm guessing this is because restuss was destroyed later in the life of the game and replaced with models that that don't render properly in the 14.1 client without some work.

There are some other things along these lines that don't work either. HOWEVER, if you're needing to do some quick work using post 14.1 data, and you're not looking to go into a production server environment with your results, then the above method works great.

Just use caution.

-Duff
 
Reply
#3
Tried this with the pre CU tre's and it works perfectly.

Just having some trouble with anything post that. I loaded the CU weapons from the Lua's that I created but they wont load in game. Seems the fulltoc tre's load ok on the server as when I try to pull a cu weapon from the frog i get no objectCRC error messages on the server, but in the client I get the message "you received a: %TO" and nothing appears.

Just wondering if you'd seen this in your trials, gonna keep playing today, see if I can see what I missed.
 
Reply
#4
Yeah, you have to update your object_crc table in the /misc folder for each item you add. Every object needs a crc in that file.
 
Reply
#5
Thanks DuffSmile
 
Reply
#6
YW, but don't thank me too quickly. it's a real PITA to keep up with that file and what is and is not in it. CRCEdit doesn't have a sort feature so it's really tough to figure out crc issues. they're easy enough to recognize tho, the core will spit out a can't find crc message, or your client will load the object but you can't see it.

One of these day's I'll write something to make it easier, but I'm too wrapped up in "discovery" mode to actually settle into "development" mode. :-)

-Duff
 
Reply
#7
The SOE scheme was really designed around dial up connections, thus why so many smaller files. Consolidating TREs is not a bad idea at all in this age of broadband. (besides you wind up knowing what is in the final TRE much more so than having it spread all over multiple TREs)

As far as the object CRCs, what I have been doing is building off the Pub 14.1 CRC and adding to it one NGE object at a time, and have my own CRC list (as the SOE one is just too big to sift through, and its useful to know exactly what is in there by virtue of adding it in there myself)

I had gone thru the gross process of extracting all the TREs having gotten sick of all the cherry picking, then put my post-Pub 14.1 stuff under version control.
 
Reply
#8
It would be nice to have a new CRC tool that would give better functionality. I thought about doing it in .net, but i don't understand any of the existing CRC code and how they're generated. I've tried, but didn't come close and bitwise operations kick my arse...

I'd like some sort of merging ability so I can maintain my own CRC file, and keep the 14.1 CRC file pure, and then merge them before packing a new .tre. A search / sort feature would be nice too. :-)

maybe if the operation is something Uli or Timbab understand they can help me out so I can put all the pieces together in .net.

-Duff
 
Reply
#9
You can use tre explorer to search for object strings already in the crc file if that's any help, I use that to check what is actually contained in the file or if something I added was successfully updated. Dose not sort tho.

On a side note, does the addition of stuff like post 14.1 stuff into that file get ignored by the core until it is called for or will it cause conflicts somewhere? Just wondering about using the final version of that file and re adding items that was taken out over the years instead of working forward and adding Cu and NGE crc's. Going to try it anyways, just wondered if anyone had tried it??
 
Reply
#10
if you try to use the very last version of the object_crc file, you'll get a CDT. I have not investigated why yet. but I did try that first.

-Duff
 
Reply
  



Forum Jump:


Browsing: 2 Guest(s)