(Link to AcmlmWiki) Offline: thank ||bass
Register | Login
Views: 13,040,846
Main | Memberlist | Active users | Calendar | Chat | Online users
Ranks | FAQ | ACS | Stats | Color Chart | Search | Photo album
12-03-21 11:53 AM
0 users currently in ROM Hacking.
Acmlm's Board - I3 Archive - ROM Hacking - Decompressing graphics
  
User name:
Password:
Reply:
 
Options: - -
Quik-Attach:
Preview for more options

Max size 1.00 MB, types: png, gif, jpg, txt, zip, rar, tar, gz, 7z, ace, mp3, ogg, mid, ips, bz2, lzh, psd

UserPost
MathOnNapkins
Posts: 153/1106
Traces are helpful, in fact that's what I did at first. But when I found that wasn't helping I just decided to look at the ram in the GSD hex editor using $18C0 as my base. It was quite easy to see what they were doing, by just repeatedly hitting the step into button.
JLukas
Posts: 15/20
Thank you MoN, that's a HUGE help. Especially the part with the grid layout, it just didn't occur to me to think of the data as a square. I'm going to look at the trace again now that I got the idea of what's going on. Thanks again for the lesson!
MathOnNapkins
Posts: 148/1106
ChaoJeff: Snes9x should run Sim City just fine, last I checked. Are you talking about Sim City 2000, a much later release? B/c that would make a whole lot more sense. I checked and it ran one copy of Snes2000 ("simcity2002.zip") I had just fine, but the "Simcity 2000 (U).zip" didn't seem to work. You might have just gotten a bad dump like I did. I'm running GSD version 9 release 8, based on Snes9x version 1.43.




JLukas: The compression is nothing special really. The ratio of compressed to decompressed data is always 3:4. Basically each sprite tile consists of 0x18 bytes, which can be divided into 3 byte subsections. When I look at 4bpp graphics in a hex editor, to me the most intuitive way is to view each 8 pixel line as a square. Luckily the developers at Squaresoft shared my point of view so it was fairly easy to understand what they were doing. Demonstration using random hex values:

$000000: FF 7F 0F 07 FF 7F 0F 07 FF 7F 0F 07 FF 7F 0F 07
$000010: 3F 1F 03 01 3F 1F 03 01 3F 1F 03 01 3F 1F 03 01

(For simplicity's sake, #$XX will refer to a literal value, whereas $XX will refer to a memory location with the given address.)

The first line of the image data is determined by the values at $00, $01, $10, and $11. The next line is determined by that of $02, $03, $12, and $13. If you look at the example above you can see that each line is therefore a 2 byte by 2 byte square.

Suppose we wanted the color index of the first (top left pixel). We'll take {FF 7F, 3F 1F} and examine the left most bit (MSB) of each byte. With #$FF the left most bit has value 1. With #$7F the MSB is 0, same for #$3F and #$1F. Thus the color index for the top left pixel is 1. Why? The MSB of $00 corresponds to bit zero of the color index, the MSB of $01 corresponds to bit one of the color index, the MSB of $10 ... corresponds to bit two of the color index. We can then see that the color index for the next pixel to the right would be 1 + (1 << 1) + (0 << 2) + (0 << 3) = 3. (Note: instead of using the MSB, for the second pixel we use the next bit to the right. For the third pixel you use the next bit to the right after that, and so on.)

Now what if each line always looked like this:

XX YY
ZZ 00, where the letters are arbitrary bits, but the fourth byte is always zero. This effectively reduces the size of the tile's colors to 3bpp (ranging from 0 to 7). But that leaves lots of redundant space, so why not compress the data into 3 byte chunks instead of 4 byte chunks? That's exactly what squaresoft did, but they made it a little more complicated.

They store it in a 3bpp format with color indices ranging from 0 to 6. 7 corresponds to 0 before any addition is done ( that's coming up). They made it more complicated by making some of the sprites have all their indices increase by 0, 5, or A, depending on the sprite. Why would you do this? It's easy to see - Tiles of the first type use the color 0 (transparent) and colors 1 through 6. Tiles of the next type use the color 0, and colors 5 through B, the last type uses the color 0, and colors A through F. Thus you only need one palette entry (16 colors) to handle three different kinds of sprites. Very economical, b/c you can not only save space on graphics by compressing them in the rom, you can partition a palette entry as well.

Anyways, when I did my decompression routine, I ignored the possibility that the finished product might have 5 or A added to all its indices. Why? B/c the relationship between all the color values would stay the same, and thus still be recognizeable in the tile editor. In the rom the tiles are stored in a neutral format that doesn't really know if it has 0, 5, or A added to them later, so it doesn't really matter.

I know I'm rambling quite a bit, but here's a diagram of what happens:

Sample compressed first line of a tile: FF F0 F0

First what happens is this gets written to the rectangular grid:

FF F0
F0 00

Then we go through each pixel, starting from left to right, and calculate the color index. The color indices going from left to right should be 7, 7, 7, 7, 1, 1, 1, 1

The decompression algorithm asks us to change 7's into zeroes. Anything else gets left alone. The result:

0F 00
00 00

Recompressing is simply changing sevens back into zeroes, then ignoring the fourth byte and packing it back into three byte segments. The LSR, ROL, ASL bullcrap is basically an algorithm that does that. It's probably not the cleanest way to do it. If I were writing code to handle this, I think I'd probably use TSB and TRB and what not.

The code in the .cpp source file basically does what the game does, just in C/C++ syntax.
ChaoJeff482
Posts: 6/16
Originally posted by Jahakob
So you're saying that I wont be able to decompress it with Lunar Compress? Well, anyway, I can't. I get "Either decompression failed, the format isn't supported, or the compressed structure has a size of 0" so I think you're saying that I can't use this program anymore...

Zsnes and snes9x can decompress the graphics, why is it impossible to build some kind of decompressor based on that technology? I'm not complaining, I'm just curious.

Thank you, anyway. You've actually found it and that should be worth something, right?



Problem, the latest SNES 9X emulator will not Support Sim City and Crashes the program when ever it is loaded, do you think it has something to do with the compression on it is very impossible to debug on it because of the Functions of the game possibly?
JLukas
Posts: 14/20
Wow, great job MoN, you sure made quick work of that. If possible, can you give a quick description of the compression?

For example,

I see the first routine at D0/CA8B that loads $24 bytes of icon data and stores it into RAM at the 00:18xx addresses.

Then it goes to the D0/CAD6 subroutine where it does all the ASL, LSR, ROL stuff - can you explain what it's doing?

Then the routine starting at D0/CAB9 places the now decompressed data into the 00:19xx addresses, if I'm understanding this right.
MathOnNapkins
Posts: 147/1106
Not a big deal, really. So I take it you figured out how to work it pretty well.
Jahakob
Posts: 10/10


Serisously though, this is the best late christmas present I've ever received. Thank you. I can't think of anything else to say, it is really awesome of you. If you ever need some graphics or something (yea, I know it's lame but at least I'm offering you something for this, like the poor man who offered his entire fortune of 1 cent...) just ask me and I'll do it at once.
I can't find a suitable emoticon for my gratitude, thank you.
MathOnNapkins
Posts: 146/1106
Yep, about 95% now. I've gotta work out a bug in the recompression... but it decompresses just find apparently. And yes it will extract all 0xB0 (176) sprites from that region. All menu items basically.

[update] so yeah I'm done as far as I can tell. Demonstration picture:


Get the program and source at http://mypage.iu.edu/~dwaggone/secretmana.rar

Jahakob
Posts: 9/10
HyperHacker, I think I understand now. Well, since I'm new to this I understand the "basics", if even that, but still. I think that instead of reading about the 65816 processor and the SNES I'll start with the 6502 processor and the assembly language. There are so much things I don't know about so I think it will do me good if I start at the bottom, if my aspiration is to be able to hack and/or translate roms, GBA, GBC, SNES, N64. I'd also like to take a look at the PCX, even if that isn't quite the same thing, FF7 in swedish would be quite awesome. Translating hasn't really taught me anything, I know how to mess around with a hex-editor to find uncompressed (not encoded, whatever) text in a rom, it's nothing...

MathOnNapkins: Do that mean that they have decompressed the graphics in two or more ways? I can't see how that is effecient, is it?
90%, that's fast! What do you use for those sort of things? C++, C, VB?
I'll stick with the colors the original uses, I'm not going to paint them in the swedish national colours, I'm just going to translate them, if that is what you're talking about. =)
Extracting all icons? Do you mean like, clothes, weapons and equipment-icons? If that is what you mean, it would certainly be great if I could alter them, not necessary, but great nevertheless.
MathOnNapkins
Posts: 144/1106
Bleh... that turned out uglier than expected. I got sort of wrapped up in this and said, hey, why not just extract all the icons that are available? So figuring out how the icon sets were split up was what I looked at next. I was also sort of wrong about how they were compressed. It's a little funkier than what I said earlier.

[update]

Anyways I'm about 90% done with a program that will do this. It's funny b/c if I don't mess with the ASL, ROL, and other crap, it still decompresses fine, but the color indices are slightly different. If you're willing to make a backup and stick with the colors the original uses, it might not be so hard to recompress or decompress this stuff.
HyperHacker
Posts: 681/5072
Well, no, not quite. The SNES processor runs the program code that decompresses the graphics. However, it also does almost everything else from checking the controller to making you die when you touch something bad to what the hell ever. It's essentially impossible for the emulator to know what task each piece of code performs. To write an actual decompressor you need to examine the code that does it (or figure out the format) and write a program that does the same.
Jahakob
Posts: 8/10
I see, the game decompresses itself to the memory, it's not the processor that does it. So you have to find the way it is compressed and how it decompresses itself in the rom, not the snes-processor. Am I right? Possibly not, but at least I understand that it doesn't work if the decompressor isn't hand-made for that particular game.

It would be amazing if you could create a program that can do that. I'm gobsmacked, I never thought that I would get this kind of help. You've all been very patient and helpful, thank you once again, I can't say that enough, can I?
MathOnNapkins
Posts: 141/1106
The "technology" as you put it, is probably something any working SNES emulator would have. Why? okay let me try to explain this. The processor on the SNES has no built in graphics compression. So when you program a game, you have to manually code functions that will decompress the tiles and put them into memory. Some types of compression are more common than others, and hence become easy to recognize across different games.

When something like Lunar Compress is built, it is built to handle most types of decompression/recompression. But the way a programmer compresses his graphics is up to him - it doesn't have to be any algorithm people already know about. I looked at the code last night and it does appear to be decompressing it, but I didn't have time to go in depth last night, and i have to go to work right now.... so you'll have to wait until then. I might be able to make a quick program to work on them.
Jahakob
Posts: 7/10
So you're saying that I wont be able to decompress it with Lunar Compress? Well, anyway, I can't. I get "Either decompression failed, the format isn't supported, or the compressed structure has a size of 0" so I think you're saying that I can't use this program anymore...

Zsnes and snes9x can decompress the graphics, why is it impossible to build some kind of decompressor based on that technology? I'm not complaining, I'm just curious.

Thank you, anyway. You've actually found it and that should be worth something, right?
JLukas
Posts: 13/20
I found the icon offsets. They are 96 bytes each, the first 48 of which contain the icon text:

128760 1287BF EQUIP icon

12C360 12C3BF Targeting icon
12C3C0 12C41F STAT icon
12C420 12C47F LEVEL icon
12C480 12C4DF ACT icon
12C4E0 12C53F EDIT (controller) icon
12C540 12C59F EDIT (window) icon

In VRAM, the range is from DC00-DFFF (there's one other unrelated icon in there, too.)

The problem is that there is some sort of compression (not the same as the title screen) - the icons don't appear correctly in a graphics editor.

For anyone interested in looking at the routines, the routine that loads the icons into RAM is at D0CA8B-D0CAB6. The decompression routine follows.

You might want to try changing around the hex values to see if you can figure it out. If you place FF for all the first 48 bytes, it won't display any text above the icon graphic.
ChaoJeff482
Posts: 5/16
Well I have tested it out, but it seems Windows refuses to do the command correctory and keeps giving me a message that keeps saying "is not recognized as an internal or external command"

Or am I doing this whole thing wrong, such as I have to create a notepad file in order to for the decomp.exe program to read the directory?
Jahakob
Posts: 6/10
Oh, okay. That's a shame...
Maybe you could tell me how you found the title-screen info? With printscreens or something? I got that snes9x-debugger, if that is what you're using. I know I'm asking for much but it would be very helpful.
If they are compressed the same way I maybe could find them somehow, if I look through large parts of the rom?
Thank you in advance, JLukas, you have been very helpful either way.
JLukas
Posts: 12/20
Thanks for catching the mistake MoN

I tried to find the icons, but that one is beyond my ability. I can't find where it's getting the data from, but it's compressed as far as I can tell. It may or may not use the same compression routine as the title screen.
Jahakob
Posts: 5/10
Wow, thank you. It worked. Thaaank you, both of you! I owe you so much, thank you.



It says 'Swedish version'.

If I get the icons in game this whole decompressing-bit would be over with.



Thank you once again. =)
MathOnNapkins
Posts: 140/1106
My guess would be to try 1CE800, as 1CE8000 is too large an offset for any SNES game (24 bit addressing). I think he made a typo.
This is a long thread. Click here to view it.
Acmlm's Board - I3 Archive - ROM Hacking - Decompressing graphics


ABII

Acmlmboard 1.92.999, 9/17/2006
©2000-2006 Acmlm, Emuz, Blades, Xkeeper

Page rendered in 0.009 seconds; used 383.23 kB (max 450.25 kB)