 |
| Register | Login | |||||
 |
Main
| Memberlist
| Active users
| Calendar
| Chat
| Online users Ranks | FAQ | ACS | Stats | Color Chart | Search | Photo album |
|
| | ||
|
| Register | Login | |||||
|
Main
| Memberlist
| Active users
| Calendar
| Chat
| Online users Ranks | FAQ | ACS | Stats | Color Chart | Search | Photo album |
|
| | ||
| 0 users currently in Programming. |
| Acmlm's Board - I3 Archive - Programming - Opening 4MB files...or well, have a seamless hex editor... |
New poll |  |  |
| Add to favorites | Next newer thread | Next older thread |
| User | Post | ||
neotransotaku Sledge Brother Liberated from school...until MLK day                   Since: 11-17-05 From: In Hearst Field Annex... Last post: 6304 days Last view: 6302 days |
| ||
| I'm in the midst of writing my own hex editor to do translation because all the other ones out there do not do what I want (i.e. Japanese Tables with mixed ASCII/Shift-JIS/EUC support in one file--I don't like Unicode because no program right now does do mixed ASCII/Unicode--if they can, then I can use WIndHex and I don't need to program anymore)
Alright, my problem is this. I want to efficiently open files. For people changing the text in a ROM, only a small portion of the file will ever be active on the screen (say 4-8KiB) at a given time. I know I could just open the file in these small chunks, but still allow seamless scrolling through a file like what WindHex and Hex Workshop allow? So basically, anyone have any approaches in dealing with this problem? |
|||
MisterJones Tooky    Since: 12-08-05 From: Mexico Last post: 6392 days Last view: 6328 days |
| ||
| Why not map it on memory? | |||
|
neotransotaku Sledge Brother Liberated from school...until MLK day Since: 11-17-05 From: In Hearst Field Annex... Last post: 6304 days Last view: 6302 days |
| ||
| I've thought about that but that's too ugly--and it won't scale for 16, 32MB files. I already tried loading in an entire 256KiB file and that took 10 or so seconds... | |||
|
sloat Since: 11-18-05 From: Delaware, US Last post: 6405 days Last view: 6405 days |
| ||
| I believe MisterJones means make a file mapping object, which would be much more efficient. It's also how Hex Workshop can open 700meg files with little or no lag.
Check out CreateFileMapping and MapViewOfFile. |
|||
|
neotransotaku Sledge Brother Liberated from school...until MLK day Since: 11-17-05 From: In Hearst Field Annex... Last post: 6304 days Last view: 6302 days |
| ||
| Hmm...memory mapping...that sounds better than just merely reading the contents into memory.
Let's see if Java has an equivalent... |
|||
Disch Red Cheep-cheep Since: 12-10-05 Last post: 6582 days Last view: 6582 days |
| ||
Originally posted by neotransotaku Oooh... no WONDER it was taking 10 seconds for a 256k file. |
|||
|
neotransotaku Sledge Brother Liberated from school...until MLK day Since: 11-17-05 From: In Hearst Field Annex... Last post: 6304 days Last view: 6302 days |
| ||
actually...i was just guessing 
I tried loading all that in a few months ago and don't remember exactly what happened. I tried the samething again today and it was seamless. I think the difference was this machine was Athlon Sempron 2800 w/ 1GiB of RAM and the other machine was P4 2.8GHz (no H/T) and only 512-32 MiB of RAM. So yeah, it appears I do not have much of an issue (as I tried a 32MiB file and it took 3 secs) but yeah...it seems so inefficient to load everything into memory... Side Note--I would obviously rather program in C/C++. Problem is, I never have been able to pick up a nice windowing toolkit as convenient as Java--I've tried both MFC and WinAPI but the books I've read present way too much information right off the bat (or too difficult to use) |
|||
sp Micro-Goomba Since: 11-25-05 Last post: 6645 days Last view: 6645 days |
| ||
| Reading the file in chunks is the way to go in my opinion. Only read what the user actually wants to see. Assuming you have a standard hex window that's never more than like 1 KB of data. This amount of data can be read from a file without any speed problems right before you display it.
Here's the tough part though. You need to figure out a way to handle changes to the bytes. If the user already made some changes you obviously can't re-read the file from disk. You have to create a more or less eleborate system that tracks these changes and makes sure the changed data is displayed instead of the original data. There are several approaches to solve this problem. I'm not going to mention them though. Not because I don't want to help but because I'm hoping you can come up with a cool and new solution yourself. That way I could learn something too. |
|||
MathOnNapkins  1100  In SPC700 HELL Since: 11-18-05 Last post: 6302 days Last view: 6302 days |
| ||
| Or you could just give him an answer and let him see if it has any flaws/improvements to be made. Seriously, crap like what he's going through is why I quit trying to make a hex editor about 2 years ago. I just didn't want to deal with it. Well... that and I was trying to make one from scratch in Delphi and didn't know shit about Turbo Pascal or its OOP framework.
Either way, I don't know about memory mapped file i/o. I remember looking into it a few times for various projects, and it seemed like if you altered the memory mapped file it would cause instant (i.e. irreversible if you don't keep track) changes to the file on disk. If I'm wrong about that somebody correct me please. |
|||
|
neotransotaku Sledge Brother Liberated from school...until MLK day Since: 11-17-05 From: In Hearst Field Annex... Last post: 6304 days Last view: 6302 days |
| ||
| Alright, this talk about use filemapping has gotten me intrigued and I have pulled out that MFC book that I was talking about, dusted it off, and started going through it again. Played around with MFC, got a little bit of the hang of it and tried to use filemapping. After some work, I finally got a succesful mapping. Question I have right now is this: What is the proper way to fully close a map file? So, I have done the following:
HANDLE hFile = CreateFile(...); HANDLE hFileMap = CreateFileMapping(hFile, ...); LPVOID lpRawdata = MapViewOfFile(hFileMap, ...); |
|||
|
sloat Since: 11-18-05 From: Delaware, US Last post: 6405 days Last view: 6405 days |
| ||
Originally posted by MathOnNapkins With Windows, it's more like a copy gets loaded in to the process address space. You use the FlushViewOfFile to save changes. So you can modify to your heart's content and not have to save it. To close it, first you call UnmapViewOfFile, then CloseHandle on the file mapping object, then CloseHandle on the file. |
|||
|
neotransotaku Sledge Brother Liberated from school...until MLK day Since: 11-17-05 From: In Hearst Field Annex... Last post: 6304 days Last view: 6302 days |
| ||
Originally posted by sloatOriginally posted by MathOnNapkins I'm not sure if developing this application on Win98 makes any difference but unless I use 'FILE_MAP_COPY' with MapViewOfFile, any writes will be reflected to the file. |
|||
|
MisterJones Tooky Since: 12-08-05 From: Mexico Last post: 6392 days Last view: 6328 days |
| ||
Originally posted by neotransotaku Programming directly on win32 can be a pain for something a little too large, but then, you can encapsulate everytinhg in objects, and make it much simpler. 
Good reads: http://www.winprog.org/tutorial/ http://www.functionx.com/win32/ http://www.foosyerdoos.fsnet.co.uk/ <-- great resource A way to encapsulate everything: http://www.relisoft.com/Win32/ if it seems a little too much, try http://www.relisoft.com/book/index.htm first
Avoid mfc as much as you can. I'd rather go with something like wtl, that one from relisoft, or wxWidgets (which is cross-platform) As for file mapping: http://www.flipcode.com/articles/article_filemapping-pf.shtml (edited by MisterJones on 12-22-05 01:43 PM) |
|||
FreeDOS + Giant Red Koopa Legion: freedos = fritos  Since: 11-17-05 From: Seattle Last post: 6302 days Last view: 6302 days |
| ||
| Perhaps I'm just beating on a dead horse, but could you possibly use KHexEdit as a base, if it doesn't already fit your needs? I don't have a problem with opening 4MB files, and with a very low memory footprint. | |||
|
neotransotaku Sledge Brother Liberated from school...until MLK day Since: 11-17-05 From: In Hearst Field Annex... Last post: 6304 days Last view: 6302 days |
| ||
| KHexEdit? it looks good. Seems like it has a bunch of stuff implemented for me already but I'm doing windows development and since it is KDE dependant, doesn't look like it is cygwin friendly... (edited by neotransotaku on 12-23-05 02:06 AM) |
|||
|
FreeDOS + Giant Red Koopa Legion: freedos = fritos Since: 11-17-05 From: Seattle Last post: 6302 days Last view: 6302 days |
| ||
| Well, if you're really daring, KDE has in CVS very new and very unstable (possibly uncompilable) native Windows support.... so that means your real hope is either using the existing KDE-Cygwin (old version of KDE, but it works), using a free OS, somehow reworking KHex to Win32, or writing from scratch (x_x) | |||
|
neotransotaku Sledge Brother Liberated from school...until MLK day Since: 11-17-05 From: In Hearst Field Annex... Last post: 6304 days Last view: 6302 days |
| ||
I could use some of the logic...I've started the MFC path :x so, the windowing is now taken care of. However, I would like to see some alternative ways to do rendering of the bytes. Because I'm working with C/C++, printf is pretty nice (a lot better than calling one function to render one byte at a time)  |
|||
HyperHacker Star Mario Finally being paid to code in VB! If only I still enjoyed that. <_< Wii #7182 6487 4198 1828 Since: 11-18-05 From: Canada, w00t! My computer's specs, if anyone gives a damn. STOP TRUNCATING THIS >8^( Last post: 6303 days Last view: 6303 days |
| ||
Originally posted by MathOnNapkins Hex Workshop has a simple workaround to this: It works on a temporary copy of the file. |
|||
|
neotransotaku Sledge Brother Liberated from school...until MLK day Since: 11-17-05 From: In Hearst Field Annex... Last post: 6304 days Last view: 6302 days |
| ||
Originally posted by HyperHackerQuestion is...how is it done? it obviously doesn't do that if you take a look at an HD image...Originally posted by MathOnNapkins well, i already know how it is done so yeah... |
|||
|
FreeDOS + Giant Red Koopa Legion: freedos = fritos Since: 11-17-05 From: Seattle Last post: 6302 days Last view: 6302 days |
| ||
| Perhaps it's done via copy-on-write. In other words, copy only the bits that changed. |
| Add to favorites | Next newer thread | Next older thread |
| Acmlm's Board - I3 Archive - Programming - Opening 4MB files...or well, have a seamless hex editor... |
| |
