We want, ney demand, a 'leasons learned' briefing!
What was the thing(s) you figured out, discovered or uncovered that allowed you to get the algorithm working?
Some vagueness or ambiguity in the spec? Some bit/byte ordering issues in the code?
What are the things that might trip up the next person who attempts this and finds themselves reading through this thread?
It was those hard coded values, they had indistinct naming (even in the docs) which got me confused, after I moved the related stuff to their own files and renamed them to match the files (I had an #if 1 statement for the implied code data at one point) I was able to see & compare to the tables provided here:
Understanding gzip
With that I was able to experiment until I got it right. Still think the design of the encoding leaves much to be desired :|
The bit of RFC 1951 that I came unstuck on for a while was this:
The meaning of codes 17 and 18 just escaped me for hours. Now I understand what it is trying to say I can understand it perfectly.Code:0 - 15: Represent code lengths of 0 - 15 16: Copy the previous code length 3 - 6 times. The next 2 bits indicate repeat length (0 = 3, ... , 3 = 6) Example: Codes 8, 16 (+2 bits 11), 16 (+2 bits 10) will expand to 12 code lengths of 8 (1 + 6 + 5) 17: Repeat a code length of 0 for 3 - 10 times. (3 bits of length) 18: Repeat a code length of 0 for 11 - 138 times (7 bits of length)
Oh, and a lot of the jargon used is from coding/communication theory, and has a subtly different meaning than the same jargon used in the wider programming domain.
It's not so much the jargon that was the problem for me but just understanding where to put the implied data and how to use it, that problem is now over and I will always have my upload for reference... provided gitlab has appropriately prepared for that predicted solar flair or something that supposed to happen in 2025 I thnk... or was it 2023? Either way it's supposed to effect every electronic device in the world so I'm considering just doing a mass print of my code as a fallback
For anyone that was looking through my deflation code I've done a cleanup up, the main loop is now easier to understand and the deflation types are properly separated into functions (made it convenient to use the type as an index to a function instead of if statements)
Gitlab - Awsdert - Deflate
Last edited by awsdert; 08-06-2021 at 12:10 PM. Reason: Undescriptive name pulled for url
Right, this is potentially the final edit to the localised version of the zlib code, before I try to plug it into my PNG code I'd like anyone interested to have a glance through the code looking for any problems or potential problems they see (besides somewhat messy code)
GitLab - Awsdert - Deflate
I made it so the allocator can be defined by the developer, same with output files or streams (like stdout or stderr)
Last edited by awsdert; 08-08-2021 at 07:54 AM. Reason: Forgot url only pulls a poor description of linked content
By the lack of responses I can see that either no-one was interested or they didn't find any problems, I'm betting the former, oh well I just deal with them as they crop up during the port process, I'll leave the code up there for others to reference though, even if the code is messy it at least makes it easier to understand the flow of analysis for the zlib format.
Perhaps some of us do not have accounts with GitLab, and prefer NOT to create yet another account we don't need or want! If you want people to download your code, then make it publicly, and freely accessible.Originally Posted by awsdert
It is though, you download the same way you download with github, just instead of the domain being github.com it gitlab.com, literally no difference in download method
Edit: Didn't realise gitlab defaulted to private access, changing it now
Last edited by awsdert; 08-08-2021 at 05:21 PM.
Are you sure that this is correct?
It might pay to run some test data though it and verify... here's the section of the RFC:Code:int ExpandZlibArchiveType0( ZLIB *zlib ) { STREAM *Stream = zlib->Stream; BUFFER *Into = GrabBuffer( zlib->Buffers, zlib->Into ); uchar *into = Into->addr; uint leng = StreamBits( Stream, 16, true ); FlipBits( &leng, 16 ); Stream->have = Stream->used + (leng * 8); while ( Stream->used < Stream->have ) into[Into->used++] = (uchar)StreamBits( Stream, 8, true ); return 0; }
Code:3.2.4. Non-compressed blocks (BTYPE=00) Any bits of input up to the next byte boundary are ignored. The rest of the block consists of the following information: 0 1 2 3 4... +---+---+---+---+================================+ | LEN | NLEN |... LEN bytes of literal data...| +---+---+---+---+================================+ LEN is the number of data bytes in the block. NLEN is the one's complement of LEN.
Last edited by hamster_nz; 08-08-2021 at 05:55 PM.
Is this supposed to happen?
Code:$ gzip < main.c > a.gz $ ./a.out -f a.gz Segmentation fault
No it's not, I'll check it out later but for now I'm working on integrating my zlib code into my png code, weird though, I was sure it was working correctly as I tested on the examples given in the reference document I gave in a prior post (I've close th tab now so can't copy the url like I did before), anyways took me all day to fix all the compile time bugs during the integration so gonna take a break tonight, there's a segfault now but again I will leave that for 2mw, if anyone feels like looking through the png code to find potential causes for the segfault then here's the link:
Files * 5cfb24d92752ca48e7ef3bd3ec93bbd603d3ef26 * Lee Shallis / glEngine * GitLab
Not expecting anyone to do so but I'm not psychic, I could be wrong with my assumption, bonus if there is, no skin off of my nose if there isn't.
You were right I had implement that wrong, should be fixed now, just struggling to understand why type 1 deflation is not producing the expected string now, rather it comes out empty
Fixed the source of the bug, I haven't directly handled the "End Block" though so I think that is the last bug if any, for now though, back to PNG code
I tried this - output is wrong too:
[code]
$ echo "aaaaaabbbbbaaaaacccccdddddd" | gzip > a.gz
$ ./a.out --verbose -f a.gz
main.c:28: ArgHasIssue( '--verbose', '(null)' )
main.c:9: ArgHasIssue( '-f', 'a.gz' )
LoadGzipDetails( 0x7ffff68bc7e0, 28 )
Gzip Header: magic = 1F8B, format = 08, flags = 00000000, xflags = 00, system = 03
Flag States: TEXT = false, HCRC = false, MORE = false, NAME = false, NOTE = false, RESERVED = 0
Expected Size = 28
gzip.c:128: Stream Details: ptr = 0x7ffff68bc7e0, byte = 33, took = 264, used = 80, have = 232
Zlib Expansion Result:
aaaaaa
