Thread: Increment gone wrong

  1. #46
    Registered User
    Join Date
    Sep 2020
    Posts
    291
    We want, ney demand, a 'leasons learned' briefing!

    What was the thing(s) you figured out, discovered or uncovered that allowed you to get the algorithm working?

    Some vagueness or ambiguity in the spec? Some bit/byte ordering issues in the code?

    What are the things that might trip up the next person who attempts this and finds themselves reading through this thread?

  2. #47
    Registered User awsdert's Avatar
    Join Date
    Jan 2015
    Posts
    1,398
    It was those hard coded values, they had indistinct naming (even in the docs) which got me confused, after I moved the related stuff to their own files and renamed them to match the files (I had an #if 1 statement for the implied code data at one point) I was able to see & compare to the tables provided here:

    Understanding gzip

    With that I was able to experiment until I got it right. Still think the design of the encoding leaves much to be desired :|

  3. #48
    Registered User
    Join Date
    Sep 2020
    Posts
    291
    The bit of RFC 1951 that I came unstuck on for a while was this:

    Code:
                   0 - 15: Represent code lengths of 0 - 15
                       16: Copy the previous code length 3 - 6 times.
                           The next 2 bits indicate repeat length
                                 (0 = 3, ... , 3 = 6)
                              Example:  Codes 8, 16 (+2 bits 11),
                                        16 (+2 bits 10) will expand to
                                        12 code lengths of 8 (1 + 6 + 5)
                       17: Repeat a code length of 0 for 3 - 10 times.
                           (3 bits of length)
                       18: Repeat a code length of 0 for 11 - 138 times
                           (7 bits of length)
    The meaning of codes 17 and 18 just escaped me for hours. Now I understand what it is trying to say I can understand it perfectly.

    Oh, and a lot of the jargon used is from coding/communication theory, and has a subtly different meaning than the same jargon used in the wider programming domain.

  4. #49
    Registered User awsdert's Avatar
    Join Date
    Jan 2015
    Posts
    1,398
    It's not so much the jargon that was the problem for me but just understanding where to put the implied data and how to use it, that problem is now over and I will always have my upload for reference... provided gitlab has appropriately prepared for that predicted solar flair or something that supposed to happen in 2025 I thnk... or was it 2023? Either way it's supposed to effect every electronic device in the world so I'm considering just doing a mass print of my code as a fallback

  5. #50
    Registered User awsdert's Avatar
    Join Date
    Jan 2015
    Posts
    1,398
    For anyone that was looking through my deflation code I've done a cleanup up, the main loop is now easier to understand and the deflation types are properly separated into functions (made it convenient to use the type as an index to a function instead of if statements)

    Gitlab - Awsdert - Deflate
    Last edited by awsdert; 08-06-2021 at 12:10 PM. Reason: Undescriptive name pulled for url

  6. #51
    Registered User awsdert's Avatar
    Join Date
    Jan 2015
    Posts
    1,398
    Right, this is potentially the final edit to the localised version of the zlib code, before I try to plug it into my PNG code I'd like anyone interested to have a glance through the code looking for any problems or potential problems they see (besides somewhat messy code)

    GitLab - Awsdert - Deflate

    I made it so the allocator can be defined by the developer, same with output files or streams (like stdout or stderr)
    Last edited by awsdert; 08-08-2021 at 07:54 AM. Reason: Forgot url only pulls a poor description of linked content

  7. #52
    Registered User awsdert's Avatar
    Join Date
    Jan 2015
    Posts
    1,398
    By the lack of responses I can see that either no-one was interested or they didn't find any problems, I'm betting the former, oh well I just deal with them as they crop up during the port process, I'll leave the code up there for others to reference though, even if the code is messy it at least makes it easier to understand the flow of analysis for the zlib format.

  8. #53
    Registered User rstanley's Avatar
    Join Date
    Jun 2014
    Location
    New York, NY
    Posts
    850
    Quote Originally Posted by awsdert View Post
    By the lack of responses I can see that either no-one was interested or they didn't find any problems, I'm betting the former, oh well I just deal with them as they crop up during the port process, I'll leave the code up there for others to reference though, even if the code is messy it at least makes it easier to understand the flow of analysis for the zlib format.
    Perhaps some of us do not have accounts with GitLab, and prefer NOT to create yet another account we don't need or want! If you want people to download your code, then make it publicly, and freely accessible.

  9. #54
    Registered User awsdert's Avatar
    Join Date
    Jan 2015
    Posts
    1,398
    Quote Originally Posted by rstanley View Post
    Perhaps some of us do not have accounts with GitLab, and prefer NOT to create yet another account we don't need or want! If you want people to download your code, then make it publicly, and freely accessible.
    It is though, you download the same way you download with github, just instead of the domain being github.com it gitlab.com, literally no difference in download method

    Edit: Didn't realise gitlab defaulted to private access, changing it now

  10. #55
    Registered User
    Join Date
    Sep 2020
    Posts
    291
    Are you sure that this is correct?

    Code:
    int ExpandZlibArchiveType0( ZLIB *zlib )
    {
        STREAM *Stream = zlib->Stream;
        BUFFER *Into = GrabBuffer( zlib->Buffers, zlib->Into );
        uchar *into = Into->addr;
        uint leng = StreamBits( Stream, 16, true );
        FlipBits( &leng, 16 );
        Stream->have = Stream->used + (leng * 8);
    
    
        while ( Stream->used < Stream->have )
            into[Into->used++] = (uchar)StreamBits( Stream, 8, true );
    
    
        return 0;
    }
    It might pay to run some test data though it and verify... here's the section of the RFC:

    Code:
          3.2.4. Non-compressed blocks (BTYPE=00)
    
             Any bits of input up to the next byte boundary are ignored.
             The rest of the block consists of the following information:
    
                  0   1   2   3   4...
                +---+---+---+---+================================+
                |  LEN  | NLEN  |... LEN bytes of literal data...|
                +---+---+---+---+================================+
    
             LEN is the number of data bytes in the block.  NLEN is the
             one's complement of LEN.
    Last edited by hamster_nz; 08-08-2021 at 05:55 PM.

  11. #56
    Registered User
    Join Date
    Sep 2020
    Posts
    291
    Is this supposed to happen?

    Code:
    $ gzip < main.c > a.gz
    $ ./a.out -f a.gz
    Segmentation fault

  12. #57
    Registered User awsdert's Avatar
    Join Date
    Jan 2015
    Posts
    1,398
    Quote Originally Posted by hamster_nz View Post
    Is this supposed to happen?

    Code:
    $ gzip < main.c > a.gz
    $ ./a.out -f a.gz
    Segmentation fault
    No it's not, I'll check it out later but for now I'm working on integrating my zlib code into my png code, weird though, I was sure it was working correctly as I tested on the examples given in the reference document I gave in a prior post (I've close th tab now so can't copy the url like I did before), anyways took me all day to fix all the compile time bugs during the integration so gonna take a break tonight, there's a segfault now but again I will leave that for 2mw, if anyone feels like looking through the png code to find potential causes for the segfault then here's the link:

    Files * 5cfb24d92752ca48e7ef3bd3ec93bbd603d3ef26 * Lee Shallis / glEngine * GitLab

    Not expecting anyone to do so but I'm not psychic, I could be wrong with my assumption, bonus if there is, no skin off of my nose if there isn't.

  13. #58
    Registered User awsdert's Avatar
    Join Date
    Jan 2015
    Posts
    1,398
    Quote Originally Posted by hamster_nz View Post
    Are you sure that this is correct?

    Code:
    int ExpandZlibArchiveType0( ZLIB *zlib )
    {
        STREAM *Stream = zlib->Stream;
        BUFFER *Into = GrabBuffer( zlib->Buffers, zlib->Into );
        uchar *into = Into->addr;
        uint leng = StreamBits( Stream, 16, true );
        FlipBits( &leng, 16 );
        Stream->have = Stream->used + (leng * 8);
    
    
        while ( Stream->used < Stream->have )
            into[Into->used++] = (uchar)StreamBits( Stream, 8, true );
    
    
        return 0;
    }
    It might pay to run some test data though it and verify... here's the section of the RFC:

    Code:
          3.2.4. Non-compressed blocks (BTYPE=00)
    
             Any bits of input up to the next byte boundary are ignored.
             The rest of the block consists of the following information:
    
                  0   1   2   3   4...
                +---+---+---+---+================================+
                |  LEN  | NLEN  |... LEN bytes of literal data...|
                +---+---+---+---+================================+
    
             LEN is the number of data bytes in the block.  NLEN is the
             one's complement of LEN.
    You were right I had implement that wrong, should be fixed now, just struggling to understand why type 1 deflation is not producing the expected string now, rather it comes out empty

  14. #59
    Registered User awsdert's Avatar
    Join Date
    Jan 2015
    Posts
    1,398
    Quote Originally Posted by awsdert View Post
    You were right I had implement that wrong, should be fixed now, just struggling to understand why type 1 deflation is not producing the expected string now, rather it comes out empty
    Fixed the source of the bug, I haven't directly handled the "End Block" though so I think that is the last bug if any, for now though, back to PNG code

  15. #60
    Registered User
    Join Date
    Sep 2020
    Posts
    291
    I tried this - output is wrong too:

    [code]
    $ echo "aaaaaabbbbbaaaaacccccdddddd" | gzip > a.gz

    $ ./a.out --verbose -f a.gz
    main.c:28: ArgHasIssue( '--verbose', '(null)' )
    main.c:9: ArgHasIssue( '-f', 'a.gz' )
    LoadGzipDetails( 0x7ffff68bc7e0, 28 )
    Gzip Header: magic = 1F8B, format = 08, flags = 00000000, xflags = 00, system = 03
    Flag States: TEXT = false, HCRC = false, MORE = false, NAME = false, NOTE = false, RESERVED = 0
    Expected Size = 28
    gzip.c:128: Stream Details: ptr = 0x7ffff68bc7e0, byte = 33, took = 264, used = 80, have = 232
    Zlib Expansion Result:
    aaaaaa

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. hex increment
    By davidx in forum C Programming
    Replies: 3
    Last Post: 10-19-2019, 07:06 AM
  2. Two pre increment in one expression
    By h255874 in forum C Programming
    Replies: 4
    Last Post: 09-21-2019, 08:47 AM
  3. Post Increment an Pre Increment operators in c++
    By anil_ in forum C++ Programming
    Replies: 4
    Last Post: 11-12-2011, 08:27 PM
  4. can't get loop to increment
    By rivkyfried1 in forum C Programming
    Replies: 2
    Last Post: 10-11-2010, 04:03 AM
  5. Post increment and pre increment help
    By noob2c in forum C++ Programming
    Replies: 5
    Last Post: 08-05-2003, 03:03 AM

Tags for this Thread