writing newlines to stdout file descriptor

Printable View

06-10-2022
awsdert

writing newlines to stdout file descriptor

I get this:

Code:

... pawPrintf( "%u", 10 ) gave: 10pawPrintf( "%d, %s", did, "abc" ) gave: 2, abc ...

From this:

Code:

pawd test_Printf() { #define PRINTF( ARGS, ... ) \ do \ { \ puts( "pawPrintf( " #ARGS ", " #__VA_ARGS__ " ); gave:" ); \ did = pawPrintf( ARGS "\n", __VA_ARGS__ ); \ } \ while(0) pawd did; puts("Attempting pawPrintf...\n"); PRINTF( "%u", 10 ); if ( did < 0 ) return -1; PRINTF( "%d, %s", did, "abc" ); return -(did < 0); #undef PRINTF }

Because I don't want my library to rely on CRT (and all the problems it comes with) I'm trying to remove reliance on standard C libraries (with exceptions for things like limits.h), I'm starting off with the file descriptors since they're the native format for linux/unix/osx which makes it convenient for me to test the wrapper functions work correctly. I've tried all I can think of but I still haven't been able to get the newline character to print, or at least produce the expected output, anyone have any familiarity with this, or any ideas to try?

Edit: For reference the internal calls fob things down to here:

Code:

PAW_API pawzd pawFdPrintv( PAWFD fd, void *inf, paws args, va_list va ) { PAWS STR = {{0}}; pawzd did = pawInitStringv( &STR, args, va ); if ( did >= 0 ) { paws str = pawSeekBufferHead( &STR ); did = pawFdPrint( fd, inf, str, did ); pawTermString( &STR ); } return did; }

I used PAWFD since I need the type name to be consistent across systems, on windows it's HANDLE instead of int, the final functions ares these:

Code:

PAW_API pawzd pawFdPrint( PAWFD fd, PAWOPS *ops, void const *src, pawzd cap ) { (void)ops; if ( cap >= 0 ) return write( fd, src, cap ); return -1; } PAW_API pawzd pawFdParse( PAWFD fd, PAWOPS *ops, void *dst, pawzd cap ) { (void)ops; if ( cap >= 0 ) return read( fd, dst, cap ); return -1; } PAW_API void pawFdFlush( PAWFD fd ) { fsync(fd); }

Don't have anything for opening & closing yet, figured I'd first make sure I can output correctly before focusing on extras
06-10-2022
awsdert

Never mind, turns out I had been returning a count that was 1 less than the actual count somehow, after abandoning the separate count I was doing - which added up the returned counts from the specifier handlers - and instead returning the count built into the buffer object I was using for the string the output started getting the newline I was expecting.
06-10-2022
hamster_nz

Traditionally DOS/Windows uses CR+LF as a line terminator, UNIX/Linux uses just the LF character.

Try just writing "Line 1\r\nLine 2\r\n" and see if it get what you are looking for, vs "Line 1\nLine 2\n".
06-11-2022
awsdert

Quote:

Originally Posted by hamster_nz

Traditionally DOS/Windows uses CR+LF as a line terminator, UNIX/Linux uses just the LF character.

Try just writing "Line 1\r\nLine 2\r\n" and see if it get what you are looking for, vs "Line 1\nLine 2\n".

Thanks but I'm already aware of those differences, guessing I didn't make it clear but I sorted the issue after finding out I had been returning the wrong count of printed characters from pawInitStringv (where the actual work is done, I got it down to just 2 calls to pawAlloc - callback pointer that can be overridden by user/developer), the count that was being returned was 1 less than the amount stored into the temporary buffer (which had the correct count attached to it anyways), somehow the amounts returned by the callbacks being utilised by pawInitStringv for the specifiers were either wrong (which I doubt) or the starting count was wrong (more likely given it was only 1 off). I chose to just abandon adding up the amounts returned by those functions and instead just return the count attached to the buffer they join their strings onto.

Anyways the \n difference will be dealt with via a macro that I'll define for it, something like PAWS_NL, ah since it's not super obvious from snippets alone, I'm arranging my integer and string typedefs to match the specifiers, so pawd would match signed int, paws / PAWS_* would be for char const *, etc paw/Paw/PAW is just the library prefix (standing for Platform API Wrapper), as for pawLf I chose pawjf instead for now, since j is being used to represent biggest similar type on integers, strings & floats in paw, L will still work as a modifier for that particular type but to enable macros like PAWLF_MAX I need to make the supported modifier a unique letter, j suffices until I think of a suitable one or the standard comes out with one for me. Additionally instead of some nonsense like register_printf_function I chose to make a custom specifier that allows you to change the function list in place, I'm using %! for now since that seemed most suitable to me since ! implies it is important and and should have special attention, I'd say that is difficult to confuse.

Edit: Since I brought it up, what do you think of these modifiers?

Code:

typedef char pawc; typedef wchar_t pawlc; typedef char16_t pawhc; typedef char32_t pawjc; typedef char const *paws; typedef wchar_t const *pawls; typedef char16_t const *pawhs; typedef char32_t const *pawjs; typedef char pawsi[bitsof(pawjd)]; #if CHAR_BIT >= 8 typedef char pawzc; typedef char const *pawzs; #define PAWZC_C( LITERAL ) LITERAL #define PAWZC_MAX CHAR_MAX #define PAWZC_MIN CHAR_MIN #else typedef wchar_t pawzc; typedef wchar_t const *pawzs; #define _PAWZC_C( LITERAL ) L##LITERAL #define PAWZC_C( LITERAL ) _PAWZC_C( LITERAL ) #define PAWZC_MAX WCHAR_MAX #define PAWZC_MIN WCHAR_MIN #endif #ifdef PAW_ON_WINDOWS typedef TCHAR pawtc; typedef TCHAR const *pawts; #define PAWTC_C( LITERAL ) TEXT( LITERAL ) #define PAWTC_MAX TCHAR_MAX #define PAWTC_MIN TCHAR_MIN #else typedef char pawtc; typedef char const *pawts; #define PAWTC_C( LITERAL ) LITERAL #define PAWTC_MAX CHAR_MAX #define PAWTC_MIN CHAR_MIN #endif

Since char32_t is currently the biggest I decided it was the best choice for the j modifier, char16_t is usually half it's size making it a natural choice for the h modifier, char8_t doesn't exist yet so I went with the next best thing, since I knew char was always chosen for it when it is >= 8 bits I just followed suit there, wchar_t seemed like the next best thing to it on the rare systems that char is not at least 8 bits. Because TCHAR exists on windows I figured I'd avoid confusion on the t modifier and just reserve it for that, that naturally meant a suitable type was needed for linux and apple, as far as I know char is the internal type used in both so I felt it fine to just use it for the t modifier there. I used the z modifier because z doesn't have a consistent size across systems anyways so seemed suitable for a guaranteed 8 bit char, though as I was writing this post I realised I coulda used the hh modifier instead as it would usually be half char16_t's size. l was already reserved for wchar_t/wint_t so I just rolled with it. Don't like that lc doesn't necessarily refer to the same size and sign as wchar_t, why is that the case anyways?
06-11-2022
Schol-R-LEA-2

Quote:

Because I don't want my library to rely on CRT (and all the problems it comes with) I'm trying to remove reliance on standard C libraries

If it wouldn't be too much to ask (and assuming you haven't answered this before), why do you see the C standard I/O as being specific to consoles? It isn't; quite the contrary, it is designed primarily for file streams, with the functions which default to stdin and stdout being special cases for handling common uses (hence why standard C is designed exclusively for stream-buffered I/O, requiring additional libraries such as Curses for more complex console handling). Could you elaborate on what you mean here, please?

It should be noted that - as often comes up when discussing text mode in operating systems development, among other places - in consoles/terminals/CRT, the concept of newline differs from how it is represented in a file stream. In the stream (and in a file) a newline is a character or combination of characters, but when displayed to a console (or historically, printed out to teletype), that representation has to be replaced with the actions of returning the display cursor (or print head) to the beginning of the line of text (Carriage Return) and advancing the cursor or print head one line down the page (Line Feed). While this won't come up when working exclusively with files, it will come up if you then try applying the file stream functions to a console output.
06-11-2022
hamster_nz

Quote:

Originally Posted by Schol-R-LEA-2

If it wouldn't be too much to ask (and assuming you haven't answered this before), why do you see the C standard I/O as being specific to consoles?

I think in this case CRT is "the C Runtime", not "Cathode Ray Tube". But then again Google shows "Critical Race Theory" for CRT. A library that does I/O but doesn't build on the standard C library seems very hard to integrate with. Each to their own I guess.

OP: Your first reply wasn't there while I was writing mine, although mine was most likely sitting on my phone unsaved for a while.

TBH I didn't look at the code too much, just enough to see that the problem was not in the code posted, and made a stab in the dark about the CR/NL thing as being a possible cause.

I've got no opinion on the list of modifiers - it doesn't address any of my needs and I have no intention of using it, so who am I to comment? My lack of opinion isn't agreement or disagreement, approval or disapproval, just apathy.

Where I do have an opinion is that too much reliance on the preprocessor is a bad thing. It makes code very opaque, harder to write and harder to debug. There are times where using the preprocessor in such ways makes sense, but they are few and far between.
06-11-2022
awsdert

Quote:

Originally Posted by Schol-R-LEA-2

If it wouldn't be too much to ask (and assuming you haven't answered this before), why do you see the C standard I/O as being specific to consoles? It isn't; quite the contrary, it is designed primarily for file streams, with the functions which default to stdin and stdout being special cases for handling common uses (hence why standard C is designed exclusively for stream-buffered I/O, requiring additional libraries such as Curses for more complex console handling). Could you elaborate on what you mean here, please?

It should be noted that - as often comes up when discussing text mode in operating systems development, among other places - in consoles/terminals/CRT, the concept of newline differs from how it is represented in a file stream. In the stream (and in a file) a newline is a character or combination of characters, but when displayed to a console (or historically, printed out to teletype), that representation has to be replaced with the actions of returning the display cursor (or print head) to the beginning of the line of text (Carriage Return) and advancing the cursor or print head one line down the page (Line Feed). While this won't come up when working exclusively with files, it will come up if you then try applying the file stream functions to a console output.

Er, no I don't see the C library as being exclusively for consoles, that just happens to be the easiest place to start since what I'm aiming to move onto before I take the detachment from CRT seriously is wrapping OpenGL & Vulkan and trying out some stuff I have in my head for rotation without PI & trigonometry. Additionally I needed custom specifier support anyways so it just made sense to start where I could see the results clearly.

As for the newline sequence, why'd you think I gave the example

Code:

#define PAWS_NL "\n"

instead of

Code:

#define PAWC_NL '\n'

in my previous post (probably from the edit down)? It's precisely because of that sequence deal that I didn't try to put it in character form and instead chose string form.

Edit: Just looked at my previous post and now realise I didn't elaborate on the content of PAWS_NL, though given the edit it was relatively easy to work out
06-11-2022
awsdert

Quote:

Originally Posted by hamster_nz

I think in this case CRT is "the C Runtime", not "Cathode Ray Tube". But then again Google shows "Critical Race Theory" for CRT. A library that does I/O but doesn't build on the standard C library seems very hard to integrate with. Each to their own I guess.

OP: Your first reply wasn't there while I was writing mine, although mine was most likely sitting on my phone unsaved for a while.

TBH I didn't look at the code too much, just enough to see that the problem was not in the code posted, and made a stab in the dark about the CR/NL thing as being a possible cause.

I've got no opinion on the list of modifiers - it doesn't address any of my needs and I have no intention of using it, so who am I to comment? My lack of opinion isn't agreement or disagreement, approval or disapproval, just apathy.

Where I do have an opinion is that too much reliance on the preprocessor is a bad thing. It makes code very opaque, harder to write and harder to debug. There are times where using the preprocessor in such ways makes sense, but they are few and far between.

Well I consider apathy close enough to agreement in that you're not disagreeing so that's good enough for me, I wouldn't expect the library to solve your needs when I'm currently writing it for my own, though speaking of what your needs would be, I'm curios, what are they, would be nice to have them in the back of my mind while designing the library.

As for reliance on the pre-processor, nah I'm not over-reliant on it, it's just a lot more convenient for particular scenarios, for example the newline sequence I spoke of, or making debug only flags (example below), or system specific calls/typedefs.

As for your assumption that I meant the C runtime, yeah that's right, rather I didn't expect anyone to assume any other meaning in this forum since I figured that besides newbies, anyone in this forum would be rather familiar with the problems the windows CRT has due to multiple implementations.

Code:

#define PAW_O__DBGMEM 0x8000 #ifdef _DEBUG #define PAW_O_DBGMEM PAW_O__DBGMEM #else #define PAW_O_DBGMEM 0 #endif
06-12-2022
hamster_nz

Quote:

Originally Posted by awsdert

... though speaking of what your needs would be, I'm curios, what are they, would be nice to have them in the back of my mind while designing the library.

My needs are modest. I only wish that C's bit-level operations were more flexible. I work in an hardware R&D job so a good day's work is a lot of research and if there is some C code it is something like this:

Code:

#include <stdio.h> #include <stdint.h> #include <stdlib.h> uint32_t tausworthe(void) { static uint32_t s0=0xFFFFFFFFU, s1=0xFFFFFFFFU, s2=0xFFFFFFFFU; s0 = ((s0 & 0xFFFFFFFE) << 12) ^ (((s0 <<13) ^ s0) >> 19); s1 = ((s1 & 0xFFFFFFF8) << 4) ^ (((s1 << 2) ^ s1) >> 25); s2 = ((s2 & 0xFFFFFFF0) << 17) ^ (((s2 << 3) ^ s2) >> 11); return s0 ^ s1 ^ s2; } int main(void) { for(int i = 0; i < 1000; i++) { uint32_t x = tausworthe(); } while(1) { uint32_t x = tausworthe(); fwrite(&x,4,1,stdout); } return 0; }

It passes all the "Dieharder" randomness tests, (but sadly only scores "weak" on the two of them) and can be implemented in hardware to generate multi-GB/s streams of pseudo-randomness.

Tomorrow I think I will be working on fast approximations for some transcendental functions over a limited range.

Quote:

...I figured that besides newbies, anyone in this forum would be rather familiar with the problems the windows CRT has due to multiple implementations.

I first programmed C for DOS on 8086s, and on Windows C since the 80286 days. I now only write C on Linux, but sometimes port to Windows. As far as I know, current C runtimes and standard C libraries follow the standards.
06-12-2022
awsdert

Quote:

Originally Posted by hamster_nz

My needs are modest. I only wish that C's bit-level operations were more flexible. I work in an hardware R&D job so a good day's work is a lot of research and if there is some C code it is something like this:

Code:

#include <stdio.h> #include <stdint.h> #include <stdlib.h> uint32_t tausworthe(void) { static uint32_t s0=0xFFFFFFFFU, s1=0xFFFFFFFFU, s2=0xFFFFFFFFU; s0 = ((s0 & 0xFFFFFFFE) << 12) ^ (((s0 <<13) ^ s0) >> 19); s1 = ((s1 & 0xFFFFFFF8) << 4) ^ (((s1 << 2) ^ s1) >> 25); s2 = ((s2 & 0xFFFFFFF0) << 17) ^ (((s2 << 3) ^ s2) >> 11); return s0 ^ s1 ^ s2; } int main(void) { for(int i = 0; i < 1000; i++) { uint32_t x = tausworthe(); } while(1) { uint32_t x = tausworthe(); fwrite(&x,4,1,stdout); } return 0; }

It passes all the "Dieharder" randomness tests, (but sadly only scores "weak" on the two of them) and can be implemented in hardware to generate multi-GB/s streams of pseudo-randomness.

Tomorrow I think I will be working on fast approximations for some transcendental functions over a limited range.

I first programmed C for DOS on 8086s, and on Windows C since the 80286 days. I now only write C on Linux, but sometimes port to Windows. As far as I know, current C runtimes and standard C libraries follow the standards.

Welp that bit operation flexibility is beyond my ability to fix, for that particular example I'd say a temporary macro would be suitable:

Code:

uint32_t tausworthe_via_pointers(uint32_t *s1, uin32_t *s2, uint32_t *s3) { #define TMP(S, S_AND, LSH, S_LSH, RSH) *S = (((*S) & S0_AND) << LSH) ^ ((((*S) << S0_LSH) ^ (*S)) >> RSH) TMP(s0,0xFFFFFFFE,12,13,19); TMP(s1,0xFFFFFFF8, 4, 2,25); TMP(s2,0xFFFFFFF0,17, 3,11); #undef TMP return (*s0) ^ (*s1) ^ (*s2); }

Not ideal but I cannot think of a better way to do it, as for those extra tests, maybe doing the same ops but with reverse values might work? Here's what I mean using the example I gave combined with a modified version of your example:

Code:

uint32_t tausworthe() { static uint32_t s1 = UINT32_MAX, s2 = UINT32_MAX, s3 = UINT32_MAX; tausworthe_via_pointers( &s1, &s2, &s3 ); return tausworthe_via_pointers( &s3, &s2, &s1 ); }

As for the current C runtimes, to my knowledge it's not the standards they follow that's the issue (well except in certain cases like scanf), it's that there's multiple versions of those runtimes causing ABI issues
06-12-2022
hamster_nz

2 Attachment(s)

Once tested in C, it gets re-written into a hardware description language (HDL):

Code:

if rising_edge(clk) then random <= s0 XOR s1 XOR s2; s0 <= (s0(19 downto 1) & s0(31 downto 19)) XOR (zero(31 downto 13) & s0(18 downto 6)); s1 <= (s1(27 downto 3) & s1(31 downto 25)) XOR (zero(31 downto 7) & s1(29 downto 23)); s2 <= (s2(14 downto 4) & s2(31 downto 11)) XOR (zero(31 downto 21) & s2(28 downto 8)); end if;

So using the preprocessor isn't an improvement, nor is using pointers.

Code:

TMP(s2,0xFFFFFFF0,17, 3,11);

By itself this line is pretty ambiguous, and can't be mapped to hardware.

The rest of my design flow is that the HDL is verified in simulation against the C implementation, then converted to hardware and sort-of-implemented in silicon of an FPGA..

In case you are interested, attached is the resulting logical h/w schematic and a snippet of the FPGA die where the PRNG is placed.

The purple bits is logic resources that makes up the random number generator, the green lines are active wires in this design, and the gray lines are possible connections.
06-12-2022
awsdert

Quote:

Originally Posted by hamster_nz

Once tested in C, it gets re-written into a hardware description language (HDL):

Code:

if rising_edge(clk) then random <= s0 XOR s1 XOR s2; s0 <= (s0(19 downto 1) & s0(31 downto 19)) XOR (zero(31 downto 13) & s0(18 downto 6)); s1 <= (s1(27 downto 3) & s1(31 downto 25)) XOR (zero(31 downto 7) & s1(29 downto 23)); s2 <= (s2(14 downto 4) & s2(31 downto 11)) XOR (zero(31 downto 21) & s2(28 downto 8)); end if;

So using the preprocessor isn't an improvement, nor is using pointers.

Code:

TMP(s2,0xFFFFFFF0,17, 3,11);

By itself this line is pretty ambiguous, and can't be mapped to hardware.

The rest of my design flow is that the HDL is verified in simulation against the C implementation, then converted to hardware and sort-of-implemented in silicon of an FPGA..

In case you are interested, attached is the resulting logical h/w schematic and a snippet of the FPGA die where the PRNG is placed.

The purple bits is logic resources that makes up the random number generator, the green lines are active wires in this design, and the gray lines are possible connections.

When you're just experimenting with C code or making a software version (albeit not as fast as hardware) the preprocessor macros & the pointers are useful, besides you can always just preparse the file after the experiments to unroll the macros before parsing the result through the intended parser, I bet it would be a ton easier to offload macro support etc into a proper compiler and focus whatever parser you're using on the just the preparsed output. Anyways I concede your point about the parser have a harder time with macros, though if you meant to the human eye I would beg to differ in the example I gave (besides the fact I made typos in the names), I mean the code is literally right there in front of the developer, if they really can't make heads or tails of what the macro does despite that then they probably shouldn't be dealing with making hardware versions of the code until much later (like maybe a year or 2 later).

As for the diagrams, while I do find them interesting and have a rough understanding of what they're doing, I'm more into code than circuits so I don't fully grasp them (also I'm too lazy to check whatever it is I can sense I don't understand despite only giving them a minor look for 10 or so seconds), thanks anyways, it's still appreciated, pity I don't seem to be of help in this case. The only other idea I have for fixing those 2 weak tests is to maybe lookup an existing random number generator that does pass them then pass the output of your tausworthe function/circuit and see if that resolves it, by the sounds of it your being paid to design & make the circuit, I imagine the contractor or whoever won't mind a minor increase in the circuit's price so long as it doesn't require some additional license fee.