I wrote a pair of programs to allow someone to send arbitrary binary files over a specific radio mode with a restricted character set, where each character is represented with some Huffman code. One program is an encoder, the other is a decoder.
The programs work fine on Linux. It operates exclusively over stdin and stdout. I fuzzed them using 1 GiB random files. However, when I try to compile it for Windows (using either cl.exe or x86_64-w64-mingw32-gcc (on WSL), the programs refuses to output anything to any output stream.
I tried removing all outputs to stderr, tried to do two putchar() calls with '\r' '\n' (thinking it was an issue with LF vs CRLF), fflush() every time I putchar(), but it still won’t work. The only thing that worked was completely commenting out the main while loop in the encoder program, in which I managed to get the initial printed string FILE:.
With the loop not commented out, I won’t even get the first printf() with an immediate fflush(stdout) afterwards. The program, however, will halt. Below is one of the programs, which I want to debug first.
For sanity checking, I tried to compile a simple Hello World! program that uses both printf() and putchar(), and it works fine on Windows, but my specific program does not. I’m at a loss as to where to check first.
js8file_enc.c
#include "js8file_enc.h"
// From stdin
bool fill_buffer(bool * bitbuf, size_t * buf_size, bool ** bitbuf_read_head_ptr){
// Justify remaining buffer back to the beginning
if(*buf_size > 0){
memmove(bitbuf, *bitbuf_read_head_ptr, *buf_size);
}
// Read from stdin (indeterminate length), keep track of buffer size
char buf[BUF_SIZE];
// The max bytes here is sizeof(buf) - ceil(*buf_size/8) expressed in a roundabout way.
// This will fill our byte buffer as much as possible for the bit buffer to be near-full in the next step.
size_t increase = fread(buf, 1, sizeof(buf) - ((*buf_size+8-1)/8), stdin);
// Extract bits from byte buffer
for(size_t i = 0; i < increase; i++){
for(int j = 0; j < 8; j++){
bitbuf[*buf_size] = (buf[i] >> (7 - j)) & 1;
(*buf_size)++;
}
}
*bitbuf_read_head_ptr = bitbuf;
return increase > 0;
}
int main(){
printf("FILE:");
bool bitbuf[BUF_SIZE*8];
bool * bitbuf_read_head = bitbuf;
size_t buf_size = 0; // In bits
while(true){
if(buf_size < 8){ // 8 bits, length of longest key
if(!fill_buffer(bitbuf, &buf_size, &bitbuf_read_head)){
// There may still be bits remaining (<8), encode the rest
while(buf_size > 0){
for(int i = buf_size; i > 0; i--){
if(LUT[i] == NULL){
continue;
}
uint8_t key = 0;
for(int j = 0; j < i; j++){
key |= bitbuf_read_head[j] << (i - j - 1);
}
if(LUT[i][key] != 0){
putchar(LUT[i][key]);
bitbuf_read_head += i;
buf_size -= i;
break;
}
}
}
break;
}
}
int max_key_size = (buf_size < 8) ? buf_size : 8;
// Produce keys in descending order, check for first match, write to stdout, then move buffer head.
// If there are no codewords of length n (that is, LUT[n] is NULL), then we skip to the next length.
for(int i = max_key_size; i > 0; i--){
if(LUT[i] == NULL){
continue;
}
uint8_t key = 0;
for(int j = 0; j < i; j++){
key |= bitbuf_read_head[j] << (i - j - 1);
}
if(LUT[i][key] != 0){
putchar(LUT[i][key]);
bitbuf_read_head += i;
buf_size -= i;
break;
}
}
}
fflush(stdout);
fprintf(stderr, "<EOF>\n");
fflush(stderr);
return 0;
}
js8file_enc.h
#include "ref.h"
// Indexed by codeword length (.bits)
const unsigned char * LUT[9] = {
[0] = NULL,
[1] = (unsigned char[0b10]){
[0b0] = ' ',
[0b1] = 'E'
},
[2] = NULL,
[3] = NULL,
[4] = (unsigned char[0b10000]){
[0b1101] = 'T',
[0b0011] = 'A'
},
[5] = (unsigned char[0b100000]){
[0b11111] = 'O',
[0b11100] = 'I',
[0b10111] = 'N',
[0b10100] = 'S',
[0b00011] = 'H',
[0b00000] = 'R'
},
[6] = (unsigned char[0b1000000]){
[0b111011] = 'D',
[0b110011] = 'L',
[0b110001] = 'C',
[0b101101] = 'U',
[0b101011] = 'M',
[0b001011] = 'W',
[0b001001] = 'F',
[0b000101] = 'G',
[0b000011] = 'Y'
},
[7] = (unsigned char[0b10000000]){
[0b1111011] = 'P',
[0b1111001] = 'B',
[0b1110100] = '.',
[0b1100101] = 'V',
[0b1100100] = 'K',
[0b1100001] = '-',
[0b1100000] = '+',
[0b1011001] = '?',
[0b1011000] = '!',
[0b1010101] = '"',
[0b1010100] = 'X',
[0b0010101] = '0',
[0b0010100] = 'J',
[0b0010001] = '1',
[0b0010000] = 'Q',
[0b0001001] = '2',
[0b0001000] = 'Z',
[0b0000101] = '3',
[0b0000100] = '5'
},
[8] = (unsigned char[0b100000000]){
[0b11110101] = '4',
[0b11110100] = '9',
[0b11110001] = '8',
[0b11110000] = '6',
[0b11101011] = '7',
[0b11101010] = '/'
}
};
ref.h
#include <stdio.h>
#include <stdbool.h>
#include <string.h>
#include <stdlib.h>
#include <stdint.h>
#include <sys/types.h>
// DEFAULT SETTING: 524288
// 512 KiB, packed (bytes read from stdin)
// 4 MiB, unpacked (512 KiB worth of bits stored in bitbuf)
// Changed for debugging purposes
#define BUF_SIZE 524288
typedef const struct codeword{
const uint8_t key;
const uint8_t bits;
} codeword;
EDIT: The first answer states that the default Windows stack size is 1 MB. I did not know this, and I’ll test out this fix.
>Solution :
This line in main():
bool bitbuf[BUF_SIZE * 8];
Where BUF_SIZE is defined in ref.h as 524288. That’s going to allocate 4 megabytes off the stack on startup. That crashes your program before it can even get to the printf("FILE:") statement.
By default, the VC compiler only allows 1MB of stack size. And if you’re compiling your program as 32-bit, you’ve already exhausted all the memory.
Solutions:
-
Change the above line to be:
bool* bitbuf = malloc(BUF_SIZE*8);(Don’t forget to
freeit when done)
OR
- Use the compiler and linker flags for increasing stack size (I can look this up if you want), but the above solution of using a dynamic memory allocation is better.
Similarly, the char buf[BUF_SIZE]; in fill_buffer is taking up 512KB of stack.