Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Mounting memory buffer as a file without writing to disk

I have a server and needs to feed data from clients to a library; however, that library only supports reading files (it uses open to access the file).

Since the data can get pretty big, I rather not write it out to a temporary file, read it in with the library then delete it afterwards. Instead I would like to do something similar to a ramdisk where there’s a file in which the content is actually in memory.

However, there can be multiple clients sending over large data, I don’t think constantly calling mount and umount to create a ramdisk for each client is efficient. Is there a way for me to mount an existing memory buffer as a file without writing to disk?

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

  • The library does not support taking in a file descriptor nor FILE*. It will only accept a path which it feeds directly to open
  • I do have the library’s source code and attempted to add in a function that uses fmemopen; however, fmemopen returns a FILE* with no file descriptor. The internals of the library works only with file descriptors and it is too complex to change/add support to use FILE*
  • I looked at mmap, but it appears to be no different than writing out the data to a file
  • Using mount requires sudo access and I prefer not to run the application as sudo
bool IS_EXITING = false;
ssize_t getDataSize( int clientFD ) { /* ... */}

void handleClient( int clientFD ) {
  // Read in messages to get actual data size
  ssize_t dataSize = getDataSize( clientFD );

  auto* buffer = new char[ dataSize ];

  // Read in all the data from the client
  ssize_t bytesRead = 0;
  while( bytesRead < dataSize ) {
    int numRead = read( clientFD, buffer + bytesRead, dataSize - bytesRead );
    bytesRead += numRead;
    
    // Error handle if numRead is <= 0
    if ( numRead <= 0 ) { /* ... */ }
  }
  
  // Mount the buffer and get a file path... How to do this
  std::string filePath = mountBuffer( buffer );

  // Library call to read the data
  readData( filePath );

  delete[ ] buffer;
}

void runServer( int socket )
  while( !IS_EXITING ) {
    auto clientFD = accept( socket, nullptr, nullptr );
    // Error handle if clientFD <= 0
    if ( clientFD <= 0 ) { /* ... */ }

    std::thread clientThread( handleClient, clientFD );
    clientThread.detach( ); 
  }
}

>Solution :

Use /dev/fd. Get the file descriptor of the socket, and append that to /dev/fd/ to get the filename.

If the data is in a memory buffer, you could create a thread that writes to a pipe. Use the file descriptor of the read end of the pipe with /dev/fd.

Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading