My institution is slowly transiting from SAS to R, most of the code is written in arrow/dplyr or data.table, using the .parquet format as its main storing format.
In my own work I am usually dealing with storing and analysing data from 1 million to 10 million rows and up to 150-200 columns.
Parquet format is great for this kind of usage, but an unusual error has been occuring recently, and I couldn’t find any ressource on the internet :

library(arrow)
library(tidyverse)

open_dataset(data_error)

Error in `open_dataset()`:
! IOError: Error creating dataset. Could not read schema from 'path/example.parquet'. 
Is this a 'parquet' file?: 
Could not open Parquet input source 'path/example.parquet':
Couldn't deserialize thrift: TProtocolException: Exceeded size limit

The same would happen with the function read_parquet.

What is data_error ?

data_error is just a typical data.frame, extracted from a bigger data source (let’s call it data_clean), through a few data.table processes and saved by write_parquet unpartitionned. Please not that this error does not occur if the parquet file is partionnized.

This error first happened on a program on data.table that I didn’t write, and I’m not familiar enough with data.table to understand the underlying issue.

repex :

library(arrow)
library(data.table)
# Seed
set.seed(1L)
# Big enough data.table 
dt = data.table(x = sample(1e5L, 1e7L, TRUE), y = runif(100L)) 
# Save in parquet format
write_parquet(dt, "example_ok.parquet")
# Readable
dt_ok <- open_dataset("example_ok.parquet")
# Simple filter 
dt[x == 989L]
# Save in parquet format
write_parquet(dt, "example_error.parquet")
# Error
dt_error <- open_dataset("example_error.parquet")

Thank you all for your help !

>Solution :

The culprit is that once you call dt[x == 989L], an index is created in the data.table.

set.seed(1L)
dt = data.table(x = sample(1e5L, 1e7L, TRUE), y = runif(100L))
attr1 <- attributes(dt)
dt[x == 989L]
attr2 <- attributes(dt)
str(attr1)
# List of 4
#  $ names            : chr [1:2] "x" "y"
#  $ row.names        : int [1:10000000] 1 2 3 4 5 6 7 8 9 10 ...
#  $ class            : chr [1:2] "data.table" "data.frame"
#  $ .internal.selfref:<externalptr> 
str(attr2)
# List of 5
#  $ names            : chr [1:2] "x" "y"
#  $ row.names        : int [1:10000000] 1 2 3 4 5 6 7 8 9 10 ...
#  $ class            : chr [1:2] "data.table" "data.frame"
#  $ .internal.selfref:<externalptr> 
#  $ index            : int(0) 
#   ..- attr(*, "__x")= int [1:10000000] 17660 25871 28519 270694 275019 419020 437190 615578 628622 739696 ...

Notice the addition of the index attribute.

The default action of arrow is to store attributes; one nice side-effect of this is that dt_ok will actually be of class data.table:

head(dt_ok) |> collect() # assuming dplyr::collect is visible?
#        x         y
#    <int>     <num>
# 1: 24388 0.4023457
# 2: 59521 0.9142361
# 3: 43307 0.2847435
# 4: 69586 0.3440578
# 5: 11571 0.1822614
# 6: 25173 0.8130521

The file size is also adversely affected (not sure if you are aware of this):

file.info(list.files(pattern = "*parquet"))
#                            size isdir mode               mtime               ctime               atime  uid  gid uname grname
# example_error.parquet 209818297 FALSE  664 2024-02-12 11:34:05 2024-02-12 11:34:05 2024-02-12 11:34:05 1000 1000    r2     r2
# example_ok.parquet     25744071 FALSE  664 2024-02-12 11:33:58 2024-02-12 11:33:58 2024-02-12 11:33:58 1000 1000    r2     r2

Clearly the _error file has something more. The normal efficiency of binary-data-storage in parquet files is not afforded to R attributes, so it makes sense that 10Mi values in a vector stored less-efficiently would take up that space.

If we remove the index, the problem goes away. One way to remove the index is to manually set the order:

write_parquet(dt, "example_error.parquet")
setorder(dt)
names(attributes(dt))
# [1] "names"             "row.names"         "class"             ".internal.selfref"
write_parquet(dt, "example_ok2.parquet")
open_dataset("example_error.parquet")
# Error in open_dataset("example_error.parquet") : 
#   IOError: Error creating dataset. Could not read schema from '.../example_error.parquet'. Is this a 'parquet' file?: Could not open Parquet input source '.../example_error.parquet': Couldn't deserialize thrift: TProtocolException: Exceeded size limit
open_dataset("example_ok2.parquet")
# FileSystemDataset with 1 Parquet file
# x: int32
# y: double
# See $metadata for additional Schema metadata

My immediate thought is that this is a bug, perhaps due to the size of the attribute. For demonstration, if we instead repeat this with 100 rows, we have no problem.

set.seed(1L)
dt = data.table(x = sample(1e5L, 1e2L, TRUE), y = runif(100L))
write_parquet(dt, "small1.parquet")
open_dataset("small1.parquet")
# FileSystemDataset with 1 Parquet file
# x: int32
# y: double
# See $metadata for additional Schema metadata
dt[x == 8229L]
#        x       y
#    <int>   <num>
# 1:  8229 0.62041
str(attributes(dt))
# List of 5
#  $ names            : chr [1:2] "x" "y"
#  $ row.names        : int [1:100] 1 2 3 4 5 6 7 8 9 10 ...
#  $ class            : chr [1:2] "data.table" "data.frame"
#  $ .internal.selfref:<externalptr> 
#  $ index            : int(0) 
#   ..- attr(*, "__x")= int [1:100] 35 50 21 78 98 33 9 77 88 94 ...
write_parquet(dt, "small2.parquet")
open_dataset("small1.parquet")
# FileSystemDataset with 1 Parquet file
# x: int32
# y: double
# See $metadata for additional Schema metadata

I suggest (request, even) that you submit a bug report.