I am using ElasticSearch, here we are creating the day wise index and huge amount of data is being ingested every minute. wanted to export few fields from index created every day to Google cloud storage. am able to achieve this with output file as json as shown below:

input {

 elasticsearch {

    hosts => "localhost:9200"

    index => "test"


    query => '

    {
    "_source": ["feild1","filed2"],

    "query": {

    "match_all": {}

    }

    }

  '

  }

}


output {
   google_cloud_storage {
     bucket => "bucketName"
     json_key_file => "creds.json"
     temp_directory => "/tmp"
     log_file_prefix => "logstash_gcs"
     max_file_size_kbytes => 1024
     output_format => "json"
     date_pattern => "%Y-%m-%dT%H:00"
     flush_interval_secs => 600
     gzip => false
     uploader_interval_secs => 600
     include_uuid => true
     include_hostname => true
   }
}

However how to export it as CSV file and send it to Google Cloud Storage

>Solution :

You should be able to change output_format to plain but this setting is going to be deprecated

You should remove output_format and use the codec setting instead, which supports a csv output format

google_cloud_storage {
   ...
    codec => csv {
        include_headers => true
        columns => [ "field1", "field2" ]
    }
}