how to split a csv file into small chunks with headers using powershell

February 2, 2023

I am using the following code to split the large csv file into chunks. But the headers are not appending output files.

Here is the powershell script:

$InputFilename = Get-Content 'C:\Users\Sridhar\Downloads\Leads.csv'
$destinationPath = 'C:\Users\Sridhar\Downloads\'
$OutputfilenamePattern = 'leads_'
$LineLimit = 500
$line = 0 
$i = 0
$file = 0
$start = 0
$header = $InputFilename[0]
while ($line -le $InputFilename.Length){
 if($i -eq $LineLimit -Or $line -eq $Inputfilename.Length){
    $file++
    $Filename = "$destinationPath$OutputfilenamePattern$file.csv"   
    $InputFilename[$start..($line-1)] | Out-file $Filename -Force -Encoding utf8
    $start =$line;
    $i = 0
    Write-Host "$Filename"
}
$i++
$line++
}

>Solution :

For a compete explanation, see: Mastering the (steppable) pipeline

$BatchSize = 500
Import-Csv .\Leads.csv |
    ForEach-Object -Begin {
        $Index = 0
    } -Process {
        if ($Index % $BatchSize -eq 0) {
            $BatchNr = [math]::Floor($Index++/$BatchSize)
            $Pipeline = { Export-Csv -notype -Path .\leads_$BatchNr.csv }.GetSteppablePipeline()
            $Pipeline.Begin($True)
        }
        $Pipeline.Process($_)
        if ($Index++ % $BatchSize -eq 0) { $Pipeline.End() }
    } -End {
        $Pipeline.End()
    }