Bash for loop treat all files at once instead of one after another

I’m facing a weird issue and can’t figure out where its from

I have 2 similar functions :

TMP_DIR="/tmp/folder"
OUTPUT_DIR="output"

func1()
{
   for file in `ls $TMP_DIR/*_INPUT_VLAN.csv`
   do
     local hostname=$(echo $file | awk -F '_INPUT' '{print $1}' | cut -d "/" -f 4)
     OUTPUT=$(echo $hostname"_net_conf.txt")
     IFS=";"
     cat $file | sed '1d' | while read f1 f2 f3 f4 f5 f6 f7 f8
     do
       echo "some things with $f1 $f2 .." >> $OUTPUT_DIR/$OUTPUT
     done
   done
}

func2()
{
   for file in `ls $TMP_DIR/*_INPUT_ROUTES.csv`
   do
     local hostname=$(echo $file | awk -F '_INPUT' '{print $1}' | cut -d "/" -f 4)
     OUTPUT=$(echo $hostname"_net_conf.txt")
     IFS=";"
     cat $file | sed '1d' | while read f1 f2 f3 f4
     do
       echo "some things with $f1 $f2 .." >> $OUTPUT_DIR/$OUTPUT
     done
   done
}

My input files look like this :

file1 : HOSTNAME1_INPUT_VLAN.csv

int;type;vxid;name;id;ip;mask;comment
int1;int;0;vlan1;1;192.168.1.1;255.255.255.0;VLAN Network 1
int2;ext;1;vlan2;1;192.168.2.1;255.255.255.0;VLAN Network 2
int3;int;2;vlan3;1;192.168.3.1;255.255.255.0;VLAN Network 2

file2 : HOSTNAME2_INPUT_VLAN.csv

int;type;vxid;name;id;ip;mask;comment
int1;int;0;vlan1;1;192.168.1.1;255.255.255.0;VLAN Network 1
int2;ext;1;vlan2;1;192.168.2.1;255.255.255.0;VLAN Network 2
int3;int;2;vlan3;1;192.168.3.1;255.255.255.0;VLAN Network 2

file1 : HOSTNAME1_INPUT_ROUTES.csv

Network;Interface;Gateway;Comment
0.0.0.0/0;int1;gateway1;Comment 1
net_1;int2;gateway2;Comment2
net_2;int2;gateway2;Comment2,1
net_3;int3;gateway3;Comment3

file2 : HOSTNAME2_INPUT_ROUTES.csv

Network;Interface;Gateway;Comment
0.0.0.0/0;int1;gateway1;Comment 1
net_1;int2;gateway2;Comment2
net_2;int2;gateway2;Comment2,1
net_3;int3;gateway3;Comment3

When I run the script, the first function succeed but it looks like the second one is not doing the for loop and try to cat both files at once :

cat: '/tmp/folder/HOSTNAME1_INPUT_ROUTES.csv'$'\n''/tmp/folder/HOSTNAME2_INPUT_ROUTES.csv': No such file or directory

If I had some echos in the functions to see what happens, I see that both files are proceed at the same time

func1()
{
  for file in `ls $TMP_DIR/*_INPUT_VLAN.csv`
  do
    local hostname=$(echo $file | awk -F '_INPUT' '{print $1}' | cut -d "/" -f 4)
    OUTPUT=$(echo $hostname"_net_conf.txt")
    echo "Hostname for $file will be : $hostname"
    echo "Ouput for $file will be located in : $OUTPUT_DIR/$OUTPUT"
  done
}

function2()
{
  for file in `ls $TMP_DIR/*_INPUT_ROUTES.csv`
  do
    local hostname=$(echo $file | awk -F '_INPUT' '{print $1}' | cut -d "/" -f 4)
    OUTPUT=$(echo $hostname"_net_conf.txt")
    echo "Hostname for $file will be : $hostname"
    echo "Ouput for $file will be located in : $OUTPUT_DIR/$OUTPUT"
  done
}

Output :

Hostname for /tmp/folder/HOSTNAME1_INPUT_VLAN.csv will be : HOSTNAME1
Ouput for /tmp/folder/HOSTNAME1_INPUT_VLAN.csv will be located in : output/HOSTNAME1_net_conf.txt

Hostname for /tmp/folder/HOSTNAME1_INPUT_ROUTES.csv
/tmp/folder/HOSTNAME2_INPUT_ROUTES.csv will be : HOSTNAME1
HOSTNAME2
Ouput for /tmp/folder/HOSTNAME1_INPUT_ROUTES.csv
/tmp/folder/HOSTNAME2_INPUT_ROUTES.csv will be located in : output/HOSTNAME1
HOSTNAME2_net_conf.txt

cat: '/tmp/folder/HOSTNAME1_INPUT_ROUTES.csv'$'\n''/tmp/folder/HOSTNAME2_INPUT_ROUTES.csv': No such file or directory

Both functions do the for loop in the same why, I don’t get why the func2 doesn’t proceed as excepted.

I find a workaround by listing files in directory, storing in a tmp file and reading the tmp file with a while loop though, but I wanna know why the original for loop is not not working

Thanks in advance for your help

>Solution :

func1 sets IFS to ; so the output of ls $TMP_DIR/*_INPUT_ROUTES.csv in func2 isn’t being split on newlines anymore. IFS controls how word splitting is performed so you need to be very careful when modifying it.

Some general recommendations:

• Don’t parse ls. Use globs instead.

  for file in "$TMP_DIR"/*_INPUT_VLAN.csv ; do
    ...
  done

• Try to localize your IFS changes using scoped environment variables or subshells where possible. For example if you just need IFS changed for your while-loop then use this. The change to IFS will be limited to your read command this way.

cat $file | sed '1d' | while IFS=';' read f1 f2 f3 f4 f5 f6 f7 f8
     do
       echo "some things with $f1 $f2 .." >> $OUTPUT_DIR/$OUTPUT
     done

• Run your script through shellcheck to look for common issues. It will highlight many common issues like the ls parsing problem.