Linux dos2unix command syntax — removing hidden Windows characters from files

While editing files on a machine running some form of Windows/MacOS and uploading them to a Linux server is convenient, it can cause unforeseen complications. Windows- and MacOS-based text editors put special characters at the end of lines to denote a line return or a new line. Usually harmless, some applications on a Linux server cannot understand these characters, and so they can cause the service not to respond correctly. There is a simple way to correct this problem: dos2unix.

Key points

By reading this article, here are main points you will get out of it:

• Understanding the dos2unix command.
• Being familiar with what the Windows hidden characters look like.
• Understanding the unix2dos command.
• Knowing the benefits of using the unix2dos and dos2unix commands.
• Using the dos2unix command.
• Commonly used modes for the dos2unix command.
• Common issues when using the dos2unix command.

Understanding the dos2unix command

Windows and MacOS use the same format as DOS, where at the end of each line, there are two characters that signify the end of the line:

1. Carriage Return (\r)
2. Followed by Line Feed (\n)

While UNIX files only use the Line Feed (\n). In this regard, dos2unix will help you convert files from DOS line endings (\r + \n) to UNIX endings (\n). Usually, these characters will not appear in most Windows/MacOS text editors, but they will appear in Linux when the concatenate (cat) or vi improved (vim) commands are used.

What do the hidden characters look like?

Below is an example from a php.ini file showing what the ^M hidden characters look like:

;;;;;;;;;;;;;;;;;;;^M
; Resource Limits ;^M
;;;;;;;;;;;;;;;;;;;^M
;^M
max_execution_time = 30^M
max_input_time = 60^M
memory_limit = 32M^M

Here is another example showing what the ^M hidden characters look like:

[root@host ~]# cat -v text.txt
Hello,^M
This is a test file.^M

You can use the octal dump (od) command to show each character in a text file, including the end of the line characters, to fully understand the format of the file. Here is an example:

[root@host ~]# od -bc text.txt
0000000 110 145 154 154 157 054 015 012 124 150 151 163 040 151 163 040
         H   e   l   l   o   ,  \r  \n   T   h   i   s       i   s
0000020 141 040 164 145 163 164 040 146 151 154 145 056 015 012
         a       t   e   s   t       f   i   l   e   .  \r  \n

Understanding the unix2dos command

Sometimes, you might need to transfer a file from a Linux environment to a Windows/MacOS one, and you might need to convert the files from UNIX to the DOS format. This scenario is where the unix2dos command comes in handy. The unix2dos command is included in the dos2unix package and works in the same way and format as the dos2unix command.

Benefits of using the unix2dos and dos2unix commands

The unix2dos and dos2unix commands are a simple yet powerful set of tools for Linux users, allowing the conversion of text files from DOS/MAC to Unix format in just seconds, which is essential for file transfers between various operating systems. The commands have multiple flags, which helps when including them in scripts and automation:

• Flags for the unix2dos command
• Flags for the dos2unix command
  

How to use the dos2unix command

We will take a look at a dos2unix command example and how the conversion is done. Again, here is that example from the same php.ini file used above showing what the hidden characters look like:

;;;;;;;;;;;;;;;;;;;^M
; Resource Limits ;^M
;;;;;;;;;;;;;;;;;;;^M
;^M
max_execution_time = 30^M
max_input_time = 60^M
memory_limit = 32M^M

As stated before, these characters will not appear in most Windows text editors but will appear in Linux command line utilities. Removing them is rather painless. Just run the command dos2unix:

[root@host ~]# dos2unix /path/to/file

In the above example looking at the php.ini file, the command will look something like this:

[root@host ~]# dos2unix /usr/local/lib/php.ini
dos2unix: converting file /usr/local/lib/php.ini to UNIX format ...

That’s it. If the command has worked correctly, the file in question shouldn’t have any of the hidden characters in it, like the following:

;;;;;;;;;;;;;;;;;;;
; Resource Limits ;
;;;;;;;;;;;;;;;;;;;
;
max_execution_time = 30
max_input_time = 60
memory_limit = 32M

Commonly used modes for the dos2unix command

Here, we will discuss some of the dos2unix command syntax and flags that will allow you to change the behavior of the command based on your needs.

Conversion mode

The -c flag allows you to set the conversion mode. You can choose between ASCII, 7bit, ISO, or Mac, with ASCII format being the default one:

dos2unix -c ascii text.txt

Force conversion

The -f flag allows you to force the conversion of the TXT file:

dos2unix -f text.txt

Keep the original file

By default, dos2unix will convert the file and overwrite the original one. The -n flag allows you to keep the original file in place and create a new file with the converted format:

dos2unix -n text.txt text1.txt

Recursive mode

The -r flag converts files in recursive mode, allowing the conversion of all the files in one directory.

dos2unix -r directory/

Common issues using the dos2unix command

You might encounter different issues when running the dos2unix command, like with any tool. Despite it being straightforward and easy to use, you might confront issues during the conversion process. We will go over a few typical issues that might occur and discuss solutions to troubleshoot them.

File permissions issue

You need to make sure that the file you want to convert has the appropriate permissions to allow you to do so. The chmod command will help you change the permissions and edit the file. In this regard, you can refer to the Basic file permissions article discussing how to use the chmod command.

The "dos2unix: command not found" issue

When issuing the dos2unix command, you might encounter the "dos2unix: command not found" error. This is due to the fact that dos2unix is not included by default in all Linux distros, so you will need to install it on your machine using the following commands.

For CentOS/AlmaLinux-based distributions of Linux, use this command:

yum install dos2unix

For Ubuntu/Debian-based distros, use this command:

apt-get install dos2unix

For Arch Linux-based distros, use this command:

pacman -S dos2unix

Binary files

So, dos2unix is not intended to work with binary files. Making sure that the file you want to convert is a text-based is crucial, as trying to convert a binary file will corrupt it.

Wrapping up

The dos2unix command is a simple way to ensure that files edited and uploaded from a Windows/MacOS machine to a Linux machine work and behave correctly — and vice versa with the unix2dos command. Since DOS format line endings are different from Unix format line endings and are used by Windows and MacOS, it becomes essential to understand the dos2unix and unix2dos commands — and the flags that can be used when issuing them.

If you are uncomfortable running the commands documented in this help article yourself or have any questions or concerns, please give our professional support staff a call.