Binary v. Text

Machine-readable data - from binary to text

 (2-byte-number)(4-byte-number)(1-byte-character)(2-byte-number)....

 (4-byte-no)(1-byte-char)(2-byte-no)(4-byte-no)....

 (1-byte-character)(1-byte-character)(1-byte-character)....

• Binary file
• Text file
• Plain text
• Of course, "text" files, like all files, are ultimately stored in binary as 1s and 0s.

• ASCII and other characters

• Display whether files are binary or text:

    file *
  

  See "man file".

Text format is less efficient, but easier to work with

• Binary format: First 2 bytes of the file are:
  10010000 10000000
  You just have to "know" that the first 2 bytes are to be read together as defining an integer. If we interpret them that way, they define the number:
  36992

• Readable text format: First 16 bytes of the file are:
  00111100 01101110 01110101 01101101 00111110 00110011 00110110 00111001 00111001 00110010 00111100 00101111 01101110 01110101 01101101 00111110
  which translates byte-by-byte as the characters:
  60 110 117 109 62 51 54 57 57 50 60 47 110 117 109 62
  that is, the characters:
  '<' 'n' 'u' 'm' '>' '3' '6' '9' '9' '2' '<' '/' 'n' 'u' 'm' '>'
  i.e. when displayed in a text editor this file will read:
  <num>36992</num>
  All this to make the number human readable.

• Binary format: Write a program to edit it.

• Text format: Use text editor to edit it.

"Human readable" machine data

It is a much less efficient format - might take 16 1-byte characters to display the 2-byte-number - and this is why binary was so popular in the past. But can use such schemes now because machines more powerful, disk space bigger, bandwidth better.

HTML showed it could be done.
XML took the idea much further.
"Like HTML, XML files are text files that people shouldn't have to read, but may when the need arises."

Secret-format binary (Microsoft Office)

If you use binary Microsoft Word .doc, it is hard to write scripts to manipulate your files as you can with text files. Instead you may have to point and click inside other people's menus. This is the main reason why I never used Microsoft Word.

If you use Microsoft Word .docx, the raw data is text format (XML). This is easier to script, but there are still issues.

• Old post sums it up: What's So Bad About Microsoft?
  • Closed, secret file formats

• Alternative View of the Microsoft Monopoly - 1999 discussion on secret file formats.
  • Interesting argument that: "The file formats of MS Office were designed by Microsoft to be difficult to reverse engineer and to be as closely tied as possible to the Microsoft platform. This does not translate to a good standard. If a standard is to be decided for Word Processing it should be human readable, easily understandable, cross platform, and leave room for upgrades with bidirectional compatibility. The Office formats have no concept of expandability and are neither forward nor backward compatible because Microsoft always intends to replace the format with something incompatible in the next release to force users to upgrade. The Office file formats have no concept of interoperability because Microsoft's primary concern is forcing people to use Microsoft Office on Microsoft Windows. The Office file formats are not easy to implement or understand because part of their purpose is to delay competitors from reverse engineering them."

The end of Office secret format

• See the debate on how and why Office secret format ended in Office 2007 on.

• DOCX is text, spec is public: Office Open XML.
• DOC is binary, but spec is now public.

1. Rename file.docx to file.zip
2. Unzip it
3. See document.xml