The Case of the Secret Compartment in the plain-text file


Hidden data somewhere in a big opaque file? Nobody would be surprised.

But hiding in a very short plain-text file? It sounds impossible until you use Amazon’s AWS tools:

% echo hello > hello
% aws s3 cp hello s3://indiecomp-test/hello > out
% cat out 
upload: ./hello to s3://indiecomp-test/hello

Seems all very straightforward, right? The trouble was, this line never matched my regular expression looking for “upload:” at the beginning of the line. After hair pulling, “od” to the rescue:

% od -c out
0000000 C o m p l e t e d   6   B y t e
0000020 s / 6   B y t e s   ( 5   B y t
0000040 e s / s )   w i t h   1   f i l
0000060 e ( s )   r e m a i n i n g \r u
0000100 p l o a d :   . / h e l l o   t
0000120 o   s 3 : / / i n d i e c o m p
0000140 - t e s t / h e l l o 
0000160                           \n
0000176

What? My file started with “upload:” not with “Completed 6 Bytes”  … until we keep reading and we come across \r — which apparently, on my terminal, puts the cursor back at the beginning of the line, and “upload” overwrites what was there before.

In all the years of using command-line tools, I have never seen anything like this. Can somebody explain to me how this is a good idea? You know, dear Amazon programmers, there are accepted ways of emitting progress messages. This is not one of them, as any shell programmer will tell you who just wasted some otherwise perfectly fine time on this nonsense.