Bash: Piping data into a command using heredocs
I’ve been playing around with some data modelled in neo4j recently and one thing I wanted to do is run an adhoc query in the neo4j-shell and grab the results and do some text manipulation on them.
For example I wrote a query which outputted the following to the screen and I wanted to sum together all the values in the 3rd column:
| ["1","2","3"] | "3" | 1234567 |
| ["4","5","6"] | "6" | 8910112 |
Initially I was pasting the output into a text file and then running the following sequence of commands to work it out:
$ cat blah2.txt| cut -d"|" -f 4 | awk '{s+=$0} END {print s}'
10144679
One way to avoid having to create blah2.txt would be to echo the output into standard out like so:
$ echo "| ["1","2","3"] | "3" | 1234567 |
| ["4","5","6"] | "6" | 8910112 | " | cut -d"|" -f 4 | awk '{s+=$0} END {print s}'
10144679
But it gets a bit confusing as the number of lines of results increases and you have to keep copy/pasting the cut and awk parts of the chain around which was annoying.
One of the things I read on the bus this week was a blog post going through a bunch of bash one liners and half way through it covers piping data into commands using heredocs which I’d completely forgotten about!
A simple example could be to send a simple message to cat which will output the message to standard out:
$ cat <<EOL
heredoc> hello i am mark
heredoc> EOL
hello i am mark
That works if we want to pipe data into a single command but I didn’t know how we’d be able to pipe the output of that command to another command.
In fact it’s actually reasonably simple:
$ cat <<EOL | cut -d"|" -f 4 | awk '{s+=$0} END {print s}'
pipe pipe heredoc> | ["1","2","3"] | "3" | 1234567 |
pipe pipe heredoc> | ["4","5","6"] | "6" | 8910112 |
pipe pipe heredoc> EOL
10144679
And now I have no need to create random text files all over my machine!
About the author
I'm currently working on short form content at ClickHouse. I publish short 5 minute videos showing how to solve data problems on YouTube @LearnDataWithMark. I previously worked on graph analytics at Neo4j, where I also co-authored the O'Reilly Graph Algorithms Book with Amy Hodler.