10: Command-Line Superpowers: Connecting Simple Tools for Complex Solutions

Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Jim (00:05):
Welcome and thanks for tuning in to Runtime Arguments.
We've got a great episode foryou today on using the command
line shells.
I can hardly wait to get intoit, but first let's take care of
a few things.
This is episode 10.
Can you believe it, Wolf?
Do you believe that?
Uh that's I actually kind ofdon't.

(00:25):
It's not our 10th episode.
Uh we're programmers.
We start counting at zero.
So while it's labeled episode10, it's actually our 11th
episode.
Um I want to thank you for forlistening today, for tuning in.
And um we we we like feedback uhand we get feedback.

(00:48):
We just want more feedback.
Um in a minute, I'm gonna goover a couple of items from from
two of our previous episodes.
Um but we're wondering if if youcan give us some feedback.
We're asking for some specificfeedback here.
We would like to know how longyou would like our episodes to

(01:08):
be.
We've had some short episodes.
The shortest was 27 minuteslong, and we've had some long
episodes.
Our longest was the most recentepisode on data types.
That was an hour and 27 minutes.
I think we're shooting forsomewhere in between, but what
do you want?
Well if you want us to go morein depth, we'll do it.
Wolf, do you have uh you want toelaborate a bit on that?

Wolf (01:31):
Yeah, that's uh that's my feeling.
It's about depth.
There are some topics that arevaluable and we want to talk
about, but the they're not justall that deep.
And if there was a topic likethat, I I wouldn't want it to
run all that long.
Uh to me, the the feedback I'dlike to hear is if a topic does

(01:56):
have some depth, should we divein and spend a little more time
to make sure we get all theparts that you might care about?
Or should we cut it off so thatit's short enough for you to
consume?
Um I I really think that the uhthe whole point of our podcast

(02:18):
is to give the listener, giveyou the information that you
want.
Do you want us to go longer andgive you more?
Or uh are we giving the rightamount?
Or maybe too much.
Uh that's the feedback we'dlike.
Write us, tell us.

Jim (02:35):
Yeah, so please send us your feedback at feedback at
runtimearguments.fm.
I'll include that and more linksuh in the show notes.
Uh previous episode feedback.
Uh uh last episode was aboutdata types.
And uh I made a comment about uha 64-bit machine.

(02:57):
Uh, I think both Wolf and I weresurprised that uh an int, if
you're programming in C, an inton a 64-bit machine is not 64
bits long.
It's typically 32 bits, and thatreally surprised me, and I guess
I should have thought about itmore.
Um it turns out that most of thetime you don't need 64-bit

(03:19):
integers.
32-bit is probably overkill alot of times.
You know, think about it.
You're writing a uh a for loopin C, and you create a uh you do
int i, and then you do this fori equals zero till i is less
than some kind of small number,uh, i plus plus.
It certainly doesn't need to be64 bits.

(03:40):
And a lot of the integers youdeclare in your program are like
that.
They don't need to be reallylarge numbers.
The reason why this is importantis uh you can fit a lot more
32-bit integers in the sameamount of memory as as uh a lot
more than you can 64-bitintegers.
So the processor cache is moreefficient if you're using

(04:00):
shorter integers.
Um if you're storing thesethings uh uh on the disk, uh
shorter integers take lessspace.
It just makes a lot of sense.
Use the right size for whatyou're doing.
So I think the compiler writersthey decided uh let's make it
into only 32 bits because that'ssuitable for almost anybody.

Wolf (04:22):
And you can always ask for more.

Jim (04:25):
Yeah, if you need 64 bits, ask for a long.
If you need 128 bits, ask for along, long.
Uh there's ways to get the sizeyou need.
Uh and a and a processor, even a64-bit processor, will process
32-bit integers very, veryquickly.
I suspect faster than it wouldprocess a 64-bit integer.

(04:47):
Um anyway.
Uh the other episode, quite awhile back, we did an episode on
VPNs uh connecting safely.
Um and it came to my attention,somebody sent me a link about uh
uh we we tried to make thedistinction between VPNs and VPN
services, where VPN is somethinglike uh uh OpenVPN or WireGuard

(05:09):
or one of those, and that'swhere you connect your machine
to another machine or yourmachine to your network at work,
something like that.
Then there's VPN services.
That's something I said thatreally should be more like an
anonymizer service.
You use that to connect out tothe world, and you don't want
people to know who you are.
Maybe you're trying to watch TVwhen you're in a foreign

(05:30):
country, but you want to accessyour local uh American uh
content, so you use one of theseVPN services.
Well, it turns out that at leastone of the VPN service companies
is doing something I think ispretty nasty.
They're taking screenshots, uh,screen captures uh of the web
pages that you're browsing.

(05:52):
They claim it's to uh feed intotheir AI analys an analyzer and
uh protect you better.
I I think that's pretty creepy.
If they're actually grabbingyour screenshots, it doesn't sit
well with me.
Um there's an article about it.

(06:14):
Uh I'm gonna include the link tothat article in the show notes
uh in case you want to read moreabout it.
I I think it's pretty icky.
Um yeah.
Uh that's all I'm gonna sayabout that.

Wolf (06:27):
Makes you want the other kind of VPN.
Yeah.

Jim (06:31):
Yeah.
Yeah.
That's just uh, you know, theyall could be doing that,
especially the ones that are uhplugins to your browser that
that claim they give you safebrowsing.
I don't know.

Wolf (06:44):
So a thing you just said having to do with the size of
integers, and if this turns outto be completely uninteresting
to people, you can edit it out.
But I just have to say thisthing because it bothers me all
the time and has for decades.
And that is this.
Uh in C and C ⁇ , there is adifference between plus plus i

(07:10):
and i plus plus.
When you say plus plus i, youmean the easy thing.
You mean increment the variableand then use the value.
It's simple.
When you say i plus plus, whatyou mean is take the value the
variable has now so that you canuse and test it, and then

(07:35):
without revealing the result toanyone, increment that integer.
And every single fricking loopin C and C is written uh for I
get zero i less than n i plusplus.
And that is wrong.

(07:56):
That is absolutely backwards.
They're not using the value,they're asking the compiler to
do the harder thing, which meanssomething different and doesn't
do what they want or are askingfor.
So it turns out that most C andC compilers are smart enough to

(08:16):
say, oh, you said the wrongthing.
I'll do the right thing anyway.
It still pisses me off.
When you write code, you shouldsay what you mean.
That's my opinion.

Jim (08:25):
You know, I I write code like that, and I do the I plus
plus.
And to me, it makes perfectsense.
It means use the value and thenincrement it for the next time
around.
I I I don't understand the theissue there.
Um you know, one's prefix uhincrementing and one's postfix
incrementing.
Is that the right word for that?

Wolf (08:47):
That is the right word, and the question is how you have
written your loop.
Um is it less than n or is itless than n plus one?
Or you know, you control, areyou using that uh increment the
right way?
Okay.
Um I just hate it when peopledon't think about it.

Jim (09:05):
Uh okay.
All right.
So uh today we're here to talkabout command line shells.
Wolf, let's get into it.

Wolf (09:16):
Um the world is changing.
In the old days, um, we didn'thave GUIs and all these crazy um
interesting editors and whatnot.
What we had were we had a wholebunch of independent little
tools, and when you got a Unixand years later, Linux

(09:39):
distribution, a big part of itwas that collection of tools.
A lot of them were standard,whatever.
But most programmers today arecomfortable in their IDE, their
integrated developmentenvironment, but they still
click through file managers andmanually handle repetitive tasks

(10:00):
outside of their IDE.
Um you might know basic commandline navigation within a shell.
Uh maybe you even write theoccasional script, but you are
missing the real, actualsuperpower of command line
shells.
And it's less about the shelland the existence of all of

(10:24):
these tools that it had takendecades to gather together.
Um, and that is composition.
The art of connecting existingtools to solve problems way
harder than you are probablyfacing this very instant.
Uh, and maybe you have facedsome hard problems where this

(10:47):
would have been the rightanswer, but solving the problem
without writing any new code.
Um this episode is about howthinking like a conductor, like
an orchestra conductor, whereeach player in the orchestra,

(11:08):
they know how to do their job.
And as the conductor, you're notgonna play the song, you're
gonna combine the skills of allthose individual players who are
absolute experts, and the resultof combining them is gonna be
something magnificent.

(11:29):
Did you have to write something?
Absolutely not.
Um, so we're gonna show you howto orchestrate these simple,
individual, specialized toolsinto powerful workflows that
handle all this ridiculous,tedious work that right now

(11:52):
you're doing by hand.
Um, whether you're a Pythondeveloper who reaches for
scripts too quickly, or someonewho's been intimidated by the
terminal, when we're done, youare gonna have an understanding,
or the start of one, of how tocompose solutions from tools

(12:15):
that already exist on yourcomputer.
And and I mean whatever computeryou use.
I don't care if you're usingMac, Linux, Unix, Windows, uh I
don't care.
Uh every single computer thathas made it to today is covered
with these tiny tools that dojust the right thing, just the

(12:39):
right way.
And when we're done, you'regonna walk away and know how to
stick them together to solveyour problem.
Uh we've got five um, I don'twant to say takeaways.
We do have a takeaway, but uh Ifeel like we have five um

(12:59):
specific uh courses of action,things you can do, things you
can try this week to startbuilding your personal
superpower toolkit.
Um this I've talked about a lotof things since I was Jim.
Um Jim and I have uh specificgoals for this podcast.

(13:25):
For me, one of my goals is Iwant you, the listener, to be
better.
I want you to solve harderproblems, I want you to solve
them quicker and easier, I wantyou to get past the stupid stuff
and into the part of your jobthat's interesting and unique.

(13:47):
And this episode is a whole setof tools on how to do exactly
that that have always beenthere, have always been
available to you, and you justdidn't know about.
Um that's the plan.
So, um anything uh specific youwant to interject there, Jim?

Jim (14:10):
Well, you kind of covered the whole musical composition
thing.
I was gonna ask about that.
Um I I I think your analogythere uh uh about a musical
conductor uh conducting the theexperts to to make the magic
really, really fits what we'retalking about here.

Wolf (14:29):
Uh yeah, we're gonna talk about specific tools, and uh I I
I I want to say right up front,um everybody who's listening to
this podcast, um, the fact thatyou're listening, I think,
probably puts you in thecategory of a pretty damn good

(14:51):
programmer.
You want to learn stuff.
Just wanting to learn raises youup in the echelon.
But I'll bet, um you by yourselfcould not have written VS Code.
You by yourself could not havewritten Vim or Pycharm or uh

(15:14):
Helix.
You can do lots of great thingsand you have specialized
knowledge for the problem you'reworking on, but the fact is
you've got tools that are betterthan you.
And it turns out there's anentire set of tools on your
machine that are better thanyou.

(15:35):
You don't have to write them.
All you have to do is connectthem.
Um so let me uh give you the andI'm I'm making air quotes that
you can't see because this is anaudio podcast.
Someday maybe we'll fix that.
I don't know.
But here's a problem programmerscomfortable with Python or maybe

(15:56):
JavaScript or or whatever, uh,but doing repetitive tasks
manually.
Uh for instance, a thing thatJim and I do when we start a
podcast.
We're looking in our GUIrecording app, uh, which happens
to be audio hijack, and in audiohijack, there's a set of files

(16:18):
that were the recordings fromlast time.
We grab those files with themouse and we throw them in the
trash because we don't want themanymore.
That's a manual action.
Um if you're doing one manualaction, maybe that's okay.
Uh if you're doing somethingover and over again, let's fix
that.
Um so you're probably already apower user in your IDE.

(16:43):
Uh you can use autocomplete, youcan use refactoring, there's
tons of integrated tools.
Among the two most importantthings that your IDE almost
certainly does for you is itprovides good access to a source
code control system.
I happen to use Git, there areothers, uh, but Git is a command

(17:06):
line tool, it's just that yourIDE really knows a lot about it
and how to give you an interfaceonto it.
Um but outside your IDE, you'reclicking through file managers,
manually renaming files, copyingpaths.
Here's the promise of thisepisode.

(17:30):
You are going to learn toconduct an orchestra of existing
tools instead of writing newprograms.
Now I know you could write a newprogram.
Uh I I'm working on a GUI app,and our team is small.
A thing we do is build it, weturn it into an actual app.

(17:53):
The thing that builds it for usis a script.
Um one of the guys in our teamwrote that script.
It's a Python script that buildsan application.
That's nice.
He wrote an answer in Python.
But you don't always have to dothat.
Sometimes the solution to yourproblem is so simple that you

(18:16):
can just connect together threeexisting things.
So here's the core thesis ofthis entire episode.
The command line isn't aboutlearning another programming
language, it's aboutcomposition.

Jim (18:35):
Tell me more.
I want to get into this.
This is this is really this isreally neat stuff.

Wolf (18:42):
Um well, let me tell you the traditional view, because uh
I think uh programmers fallalong a scale, and I think most
modern programmers um are veryattuned to their GUI tools.
They know how to use them very,very well.

(19:04):
And sometimes uh they have towrite some command on the
command line.
Sometimes you can't get awayfrom saying awk or grep or
something.
Maybe you could, I don't know.
But what I'm saying is you'renormally a GUI programmer, and
sometimes you have to type asingle command on the command

(19:27):
line.
Some weird thing that Git cando, but your IDE doesn't give
you any access to it.
Uh a person like that isn'tusing composition at all.
Yeah, they're using the commandline, but they're just using it
to execute one weird command.
Um, and learning how to callthat command is actually the

(19:49):
hard thing.
Making it act act on the rightfiles.
Um okay, that's the traditionalview.
The reality is that the commandline.
Is about connecting existingcapabilities where each tool you
mention, and it's not just goingto be Git, it's gonna be five

(20:11):
things, it's gonna be twothings, it's gonna be eight
things.
Each one of those tools does onejob, and it does that job
perfectly.
The reason we know it does itperfectly is because it's
probably been around for 30years, and every time it doesn't
do it perfectly, somebody getspissed off and it gets fixed.

(20:33):
Um the Unix philosophy uh is doone thing and do it well.
But the magic, and they don'tput this in the quote, the magic
is the connections.
Um let me let me give you asimple story to start you out.

(20:56):
Um this is one place where I'm alittle sad that we're an
audio-only podcast, because if Icould show you pictures, um I
think things would be easier tosay.
But let me tell it to you like astory.
Um here's the story.

(21:16):
I want to find, in whateverfolder I'm in, maybe my project,
I want to find all of the Pythonfiles that aren't tests.
Um so here's how compositionworks.
I tell the computer, find me allPython files in this project.
It turns out there's a tool forthat.

(21:37):
Uh it's a built-in tool, it'savailable on every platform.
The name of that tool is Find.
There's open sourcealternatives, but what tool you
use doesn't really matter atthis moment.
When you're starting in withthis stuff, start with the with
the basics, the things you have.
Your shell doesn't matter, yourterminal doesn't matter, the

(21:58):
commands don't matter, as longas they do the thing you want.
And find, in this case, does thething you want.
You can tell find, look at thisdirectory hierarchy, and give me
a list that includes every andonly Python files.
The thing it gives you back is ahunk of text.

(22:21):
Um, and in that text, each lineis a full path to a Python file.
Um then say filter out any thathave test in their name.
This filter operation can bedone with a tool named grep or
several others.

(22:42):
Um they can be run in reverse.
When you say grep minus v sumpattern, it will repeat back to
you the lines that you sent intoit that don't match your
pattern.
So this is a simple uh command,it's grep minus v test.
So you start with find.

(23:03):
Uh if I'm remembering thiscorrectly, it would be find
minus e dot pi, or maybe it'spi.
That's find all the filesstarting here that end with pi.
Um and then you use the pipesymbol because what that
produces is a hunk of text, andthe pipe symbol goes into grep,

(23:25):
you say grep minus v test.
What you get out of that is thatsame list of files you found,
but all the ones that had testin the name are gone.
Uh there's no blank lines, it'sstill all squashed together.
It's just the Python files thataren't tests.

(23:48):
Um, and then I say, count howmany are left.
One of the most basic Unixcommands or Linux is WC, word
count.
Uh WordCount uh does three jobs,it can do them all at once or
one at a time.
Uh word count can tell you thenumber of characters that you
piped into it, the number ofwords you typed into it, or the

(24:12):
number of lines you typed intoit.
Because the thing I just sentinto it was one file per line.
If I say WC minus L, the thingit's going to spit out is a
single number.
And that number is the number ofnon-test Python files.

(24:34):
Three tools, three specialists,each perfect at their job,
composed into something morepowerful.
The only thing you had to know,aside from that you wanted that
information, was you had to knowthe name of the tool that did
that job, and you can start witha pretty small group and have

(24:57):
just the right tools to do whatyou need.
And the character you wanted toconnect those with, which I
already said is the is thevertical bar.
Um looks like you have aquestion on your mind, Jim.

Jim (25:10):
Yeah, yeah.
This I'm thinking how simple itis what you just said, right?
The the find command pipesthrough the grep command, pipes
through the word count command.
Boom, you're done.
You got three three commandsthat are all great at their job.
How would you even attempt to dosomething like that in Python?
You know, you'd have to open adirectory, you'd have to start
reading the entries in thedirectory, and every entry that

(25:32):
you read that is a directory,you'd have to recursively
descend into that subdirectoryand keep on going, looking for
files that match a pattern andcounting along the way.
It's much harder to grasp.
A lot of programmers don't evenwork on things like that.
Um because it's it's clumsy.

(25:54):
But what you just talked aboutdoing in shell composition is is
is beautiful.
I I like it.
It's a one-liner.
Yeah.
Or, you know, you could youcould put a backslash after the
pipe symbols if you want tospread it out into three lines,
but it's it's it's you can graspwhat each step is doing.
It's beautiful.

Wolf (26:15):
Yeah, and um the so we're doing this all in a shell.
Um, and the thing that shellsspecialize in, the problem that
they address, is they know howto find and run programs.
If you want to find and run aprogram uh from inside your

(26:39):
Python script, um it's actuallya lot of work.
Uh you have to you have to startstuff up and launch them and
connect it.
It's terrible.
I don't like it.
Um anyway, uh a thing about thisis that once you've got that
pattern, um find uh grep wc.

(27:02):
Once you know that, it's easy tochange your mind about how you
want to run that.
Do you want to look atJavaScript files instead of
Python?
Do you want to look for backupfiles instead of test files?
Uh you can add on more filters.

(27:22):
Uh for instance, you can findall the files that are backups
but aren't tests.
Umce you know just a little bit,suddenly you have this new power
to stick things together.

Jim (27:41):
It's real common to chain these filters together, too.
You you talked about one filter,grep.
You could pipe the output ofgrep into another grep looking
for a different pattern, andthen take that and pipe it
through yet another grep.
You you could have three or fourgreps in a row doing this this
pattern matching for you.
And each one is clearly doingone thing, uh and it it it's

(28:02):
easy to follow.

Wolf (28:04):
And I have actually done that?
That particular thing that youjust said is so important to me.
I have an alias.
Um, my alias is the word did.
So if I want to know the lasttime I did git add, I say um I

(28:25):
use the special notation to grabarguments and stuff.
But essentially, what I do is Icall history.
History gives me a list of everycommand I've ever issued.
We'll talk more about thatlater.
Then I pipe that into grep andfind every place I used the
phrase git add.

(28:45):
And then I pipe the results ofthat into git, into uh grep, but
this time with minus v, and Itake out anything that started
with the word did.
Because I only care about placeswhere I actually did a git add,
not this command where I justhappened to search for one.

(29:06):
Um, and then what it spits outis a list of uh commands with
command line numbers, becausethat's how I uh store my
history, of places where I saidgit add.
And because I know the number,and I'll talk about this later,
there's special things I can dowith the history, like
re-execute or edit or all kindsof things like that.

(29:29):
But let me move us on to a morecomplicated composition story.
Um and this is something that isalmost certainly going to impact
people who are trying to runlocal web servers or processes
that do things in thebackground, whatever.
Um this is about finding andstopping runaway processes.

(29:54):
So let's not worry about theexact command yet.
Let's just talk about thethinking.
Um, the first step is show meall the running processes.
Take my word for it, there's acommand that does this.
Then filter for only the Pythonones.
It turns out that these runningprocesses have a command line

(30:16):
that says exactly what isrunning.
Python is going to be in thatcommand line if it's a Python
program.
Then extract just the process IDnumbers.
Then kill every one of thethings that you had that were
left.
Um that's exactly the same kindof thing we just did.

(30:40):
It's a sequence of individualcommands good at the one thing
they do that end up with theresult you wanted, which for
sure your IDE did not providefor you.
Um there's a pattern.
Notice the pattern.

(31:02):
Each tool does its specialty andpasses the results on to the
next.

Jim (31:08):
Alright, so I I get it.
You you you get a list of theprocesses that are running, you
pipe it through grep to filterit, you pipe it through a few
more things, and eventually youfinally run kill on the process
ID.
Uh, that's kind of scary though,uh, kind of dangerous for people
to be killing processes.
Uh, do they need to learn how todo complex stuff like that right

(31:29):
away?

Wolf (31:30):
My feeling is that you uh you probably want to crawl
before you walk.
I feel like uh you can build apipeline, that's what we call
these things, pipelines,whenever there's more than one
command involved and you saythem together in a row and you
send the output from one intothe input of another.

(31:52):
That's a pipeline.
I feel like there are simplecommands, and you can start with
a two-command long pipeline andstill have something useful that
solves a real problem you have.
Yeah, I'm giving complicatedexamples to help you understand

(32:14):
just the range of power, andbelieve me, I'm not even close
to showing you what you can do,uh, but I want you to start one
at a time, two at a time.
Um, and we're gonna talk about apath to learning this stuff uh a
little bit later.
But the complexity is gonnabuild naturally as you get

(32:36):
comfortable.
Um, this is just like I I do anexercise program every other day
or three times a week, whatever.
And and a thing the guy on thefitness video says multiple
times per session, every singletime, is he says, do what feels
right for you.

(32:57):
Yeah, we're gonna say somecomplicated things, but for
sure, start easy.
This actually is easy stuff, andthere actually are easy
commands, and if you startthere, you're going to feel
confident and knowledgeable andready to move on quickly.

(33:17):
That's my feeling.
Um, in fact, why don't I rightnow hit on the very most
important.
Remember, I said there's um,gosh, it might be in the
hundreds of individual tools,not just the ones that come with

(33:38):
your OS, but that you candownload because they do new and
interesting things, or becausethey do existing things better.
Let me hit on just the five orsix basics under each of the
most important parts of youdoing composition.

(33:59):
First, let me talk about uh textprocessors and filters.
We already talked about grep.
Grep looks at individual lines,decides if it matches a pattern,
and then, depending on whatyou've said, either lets that
line go through or stops theline from going through.
That's a simple explanation.

(34:20):
Um we used the pattern test.
That's simple.
It turns out grep can do a lotmore complicated things than
that, but don't worry about it.
Just worry about does it havetest in it?
Um there is a tool called sort.
Um, you hand it messy data, uh,and it it does what you think.

(34:42):
Uh you can it might be that thedata you're giving it isn't just
a list of individual words.
They might be full-on lines, soyou need to tell sort things
like sort by the third column,treat it numerically.
You know, sort can becomplicated, but it's one of the

(35:03):
basics.
If you just use it with the wordsort, it's gonna do what you
want, and it's key to gettingsome things done.
Unique.
Uh it's spelled funny, U-N-I-Q.
Uh unique does two interestingthings.
Um, it does the thing you thinkit does.

(35:24):
If you've got a list of 20 itemsand seven of them are
duplicates, uh the thing itprints out is a lit I don't I
don't know if I'm getting themath right, but 13 items.
Essentially, every item in theoutput is the only case of that
item.
Everything that was a duplicateis removed, but they don't

(35:45):
remove enough that that wholething is gone.
So unique can be super useful.
That's a thing I use with gitall the time.
Uh, git has several differentcommands that will list files
that you care about.
Umified and you need to fix, orwho knows what.
Um it turns out a couple of thedifferent ways that you can ask

(36:10):
for these files will say aparticular file name multiple
times.
If I pipe that into unique, Ican get rid of all the extras.
Um and that turns out to beuseful.
There's a pair of commands thatI use all the time.
Um, head and tail.
Um they are amazingly simple, asalmost all of these commands

(36:34):
are.
Head, you pipe in a file, thatfile might be gigantic, three
gigabytes, who cares?
Head returns to you, prints out,um, just some number of the
initial lines of the file, andit makes sure not to waste time

(36:57):
worrying about the rest of thefile.
Tail is exactly the opposite.
Send in some big file, tell ityou want the last ten lines, or
whatever, they take parameters,um, and get just the last ten
lines.
Um head and tail are superuseful.
Let me give you an example.
Um if you want to know the threebiggest files in a given

(37:24):
directory, be in that directoryand say ls.
Ls lists the files there, and ittakes an option that will list
them by size.
Uh the largest first, the thingit produces is a list in the
form of text.

(37:45):
You send that into head, and yousay just the top three.
And what you get back is thethree biggest files in that
directory.
Um that's composition.
It's simple composition.
Um and finally, one we talkedabout using, WC.
WC gets used all over the place.

(38:06):
It counts lines, words, andcharacters.
Um, it can do them all at once,it can do them one at a time.

Jim (38:14):
Uh you know what, there's two commands you missed that I I
think are incredibly important.
Uh the first one is diff.
I use diff all the time.
What that you feed it two files,and it gives you the difference.
I I guess you can't really usethat in a in a pipeline, though,
right?
I mean the output you could usein a pipeline.

(38:35):
I'm not sure that you could usethe input in a pipeline.
Anyway, diff is an incrediblyimportant tool to me.

Wolf (38:41):
There um just to look ahead, stuff I'm actually not
going to discuss in thisepisode.
It turns out that there are waysto uh use whatever shell you're
using to uh specify a process,and that process has output, and

(39:05):
the characters you put aroundcalling that process turn the
output that it produces into aparameter, an argument, a text
file input into some othercommand.
So the thing you want.
That's right.
The thing you pro actuallyprocess substitution, but the

(39:27):
thing you want, you can have.
Yes.
And the output of diff, you canhave.
And you can do the right thingwith it.
It's complicated.
That's the main reason I didn't.

Jim (39:38):
Yeah, diff in a pipeline would usually be the first thing
in the pipeline that's going toget you the data that then you
can filter.
Uh the other command is more.
Uh or or it's uh its friendless.
They both do the same thing.
I think less is an open sourceversion, more is uh kind of a
standard Unix command.

Wolf (39:56):
You should use the word.
They are pagers.
Page or whole job.

Jim (40:01):
It'll give you a page at a time.
So you might have a command thatspits out an awful lot of data.
Maybe you're running this finethrough grep and it's spitting
out you know a thousand linesand you want to see it uh uh in
a page at a time.
Pipe that into page into uh moreand it'll show you one page at a
time.

(40:21):
And it's got there's fancy stuffto it that you can search and
and that kind of stuff, but uhjust more is incredibly useful
in a pipeline.

Wolf (40:30):
I totally agree.
Um I tried to keep this listshort because I want people to
play with the easiest things.
But though but the tools younamed, 100% useful, 100% on my
list.
Um I absolutely use them and inexactly the way you just said.

(40:53):
Um so those are the filters, butthere's also the data sources.
Um the things we just talkedabout always take something in
and then spit something out.
That's what I mean when I sayfilter.
They make some change.
Uh but there's also things thatjust produce data from scratch.

(41:17):
For instance, find.
When we used find in our exampleabove, we didn't give it any we
didn't pipe anything into it.
All we said was tell us all thePython files.
And then it produced something.
It produced a list of full pathsthat ended with.py.

(41:38):
Um so that's a producer.
Um, it's a data source.
Ls, we talked about that.
Whatever directory you're in, orif you give it special
parameters to make it lookelsewhere, ls will give you a
list of the files in thatdirectory.
Uh PS, that's the command wedidn't name when we talked about

(42:02):
handling runaway processes.
PS lists running processes onyour system.
Uh you may need that, probablynot right at the end of this
episode, but but soon.
Cat, um cat is a tool I use allthe time.
What cat does is you give it afile, and uh from that file

(42:26):
name, which is almost always aparameter, you say cat foo.py,
whatever, um, it prints theresult to your screen.
If you then piped that into moreor less or whatever you wanted
to use, or sent it into youreditor, or uh made it be one of
the things that you wanted to gointo diff.

(42:46):
Um cat has a super simple job,but that job is incredibly
useful.
And finally, something I amgoing to talk about later,
because I think this is soimportant, and that is uh, and
it's built into every shell thatyou might use.
Uh and like I said earlier, whatshell you start with um really

(43:08):
doesn't matter all that much atthis level.
Later you'll have ideas aboutwhy you want one or the other,
but for now it doesn't matter.
They all provide a historycommand.
Um history produces an orderedlist from the oldest you have
allowed it to remember.

(43:30):
If you make it be that historystores everything in memory,
then it's only the stuff you'vedone this session.
If you tell history it's allowedto save things in a file, then
it's every command you've everissued up to some limit.
Um history uh is another datasource.

(43:50):
It produces lists of commandsyou have executed.
So that's the important thingfor what I'm talking about right
this minute.
The thing I'm going to talkabout is as you are learning
about composition, you're goingto say things to yourself like,
what the hell did I doyesterday?

(44:10):
I remember that I did a thing,and I history what you did.
Yeah.
Um, okay, and finally, so wetalked about the processors and
filters, we talked about thedata sources, and finally,
there's the things that connectall those things together.

(44:34):
What you use to build apipeline.
Um, and there's a couple ofthings.
I think of the three things I'mabout to say, the most important
things are pipes.
A pipe is uh usually expressedas the vertical bar in some
programming languages.

(44:55):
That's the OR character, or halfan OR character, or it's bitwise
OR.
It's the vertical or.
Um and what the pipe does is itsays take whatever the output of
the thing on my left was and useit as the input to the thing on
my right.
Absolutely the most important uhfor building a pipeline

(45:20):
character.
Uh and the second most importantthing is redirection.
When you run a command, um thedefault thing that that command
does is it works on what we callstandard in, whatever you've
sent into it, maybe using thatpipe command, and it prints

(45:44):
stuff to the screen, and itmixes in with that anything that
you've printed as an error.
These three differentpossibilities are called
standard in, that's the thingthat went into the tool,
standard out, that's what gotprinted to your terminal, and

(46:04):
standard error.
That's only things that you haveexpressed as errors, and without
special handling, they just mixtogether on your screen.
It turns out there are specialcharacters using the greater
than sign and the less thansign, and these specific files
have magic numbers.

(46:26):
Uh if you use those numbers andampersand i I don't care.
You don't need to do thosethings this minute.
Um and when you do need to dothem, um you'll know where to
look.
It'll be it'll be easy to figureout.
So those three collections oh, Iforgot to name the third thing

(46:46):
in command composition.
Um we were talking about diff umcommand substitution, um, where
you can say, running thiscommand produces output that
should be considered a parameterto this function.

(47:09):
It's advanced.
Some shells suck at it.
You're not going to use it inthe next six months, but I'm
telling you so you know it'sthere.
Um so real composition storiesum that exhibit this pattern

(47:33):
because this is the change Iwant to make in you.
I want you to stop thinkingabout the problem you're having
as one of these two things.
I don't want you to think, oh myGUI must do something for this
in a menu.
I don't want you to think that.
And I also don't want you tothink the thing on the other

(47:55):
end.
Oh, I could just write a Rubyprogram that solves this.
Both of those are almostcertainly wrong.
What I want you to do is I wantyou to think about what tools
you know and how you mightcombine them so the total amount
of code you end up writing iszero or close to zero.

(48:20):
Um for instance, finding some ofthe large files that are eating
up disk space.
Uh this is one I actually justtalked about.
Um find all the files in thisdirectory tree.
I said do that with ls.
Then sort them by size, largestfirst, you could do that with
sort.
Then show me just the top ten.

(48:42):
You can do that with head.
Um you only need three tools todo this, and it answers your
question, and it took you twominutes.
Um that's gonna be in yourhistory, if you care about it
later.
If you don't trust history yet,you can copy that one line of

(49:03):
shell scripting and paste itinto your uh text file of
interesting saved pipelines.
Um It's an easy answer.
A hundred times easier thaneither of the two extremes I
gave right before this.

unknown (49:22):
Uh-uh.

Jim (49:24):
Alright.
How do I tell these tools what Iwant?
You're talking about using findto uh uh to find largest files,
to find big files, right?
Um how do I tell it uh that Ionly want files over a certain
size?
How does that work?

Wolf (49:41):
Okay, the beauty of the command line and of these
individual tools, so far I'veI've talked about very simple
ones where you don't have totell them uh very much about how
to do their job.
But the beauty of these tools issomething you don't get uh from

(50:04):
a GUI.
Uh and by GUI I mean VS Code oron the Mac the Finder or in
Windows the Explorer or umZolfin in KDE, whatever.
Um in those you double-click ona program and the program runs,
and that's all you got to tellit.
But on the command line, thereare lots of ways to fine-tune

(50:27):
what you want a specific commandto do.
Uh you can tell you can use uhIO redirection, like we just
talked about, and that almostdoesn't involve the tool at all.
But also many of these toolstake arguments and options and

(50:48):
parameters.
For instance, when you say gitadd foo.py, uh foo.py is an
argument to git add.
You're telling it, I want you todo something to foo.py.
Lots of the more and moreinteresting commands that are
available to you take argumentsor flags.

(51:12):
It turns out that um the findcommand knows specifically about
the notion of file size and itunderstands numbers, you can
tell it, only show me files thatare greater than a hundred

(51:32):
megabytes.
And by the way, um if you areusing Git and you're building a
Git repo and you have a file,and that file is a hundred
megabytes, please do not addthat uh file to your commit
because even removing it lateris not going to make your repo
smaller.

Jim (51:53):
All right, everybody raise your hand if you've made that
mistake before.
I know I have.

Wolf (52:01):
Um anyway, uh find is a pretty complicated command.
You can use it pretty easily,but it will do more and more and
more sophisticated things.
It'll find things by pattern, bysize, by file type, by location,
by um uh actual whether it's adirectory or it is a file, uh,

(52:26):
or that it's one of either, orthat it's a link, that it's a
symbolic link or a hard link.
It it can know these things.
Um, it can know how recently itwas modified or created.
It can get you only the onesthat are older than that.
There's a ton of things find cando.
Do I want you to start withthese?

(52:48):
Absolutely not.
Um I don't even know if I wantyou to start with find.
Um, but I want you to know this.
Um everybody uses find or FD orsome tool that's not unlike
these.
And there are ways to know howto call these things and get

(53:14):
what you want.
Um there's a thing built intoyour shell called man, M-A-N.
That's short for manual.
Um man is just another tool,just like any of the ones I've
talked about so far.
Uh, but man uh prints outdocumentation.
There's uh a a couple otherslike that info.

(53:36):
That doesn't matter.
Almost certainly what you'regoing to do is you're gonna
Google for the find command.
Or maybe you're gonna useChatGPT or whatever particular
AI you like.
It turns out I have a thing thatI really like.
Um it is a command line tool,it's non-standard, um, but it's

(53:58):
pretty easy to install, andthere's multiple um
implementations of it.
The name that covers all of themis TLDR.
That stands for too long, didn'tread.
And what TLDR does is it's likeman the good parts.

(54:22):
When you look at the man pagefor something, it might be 20
pages of dense text.
But when you run TLDR aboutfind, it will print one
screenful, maybe a little more,that shows maybe six different
ways to call find.

(54:42):
And each way it shows the exactcommand with the exact arguments
and file names, and then a shortdescription, what does that
mean?
Um I TLDR solves most of myproblems, and if they don't,
then I'll I'll go to the AI.

(55:03):
Um I feel like like that worksfor me.
Um anyway, uh you you've gotwhat looks like a question on
your face, Jim.

Jim (55:15):
You know, you you're talking about a long chain of
tools here.
Uh for for somebody just gettinginto this, um, how how do they
know how to make the distinctionbetween what they should do as a
as a uh composition or shouldthey just break out Python or
Perl or whatever?
How do they know?

Wolf (55:37):
Um well I have a rule for myself.
Uh let me tell you what my ruleis.
Maybe this works for you too.
If the things you want to doalready exist and all you have
to do is connect them, thenbuilding a command pipeline is

(56:00):
almost certainly the rightthing.
If it's something incrediblysimple, like deleting a single
file uh inside VS Code, maybeyou just want to right-click on
that file with the mouse uh anduse your GUI.
If it's editing a Photoshopfile, yeah, that's a GUI task.

(56:22):
If you have to make up new datastructures, new algorithms, new
things that don't exist rightnow, maybe you want to write a
Python program or a Pro programor a Java program, whatever,
whatever your forte is, that'swhat you want.
Um a lot of problems can besolved with these uh pipelines,

(56:48):
and the reason so many problemscan be solved with them is
because these tools have evolvedas we've come along.
That's all we had to start with.
And nobody deletes them, theykeep adding new ones as we come
up with new tasks we want tosolve.

(57:09):
So my feeling is if you canunderstand what you want, and
it's obviously not a simple GUItask, probably start with the
command pipeline.
I want you to succeed with that.
If you can't, you can't give upand write a program or or do

(57:30):
whatever you have to do.
But I want you to start with thething that is going to be the
easiest for you.
Maybe not right this minute, butit will be the easiest for you
sooner or later.

Jim (57:47):
Um But this is um uh you're talking about a chain of uh
programs, right?
Um wouldn't you say that it'smore maintainable?
If you're gonna do it as a shellscript or or a shell pipeline,

(58:08):
it's more maintainable than aPython script uh because each
piece is simple, testable, andyou can modify individual steps
easily.

Wolf (58:17):
Is that uh sound about right?
Those words that you just said,I 100% agree with.
I love Python, and in fact, um Ihave two major shells I use.
I use bash and z shell.
Uh those those things don'tmatter.
Um but the thing I'm going tosay is, I love Python more than

(58:39):
I love bash and z shell.
But it's just better to write acommand pipeline at least half
the time.
Um it is the right answer.
So often, you really ought tostart with the right answer.
Because the whole thing we do asprogrammers, this is our job.

(59:03):
The first thing we have tounderstand is the problem, and
the second thing we have tounderstand is what are the right
tools to address that problem?
And, you know, I love Python,but the fact is Python isn't
always the right tool.
Um, and there's just a ton oftasks that are part of our job

(59:30):
where building a commandpipeline is the right answer.
Um yeah, I don't I don't thinkyou sh you uh ship shell scripts
to end users, but so much of ourjob is doing these specific
things.
Like for instance, I've got athousand um ping files, and

(59:56):
because I'm sending them to myvalidation team, they all need
To be resized and turned intoJPEGs and have better names.
Um, I can write a Python scriptto do that.
It's it's easy enough.
It might it might take me threehours.
I don't know.
Um it turns out that there is auh tool called ImageMagic that

(01:00:19):
does every single one of thosethings.
We'll do it instantly, and we'lldo it to a list of files.
All I have to do is build theright pipeline.
Um I think um this this wholething uh is it's a s it's an

(01:00:43):
in-between place.
Yeah, we are great at writingprograms.
That's why we do what we do.
We're not just great at it, welove it.
We want to do it.
And we're good at using modernGUIs.
Uh, you know, with theauto-predict and the refactoring

(01:01:03):
and the complete and the look upthe definition.
Uh that stuff's so great it'sit's almost fun to use.
Here's a new level.
Something in between.
I promise you it works becauseold guys like us have been using
it since before there were GUIs.
This stuff works.

(01:01:26):
Once you know how it works, howyou can exploit it to get what
you want, once you know thosethings, uh I'm gonna use the
word from the from the intro.
This will be a superpower foryou.
Your teammates will be writing agoddamn Python program that

(01:01:48):
takes them two days, and youwill have the answer in ten
minutes, because you hookedtogether the commands in a
pipeline that did exactly theright thing.
Um, and once you understand thebasics, once you understand how
to connect things, each new toolthat you learn, and and you can

(01:02:14):
learn them as fast or as slow asyou want, it multiplies with
every other thing you alreadyknow.
Um those five core tools Italked about, uh, just those can
create potentially hundreds ofuseful combinations.

(01:02:35):
Uh if you learn one more tool,suddenly all your existing
knowledge becomes even morepowerful.
Um we've told you a lot about umwhat you're getting, what we're
offering you.

(01:02:55):
But let me give you a path.
Um I feel like the mostimportant thing, and I said the
words before, is to crawl beforeyou walk.
I think this is all doable.
I think this is all totallywithin your skills skill set,
and I think this will give yousuperpowers.

(01:03:18):
Um I think uh the things to knowto start off is don't abandon
your current tools.
If you're a VS Code user, use VSCode.
VS Code is your friend.
If you use PyCharm, use PyCharm.
I don't want you to make yourlife worse.

(01:03:39):
I want you to add things thatmake your life better.
Um a thing all of these toolswill let you do is you can
right-click in your filemanager, um, and that might mean
Windows Explorer, or it mightmean the Finder, or it might
mean Dolphin, or I I think itactually works in VS Code.

(01:04:03):
Click there, right-click there,and say open terminal here.
Um from that point, learn basicfile navigation.
Maybe you already know, maybeyou know what C D does, and
maybe you know what LS does, orin the case of Windows, uh dir.

(01:04:25):
Um but that's where you start.
It's it's the file system.
Start there.
Umce you're comfortable withthat, you can get a terminal,
you can type a command, you canhit return and see what that
command does.
The next thing to do is scratchan itch.

(01:04:48):
Uh pick one thing that you doover and over again that annoys
you.
Uh for instance, finding filesmodified recently, checking
what's using disk space, seeingwhat processes are consuming
CPU.
Uh, these are separate things.
I'm just saying one of thoseprobably annoys you.

(01:05:10):
Um learn just the compositionpattern that solves that
specific problem.
Um and then but keep using theGUI for everything else.
Um I mean, the GUI works foryou.
You are a programmer, you'redoing the right thing.
Um and the goal for this is toexperience your very first.

(01:05:34):
Holy crap! This is so muchfaster.
Um that's what I want you tofeel.
That's what I want you to try tosuppress saying.
Um Good things are gonna happento you.
I want them to happen to you.
Uh the next thing to do is uhyou've done one tool.

(01:05:56):
Um, you've you've done the filesystem.
Now start hooking thingstogether.
Uh chain a couple of the toolsyou already use.
Uh use ls, or whatever it is onyour machine, could be dir, uh,
to find just the text files orlog files, and then pick out

(01:06:20):
just the first couple.
Maybe they're the mostinteresting.
Maybe you've arranged by date,so they're the most recent.
Um that's a chain, that's uh apipeline, that's useful.
Uh practice taking some existingcomposition, something you've
already written.

(01:06:42):
Update the search terms, addmore filters, try different
sorting.
Um, like I said, uh maybe you'rebuilding up a text file where
you've pasted one of thesecommands on each new line, grab
it out of there.
Maybe you've advanced all theway into using history.

(01:07:03):
Um if you use history, um youruh shell almost certainly
supports the up arrow key to goback to previous commands, or
maybe control R.
Ctrl R lets you type a patternto find a command.
Uh I re excuse me, I recommendstarting with a text file that

(01:07:25):
you build yourself.
That way you can keep the onesthat work and not add the ones
that don't work.
Um modify, change the terms, addmore filters, try different
sorting.
The whole goal of this isthinking.
Thinking in terms of this tool,then that tool.

(01:07:49):
Instead of individual commands.

Jim (01:07:53):
Uh how do you remember all of these combinations?
There's there's an infinitenumber of possibilities.

Wolf (01:08:02):
So uh this is where I feel like history is your friend.
And I all I already mentionedthe notion that you take the
ones that are particularly goodfor you, that you liked, you're
proud of, that had the rightresults, and that you add those
to a text file that you takecare of.

(01:08:24):
Maybe even check it into Git, Idon't know.
Um, but every single shell, nomatter what you're using, has
history.
They will remember the commandsthat you've entered, and you can
find the things you've donebefore.
Um, it turns out once you getgood at this particular

(01:08:47):
facility, which is not what thisepisode is about, um, you can
edit commands that you'vealready written straight out of
history.
Uh you can copy and paste them.
Like I said, start with a textfile.
That's that's my feeling.
Um I think we already talkedabout getting help with the

(01:09:10):
individual commands, and I thinkthat's pretty personal.
Uh you might be a Google personand you want to use Google to
find the answer.
And believe me, Google knows theanswer.
Um, Google often knows theactual manual page for a
specific tool.
It'll take you right there.

(01:09:31):
I don't really like manual pagesbecause they're so long and
dense.
I really like that TLDR tool.
Too long didn't read.
On my machine, uh you have toinstall it, no matter what.
On my machine, which is a Mac, Iused Homebrew.
You can use Homebrew on Linuxalso, or you could get it
through apt.

(01:09:51):
Uh and there's a bunch ofdifferent implementations.
But TLDR is good.
Um, whatever your favorite AI isgood.
Um so from solving problems withcommand pipelines to um building

(01:10:12):
up a library that does things.
Remember, I talked about addingthem to a text file?
Um, annotate that text file.
Uh, you're going to end up witha lot of pipelines in there.
Say what they do, say why youneeded them in comments.
Uh most shell comments startwith the pound sign.

(01:10:36):
Um and as you look at these, uhyou're gonna start recognizing
patterns.
You're going to see how thingsyou did two weeks ago or last
week are like the problem youwant to solve today.
And that's gonna give you ideas.
How am I gonna do this?

(01:10:57):
You are on the path.
Once you start seeing oldthings, understanding what they
mean, and how that applies tothe problem you're on right now,
you have a whole new way ofsolving problems.
A way the other people on yourteam don't have.

(01:11:17):
You are better than them now.
I wouldn't take that lightly.
Um what's going to happen isyou're going to stop thinking to
yourself, how how did I do this?
And start thinking, what have Idone that's similar?

(01:11:39):
Um.
And I don't know how far in thefuture this is, because I feel
like I've I've covered the topicof the thing I want you to know.
Um, but these pipelines arereally automation.
But they're automation thathappens right then.

(01:12:02):
If you saved them into anexecutable script, um you could
do things to them that made thembe real, full-on, let's say
you're using bash, bash scripts.
Instead of just running them ina directory, you could make them
have arguments and use them onhundreds of files and do all

(01:12:22):
that.
I think that's out of scope.
But I am saying, once you havebeen working on that history,
once you're thinking this way,once the default in your mind is
what things already do this thatI can just stick together, the
next step for you is making thecomputer run scripts.

(01:12:48):
Um you look like you've gotquestions on your mind, Jim.

Jim (01:12:52):
Yeah, there's a there's a lot to unpack there.
Uh, a couple of things I wantedto ask.
One is uh everything you'retalking about just sounds like
Unix in Linux.
What do you do if you're aWindows developer?
So help them out.

Wolf (01:13:07):
On Windows, um, I can name first of all, there's WSL.
Uh that doesn't answer thequestion because WSL is
typically just Linux or Unix, sothat's not satisfying.
But um there are at least threedifferent shells available to
you on Windows.

(01:13:27):
There are some things thataren't the same.
Um but uh the three things thatyou might care about are uh you
might care about command.exe.
That's the thing everybody getsby default.
You might care about umPowerShell.
Um if you don't havecommand.exe, it's probably

(01:13:50):
because you specifically askedfor PowerShell.
And finally, um you can havebash.
Um there's several differentways to get it.
I have bash on my Windowsmachine, not under WSL, provided
by uh get bash for Windows.

(01:14:11):
Um bash is gonna be just likeeverything we've already talked
about, but key to know is thatthe things I said, um
command.exe and um PowerShell,the words are different.
The names of the commands, butthe idea of hooking them

(01:14:34):
together into a pipeline, uh,that's the same.
Um I they they might bedifferent characters.
I don't I don't use either ofthose all that much.
But the point is, um whatevershell you're using, you can play
with and grow into thissuperpower.

Jim (01:14:56):
You know, we we we talked about uh you run a command to
produce some data, uh you pipethat into another command to
filter the data, maybe a commandto sort the data, uh, another
command to unique it to you knowget out duplicates.
Uh each step along the way,you're not changing the original

(01:15:18):
data, you're creating new datafrom the old data.
That sounds like immutability tome.
Right?
Uh you've got separate stepsthat each one of them does
something specific.
Man, this is starting to soundlike functional programming.
Uh we covered that back inepisode 100.

Wolf (01:15:38):
100% right.

Jim (01:15:39):
This is have we been doing functional programming all
along?

Wolf (01:15:44):
You totally hit it on the nose.
Um that's what it is?
Yes.
Uh command pipelines are almostalways functional programming.
Now it turns out there are somecommands you can stick in there
that sort of screw it up.
Um there's a command called TT-E-E.

(01:16:05):
Um that can make it differentthan what we're what we're
discussing.
I don't think we need to discussT, but if you're interested in
that, man T.
Yeah.
Uh the main thing to know isyour observation right on
target.

Jim (01:16:21):
See, I can be right once in a while.
Oh, let's keep going.
This is good stuff.

Wolf (01:16:32):
Um Okay.
Uh I want I already talked aboutum when to use composition
versus when to actually write ageneral purpose programming
language program.
Um and basically uh the thing toknow is uh when you outgrow the

(01:16:58):
functions available to you inthese command line tools, um, if
you need some kind of special uhdata structure, uh most of these
tools, these commands, uh don'tthey might have data structures
inside them to help them do thething they do.

(01:17:21):
Diff in particular has some veryinteresting data structures.
But most of them uh don't exposethose data structures, you don't
get to use them.
Um a shell, like bash, um, doessupply some some things you can
use, like it has a hash table inuh recent versions of the shell

(01:17:47):
of bash.
Um but if you need advancedprogramming tools uh and you're
going to do things that's lessabout executing some existing
command and more about um usingalgorithms to figure out what to
do, probably you've advancedinto the territory of writing a

(01:18:09):
full-on program.
Um if you're ed and like I saidbefore, if you're going to
change the color of a pixel inPhotoshop, probably you're just
gonna use the mouse.
So those are the two ends of thespectrum, and a very large
fraction of what's in between isabout building the right

(01:18:29):
pipeline.
Um I have no problem looking ata situation, looking at a task I
want to accomplish, trying tosolve it with l what looks like
the easiest tools to me, whichare often a pipeline, and then

(01:18:52):
after getting a certaindistance, saying to myself,
yeah, maybe not.
Maybe I can't do it with this.
I guess I'll write a program.
Um that's that's certainlypossible.
I'm now I'm gonna say somethingincredibly controversial.
Um something that willabsolutely get me yelled at.

(01:19:13):
Um, and that is this.
Um all these shells, bash and uhZ shell and all of them, um they
are all full-on programminglanguages.
Um just like Perl is uh uh has alot in it that's about solving

(01:19:34):
string problems.
So strings and regularexpressions are s are are a
native thing to Perl.
If you're working on stringproblems, Perl is your answer.
It's a great language for that.
Shells, all of them, um are arereally about a couple of things.

(01:19:57):
One of those things inparticular is Is finding and
running existing binaries.
Like where is LS?
Um, you know, it turns out thatfor sometimes ls might be built
into the shell, but can't thereare lots of things that aren't.
And shells are about runningthose programs.

(01:20:17):
There's a couple other thingsthat shells are tuned for.
Um your Python program, let'ssay, uh has to open up a
library.
It is in the standard library,it's about processes, um, and it
has to open up a a sub-processto launch one of these commands,

(01:20:40):
and there's a lot of boilerplateand syntax around it, and
there's um stuff that's actuallyhappening at execution time, um,
it's harder and more costly torun an external program from a
Python script than it is to runit from the shell.
Probably that doesn't matter.

(01:21:03):
Probably that difference is verysmall.
But when you look at yourproblem, if your problem is
fundamentally about runningexternal programs, especially
many, many times, especiallymany different things, and
redirecting their I.O.

(01:21:25):
probably bash is better thanPython for writing that script.
But if your program doesn't runthat many external tools, does
do things that Python is reallygood at, then maybe Python's the
right answer.
Um if you start with one, it'sit's not that hard to move to

(01:21:50):
the one that you have decidedturns out to be the better
answer.
Um I don't know what theevidence will be for you that
it's the better answer.
Um if we go back to ourperformance and optimization
episode, uh the way you wouldknow is because you're
measuring, because of courseyou're measuring.

(01:22:13):
Um and sometimes you just likewriting the Python better.
Um I feel like, before I didwork on this episode, my feeling
like, my feeling was that ifyou're writing a bash script to
do some task, first of all, ifit's you, if you if it's an if

(01:22:37):
it's a programming task, becauselike I said, if you're shipping
it to a user, it's not gonna bea shell script.
So if it's you, if it's for yourjob, if you're doing a
programming task, if you'rewriting it in a shell script, my
old feeling used to be if youhad to have a hash table or you

(01:22:59):
had to have a loop, just writeit in Python.
That was my old feeling.
But after really thinking aboutand researching this topic, I
feel like using uh pipelines tosolve problems um extends way

(01:23:20):
further into the range where oldme would have just sss skipped
on over to Python.
I think I was wrong, and I thinkI have better answers now.
Um that's what I feel.
Um so I think the key insighthere is most I should write a

(01:23:45):
Python script for this momentsare actually probably two or
three shell tools composedtogether.
So before you fire up your IDE,ask yourself, am I really
creating a brand new capabilityhere?
Uh, or am I just connecting somethings um that already exist?

(01:24:10):
Um at this point I would talkabout history.
Um I think history is one of themost important things in your
shell.
I use it non-stop.
In fact, I use history so muchbecause I'm constantly taking
commands I've recently run andfixing them just a little bit.

(01:24:34):
Um I use it so much that Iactually use a third-party tool
called Atun.

Jim (01:24:40):
Um but the fact is that's uh A-T-U-I-N.

Wolf (01:24:45):
Yes.
Um I'm gonna actually dive in alittle bit.
That's the name of the turtle inthe Terry Pratchett books that
supports the disc world on itsback.
Um it's an amazing tool.
I can't live without it.

(01:25:06):
There are changes I want to maketo it.
History is a complicated thing,there's a lot of things you can
do with it.
You'll get there.
Um What I want is I want you totype things into a shell.
I want you to connect some basiccommands, and I want you to go
forward from there.

(01:25:27):
History's gonna help you later,um, but I don't I don't think
you need that uh right thisminute.
Let me we're getting close tothe end, and I think this is
probably our absolutely newlongest episode.
But here is your call to action.
First of all, just likeeverything else, don't switch

(01:25:52):
everything at once.
Don't abandon your currentworkflow.
Um existing file managers, uh,and you can't see me, I'm using
air quotes here.
Open terminal here feature.
Practice basic navigation, uh,especially in directories,

(01:26:14):
locations that you already know.
You know what's there, you knowwhat you ought to see.
Um, and your success metric isnavigate to one of your project
folders and list its content viacommand line.
Okay, step two.
Find one annoying repetitivething you do.

(01:26:37):
Something absolutelyfrustrating.
Uh, and I don't know about you,but I'm constantly faced with
frustrations.
And a problem with me is that ifsomething frustrates me, I will
immediately focus all power onit.
Probably that's not right.
Um something that you normallydo through the GUI.

(01:26:59):
For example, finding all thefiles you modified this week,
checking which folders are usingthe most disk space, seeing
what's consuming CPU.
Uh learn just the compositionsolution for that one specific
problem.
The things I just said are threespecific problems.

(01:27:21):
And your success metric for thisis that whatever that annoying
task is, you get the answerfaster through composition than
through clicking around.
Um Step three, this is where Iwant you to adopt thinking about
connections into your into howyour brain works.

(01:27:45):
Um this is it.
This is the primary school uhskill.
Um I want you to start thinkingthis tool, then that tool.
Uh start simple, show me thefiles, then filter for the ones
I want.
Uh practice modifying.
Change what you're searchingfor, add more filters, try

(01:28:06):
different sorting.
The success metric here is whenyou see output from a command,
you naturally think, I couldpipe this to something else.
That should be the first thingin your mind.
What else can I do with this?
Um Step four.
Uh start with the things yoursystem gives you.

(01:28:29):
There's so many options, andthey so don't matter.
Uh any terminal application, anyshell, you want to focus on
composition concepts beforetrying to pick the right tools.
Don't get distracted by tryingto find the best setup until you

(01:28:50):
prove that this wholecomposition thing works for you
at all.
Uh the superpower thinkingtransfers between any tools you
choose.
Um for instance, on macOS, thedefault terminal shell that you
will get is Z Shell.

(01:29:11):
I don't care.
Use Z Shell.
Whatever you learn in Z Shell isgonna apply to Bash, it's gonna
apply to PowerShell, it's gonnaapply to CornShell, it doesn't
matter.
The shell does not matter.
Worry about composition.
Um the success metric here isthat you're comfortable creating

(01:29:32):
basic compositions regardless ofwhich specific shell you're
using.
Um and finally, uh build yourpersonal composition reference.
Every time you do something thatworks, every time you do

(01:29:53):
something you like, stick it inyour text file.
Make sure that text file has agood name.
Write a nice clear description,like find Python files without
tests, or shows biggest files inthe project.
This becomes your personalsuperpower reference manual.

(01:30:14):
You are going to outgrow this inat least two ways.
Um first, eventually, history isalways gonna be your answer.
And second, it won't be longbefore you don't need to look at
the things you said before.
You just know.
Uh, and your success metric isthat you find some old

(01:30:36):
composition to solve a new butsimilar problem.

Jim (01:30:43):
There's a lot there.
How long do you think this wouldtypically take?
Um when when should somebodyexpect to go from a complete
beginner to uh they feel likethey have superpowers?

Wolf (01:30:57):
I think this is quick.
Um, the words I'm about to saymight not sound quick, but I
think, first of all, um we'retalking about programmers.
These are people who understandprograms.
Um, they've got knowledge, theyknow how to interact with their
computer, things are happening.

(01:31:18):
I feel like most of the peoplein this group hit their first
real, and I'm making air quotesagain.
Wow, this is so much fastermoment, uh, within a week, maybe
two, uh, of focused practice.
The genuine superpower feelingusually comes um in a month or

(01:31:42):
two, uh, when you startautomatically thinking in terms
of composition uh when youencounter a new problem.
But e everybody's different.
Um some people get completelyhooked after their first
successful pipe, others needmore time to see the value.
The key is starting with thereal problems that actually piss

(01:32:07):
you off.
Um and maybe this isn't foreveryone.
Uh maybe the GUI is exactly thething that you want.
Um I think these can help you.
Um let me um uh give you acouple of takeaways.

(01:32:29):
Um I have given you theconceptual framework, the
command line as a composition uhand orchestration.
It's not about memorizingcryptic commands.
If you have to memorize acommand, probably you're down
the wrong path.
The specialist tools alreadyexist on your computer right

(01:32:53):
now.
Uh and we know they're therebecause these are the things
experts have required and askedfor and written themselves over
the course of the last 30-40years.
Uh, all you're doing is learningto conduct the orchestra.
Every composition you createbecomes part of your growing

(01:33:17):
personal capability.
Start now, start this week.
Pick one repetitive, annoyingtask and figure out the
composition-based solution.
Um, your journey from a GUI,dependent programmer to a
composition master starts withconnecting just two simple

(01:33:42):
tools.

Jim (01:33:44):
Well, there you have it.
Uh your keys for uh gaining anew superpower.
And uh, you know, a lot of usold guys, Wolf and I have been
doing this for a long time.
We started out in this worlddoing uh uh command line
programming, uh composition,stuff like that.
Um, but I'm afraid a lot of thea lot of the younger developers,

(01:34:06):
you know, they're using VS Codeor they're using Xcode or
they're using PyCharm orwhatever, and they just don't
get exposed to this.
Follow uh what Wolf said, andyou will elevate your skills.
Uh that new superpower willreally come in handy.
So thanks for that, Wolf.
Um I I I think it's time to wrapit up.

(01:34:28):
Uh boy.
Uh we've been talking for uh 95minutes.
Uh there's some pretty goodinformation there.
Uh this is our longest episodeso far.
And as I said back at thebeginning, uh we really like
some feedback.
Uh any feedback is great, butwe're particularly interested in

(01:34:48):
feedback about uh episodelength.
We we want to give you guys thethe the amount of content that
you'd like, so tell us what thatis.
You can send that to feedback atruntimearguments.fm.
Uh if you want to get a hold ofus uh individually, I'm Jim
McQuellen.

(01:35:09):
Uh my email address is jam at uhruntimearguments.fm.
And my partner here is Wolf.
Talk to him at wolf atruntimearguments.fm.
We're both on Mastodon.
You can find us there.
Um lots of ways.
Uh we do have a website,runtimearguments.fm.

(01:35:31):
Check us out there.
Um until next time.
Uh thanks for listening.
Thanks for coming along.
And uh and Wolf, uh, gotanything to say?

Wolf (01:35:43):
I want to I I I know this is beyond my skill set, but I
want people to be smarter.
I want to be part of making themsmarter, and I feel like maybe
this episode helps.

Jim (01:36:03):
How do we top that, huh?
All right.
Thanks a lot, everybody.
Until next time.

All Episodes

Episode Transcript

Host

Jim McQuillan and Wolf

Popular Podcasts

Stuff You Should Know

Crime Junkie

On Purpose with Jay Shetty

.css-15opob5{left:0;position:absolute;top:0.8rem;} All Episodes

.css-14f5ked{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;}10: Command-Line Superpowers: Connecting Simple Tools for Complex Solutions