March 15, 2021 • 49 mins
Summary
Python is a language that is used in almost every imaginable context and by people from an amazing range of backgrounds. A lot of the people who use it wouldn’t even call themselves programmers, because that is not the primary focus of their job. In this episode Chris Moffitt shares his experience writing Python as a business user. In order to share his insights and help others who have run up against the limits of Excel he maintains the site Practical Business Python where he publishes articles that help introduce newcomers to Python and explain how to perform tasks such as building reports, automating Excel files, and doing data analysis. This is a great conversation that illustrates how useful it is to learn Python even if you never intend to write software professionally.
Announcements
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- We’ve all been asked to help with an ad-hoc request for data by the sales and marketing team. Then it becomes a critical report that they need updated every week or every day. Then what do you do? Send a CSV via email? Write some Python scripts to automate it? But what about incremental sync, API quotas, error handling, and all of the other details that eat up your time? Today, there is a better way. With Census, just write SQL or plug in your dbt models and start syncing your cloud warehouse to SaaS applications like Salesforce, Marketo, Hubspot, and many more. Go to pythonpodcast.com/census today to get a free 14-day trial.
- Your host as usual is Tobias Macey and today I’m interviewing Chris Moffitt about how Python is used to help manage business needs and processes and his work to share advice on this topic at Practical Business Python
Interview
- Introductions
- How did you get introduced to Python?
- Can you start by giving an overview of your mission at Practical Business Python?
- What was your inspiration for starting the site and what keeps you motivated?
- What are some of the kinds of problems that a business user is looking to solve for themselves?
- Why is Python a viable tool for a business user to become familiar with?
- How would you characterize the difference between the ways that a software engineer and a business user approach Python?
- What do you see as the tipping point of complexity or time investment past which a business user will pass a given project on to a software engineer?
- How much familiarity with adjacent concerns such as version control, software design, etc. do you consider useful for a business user?
- What are some of the ways that you use Python in your day-to-day?
- What are some of the onramps for integrating Python into a user’s workflow?
- What are some common stumbling blocks that business users run into when getting started with Python?
- What are some of the most interesting, innovative, or impressive ways t
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Unknown (00:13):
Hello, and welcome to Podcast Dot in It, the podcast about Python and the people who make it great.
When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So take a look at our friends over at Linode.
With the launch of their managed Kubernetes platform, it's easy to get started with the next generation of deployment and scaling powered by the battle tested Linode platform,
(00:34):
including simple pricing, node balancers,
40 gigabit networking,
dedicated CPU and GPU instances, and worldwide data centers.
Go to python podcast.com/linode,
that's l I n
o d e, today and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show. Your host as usual is Tobias Macy. And today, I'm interviewing Chris Moffett about how Python is used to help manage business needs and processes in his work to share advice on this topic at Practical Business Python. So, Chris, can you start by introducing yourself? Hi, Tobias. My name is Chris Moffett, and I blog at Practical Business Python,
(01:13):
which is
dedicated to
sharing examples of how you can use Python to solve common business problems.
I've been using Python for
probably
a dozen plus years and was formally trained in electrical engineering and computer science, but never actually been a computer scientist.
(01:34):
And most of my career has been on the business side. So that's where I've really enjoyed combining that business experience, that Python experience, and that's a lot of what I'm trying to share at Practical Business Python. Do you remember how you first got introduced to Python?
I do. Maybe like a lot of other people, I had used some other languages. I did a little bit of work as a Unix system administrator
(01:58):
and did some scripting in Pearl and
exposure to other programming languages in the past.
And someone that I was working with said, hey. You might wanna check out this Python thing. It's pretty cool language.
This was probably
in
2, 001, 2, 002,
so still relatively new. And I
(02:20):
looked at it a little bit and thought, you know, this whole white space thing,
it just didn't really resonate with me. I didn't think it was that good a fit, so I passed.
Fast forward about 6 or 7 months, and I was working on a Windows system
and needed to do some scripting
and
took another look at Python and started to play with it for about half a day or so. And suddenly, it just kinda clicked in my brain, made a lot of sense. And I've been using Python in some way, shape, or form ever since then. You mentioned that you've been using practical business Python as an avenue to help educate people on some of the ways that the language is used to solve different problems within the business context.
(03:02):
I'm wondering if you can just give a bit of an overview about your mission
of practical business Python and some of the backstory of how you got started with it and what keeps you motivated after multiple years of blogging and providing resources to that particular segment of the ecosystem?
So like I mentioned, I first got exposed to Python
a while back. So this was pre data science
(03:23):
hype, if you will. This was even probably before Django was really getting started in the web space, so I used it a little bit from an administrative
perspective.
I also did a little bit of Django work.
And then over time, I just stopped using Python. So I didn't have a real good excuse or reason to use it on a daily basis.
(03:44):
Well, I wanted to find an opportunity
to continue
to use Python, explore the Python ecosystem,
and maybe share some of that with people. So I thought, you know, maybe it would be a good idea to start a blog.
And that's the genesis of Practical Business Python. So I had this idea in
middle of 2014.
(04:06):
And I originally thought that I would maybe
spend more time talking about things like Raspberry Pi. So I was interested in that, and I thought that would be a fun thing to learn and as I'm learning, blog
about. But then I started to do a little bit more research to figure out what I wanted to write about, and that's when I discovered Pandas, started getting more involved with the data science world.
(04:29):
And
as I continued that discovery process
of learning about all these tools and figuring out how I could apply it to my own problems,
it just naturally flowed into blog articles where I could share that with people.
And then what was really interesting is the feedback I would get from people. So once I would post an article, I'd get feedback, oh, this is really helpful. This really clarified some concepts for me, And that kind of creates that feedback loop that encouraged me to continue to create content
(05:00):
and, you know, improve it over time. And
to be honest, that's really what sustains me now with the blog. I guess 2 things. 1, I enjoyed learning. I enjoy learning more about what's going on in the Python world and how you can use all these really sophisticated tools that are out there. And I enjoy
presenting it to people in a way that hopefully simplifies it and brings it down to the types of problems that they're dealing with.
(05:25):
And the feedback I get from people when they say, hey. This is great. Here's a problem I had, or you helped me save
hours of work in my day job because of Python. So I think those are the types of things that really sustain me going forward. And you mentioned that you have a degree in electrical engineering and computer science, so you're familiar with sort of the programmatic aspects. But you also mentioned that your actual day to day is more on the business side of things. So I'm wondering if you can just give a bit of context about some of the
(05:56):
specifics of your role and some of the ways that you actually end up using Python in your day to day. So my specific day to day role is I work doing strategic pricing for a medical device company here in the US.
And what that means is it's a lot of data analysis,
taking a look at maybe 3rd party or external data, as well as our internal data, and trying to figure out the most effective way to price our products and discount our products in the market.
(06:23):
So I use Python a lot to build
repeatable
and scalable data analysis processes
for myself.
And
the way I think of it is
a lot of people would use Excel to maybe model the data, do the analysis, and I try as much as possible where it makes sense to use Python and Pandas and Jupyter Notebooks
(06:46):
to do that ad hoc analysis
and build it in a way that's more repeatable so that once I've done an analysis 1 time, it's easier to
replicate
for another product or another scenario I wanna model. Excel is definitely a tool that is used pretty ubiquitously
across different businesses and is, in some ways, considered to be the most widely used programming environment that are in existence
(07:12):
For people who are used to Excel and familiar with it, there are definitely ways to use
Python within that context where you can
either read or write Excel files into Python programs and then do some manipulation and then either write it back out to another Excel file or use it in other contexts. But for
(07:33):
the people who are coming to your blog and who you interact with, what are some of the biggest problems that they are
working with and looking to solve for themselves, and what are the points where they start to hit the limits of Excel and need to be able to branch out beyond that? I think there's a couple common cases. 1 of the most common things that I hear, at least specific feedback from people,
(07:57):
are cases where there are a lot of Excel files that they're trying to manipulate.
So, typically, it's a case where maybe it's internally within the company, you're generating a lot of Excel files,
or maybe it's kind of like a b to b transaction where
customers
or business partners are sending Excel files,
(08:18):
and people are having to manage dozens or 100 or thousands of Excel files potentially
and manually
trying to copy and paste
or maybe using a little VBA to manipulate this data.
And
what I think is most people that come to my blog have
(08:38):
some background
maybe through
their university or schooling
or maybe just self educated, they understand
there has to be a better way. Like, maybe they're not computer science professionals,
but they've maybe been exposed to a little bit of programming and start to do some investigation and say, this Excel
(09:00):
pile of files that I'm trying to manipulate and manage is just that's not what computers are here for. It's not for me to be the cut and paste
and do all these VLOOKUPs so they find practical business Python
as a place to give them ideas about how they can
manipulate all those files. So I think that's a pretty common
(09:21):
use case of Excel or CSV file manipulation and maintenance.
The other use case that I think
fits well with Practical Business Python that draws in a lot of people is where there are
techniques that are maybe a little more complex
to do in Excel, but are easier to do in Python. But 1 of the specific examples I'll give is a article I wrote about fuzzy matching, so trying to match addresses. So names and addresses of accounts.
(09:51):
That's 1 of those things that is, I think, a fairly common business problem. And
if you don't have a really robust solution, most people try and do Excel to maybe do, like, a VLOOKUP and some some string matching.
But Python has libraries that can do
much more robust, sophisticated,
(10:11):
fuzzy matching.
And when people get to the point where they're like, this Excel solution isn't gonna work, it's just not sophisticated enough, I need to look for something else in the, quote, data science space,
that's where they start to kind of fall into that or at least
identify that Python might be a good solution and hopefully find practical business Python to give them some pointers. So I think those are 2 key areas.
(10:38):
And then probably the third 1 is just when the overall data just gets so big
that it won't fit in Excel. And I think we're at this interesting place
with Python versus Excel where
Excel will break down at a 1, 000, 000 or so rows, but a 1, 000, 000 rows of data,
even 2, 000, 000 rows of data is not really that big in a modern laptop that's gonna fit in your memory, and you can do manipulation
(11:02):
with Pandas pretty easily with it, whereas in Excel,
it's just not a pleasant experience. So I think that's another reason why people start to look for alternatives to Excel.
Another interesting aspect of sort of the Excel ecosystem is that because it's so ubiquitous, it has become kind of the default method of data interchange within a business where if somebody wants
(11:25):
to get a report or ask for some data from somebody else, then they will just share an Excel file through Dropbox or use Google Sheets or something like that. And so particularly for those types of environments, I'm wondering what you have seen as far as any kinds of switching costs for people who start to use Python
for being able to do that analysis and then being able to interoperate with other people across the business to say, I've done this analysis. Here's the output. And then they say, well, where's the Excel file? I wanna be able to do my own slicing and dicing on it. And if there's any sort of virality of Python within a business context once 1 person starts to go down that road. It's a really good question, and I think that is 1 of the biggest challenges. So I have struggled with taking that next step. So I can build analysis that works well for me. I can maybe train a small group of people on how to use the tools, and maybe we share Jupyter Notebooks or Python programs to do analysis.
(12:23):
But you're right. When it comes to sharing
a report you build with pandas and maybe some visualization,
there's no good way to do that in the the current ecosystem today. I mean, there are commercial offerings and there are opportunities, but it is a big challenge to figure out how do I share that.
And to be honest, what typically happens is
(12:45):
converting that report or that data back into Excel for sharing
or cutting and pasting
information
into an email or putting it in a PowerPoint.
So I feel like we haven't arrived. We don't have a perfect solution yet,
but, hopefully, we'll get some better options here in the future. But you're right. It is a big challenge, and it is something that I think is gonna take some time for us to figure out what's the best way to make
(13:13):
Python
code as interoperable and as shareable as Excel is today. Well, hopefully, Microsoft actually does commit to adding Python support within Excel, so it'll sort of lighten the load. And I think that's really interesting.
I'm kind of fascinated
with what Microsoft is doing with Python. I mean, they have certainly
(13:33):
improved
Python
or the interaction of Python and Windows so much from, you know, where we were many, many years ago where getting Python just installed on a Windows system was hard. And now it's not quite part of the base install, but it's very easy to install. And you look at Versus Code,
which I think the majority of developers are using today, and it has first class integration with Python and works really well.
(13:59):
And Microsoft is clearly supporting it. So I think the $1, 000, 000 question is, what is that gonna look like with Excel?
And what I think is also interesting is
Excel continues to evolve.
So Excel feels like it's getting more,
I'll call it, 1st class programming
capabilities. So Excel just recently launched a new Lambda functionality where you can kind of build
(14:25):
functions
within Excel that are easier to scale and not necessarily go to VBA.
Excel has started to introduce different data types.
I was just looking before we got on this call
that Excel has a new
something they're calling Power f x, which is a low code programming language for the Power platform.
(14:47):
So it feels like we're moving towards this convergence where Excel's gonna continue to get more
powerful
programming, and Python's gonna get more powerful visualization
and
sharing capabilities,
and I don't know what that ultimate
place is gonna look like 5, 10 years from now. On the point of Microsoft's investment in Python, they've even hired Guido onto their developer division. So it'll be interesting to see what comes out of that as well. And then in terms
(15:17):
of Python
as a tool in the business, you know, there are definitely points where you hit the limitations of Excel or some of the other out of the box tools
that are common in a business context. But what is it about Python itself that makes it a viable option for business users to
transition to or start to adopt for integrating into their workflows and solving the types of problems that they're coming up against? I think there's probably 2
(15:42):
main reasons.
1, Python is
for a new person to programming. Python is a language that
has a very clean syntax
and good tutorials and good reference materials to help people learn. And my experience is people generally can look at Python programming
or Python code and not be intimidated by it. They kind of look at it, and it makes sense. They understand
(16:09):
it. So it is simpler for new people to get started, and there's a lot of concepts that Python takes care of that you
maybe have to deal with in a lower left language like a c or c plus plus.
It also has that cleaner, crisp, concise syntax versus something like a Java. So you can get started with Python with minimal boilerplate,
(16:31):
and I think that really helps with adoption.
I think the second thing is the Python ecosystem.
So there are so many tools,
libraries, modules out there in the Python world that can help people solve their problems. And so if it's maybe
you're working in a specific scientific field, there might be a package out there that does a common type of analysis for you. There could be common packages to make it easier to
(17:00):
work with address information. You know, just a whole host of common problems where there's already been a lot of work done
in Python
that you can leverage. So being able to have those building blocks, I think, are 2 of the really
primary reasons why Python is good for business users. And I think the third 1
is
(17:20):
I suspect that within most organizations,
someone is using Python probably in the IT or technical organization.
So Python is so common now
that business setting, they don't know about it. But if you go to your IT team or maybe your r and d team,
someone's using Python. So there's already kind of this foot in the door, and how do you expand that to have more business users use it? Coming as somebody who is
(17:47):
a software engineer by profession,
I have certain conceptions about the ways that you structure your programs and the
ecosystem of tools around my software that I wanna be able to incorporate. So things like testing and linting and version control and deployment considerations.
But for people who are just using it to solve day to day problems likely confined to their laptop, how would you characterize the differences between the ways that a professional software engineer might approach a problem versus
(18:18):
the approach that a business user might take when addressing a problem with Python?
There is a big difference. I think something
as simple as version control. So your software engineer is gonna use GitHub or or whatever the tools.
And once you use those tools, you rely on those, and you almost don't question
why you need to have that in place.
(18:39):
Whereas a business user that doesn't have that exposure,
I think it's a lot to try and
explain how do I program and also how do I use Git to check-in and check out and update the code and see the diff. So I think that is 1 of those things where
I would hope the business user
could get to that level where they understand some of those things like version control, but I typically don't
(19:03):
focus on that in the beginning. So I think think that's 1 area where a software engineer such as yourself or others would really cringe
as the business users
continue to use the version control by file naming convention
or
maybe something like OneDrive or Dropbox or whatever to version control
or backup. So I think that's a key area. I think also the rigor around documentation
(19:30):
and testing
and like you said, linting, having consistent
standards,
I think software engineers would cringe, right, because there isn't any of that for the business users.
But my counterargument
to that is
business users already creating
formulas in Excel. They don't have a test plan for this Excel
(19:50):
document that is probably critical to what the business is doing or at least critical to the decisions the business is making. So I think we need to have some brace and support for these business users as they're trying to learn that and not expect them to
really get up to that software engineer level out of the gate,
that that's more of an aspirational goal that we could have in the future as the tools improve
(20:15):
and as the knowledge of Python and software engineering practices
improves.
A lot of the aspects of things like version control, linting, etcetera,
is largely in place because of the fact that a given software engineer is likely
(20:38):
the the, you know, linting rules for the different stylistic elements. Whereas for the business user, they might be the only person in the whole company that's ever gonna look at this code or care about what it's actually doing. So
as you said, there's certainly room for a lot of
understanding that they don't need to care about those types of things because the thing that they're caring about is
(21:00):
how many units that I shipped last month, how's that gonna impact the number of units that I ordered this month or whatever the case may be versus,
you know, I need to be able to run this code on a server spread across, you know, 3 different availability zones to make sure that it doesn't go down when I have 5, 000 users hitting it. Exactly. It's a completely different scale, but I think that is 1 of the nice things about Python. And I did mention this earlier about what I like about Python is the ceiling is so much higher. Like, if someone spends the time to learn VBA,
(21:30):
like, VBA,
it's gonna work in Excel, and you can work in the Microsoft platform and do some automation. But you're never gonna
progress and deploy, you know, massively at scale applications with VBA. Whereas Python,
if you wanna just learn a little bit of Python and a little bit of Pandas, you can do some work. But if
you wanna continue to evolve
(21:52):
and grow, I mean, the sky's the limit. Right? Python will allow you to do a lot of things. So I think that's a really
nice benefit.
I think the other thing that
Python is good at and where you can start to maybe take some baby steps for
proper software engineering skills is just starting to train people on
(22:14):
stop and think about how to assign names to your variables
and stop and think about how to name your Jupyter Notebooks.
Those very basic things
that
give an example, I was just talking with someone in my job, and I was showing them Python and kind of doing some mentoring and training them.
And they looked at 1 of my lines of code, and they were questioning about my variable name. And they said, well, the variable says this, but don't you really mean that?
(22:41):
And it's a great experience to kinda cause me to take a step back and realize, like, you really do
need to take some time and think about something as simple as naming a variable. And that's kind of the famous quote about the number of challenging things in computer science and
naming things is hard. And so I think that's where
the basic Python you can start to build in some of those concepts
(23:05):
that
a software engineer, I think, would recognize
as good practices.
The perennial joke of there are 2 hard things in computer science, naming things, cache invalidation, and off by 1 errors.
Exactly.
I've given that joke a couple of times in some of my business settings, and people just kinda look at me funny, but I like it.
(23:26):
For business users who do start down the road of
using Python to solve their specific problems,
what are some of the cases where you see a tipping point
beyond which they are better served by passing off that particular piece of software or
problem to a software engineer to build a more robust solution versus just sort of custom coding it for their use case? And, you know, in the reverse, what are some of the cases where
(23:53):
you've seen business users go to the software engineering department to say, hey. I need you to build this tool, whereas something that could actually
be easily handled by teaching them a little bit of Python versus just handing them the solution.
The first part of the question of how do you decide when it needs to get handed off, I think there are a couple things to keep in mind. So 1 will be
(24:14):
how broadly does the tool need to be shared?
So if it is for
your own consumption, your own analysis, and the output is an Excel file or an email or a PDF report,
then
maybe you don't really need a big software engineering investment.
But once you get to the point where
you need to share this
(24:36):
broadly
outside of your team across the organization,
then you start to get into concerns around reliability,
scalability,
security,
and those are the types of things where you need someone
within the organization to help set that up and manage and maintain that. So I think that's 1 big thing. The other 1 is depending on the size of the data. So if the data does start to get big and
(25:03):
you can't just have your laptop running 24 hours a day doing some sort of number crunching and serving, you know, some sort of web server for people to see the results,
you need
to work with the IT organization,
the software engineering team to build something that's a little more robust.
And then I would say the other piece to this is security.
(25:27):
So
a lot of times when you're just doing the work on your own, you're looking maybe at the organization data as a whole. But if you share it, you might have concerns about
broadly sharing that information across the organization. You need to partition it based on roles,
and that's where you certainly need
more of that
software engineering type role to help. I'd say on the flip side,
(25:50):
I think the software engineering team
would
get a lot of benefit. If more people knew how to program, I think it would give those people
better capability to work with the software engineers to build solutions that work for the organization.
And I just think that
the more
the business understands programming,
(26:11):
and I don't mean that they all need to be software engineers, but if they start to understand Python a little bit more, I think that does help
the IT organization
or the software engineers kinda build the overall
IQ, if you will, of the organization on what it takes to develop software.
And then, hopefully, you could then translate that into,
(26:33):
well, now here are some solutions where I can just provide you the data, and you can do the analysis on your own. And when it reaches that tipping point where it needs to scale, then we can have a discussion. Yeah. And it also
once somebody understands a bit more about the ways that software engineers think about problems and approach them, they can also help even just with raising the quality of bug reports rather than just the site is down. It's well, the site is down. I tried this thing. This is what's not working.
(27:01):
So
that definitely helps to improve the overall relationship.
Exactly. And I think,
you know, some part of the challenge with doing software development is the business users are gonna try and put everything in there and, you know, try and push all that functionality in there. And there needs to be some level of tension
and give and take to make sure that we're really only working on
(27:22):
the features that are gonna provide the most benefit and not spending a lot of time on features that no 1 uses.
And I think that's hard
for people that don't do any programming to understand what that trade off is like. And, hopefully,
getting a little bit of experience with Python will start to give them a little bit better feel for the other side of the table.
As far as the
(27:43):
onramps for integrating Python into a user's workflow, we kind of discussed a bit about the tipping point where they hit the limits of Excel and maybe they start to experiment a little bit with pandas or reporting. But
what are some of the useful sort of concrete steps that you can take to
help a business user actually
(28:04):
start down the path of using Python and understanding
how to
formulate their problem in a way that it can be solved using Python as a tool and
helping to
expose them to the overall ecosystem
of packages that are available for them to be able to include
in the solutions that they're trying to build for themselves?
(28:24):
1 of the things that I think is a good way to do it is to start small with a problem that's really well understood.
And typically problems that are like we talked about, maybe there's a lot of Excel file manipulation
or there's a report that goes out every day or every week
or just some sort of process that's repetitive
(28:48):
and might not be that time consuming,
but is inefficient.
So
1 of the things that I like to steer people away from is don't try and jump into
using Python to maybe build,
you know, an artificial intelligence
system to predict your inventory
levels 6 months in the future. Like, that's a really good problem, but that's a hard problem to solve.
(29:14):
Maybe it's better to start with this Excel file that every Friday I send out. It takes me 15 to 20 minutes.
It's not terribly
complicated work, but it's also not very value added work. And let's focus on this small problem
and show the users how they can use Python in this problem that they understand well so that they can just focus on learning Python
(29:37):
and not, you know, trying to figure out how to solve the problem.
And I think that's a good way to get people started
because what that does is it starts to open people's eyes
to kind of how Python works and how you can get things done with Python and how little Python you need to solve a problem.
But it also starts to show them that once you automate a task,
(30:00):
there may be benefits outside of just time savings.
And so what I've seen, and I've even had it in the past
with things that I've done. Let's say I have an Excel report
or an analysis I need to do, and it takes me an hour to do it.
It's not terribly complicated, but it's time consuming.
And because it takes me an hour, I only run it once a week or once a month.
(30:25):
But if it was a push button thing, it might be something where I wouldn't have any issue if I run that report every day or twice a day. And so sometimes I think people don't realize that by automating it, it's not just the time savings of your time at the keyboard doing the work. It's also
the new
capabilities you have to share that information
(30:47):
in a more timely fashion with the ultimate customers.
And so once you start a user down that path of starting to use Python and understanding
some of the scopes of problems that they can solve with it, what are some of the common stumbling blocks that you see them run into as they're starting to move beyond just that initial stage of, okay. Well, here's a tool. I can use it to
(31:08):
then trying to explore on their own and
be able to expand the horizons of the types of problems that they're trying to solve? I think the biggest problem is time
and willingness
to
roll up their sleeves and actually do some of the basic learning work you need to do to have that foundation so you can solve the problem.
(31:29):
So what I see is, at least today, people are receptive to the idea of using Python
because it is so popular from the data science
perspective.
So they're excited to use Python. They have a problem. They think Python can help,
and maybe they look at some Python code and think, oh,
that makes sense to me. It's really simple.
(31:51):
I should be able to use it to solve my problem, but then they don't take the time
to actually sit down and try and code and solve a problem.
And, unfortunately,
I think with programming, like with most things in life, the only way you can really learn
is to do it and troubleshoot it
and spend time, you know, kind of wrestling with the problem
(32:14):
so that you understand it.
And it is hard for people
when they have a busy job and they're working
40, 50, 60, 70 hours a week, however many it is,
and to ask them to carve out time
to learn
so that they can shave 5, 10 minutes off this report they're running. So I think it's how do you get people to buy into
(32:39):
this is a skill
that I'm gonna learn and I'm gonna have with me for life, and the benefits are gonna be worth the time investment.
So I think that's the single biggest challenge.
And then the other challenges are probably more
technical. You know, how do you get Python installed? How do you get your packages installed? How do you share your results? All those kinds of things.
(33:02):
I think those are solvable, but the biggest challenge is helping people
commit to the learning path
to actually build that capability and skill. As far as some of the kind of common roadblocks that everybody who uses Python runs into, what do you see as some of the biggest sticking points? So I'm thinking in particular about things like, I
(33:22):
have this script. It's working. I have all my dependencies installed, and maybe I've even gone down the path of figuring out virtual end. So I've got a virtual end set up, and then Python releases a new version. My version on my machine updates, and now all of the packages that I have are broken because the Python that it's linking to is no longer there. I'll be honest. You know, that's 1 of those things that I tend to gloss over, and I try to when I'm trying to train people,
(33:47):
try and keep the environments
pretty well controlled and walk them through anytime we need to install something.
But to be honest, that's not really a scalable solution. It works when you're trying to train someone, but it's not
realistic.
Right? Like you said, we've all had the situations where suddenly the environment gets
messed up in some way, and you have to blow it away, rebuild it from scratch.
(34:12):
And I have not spent the time to walk through that with people.
It's a challenge. I guess I don't, unfortunately, have a great answer for how to solve that. Well, maybe that's an idea for your next article.
Exactly.
As you have been using Python in your own work and working with people in your organization
as well
(34:33):
as working on the blog to help spread the awareness of this as a viable option for business users.
What are some of the most interesting or innovative or impressive ways that you've seen Python used by people within the business context?
1 of the things that I've
enjoyed is I've written a couple examples of case studies where people have reached out to me and said, here's how I've used
(34:56):
concepts that were present on Practical Business Python or other places and how I took that knowledge and built a solution for myself.
And I just find it fascinating
how they are able to take all these building blocks
that are laid out on the blog in other places and
apply it to their own problems.
(35:17):
And to be honest, a lot of the solutions are not
necessarily rocket science,
not extremely
sophisticated.
But what I think is interesting is how much
value someone can realize with just a little bit of Python. Like, you don't need to write 10, 000 lines of Python code to solve a real problem.
(35:39):
You can take those building blocks and then 10, 20, a 100 lines of Python really solve some challenging business problems that
you would have spent hours trying to manually do. So I think that's
the most impressive thing to me is when I see people that just
start from little or no Python knowledge and
(36:00):
learn enough to actually
solve a problem that they think adds value to their business.
We've all been asked to help with an ad hoc request for data by the sales and marketing team, then it becomes a critical report that they need updated every week or every day. Then what do you do? Send a CSV file via email?
(36:22):
Write some Python scripts to automate it? But what about incremental sync, API quotas, error handling, and all of the other details that eat up your time?
Today, there is a better way. With Census, just write SQL or plug in your dbt models and start syncing your cloud data warehouse to SaaS applications like Salesforce, Marketo, HubSpot, and many more. Go to python podcast.com/census
(36:44):
today to get a free 14 day trial and make your life a lot easier.
And so another interesting thing to think about is that Python as a language has evolved quite a bit since its early days, and there are
even more interesting things being added to the language
with each release. So things like pattern matching and the updated parser and different
(37:10):
modules in the standard library.
And a lot of these are driven by people who are very heavily into the software engineering aspects and building scalable systems or doing advanced data science or machine learning projects. But
what are some of the types of features that you have seen that have been particularly beneficial to
(37:30):
new users of the language or users who are in the business context? And what are some of the features that you would like to see added to
either the language or some of the broader ecosystem?
I think the things that business users really appreciate about Python are
simple things like
there are libraries in place, so I don't have to worry about date time manipulation.
(37:52):
And, certainly, that's not new to Python by any means,
but I think those basics for
daytime manipulation,
file manipulation,
I really enjoy Pathlib, which is newer. And I think when you're working on a Windows environment, that makes
working with Windows path names so much easier.
(38:12):
Those are some of the things I really appreciate.
I do feel like a lot of the more recent
improvements to Python have been
maybe a little more software engineering focused, like type hints, I think, are,
you know, not something that I personally spend a lot of time with just because
the programs I work on
are generally smaller in scope. And just for me, I I certainly see the value, but I don't have a lot of experience there.
(38:37):
I think the other thing that has been nice is
we have gotten better at deploying
Python and binary modules on Windows systems,
whether it's the conda or even now pip, you know, seems to work a lot better on Windows in general.
I think those are all really good improvements that I've seen over the years with Python.
(38:59):
I guess from an ecosystem perspective, I would like to continue to see improvements
in that environment management piece.
I mean, we touched on it briefly. We've all had that experience where the environment gets messed up or
there's a package you need to install,
especially with conda. I seem to have this a lot now where it goes and it's reverting packages back to old versions and
(39:23):
some packages to newer versions or just the
dependency resolver seems to take forever.
I think that is an experience that we need to figure out how to make it
easier for people to manage and not have to spend so much time
worrying about versions in their environments
and being afraid that any kind of upgrade or install is just gonna hose
(39:48):
your entire setup. Yeah. I'm definitely excited for things like the updated dependency resolver in PIP that I think just became the default in the most recent release. And then also things like PEP 585 for
introducing a pikash directory
as the default means of segmenting
or creating a cordoned area for all of your dependencies for given projects to go rather than having to worry about understanding creating a virtual end and pinning it to a particular version of Python where the libraries just go there and then whatever interpreter you're using knows to look at the PyCash directory for the dependencies you're trying to load. Yeah. I'm excited about that as well. And I do think, you know, 1 of the things that's challenging with Python is it has been around for so long. And depending on where you search, you can find
(40:33):
tutorials and examples that are recommending approaches that maybe are not the right approach to use now in the future. And so I think that's a challenging thing for new users to figure out, you know, what do I use to manage my environments? Do I use conda? Do I use PIP? Do I use 1 of these other
tools and which ones are
appropriate?
I think it would be nice if there were
(40:55):
kind of that 1 approach that we could all gel on that would work equally well on Windows and Mac and Linux systems.
In terms of deciding what tool to use for a given problem,
obviously, once you have a hammer, everything starts to look like a nail. And so there are definitely cases
where you say, oh, well, I know Python, so I'm gonna use Python for everything. But what are the cases where you would advocate for a tool other than Python for a business user who is trying to solve a problem in the business context?
(41:22):
You know, a lot of it depends on the organization and what infrastructure they
have in place. But I think there are certain things where
it's important to understand
what the capabilities are of the existing software suite. So I wouldn't recommend that you use Python to roll your own
ERP system or to build your own customer
(41:43):
relationship management system.
You're probably using
SAP or Salesforce
or you have
Tableau or some other visualization tool. So taking a look at what those
tools are in place
and what are the capabilities they have. I mean, many of these tools are
really powerful
and can do a lot of things, and
(42:05):
maybe
time is better spent in learning how to be a superuser
in those tools versus trying to use Python to roll your own solution.
So I think that is a good discussion
that an organization
needs to have, and I think there needs to be
that knowledge so that you can have some some evaluation
(42:27):
of a 1 off Python
script, a handful of Jupyter Notebooks. Is that the right solution, or is it no, we should use this
platform we already have in place and just figure out how
to use the existing
functionality
or how to customize it. So I think there's a balance. I don't know
how best to do it, but I think 1 of the big challenges and where I've used Python a lot is
(42:52):
big IT organizations
don't like to tackle some of these small problems.
So the IT organization is always gonna focus on
larger implementations
that span many months or potentially years.
And if someone says I have a problem that's working with Excel
and it takes me 2 or 3 hours a week, they're gonna say, well,
(43:14):
it's gonna take my IT folks
a 100 hours to build this, so it's not really worth our time. You go figure it out yourself. So I think there needs to be a little bit more of a give and take there where IT will
acknowledge and recognize that there are these smaller problems where a supported Python environment will be great
(43:34):
versus,
yeah, we really should stick with the big
enterprise platforms or the big solutions we have in place. As you continue to use Python in your own work and contribute new articles to the Practical Business Python site, what are some of the things that you have planned for the future of the site or some of the other types of content that you're looking forward to being engaged with or other ways that you are hoping to
(44:00):
kind of spread the awareness and spread the learnings of how to use Python for these use cases?
There's a couple things that I'm thinking about. So 1, I do enjoy the case studies
because I think that's a good way for me to get exposure to maybe other types of problems
and share those with people and hopefully
identify problems that resonate with the wide audience. So I I'd like to continue to have those case studies.
(44:25):
I'd like to continue to
keep looking at the data visualization
space. I think that's 1 area in Python where there's a lot of good libraries, but it's continually changing.
And putting together some articles to talk about how to maybe build more dash boards and present the information.
I think I'll continue to stick to some of the bread and butter
(44:49):
pandas
type activities where articles where people
maybe don't understand how to use pandas effectively or
those kinds of things. And I'm always interested in
new
libraries
that are out there that solve really challenging problems in an elegant way.
So 1 of the things I wrote about in the past was Facebook's,
(45:12):
profit library for time series forecasting.
And I think that's 1 of those areas that it's really interesting to figure out what the
state of the art is for time series forecasting
for
someone that's not a PhD in this area. And so I think there's a lot of interesting activity there, and I would like to continue on that.
(45:33):
Finally,
you know, outside the blog, but I have
done 1 course with Talk Python
on moving from Excel to Python. And I'd like to continue to build out some coursework there to
target users that are looking to start that transition into Python
from a business setting.
Are there any other aspects of your use of Python or your work on the blog or
(45:59):
the overall
use case of Python for business problems that we didn't discuss yet that you'd like to cover before we close out the show? I guess I'll follow-up on that last topic. You know, I am excited to see how
some of these more advanced data science,
machine learning, artificial intelligence,
whatever kind of category you wanna put them into,
(46:20):
how they continue to get streamlined
and
automated so that it's easier for people to apply it to their own problems. I think that's gonna be an exciting space where Python has a lot of good
capabilities now, but I think they're gonna be even more in the future that we can start to
bring down and make it easier for business users to take some of these more complex
(46:44):
approaches to solve the problems they have on a daily basis.
Well, for anybody who wants to follow along with you and get in touch, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us into the picks. And this week, I'm going to choose a couple of newsletters that I tend to read fairly regularly because they're
fairly concise and have a lot of good value and some good analysis on the different topics covered. So 1 of them is the data science roundup by Tristan Handy from Fishtown Analytics, who are the people behind the DBT project.
(47:15):
And the other 1 is called This Week in Data,
and that 1 is by somebody who's working in data science and data engineering at, I believe, DoorDash
or Shopify, something like that. So definitely useful newsletters to keep up to date with some of the things that are happening in the overall data science ecosystem and data engineering space. And so with that, I'll pass it to you, Chris. Do you have any picks this week? I'm gonna go a completely
(47:40):
nontechnical
direction. So I'm a bit of a sucker for
British
crime procedural
dramas, always looking for new shows to stream, and 1 of the ones I've been watching is called Line of Duty.
I don't know, 3 or 5 episodes of BBC
series on Amazon Prime, which I think is pretty well done. And then I'm also reading a
(48:04):
science fiction book called Out of the Dark
by David Weber, which is pretty interesting so far. I'm about halfway through. So I thought I'd
put something out there that's maybe a little bit lighter as we hopefully
get out of the quarantine time and people are
still trying to figure out how to entertain themselves at home. Well, thank you very much for taking the time today to join me and discuss the work that you've been doing on the blog and some of the ways that you use Python in your own work. It's definitely interesting to go outside of the particular problem areas that I spend most of my time in and understand some of the other ways that the language is used. So I appreciate all the time and energy you've put into that, and I hope you enjoy the rest of your day. Thank you. You
(48:46):
too.
Thank you for listening. Don't forget to check out our other show, the data engineering podcast at dataengineeringpodcast.com
for the latest on modern data management.
And visit the site of python podcast dotcom to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email host@podcastinit.com
(49:09):
with your story.
To help other people find the show, please leave a review on Itunes and tell your friends and coworkers.
Hello, and welcome to Podcast Dot in It, the podcast about Python and the people who make it great.
When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So take a look at our friends over at Linode.
With the launch of their managed Kubernetes platform, it's easy to get started with the next generation of deployment and scaling powered by the battle tested Linode platform,
(00:34):
including simple pricing, node balancers,
40 gigabit networking,
dedicated CPU and GPU instances, and worldwide data centers.
Go to python podcast.com/linode,
that's l I n
o d e, today and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show. Your host as usual is Tobias Macy. And today, I'm interviewing Chris Moffett about how Python is used to help manage business needs and processes in his work to share advice on this topic at Practical Business Python. So, Chris, can you start by introducing yourself? Hi, Tobias. My name is Chris Moffett, and I blog at Practical Business Python,
(01:13):
which is
dedicated to
sharing examples of how you can use Python to solve common business problems.
I've been using Python for
probably
a dozen plus years and was formally trained in electrical engineering and computer science, but never actually been a computer scientist.
(01:34):
And most of my career has been on the business side. So that's where I've really enjoyed combining that business experience, that Python experience, and that's a lot of what I'm trying to share at Practical Business Python. Do you remember how you first got introduced to Python?
I do. Maybe like a lot of other people, I had used some other languages. I did a little bit of work as a Unix system administrator
(01:58):
and did some scripting in Pearl and
exposure to other programming languages in the past.
And someone that I was working with said, hey. You might wanna check out this Python thing. It's pretty cool language.
This was probably
in
2, 001, 2, 002,
so still relatively new. And I
(02:20):
looked at it a little bit and thought, you know, this whole white space thing,
it just didn't really resonate with me. I didn't think it was that good a fit, so I passed.
Fast forward about 6 or 7 months, and I was working on a Windows system
and needed to do some scripting
and
took another look at Python and started to play with it for about half a day or so. And suddenly, it just kinda clicked in my brain, made a lot of sense. And I've been using Python in some way, shape, or form ever since then. You mentioned that you've been using practical business Python as an avenue to help educate people on some of the ways that the language is used to solve different problems within the business context.
(03:02):
I'm wondering if you can just give a bit of an overview about your mission
of practical business Python and some of the backstory of how you got started with it and what keeps you motivated after multiple years of blogging and providing resources to that particular segment of the ecosystem?
So like I mentioned, I first got exposed to Python
a while back. So this was pre data science
(03:23):
hype, if you will. This was even probably before Django was really getting started in the web space, so I used it a little bit from an administrative
perspective.
I also did a little bit of Django work.
And then over time, I just stopped using Python. So I didn't have a real good excuse or reason to use it on a daily basis.
(03:44):
Well, I wanted to find an opportunity
to continue
to use Python, explore the Python ecosystem,
and maybe share some of that with people. So I thought, you know, maybe it would be a good idea to start a blog.
And that's the genesis of Practical Business Python. So I had this idea in
middle of 2014.
(04:06):
And I originally thought that I would maybe
spend more time talking about things like Raspberry Pi. So I was interested in that, and I thought that would be a fun thing to learn and as I'm learning, blog
about. But then I started to do a little bit more research to figure out what I wanted to write about, and that's when I discovered Pandas, started getting more involved with the data science world.
(04:29):
And
as I continued that discovery process
of learning about all these tools and figuring out how I could apply it to my own problems,
it just naturally flowed into blog articles where I could share that with people.
And then what was really interesting is the feedback I would get from people. So once I would post an article, I'd get feedback, oh, this is really helpful. This really clarified some concepts for me, And that kind of creates that feedback loop that encouraged me to continue to create content
(05:00):
and, you know, improve it over time. And
to be honest, that's really what sustains me now with the blog. I guess 2 things. 1, I enjoyed learning. I enjoy learning more about what's going on in the Python world and how you can use all these really sophisticated tools that are out there. And I enjoy
presenting it to people in a way that hopefully simplifies it and brings it down to the types of problems that they're dealing with.
(05:25):
And the feedback I get from people when they say, hey. This is great. Here's a problem I had, or you helped me save
hours of work in my day job because of Python. So I think those are the types of things that really sustain me going forward. And you mentioned that you have a degree in electrical engineering and computer science, so you're familiar with sort of the programmatic aspects. But you also mentioned that your actual day to day is more on the business side of things. So I'm wondering if you can just give a bit of context about some of the
(05:56):
specifics of your role and some of the ways that you actually end up using Python in your day to day. So my specific day to day role is I work doing strategic pricing for a medical device company here in the US.
And what that means is it's a lot of data analysis,
taking a look at maybe 3rd party or external data, as well as our internal data, and trying to figure out the most effective way to price our products and discount our products in the market.
(06:23):
So I use Python a lot to build
repeatable
and scalable data analysis processes
for myself.
And
the way I think of it is
a lot of people would use Excel to maybe model the data, do the analysis, and I try as much as possible where it makes sense to use Python and Pandas and Jupyter Notebooks
(06:46):
to do that ad hoc analysis
and build it in a way that's more repeatable so that once I've done an analysis 1 time, it's easier to
replicate
for another product or another scenario I wanna model. Excel is definitely a tool that is used pretty ubiquitously
across different businesses and is, in some ways, considered to be the most widely used programming environment that are in existence
(07:12):
For people who are used to Excel and familiar with it, there are definitely ways to use
Python within that context where you can
either read or write Excel files into Python programs and then do some manipulation and then either write it back out to another Excel file or use it in other contexts. But for
(07:33):
the people who are coming to your blog and who you interact with, what are some of the biggest problems that they are
working with and looking to solve for themselves, and what are the points where they start to hit the limits of Excel and need to be able to branch out beyond that? I think there's a couple common cases. 1 of the most common things that I hear, at least specific feedback from people,
(07:57):
are cases where there are a lot of Excel files that they're trying to manipulate.
So, typically, it's a case where maybe it's internally within the company, you're generating a lot of Excel files,
or maybe it's kind of like a b to b transaction where
customers
or business partners are sending Excel files,
(08:18):
and people are having to manage dozens or 100 or thousands of Excel files potentially
and manually
trying to copy and paste
or maybe using a little VBA to manipulate this data.
And
what I think is most people that come to my blog have
(08:38):
some background
maybe through
their university or schooling
or maybe just self educated, they understand
there has to be a better way. Like, maybe they're not computer science professionals,
but they've maybe been exposed to a little bit of programming and start to do some investigation and say, this Excel
(09:00):
pile of files that I'm trying to manipulate and manage is just that's not what computers are here for. It's not for me to be the cut and paste
and do all these VLOOKUPs so they find practical business Python
as a place to give them ideas about how they can
manipulate all those files. So I think that's a pretty common
(09:21):
use case of Excel or CSV file manipulation and maintenance.
The other use case that I think
fits well with Practical Business Python that draws in a lot of people is where there are
techniques that are maybe a little more complex
to do in Excel, but are easier to do in Python. But 1 of the specific examples I'll give is a article I wrote about fuzzy matching, so trying to match addresses. So names and addresses of accounts.
(09:51):
That's 1 of those things that is, I think, a fairly common business problem. And
if you don't have a really robust solution, most people try and do Excel to maybe do, like, a VLOOKUP and some some string matching.
But Python has libraries that can do
much more robust, sophisticated,
(10:11):
fuzzy matching.
And when people get to the point where they're like, this Excel solution isn't gonna work, it's just not sophisticated enough, I need to look for something else in the, quote, data science space,
that's where they start to kind of fall into that or at least
identify that Python might be a good solution and hopefully find practical business Python to give them some pointers. So I think those are 2 key areas.
(10:38):
And then probably the third 1 is just when the overall data just gets so big
that it won't fit in Excel. And I think we're at this interesting place
with Python versus Excel where
Excel will break down at a 1, 000, 000 or so rows, but a 1, 000, 000 rows of data,
even 2, 000, 000 rows of data is not really that big in a modern laptop that's gonna fit in your memory, and you can do manipulation
(11:02):
with Pandas pretty easily with it, whereas in Excel,
it's just not a pleasant experience. So I think that's another reason why people start to look for alternatives to Excel.
Another interesting aspect of sort of the Excel ecosystem is that because it's so ubiquitous, it has become kind of the default method of data interchange within a business where if somebody wants
(11:25):
to get a report or ask for some data from somebody else, then they will just share an Excel file through Dropbox or use Google Sheets or something like that. And so particularly for those types of environments, I'm wondering what you have seen as far as any kinds of switching costs for people who start to use Python
for being able to do that analysis and then being able to interoperate with other people across the business to say, I've done this analysis. Here's the output. And then they say, well, where's the Excel file? I wanna be able to do my own slicing and dicing on it. And if there's any sort of virality of Python within a business context once 1 person starts to go down that road. It's a really good question, and I think that is 1 of the biggest challenges. So I have struggled with taking that next step. So I can build analysis that works well for me. I can maybe train a small group of people on how to use the tools, and maybe we share Jupyter Notebooks or Python programs to do analysis.
(12:23):
But you're right. When it comes to sharing
a report you build with pandas and maybe some visualization,
there's no good way to do that in the the current ecosystem today. I mean, there are commercial offerings and there are opportunities, but it is a big challenge to figure out how do I share that.
And to be honest, what typically happens is
(12:45):
converting that report or that data back into Excel for sharing
or cutting and pasting
information
into an email or putting it in a PowerPoint.
So I feel like we haven't arrived. We don't have a perfect solution yet,
but, hopefully, we'll get some better options here in the future. But you're right. It is a big challenge, and it is something that I think is gonna take some time for us to figure out what's the best way to make
(13:13):
Python
code as interoperable and as shareable as Excel is today. Well, hopefully, Microsoft actually does commit to adding Python support within Excel, so it'll sort of lighten the load. And I think that's really interesting.
I'm kind of fascinated
with what Microsoft is doing with Python. I mean, they have certainly
(13:33):
improved
Python
or the interaction of Python and Windows so much from, you know, where we were many, many years ago where getting Python just installed on a Windows system was hard. And now it's not quite part of the base install, but it's very easy to install. And you look at Versus Code,
which I think the majority of developers are using today, and it has first class integration with Python and works really well.
(13:59):
And Microsoft is clearly supporting it. So I think the $1, 000, 000 question is, what is that gonna look like with Excel?
And what I think is also interesting is
Excel continues to evolve.
So Excel feels like it's getting more,
I'll call it, 1st class programming
capabilities. So Excel just recently launched a new Lambda functionality where you can kind of build
(14:25):
functions
within Excel that are easier to scale and not necessarily go to VBA.
Excel has started to introduce different data types.
I was just looking before we got on this call
that Excel has a new
something they're calling Power f x, which is a low code programming language for the Power platform.
(14:47):
So it feels like we're moving towards this convergence where Excel's gonna continue to get more
powerful
programming, and Python's gonna get more powerful visualization
and
sharing capabilities,
and I don't know what that ultimate
place is gonna look like 5, 10 years from now. On the point of Microsoft's investment in Python, they've even hired Guido onto their developer division. So it'll be interesting to see what comes out of that as well. And then in terms
(15:17):
of Python
as a tool in the business, you know, there are definitely points where you hit the limitations of Excel or some of the other out of the box tools
that are common in a business context. But what is it about Python itself that makes it a viable option for business users to
transition to or start to adopt for integrating into their workflows and solving the types of problems that they're coming up against? I think there's probably 2
(15:42):
main reasons.
1, Python is
for a new person to programming. Python is a language that
has a very clean syntax
and good tutorials and good reference materials to help people learn. And my experience is people generally can look at Python programming
or Python code and not be intimidated by it. They kind of look at it, and it makes sense. They understand
(16:09):
it. So it is simpler for new people to get started, and there's a lot of concepts that Python takes care of that you
maybe have to deal with in a lower left language like a c or c plus plus.
It also has that cleaner, crisp, concise syntax versus something like a Java. So you can get started with Python with minimal boilerplate,
(16:31):
and I think that really helps with adoption.
I think the second thing is the Python ecosystem.
So there are so many tools,
libraries, modules out there in the Python world that can help people solve their problems. And so if it's maybe
you're working in a specific scientific field, there might be a package out there that does a common type of analysis for you. There could be common packages to make it easier to
(17:00):
work with address information. You know, just a whole host of common problems where there's already been a lot of work done
in Python
that you can leverage. So being able to have those building blocks, I think, are 2 of the really
primary reasons why Python is good for business users. And I think the third 1
is
(17:20):
I suspect that within most organizations,
someone is using Python probably in the IT or technical organization.
So Python is so common now
that business setting, they don't know about it. But if you go to your IT team or maybe your r and d team,
someone's using Python. So there's already kind of this foot in the door, and how do you expand that to have more business users use it? Coming as somebody who is
(17:47):
a software engineer by profession,
I have certain conceptions about the ways that you structure your programs and the
ecosystem of tools around my software that I wanna be able to incorporate. So things like testing and linting and version control and deployment considerations.
But for people who are just using it to solve day to day problems likely confined to their laptop, how would you characterize the differences between the ways that a professional software engineer might approach a problem versus
(18:18):
the approach that a business user might take when addressing a problem with Python?
There is a big difference. I think something
as simple as version control. So your software engineer is gonna use GitHub or or whatever the tools.
And once you use those tools, you rely on those, and you almost don't question
why you need to have that in place.
(18:39):
Whereas a business user that doesn't have that exposure,
I think it's a lot to try and
explain how do I program and also how do I use Git to check-in and check out and update the code and see the diff. So I think that is 1 of those things where
I would hope the business user
could get to that level where they understand some of those things like version control, but I typically don't
(19:03):
focus on that in the beginning. So I think think that's 1 area where a software engineer such as yourself or others would really cringe
as the business users
continue to use the version control by file naming convention
or
maybe something like OneDrive or Dropbox or whatever to version control
or backup. So I think that's a key area. I think also the rigor around documentation
(19:30):
and testing
and like you said, linting, having consistent
standards,
I think software engineers would cringe, right, because there isn't any of that for the business users.
But my counterargument
to that is
business users already creating
formulas in Excel. They don't have a test plan for this Excel
(19:50):
document that is probably critical to what the business is doing or at least critical to the decisions the business is making. So I think we need to have some brace and support for these business users as they're trying to learn that and not expect them to
really get up to that software engineer level out of the gate,
that that's more of an aspirational goal that we could have in the future as the tools improve
(20:15):
and as the knowledge of Python and software engineering practices
improves.
A lot of the aspects of things like version control, linting, etcetera,
is largely in place because of the fact that a given software engineer is likely
(20:38):
the the, you know, linting rules for the different stylistic elements. Whereas for the business user, they might be the only person in the whole company that's ever gonna look at this code or care about what it's actually doing. So
as you said, there's certainly room for a lot of
understanding that they don't need to care about those types of things because the thing that they're caring about is
(21:00):
how many units that I shipped last month, how's that gonna impact the number of units that I ordered this month or whatever the case may be versus,
you know, I need to be able to run this code on a server spread across, you know, 3 different availability zones to make sure that it doesn't go down when I have 5, 000 users hitting it. Exactly. It's a completely different scale, but I think that is 1 of the nice things about Python. And I did mention this earlier about what I like about Python is the ceiling is so much higher. Like, if someone spends the time to learn VBA,
(21:30):
like, VBA,
it's gonna work in Excel, and you can work in the Microsoft platform and do some automation. But you're never gonna
progress and deploy, you know, massively at scale applications with VBA. Whereas Python,
if you wanna just learn a little bit of Python and a little bit of Pandas, you can do some work. But if
you wanna continue to evolve
(21:52):
and grow, I mean, the sky's the limit. Right? Python will allow you to do a lot of things. So I think that's a really
nice benefit.
I think the other thing that
Python is good at and where you can start to maybe take some baby steps for
proper software engineering skills is just starting to train people on
(22:14):
stop and think about how to assign names to your variables
and stop and think about how to name your Jupyter Notebooks.
Those very basic things
that
give an example, I was just talking with someone in my job, and I was showing them Python and kind of doing some mentoring and training them.
And they looked at 1 of my lines of code, and they were questioning about my variable name. And they said, well, the variable says this, but don't you really mean that?
(22:41):
And it's a great experience to kinda cause me to take a step back and realize, like, you really do
need to take some time and think about something as simple as naming a variable. And that's kind of the famous quote about the number of challenging things in computer science and
naming things is hard. And so I think that's where
the basic Python you can start to build in some of those concepts
(23:05):
that
a software engineer, I think, would recognize
as good practices.
The perennial joke of there are 2 hard things in computer science, naming things, cache invalidation, and off by 1 errors.
Exactly.
I've given that joke a couple of times in some of my business settings, and people just kinda look at me funny, but I like it.
(23:26):
For business users who do start down the road of
using Python to solve their specific problems,
what are some of the cases where you see a tipping point
beyond which they are better served by passing off that particular piece of software or
problem to a software engineer to build a more robust solution versus just sort of custom coding it for their use case? And, you know, in the reverse, what are some of the cases where
(23:53):
you've seen business users go to the software engineering department to say, hey. I need you to build this tool, whereas something that could actually
be easily handled by teaching them a little bit of Python versus just handing them the solution.
The first part of the question of how do you decide when it needs to get handed off, I think there are a couple things to keep in mind. So 1 will be
(24:14):
how broadly does the tool need to be shared?
So if it is for
your own consumption, your own analysis, and the output is an Excel file or an email or a PDF report,
then
maybe you don't really need a big software engineering investment.
But once you get to the point where
you need to share this
(24:36):
broadly
outside of your team across the organization,
then you start to get into concerns around reliability,
scalability,
security,
and those are the types of things where you need someone
within the organization to help set that up and manage and maintain that. So I think that's 1 big thing. The other 1 is depending on the size of the data. So if the data does start to get big and
(25:03):
you can't just have your laptop running 24 hours a day doing some sort of number crunching and serving, you know, some sort of web server for people to see the results,
you need
to work with the IT organization,
the software engineering team to build something that's a little more robust.
And then I would say the other piece to this is security.
(25:27):
So
a lot of times when you're just doing the work on your own, you're looking maybe at the organization data as a whole. But if you share it, you might have concerns about
broadly sharing that information across the organization. You need to partition it based on roles,
and that's where you certainly need
more of that
software engineering type role to help. I'd say on the flip side,
(25:50):
I think the software engineering team
would
get a lot of benefit. If more people knew how to program, I think it would give those people
better capability to work with the software engineers to build solutions that work for the organization.
And I just think that
the more
the business understands programming,
(26:11):
and I don't mean that they all need to be software engineers, but if they start to understand Python a little bit more, I think that does help
the IT organization
or the software engineers kinda build the overall
IQ, if you will, of the organization on what it takes to develop software.
And then, hopefully, you could then translate that into,
(26:33):
well, now here are some solutions where I can just provide you the data, and you can do the analysis on your own. And when it reaches that tipping point where it needs to scale, then we can have a discussion. Yeah. And it also
once somebody understands a bit more about the ways that software engineers think about problems and approach them, they can also help even just with raising the quality of bug reports rather than just the site is down. It's well, the site is down. I tried this thing. This is what's not working.
(27:01):
So
that definitely helps to improve the overall relationship.
Exactly. And I think,
you know, some part of the challenge with doing software development is the business users are gonna try and put everything in there and, you know, try and push all that functionality in there. And there needs to be some level of tension
and give and take to make sure that we're really only working on
(27:22):
the features that are gonna provide the most benefit and not spending a lot of time on features that no 1 uses.
And I think that's hard
for people that don't do any programming to understand what that trade off is like. And, hopefully,
getting a little bit of experience with Python will start to give them a little bit better feel for the other side of the table.
As far as the
(27:43):
onramps for integrating Python into a user's workflow, we kind of discussed a bit about the tipping point where they hit the limits of Excel and maybe they start to experiment a little bit with pandas or reporting. But
what are some of the useful sort of concrete steps that you can take to
help a business user actually
(28:04):
start down the path of using Python and understanding
how to
formulate their problem in a way that it can be solved using Python as a tool and
helping to
expose them to the overall ecosystem
of packages that are available for them to be able to include
in the solutions that they're trying to build for themselves?
(28:24):
1 of the things that I think is a good way to do it is to start small with a problem that's really well understood.
And typically problems that are like we talked about, maybe there's a lot of Excel file manipulation
or there's a report that goes out every day or every week
or just some sort of process that's repetitive
(28:48):
and might not be that time consuming,
but is inefficient.
So
1 of the things that I like to steer people away from is don't try and jump into
using Python to maybe build,
you know, an artificial intelligence
system to predict your inventory
levels 6 months in the future. Like, that's a really good problem, but that's a hard problem to solve.
(29:14):
Maybe it's better to start with this Excel file that every Friday I send out. It takes me 15 to 20 minutes.
It's not terribly
complicated work, but it's also not very value added work. And let's focus on this small problem
and show the users how they can use Python in this problem that they understand well so that they can just focus on learning Python
(29:37):
and not, you know, trying to figure out how to solve the problem.
And I think that's a good way to get people started
because what that does is it starts to open people's eyes
to kind of how Python works and how you can get things done with Python and how little Python you need to solve a problem.
But it also starts to show them that once you automate a task,
(30:00):
there may be benefits outside of just time savings.
And so what I've seen, and I've even had it in the past
with things that I've done. Let's say I have an Excel report
or an analysis I need to do, and it takes me an hour to do it.
It's not terribly complicated, but it's time consuming.
And because it takes me an hour, I only run it once a week or once a month.
(30:25):
But if it was a push button thing, it might be something where I wouldn't have any issue if I run that report every day or twice a day. And so sometimes I think people don't realize that by automating it, it's not just the time savings of your time at the keyboard doing the work. It's also
the new
capabilities you have to share that information
(30:47):
in a more timely fashion with the ultimate customers.
And so once you start a user down that path of starting to use Python and understanding
some of the scopes of problems that they can solve with it, what are some of the common stumbling blocks that you see them run into as they're starting to move beyond just that initial stage of, okay. Well, here's a tool. I can use it to
(31:08):
then trying to explore on their own and
be able to expand the horizons of the types of problems that they're trying to solve? I think the biggest problem is time
and willingness
to
roll up their sleeves and actually do some of the basic learning work you need to do to have that foundation so you can solve the problem.
(31:29):
So what I see is, at least today, people are receptive to the idea of using Python
because it is so popular from the data science
perspective.
So they're excited to use Python. They have a problem. They think Python can help,
and maybe they look at some Python code and think, oh,
that makes sense to me. It's really simple.
(31:51):
I should be able to use it to solve my problem, but then they don't take the time
to actually sit down and try and code and solve a problem.
And, unfortunately,
I think with programming, like with most things in life, the only way you can really learn
is to do it and troubleshoot it
and spend time, you know, kind of wrestling with the problem
(32:14):
so that you understand it.
And it is hard for people
when they have a busy job and they're working
40, 50, 60, 70 hours a week, however many it is,
and to ask them to carve out time
to learn
so that they can shave 5, 10 minutes off this report they're running. So I think it's how do you get people to buy into
(32:39):
this is a skill
that I'm gonna learn and I'm gonna have with me for life, and the benefits are gonna be worth the time investment.
So I think that's the single biggest challenge.
And then the other challenges are probably more
technical. You know, how do you get Python installed? How do you get your packages installed? How do you share your results? All those kinds of things.
(33:02):
I think those are solvable, but the biggest challenge is helping people
commit to the learning path
to actually build that capability and skill. As far as some of the kind of common roadblocks that everybody who uses Python runs into, what do you see as some of the biggest sticking points? So I'm thinking in particular about things like, I
(33:22):
have this script. It's working. I have all my dependencies installed, and maybe I've even gone down the path of figuring out virtual end. So I've got a virtual end set up, and then Python releases a new version. My version on my machine updates, and now all of the packages that I have are broken because the Python that it's linking to is no longer there. I'll be honest. You know, that's 1 of those things that I tend to gloss over, and I try to when I'm trying to train people,
(33:47):
try and keep the environments
pretty well controlled and walk them through anytime we need to install something.
But to be honest, that's not really a scalable solution. It works when you're trying to train someone, but it's not
realistic.
Right? Like you said, we've all had the situations where suddenly the environment gets
messed up in some way, and you have to blow it away, rebuild it from scratch.
(34:12):
And I have not spent the time to walk through that with people.
It's a challenge. I guess I don't, unfortunately, have a great answer for how to solve that. Well, maybe that's an idea for your next article.
Exactly.
As you have been using Python in your own work and working with people in your organization
as well
(34:33):
as working on the blog to help spread the awareness of this as a viable option for business users.
What are some of the most interesting or innovative or impressive ways that you've seen Python used by people within the business context?
1 of the things that I've
enjoyed is I've written a couple examples of case studies where people have reached out to me and said, here's how I've used
(34:56):
concepts that were present on Practical Business Python or other places and how I took that knowledge and built a solution for myself.
And I just find it fascinating
how they are able to take all these building blocks
that are laid out on the blog in other places and
apply it to their own problems.
(35:17):
And to be honest, a lot of the solutions are not
necessarily rocket science,
not extremely
sophisticated.
But what I think is interesting is how much
value someone can realize with just a little bit of Python. Like, you don't need to write 10, 000 lines of Python code to solve a real problem.
(35:39):
You can take those building blocks and then 10, 20, a 100 lines of Python really solve some challenging business problems that
you would have spent hours trying to manually do. So I think that's
the most impressive thing to me is when I see people that just
start from little or no Python knowledge and
(36:00):
learn enough to actually
solve a problem that they think adds value to their business.
We've all been asked to help with an ad hoc request for data by the sales and marketing team, then it becomes a critical report that they need updated every week or every day. Then what do you do? Send a CSV file via email?
(36:22):
Write some Python scripts to automate it? But what about incremental sync, API quotas, error handling, and all of the other details that eat up your time?
Today, there is a better way. With Census, just write SQL or plug in your dbt models and start syncing your cloud data warehouse to SaaS applications like Salesforce, Marketo, HubSpot, and many more. Go to python podcast.com/census
(36:44):
today to get a free 14 day trial and make your life a lot easier.
And so another interesting thing to think about is that Python as a language has evolved quite a bit since its early days, and there are
even more interesting things being added to the language
with each release. So things like pattern matching and the updated parser and different
(37:10):
modules in the standard library.
And a lot of these are driven by people who are very heavily into the software engineering aspects and building scalable systems or doing advanced data science or machine learning projects. But
what are some of the types of features that you have seen that have been particularly beneficial to
(37:30):
new users of the language or users who are in the business context? And what are some of the features that you would like to see added to
either the language or some of the broader ecosystem?
I think the things that business users really appreciate about Python are
simple things like
there are libraries in place, so I don't have to worry about date time manipulation.
(37:52):
And, certainly, that's not new to Python by any means,
but I think those basics for
daytime manipulation,
file manipulation,
I really enjoy Pathlib, which is newer. And I think when you're working on a Windows environment, that makes
working with Windows path names so much easier.
(38:12):
Those are some of the things I really appreciate.
I do feel like a lot of the more recent
improvements to Python have been
maybe a little more software engineering focused, like type hints, I think, are,
you know, not something that I personally spend a lot of time with just because
the programs I work on
are generally smaller in scope. And just for me, I I certainly see the value, but I don't have a lot of experience there.
(38:37):
I think the other thing that has been nice is
we have gotten better at deploying
Python and binary modules on Windows systems,
whether it's the conda or even now pip, you know, seems to work a lot better on Windows in general.
I think those are all really good improvements that I've seen over the years with Python.
(38:59):
I guess from an ecosystem perspective, I would like to continue to see improvements
in that environment management piece.
I mean, we touched on it briefly. We've all had that experience where the environment gets messed up or
there's a package you need to install,
especially with conda. I seem to have this a lot now where it goes and it's reverting packages back to old versions and
(39:23):
some packages to newer versions or just the
dependency resolver seems to take forever.
I think that is an experience that we need to figure out how to make it
easier for people to manage and not have to spend so much time
worrying about versions in their environments
and being afraid that any kind of upgrade or install is just gonna hose
(39:48):
your entire setup. Yeah. I'm definitely excited for things like the updated dependency resolver in PIP that I think just became the default in the most recent release. And then also things like PEP 585 for
introducing a pikash directory
as the default means of segmenting
or creating a cordoned area for all of your dependencies for given projects to go rather than having to worry about understanding creating a virtual end and pinning it to a particular version of Python where the libraries just go there and then whatever interpreter you're using knows to look at the PyCash directory for the dependencies you're trying to load. Yeah. I'm excited about that as well. And I do think, you know, 1 of the things that's challenging with Python is it has been around for so long. And depending on where you search, you can find
(40:33):
tutorials and examples that are recommending approaches that maybe are not the right approach to use now in the future. And so I think that's a challenging thing for new users to figure out, you know, what do I use to manage my environments? Do I use conda? Do I use PIP? Do I use 1 of these other
tools and which ones are
appropriate?
I think it would be nice if there were
(40:55):
kind of that 1 approach that we could all gel on that would work equally well on Windows and Mac and Linux systems.
In terms of deciding what tool to use for a given problem,
obviously, once you have a hammer, everything starts to look like a nail. And so there are definitely cases
where you say, oh, well, I know Python, so I'm gonna use Python for everything. But what are the cases where you would advocate for a tool other than Python for a business user who is trying to solve a problem in the business context?
(41:22):
You know, a lot of it depends on the organization and what infrastructure they
have in place. But I think there are certain things where
it's important to understand
what the capabilities are of the existing software suite. So I wouldn't recommend that you use Python to roll your own
ERP system or to build your own customer
(41:43):
relationship management system.
You're probably using
SAP or Salesforce
or you have
Tableau or some other visualization tool. So taking a look at what those
tools are in place
and what are the capabilities they have. I mean, many of these tools are
really powerful
and can do a lot of things, and
(42:05):
maybe
time is better spent in learning how to be a superuser
in those tools versus trying to use Python to roll your own solution.
So I think that is a good discussion
that an organization
needs to have, and I think there needs to be
that knowledge so that you can have some some evaluation
(42:27):
of a 1 off Python
script, a handful of Jupyter Notebooks. Is that the right solution, or is it no, we should use this
platform we already have in place and just figure out how
to use the existing
functionality
or how to customize it. So I think there's a balance. I don't know
how best to do it, but I think 1 of the big challenges and where I've used Python a lot is
(42:52):
big IT organizations
don't like to tackle some of these small problems.
So the IT organization is always gonna focus on
larger implementations
that span many months or potentially years.
And if someone says I have a problem that's working with Excel
and it takes me 2 or 3 hours a week, they're gonna say, well,
(43:14):
it's gonna take my IT folks
a 100 hours to build this, so it's not really worth our time. You go figure it out yourself. So I think there needs to be a little bit more of a give and take there where IT will
acknowledge and recognize that there are these smaller problems where a supported Python environment will be great
(43:34):
versus,
yeah, we really should stick with the big
enterprise platforms or the big solutions we have in place. As you continue to use Python in your own work and contribute new articles to the Practical Business Python site, what are some of the things that you have planned for the future of the site or some of the other types of content that you're looking forward to being engaged with or other ways that you are hoping to
(44:00):
kind of spread the awareness and spread the learnings of how to use Python for these use cases?
There's a couple things that I'm thinking about. So 1, I do enjoy the case studies
because I think that's a good way for me to get exposure to maybe other types of problems
and share those with people and hopefully
identify problems that resonate with the wide audience. So I I'd like to continue to have those case studies.
(44:25):
I'd like to continue to
keep looking at the data visualization
space. I think that's 1 area in Python where there's a lot of good libraries, but it's continually changing.
And putting together some articles to talk about how to maybe build more dash boards and present the information.
I think I'll continue to stick to some of the bread and butter
(44:49):
pandas
type activities where articles where people
maybe don't understand how to use pandas effectively or
those kinds of things. And I'm always interested in
new
libraries
that are out there that solve really challenging problems in an elegant way.
So 1 of the things I wrote about in the past was Facebook's,
(45:12):
profit library for time series forecasting.
And I think that's 1 of those areas that it's really interesting to figure out what the
state of the art is for time series forecasting
for
someone that's not a PhD in this area. And so I think there's a lot of interesting activity there, and I would like to continue on that.
(45:33):
Finally,
you know, outside the blog, but I have
done 1 course with Talk Python
on moving from Excel to Python. And I'd like to continue to build out some coursework there to
target users that are looking to start that transition into Python
from a business setting.
Are there any other aspects of your use of Python or your work on the blog or
(45:59):
the overall
use case of Python for business problems that we didn't discuss yet that you'd like to cover before we close out the show? I guess I'll follow-up on that last topic. You know, I am excited to see how
some of these more advanced data science,
machine learning, artificial intelligence,
whatever kind of category you wanna put them into,
(46:20):
how they continue to get streamlined
and
automated so that it's easier for people to apply it to their own problems. I think that's gonna be an exciting space where Python has a lot of good
capabilities now, but I think they're gonna be even more in the future that we can start to
bring down and make it easier for business users to take some of these more complex
(46:44):
approaches to solve the problems they have on a daily basis.
Well, for anybody who wants to follow along with you and get in touch, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us into the picks. And this week, I'm going to choose a couple of newsletters that I tend to read fairly regularly because they're
fairly concise and have a lot of good value and some good analysis on the different topics covered. So 1 of them is the data science roundup by Tristan Handy from Fishtown Analytics, who are the people behind the DBT project.
(47:15):
And the other 1 is called This Week in Data,
and that 1 is by somebody who's working in data science and data engineering at, I believe, DoorDash
or Shopify, something like that. So definitely useful newsletters to keep up to date with some of the things that are happening in the overall data science ecosystem and data engineering space. And so with that, I'll pass it to you, Chris. Do you have any picks this week? I'm gonna go a completely
(47:40):
nontechnical
direction. So I'm a bit of a sucker for
British
crime procedural
dramas, always looking for new shows to stream, and 1 of the ones I've been watching is called Line of Duty.
I don't know, 3 or 5 episodes of BBC
series on Amazon Prime, which I think is pretty well done. And then I'm also reading a
(48:04):
science fiction book called Out of the Dark
by David Weber, which is pretty interesting so far. I'm about halfway through. So I thought I'd
put something out there that's maybe a little bit lighter as we hopefully
get out of the quarantine time and people are
still trying to figure out how to entertain themselves at home. Well, thank you very much for taking the time today to join me and discuss the work that you've been doing on the blog and some of the ways that you use Python in your own work. It's definitely interesting to go outside of the particular problem areas that I spend most of my time in and understand some of the other ways that the language is used. So I appreciate all the time and energy you've put into that, and I hope you enjoy the rest of your day. Thank you. You
(48:46):
too.
Thank you for listening. Don't forget to check out our other show, the data engineering podcast at dataengineeringpodcast.com
for the latest on modern data management.
And visit the site of python podcast dotcom to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email host@podcastinit.com
(49:09):
with your story.
To help other people find the show, please leave a review on Itunes and tell your friends and coworkers.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.