January 11, 2021 • 57 mins
Summary
Every software project has a certain amount of boilerplate to handle things like linting rules, test configuration, and packaging. Rather than recreate everything manually every time you start a new project you can use a utility to generate all of the necessary scaffolding from a template. This allows you to extract best practices and team standards into a reusable project that will save you time. The Copier project is one such utility that goes above and beyond the bare minimum by supporting project evolution, letting you bring in the changes to the source template after you already have a project that you have dedicated significant work on. In this episode Jairo Llopis explains how the Copier project works under the hood and the advanced capabilities that it provides, including managing the full lifecycle of a project, composing together multiple project templates, and how you can start using it for your own work today.
Announcements
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- Your host as usual is Tobias Macey and today I’m interviewing Jairo Llopis about Copier, a library for managing project templates
Interview
- Introductions
- How did you get introduced to Python?
- Can you start by describing what the Copier project is?
- How did you get involved in the project?
- Can you share some of the history of the project?
- What do you see as the most common uses for a project templating tool?
- There are a variety of different tools for scaffolding projects across a wide range of languages. What are the distinguishing features of Copier that might lead someone to choose it over the alternatives?
- Can you describe how the Copier project is implemented?
- How has the design and feature set evolved over time?
- What is the workflow for someone building a template with Copier?
- What are some of the edge cases or complexities that they might run into?
- What are the options for extensibility or integration with Copier?
- What are some of the capabilities or use cases for Copier that are often overlooked?
- What are some of the most interesting, innovative, or unexpected ways that you have seen Copier used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on and with Copier?
- When is Copier the wrong choice?
- What do you have planned for the future of the project?
Keep In Touch
Picks
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Unknown (00:13):
Hello, and welcome to Podcast Dot in It, the podcast about Python and the people who make it great.
When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So take a look at our friends over at Linode.
With the launch of their managed Kubernetes platform, it's easy to get started with the next generation of deployment and scaling powered by the battle tested Linode platform,
(00:34):
including simple pricing, node balancers,
40 gigabit networking,
dedicated CPU and GPU instances, and worldwide data centers.
Go to python podcast.com/linode,
that's l I n o d e, today and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show.
(00:56):
Your host as usual is Tobias Macy. And today, I'm interviewing Jairo Yopis about Copier, a library for managing project templates. So Jairo, can you start by introducing yourself?
Hello, everybody. I'm Jairo Jopis.
I learned programming
like 10 to 13 years ago,
just because I needed
something
(01:17):
back there in where I was working. Started with Visual Basic, PHP, JavaScript, and then Python.
Right now I am full stack
developer,
at Technitiva since
2016, and also the security
infrastructure and developer tools
responsible.
And do you remember how you first got introduced to Python?
(01:39):
Not exactly. I think it must have been like 10 years ago or or something like that.
I landed by myself
just because I was
hitting
some maintainability problems in those PHP
projects I had, and I wanted something better.
And I dive into Python 3 dot net,
(02:01):
which is a nice website for learning Python. I liked it. I started
working with it, and
suddenly my other languages became more Pythonic, so I knew this was the language.
And, well, later,
I knew Odoo, which is
ERP or framework for ERP,
(02:22):
something like that,
where it has a lot of potential and started working on it. And right now, it's my main,
business,
my daily work.
And so somewhere along the line, you ended up getting involved with the Copier project. So can you start by just giving a bit of an overview about what that project is and how you got involved with it?
(02:44):
Copier is
this kind of tools are usually called template managers
or scaffolding manager or generator.
There's some tools similar.
However, since version 3,
which is the version I helped develop,
I would say it's more a project life cycle management tool,
(03:07):
which is somehow unique.
And I got involved,
because, well, I told you before I am working with Odoo, but Huddl has a very steep
learning curve.
To develop it, you need a lot of packages, not just
from Python, but from many things you cannot pip install, although
(03:28):
it's way more than that.
And its customer needs,
specific flavor of Hadoop, so it's very difficult.
I had to develop
1 system, which is called Dutba.
It's highly opinionated
image ready to put Odoo inside it, but without Odoo.
So
(03:49):
it needs,
templates
to
configure
how do you want to build Odo for this project.
Well, I needed
some templates tool,
but when I started this project, I mean, Tudo was the image.
It had also Duda Basket Folding, which was the template.
(04:11):
This was
2017.
There were no
project life cycle management tools available.
There were
project template tools, but this was not enough for our case.
So I just started with a git repo and merge, git merge and manual patch. And
fast forward to
2019,
(04:32):
git merge is just not enough.
All the template managers says are not enough, so
they all needed something better.
Copier was not enough. So
but it was the closest 1.
So by the end of 2019,
I started doing my first pull request to Copier, and then later I became a co maintainer.
(04:56):
The full life cycle management capabilities of Copier are definitely what I find interesting. And you mentioned that when you first came to it, it didn't quite have everything that you were looking for. So I'm wondering if you can give a bit of the background of
where the project came from and the state that it was in when you first found it and what you saw as the promise
to enable you to take it to where you needed it to be?
(05:18):
Well, I have to ask the other man trainer.
He is called Ben Feller, and
he is very nice.
And I have to ask give history to get some reserve to project because I'm new in it somehow.
This project was born in
2011
by Juan Pablo Scalletti,
who I never met. The project was called Voodoo
(05:40):
back then.
It seemed like it had a joke of resurrecting project skeletons.
Then in 2018,
we can see the first comment from Ben.
And last summer,
it was
Juan Pablo left the party and Ben Feller was the new maintainer. And then,
(06:01):
so do by the end of last year, I started contributing. And
in January, I became a command center.
This contributing ID since late 2019
to
around
April 1st,
sponsored by Technitivo.
But these days, Ben is very busy, and I'm mostly maintaining
(06:21):
it on my own in my free time.
Basically the state of the project was a simple project.
I like that.
Risk following managers,
well,
these tools,
there are not many of them, at least not many serious
ones.
The other ones, all of them
had very complex
(06:43):
mindsets
or workflows
or ways to deal with the template itself.
So this was a little simple project.
You were able to make a template very easily,
and I like that. I like simple things.
So I thought that if I had to invest a lot of time improving 1 tool, I wanted to improve a tool that was simple,
(07:08):
not something to be too complex
that was gonna be very hard to prove.
Back then, before I started, contributing these new features,
the thing I liked most about Copier was how simple it was.
I saw the the the source code of the project was simple, but also
(07:30):
the template
you produced
with it
were simple also.
In terms of the actual
uses for copier
and other project templating tools, I know that there are a number of different ways that people have employed them. You mentioned the need for
being able to set up an Odoo environment and all the different packages and configuration necessary to go along with that.
(07:55):
There are projects for being able to set out the skeleton of a Python repository.
Ruby on Rails has its own built in scaffolding engine to bootstrap a project. Django has 1. I'm wondering if you can just talk through some of the
common use cases that you've seen and that you've used these templating tools for, and then maybe talk a bit about how Copier compares to the overall ecosystem of the available options.
(08:21):
We are programmers,
and we love
to, to keep our code dry. What we call dry, don't repeat yourself.
That's very nice, but sometimes you just cannot
do that.
Recently, I started 1 very, very little library that I needed for specific need. It's called Versort.
(08:42):
It's for sorting versions.
It's very small. It was like an exercise
to see how much
these
undryable things
are in a project. This project is 144
lines of Python code,
plus
111
lines of Pytest code. So it's very, very small,
(09:05):
but the total amount of lines
it's
678,
almost 700 lines
for
around 150
lines of real code.
Where are those lines from?
It's dot GitHub slash worksflows
slash,
I don't know what the yaml.
(09:27):
It's the license. It's precomicconfid.jamal.
It's Travis CI or GitLab CI or pritzhererc.jammerorreadthedocs.jammerormypy.inioypides.iniomkdocs.
That's yml. Editorconfinflak8.
Configuration
for Versus
Code. Well,
(09:48):
every new IDE,
every new automation
tool that comes to the market
says the same to you. Hey. You just have to add 1 dot 5 to your project.
But finally, you end up with a 1, 000 of dot files,
and you really need them because those are great tools
that save time for you. But how do you manage all those configurations for your projects?
(10:14):
Well, this is a very specific aspect where you happen to need a tool
that handles all of these
repetitive things for you. So when you cannot keep your code dry, you need a drying machine and this is copier.
You also asked about how does it compare
(10:34):
with other tools? I will say that there are 2 major alternatives to Copier, at least otherwise the analysis I made before
investing time on developing
it. 1 of them
is cookiecutter,
which is also written in Python, very famous in the Python world.
(10:54):
And the other 1 is Yeoman.
Both of them are
very good at what they
do, but these are just templating tools.
They only serve to bootstrap a project and many others you also mentioned because I also
looked at many other
(11:15):
alternatives, although these 2 are the biggest ones. All of them have the same problem. All you do is bootstrap the project.
So, okay, I bootstrap my project. I create these, let's say, pre commit config dot yml file.
Okay. But then
after some time, that configuration
also evolves.
(11:36):
Maybe
some CI tool requires
new
syntax,
maybe
some tool you are using with Precommit
gets a new release with a bug fix you need and all of that. So you need to manage all of that.
I'm talking just about this use case, but any other use case also
requires
(11:58):
maintenance.
Well, this is what I call day 2 operations.
It's cool to start a project. Yeah.
Everybody can do that. You can even just copy a folder you have.
But
what happens the next day, and the next week, and the next year? And what happens if you require that your framework
(12:19):
has a specific
folder structure,
and you want to provide a template to your community, which was our case, we used to do the project.
This template needs to evolve with the project.
So what do you do? How can you help people evolve their projects?
This is a key feature of Copier,
(12:43):
that as far as I know, no other templating tool has.
And Copier
relies on Git
and uses, SmartDiff
algorithm
that compares
the template evolution, the template git evolution,
with
your project evolution.
(13:05):
And then somehow tries to apply diff
just like git would do
if you
have
both stories in your git project, which is not the case.
You just have to separate completely different stories because,
yes, you you you use the bootstrapping,
but then your project evolves
(13:25):
as it want it wants. You can even change some answers to some questions.
And then this SmartDiff algorithm
that I developed,
handles all of this
and lets you actually evolve your projects.
This is, I think, the best feature of Copier, but it has another 1 that also I didn't find
(13:50):
in any of the other tools,
which is the ability to apply several
templates
to a same project.
So let's say that you develop
several languages.
For instance, you develop,
let's say, JavaScript projects
and Python projects.
If you develop
(14:10):
JavaScript stuff, you're gonna need the NPM
configurations,
JavaScript configurations. I'm not a JavaScript developer, so I might be failing
in something
I say.
Well, you need your kind of tools. See, if you develop Rust, it's the cargo stuff.
And if you develop Python, you might need to set up the py or you might prefer a pyproject.
(14:37):
Yml or a PPM, whatever.
So you have separate
scenarios
depending on the language you're using,
but
many of them
use the same tools.
Let's say you use all of these to produce
Docker images.
Okay. Then the Dockerfile is is common for everyone, or maybe the Docker ignore a file,
(15:02):
or maybe the
GitHub workflows,
things like pre commit config
that make your code pretty no matter what kind of code are you writing.
So
there's a split of needs. You may need to apply the predefined tools,
but
also the Python specific tools in a project.
(15:24):
And in a different project, you need to apply the predefined tools and the CI
specific
files also, or maybe the JavaScript ones.
So Copier lets you
apply several templates to our same project,
and this combines with a previous point. So both templates
can be applied simultaneously to the same project and make that project evolve
(15:46):
also.
So these are 2 killer features. I think Copier is,
you might like more
something
of the other projects, but these 2 features are essential
distinguishing.
And there are a couple more points.
1 of them is that you don't have to write JSON, which I think it's a big feature
(16:08):
as compared with the other tools.
I hate writing JSON. Don't know what do you think, but I hate
it. You you write Yamo on the other hand in case you like it.
The other feature is that for publishing a template,
there are some
tools
such as Yeoman
or also I think it was something like mister Bob templates for Python,
(16:32):
where you need to produce
an actual package
to make a template.
Because for instance, Yeoman is not a template manager, it's more a template framework,
which helps you build a note project,
which is your template.
But
if I'm developing Python
(16:53):
and I'm a Python developer,
and I don't know how to do a a NPM project, or I don't like JavaScript.
Why do I have to learn JavaScript,
to learn NPM, to publish my templates
as an NPM package
just to help people
(17:14):
deal with Python stuff. It makes no sense.
But on the other hand,
I like Python. I know how pip works, but also maybe somebody comes from Node and wants to use Copier and they're in the opposite situation.
So Copier,
if you want to publish a template,
you can just use a git repo,
(17:35):
and it's just enough.
I have a blog post in my blog where I explain in the in-depth
why Copier think it's way better than our alternatives.
The specific thing that intrigued me about Copier and maybe wanna reach out and learn more about it is the ability for being able to apply changes to the underlying template to projects that were generated with it, which
(18:01):
most tools such as cookiecutter or Yeoman, as you said, you can generate a baseline scaffold.
But then as you work on other projects that use some of that same structure, you find improvements or enhancements, and then you bring that back into
the project template that you're working with. But there's no way to then reflect that back out to all the other projects that it was created with. And so you have to try and figure out doing it manually in a piecemeal fashion.
(18:26):
And so you end up with a lot of drift in terms of the best practices that you've applied. And so your original project doesn't get to benefit from the learnings that you found in project number 5 or whatever. And 1 of the things that I didn't realize in what you were just saying is the ability to combine multiple templates.
You can actually
rather than having to have 1 monolithic template that covers all the concerns that you need for a project or have
(18:51):
a bunch of different conditionals that it will generate project skeletons for based on the inputs that you provide.
The input prompts, you can instead have a modular approach where you have a template that says, this is going to handle all of my
tooling for my configuration. So it will put in my pyproject.toml.
It'll configure, flakate. It'll set up my py. And then this other 1 will set up the baseline for the Django app structure. And then this other thing will handle setting up all the configuration for my CICD, and then I can apply all those together,
(19:24):
or only 1 or 2 of them, rather than having to have everything in 1 project. I think that's a pretty amazing capability as well.
I think these are very
important features for the project.
I mean,
the ones that make you choose Copier over other projects.
It has others. For instance, even if even your templates can share
(19:45):
code among themselves. So you maybe have a question, Adi.
When I say questionnaire, I mean, you ask questions to the user so you can produce the sub project you want. Okay?
So if you have a questionnaire,
you can actually combine
several questionnaires in 1.
(20:06):
So if you have several templates,
you can include them
just skids modules and combine
in the main template
several parts.
This is something that is not perfect yet, but
it also helps. So it's very good, I just say.
And so digging more into the project itself, I'm wondering if you can talk through the actual
(20:32):
copier project and how it's implemented
and how the overall design and feature set has evolved over the past year that you've been working on it.
When I since I've been working in it, I think these 2 key features
I told you were new in version 3. When I came to Copier, it was versions
2
dot
(20:53):
something. So
it's these new features is something that I am involved with, so I'm happy with it.
But the project itself, the settlement
of the project as it was when I came in
and how it is now.
It is,
API
and CLI
(21:13):
project.
So main use for copier is CLI.
You use it in your terminal.
It should be comfortable
there.
But you can actually use it as a library if you need to expand anything.
It's written for Python
3.6
or later.
Recently, it supports 3.9 also in the latest version.
(21:37):
It uses,
poetry to manage the project, which is a pleasure. I love poetry.
And
also
it's completely
typed.
Well, with static typing, it uses my pie to check that.
If you plan to use Copier as an API tool, as a library,
(21:59):
let me tell you that please use just the copy
function in the main
module because
everything else is gonna change. We'll talk about that later.
So just
use copy, which is the makes most of the work.
Something that is very, very nice about copier
(22:20):
is
how it has a very big test suite
that helps us develop safely.
And I think that the most interesting part is how do you actually
make a template?
Because
most people, I think, don't really
care about how is the tool written, but they care more about how do I write my templates,
(22:45):
which is what affects them more
directly. Right?
So
you don't need to add any logic to your templates. You can just use copier
to copy a folder.
But if you need to add logic, you have Jinja too. You don't need to add any configuration to your templates.
But if you need it to just add a copy of the the demo
(23:08):
file, which also
can
can evolve from simple to complex to multiple
includes
easily. So
it's
as complex as you need.
You don't need your template to be a git repo, although
if you use it, it's better, and you have
(23:30):
other features such as updates.
You don't need to support updates in your project. So it's something that the template
explicitly supports.
And if you don't support updates, then CopyEdge has becomes scaffolding tool, just like the
other tools we also talked about.
But if you need those updates,
(23:51):
all you need is to
allow your template to write, copy your answers file, which tracks last answers from the project.
You need to
make it git repo, and Copier will use Git tags to evolve your project.
You also don't need to program
anything,
but if you need to program something for your template,
(24:13):
Copier uses super processes
where needed.
So if you're a template's
writer,
you don't need to know Python.
You can just use any binary you want. Copier will communicate with the binary for some tasks
just using the arguments you say to hit to tweet
and some environment variables. So it's very easy to block with
(24:38):
any tooling you want.
And I said it before also, you don't need to publish a package. If you want to publish a template, you can just publish a git repo, and you're done.
To your point of being able to call out to any arbitrary
command as part of the template generation
plays into the fact of it being language agnostic. And you called out earlier the issue of
(25:02):
if you like Yeoman, for instance, but you're not a JavaScript developer, you still have to work with the NPM tool chain and things like that. I'm I'm wondering if you've looked at any
tooling around Copier to be able to package it up as a single executable so that somebody
who's interested in using its capabilities but isn't a Python native can still be able to benefit from it, and if you have any of the challenges or issues that you've come across in that process.
(25:32):
Yes. Actually, that's the main reason why this is done like that. Well,
that's 1 reason. The other 1 is that the project have this feature already when I came.
So I like that. And since I had been
checking Yeoman, I just had, for example,
I compared that and I said, okay. I like this. I like that Copier is a templating tool for the world, not just for the Python world. It's much better.
(25:59):
So if I write Rust
and I want to let
my template users
just execute
tasks against the template
with a Rust library,
or if it is written in Go, or if it's JavaScript or whatever.
I don't want these people to care about
(26:20):
anything else than just pip install
well, better, pipx install copier.
And that's it. If they do that, they
know just copier docs. I mean, there's something they will have to know, which is Jinja.
But I think it's a very simple templating language, very comfortable. So that's
(26:42):
well, at the end, you have to to have some template language.
But apart from that, you don't need to write a single Python line to make your project your template
be able to use all Copier features.
And so digging a bit more into the process of building templates for Copier,
I'm curious if you have gone down the path of taking an existing template for something such as cookie cutter and converting it into something that is compatible with Copier or any of the other frameworks. And what are some of the design considerations
(27:16):
for template authors to be able to build something that will work well with Copier and all of its capabilities?
My use case
was a very complex 1.
I mean, my use case was
so complex that I could not use
cookiecutter
or human.
So
1 good thing about how Copier has developed
(27:38):
is that I used it. I mean, I don't work gain
too much money with Copier. I just
work with other things and I needed something that
did this work. So
if you take a look at a repo that is called Dutba copier template,
you can see it's a very, very complex template
(28:00):
and it helps to make copy of what it is. But
after
Copier 3 was released,
some people started liking it, started using it and started migrating cookie cutter templates to copier templates.
And I think this is
bit of a challenge because
(28:21):
copier is a new mindset.
You cannot have a template which is compatible with both.
You can have a template which is similar,
but it's not compatible,
a 100%.
So you might have to read carefully the documentation to make your templates
match copiers.
1 differentiation
(28:41):
is, for instance, in copier, you don't need that your template is a subfolder
of your project.
The project
and the template can be the same folder.
In Qutr, for instance,
you have a main folder,
which contains your readme, your license, your whatever, and then a folder inside that repo
(29:05):
that contains the actual template.
But in Copier, it doesn't
necessarily work like that.
Actually, it didn't support that, but later we added support because somebody
was asking for it because he came from cookie cutter.
But you don't actually need that because in copier,
(29:26):
for a file to be considered a template, a Jinja template,
is masked and in a specific suffix,
which by default is dot
t m p l.
So let's say your project contains
readme.md
and then a readme.md.tmpl.
(29:48):
Copier will ignore the first 1 and just take the second 1.
So if you want to have a readme for your template,
but also a different readme
generated from your template,
you can just have these 2 files together,
then you don't actually need to have your template inside a subfolder.
So it's just a matter of taste. How do you prefer to organize your your things?
(30:12):
But it's a mindset change that if you come from cookie cutter, you you need to do. Last week, I think it was, we had an issue.
Somebody was asking us to support having several templates
in the Sangit repo,
but this is not possible for us
because Copier uses Git tags
(30:32):
to evolve your project.
So if we were to do that, you would have several templates,
which will
evolve
separately,
but actually
all would be versioned with the same version.
You would have
some confusing
things. Things like,
(30:53):
template evolving,
having a new release, but with no change.
And so
1 Git repo
will be 1 template,
and this is by design.
So we'll be like that forever, just because
we need to support updates. And and because we designed the project with a specific
goal,
it requires a specific template.
(31:15):
In terms of people who are using Copier, what are the options for extensibility
or integration with it? And
particularly, you mentioned the Python API,
just some of the interesting capabilities or features that that exposes.
I think it's a matter of of comfort
because
let's say,
(31:36):
for instance,
I use
1 CI,
internal CI,
which applies
to our company projects,
the public template we have,
plus another
private template that we also have for this kind of projects. This is 1 example of how multiple templates per project help
(32:00):
because, for instance, the public templates
will help everybody to get up and running with
quite easy. But the private 1,
makes our specific
customizations,
our internal
CICD system work,
and other would for nobody else. So it's a thing we maintain
(32:21):
internally,
and every project has both of them. So 1 CI system
just
every night checks for a new release
in both templates.
And if there's a new release,
then it updates the project.
It runs the checks.
And
if there is any conflict or any problem, it will
(32:45):
just open a pull request.
But if there's no problem, it will also merge the pull request. And so the next morning you sit
in your desk
and your project is updated because you made a release
from the template.
So
if you just use command line based CI,
(33:06):
I think you will be more comfortable using
Copier CLI
or for your daily usage of Copier, it will be more comfortable also. But if you have a specific
CI tooling that you happen to have written in Python,
I think you will be more comfortable using Copier as a library.
(33:27):
But actually you can make the same thing with both approaches.
And in terms of the actual
uses for Copier, what are some of the most interesting or innovative or unexpected projects that you've seen built with it? Well, to be honest, I have not seen many
because
I don't track copier users.
We have in our readme 1 suggestion.
(33:49):
Please type your copier templates
as copier template.
There's a topic in GitHub
labeled copier template.
So we just expect people to tag their templates,
and then
every now and then
come around and see what they're doing.
I think the more the most interesting thing I have seen there
(34:12):
is there's 1
contributor we have, very active, actually. It's called he is called. Sorry.
He's called
hang on. I'll try to say it. Okay. Timothy Matsukotelli,
and he is the main tenant of MK Doc Strings. So it was a surprise for me to see
him involved in this project that he seems to love Copilot very much. And he has a template
(34:36):
to build poetry based Python projects.
Myself,
I started developing a template, which is actually, it's just a proof of concept.
It's under the copierarch
organization
in GitHub. You can see it. It's called Auto Pretty.
And my goal is to just bundle
(34:56):
all those kind of tools
do you usually need for any kind of project.
So I'm talking about linters,
formatters,
PreCommit,
Prettier, all these kind of tools that are very basic.
I happen to have everywhere.
And it's also a project that is trying
(35:17):
a new concept of a self
templating template.
So I want to be able to have a template
to prettify things,
but to actually be able
to apply these templates to the template itself.
So I can update my dot files for the template
from the very same template.
(35:40):
Well, it's not released yet. It's just there are some alpha, you can try it. And it's not working yet, but it is like a playground I have. And at some point, I will make a better release.
And in terms of your experience of working on and with Copier and using it for your own projects, what are some of the most interesting
(36:01):
challenging lessons that you've learned in the process?
I think I learned a lot of things.
To start, I learned,
lots of tools that I didn't know.
I might seem
a little bit stupid saying this in a pocket, but I never wrote Pytest test, and it's awesome. I
like it.
(36:21):
I started using Poetry. I liked very much. Also, this is the first project where I am deeply involved
that has static typing,
and it's very good. It helps a lot. So
that's something I learned.
Something I also learned is that sometimes you need
(36:41):
to test the template.
At first,
since I was the only 1 developing my template and using it,
nobody else
would use it.
Well, I just
started releasing versions and versions of the template without any problem when anything went wrong.
But later, when this template became a more critical part
(37:04):
of our toolkit
in the workplace,
and all my colleagues
started using it, and the CI system started updating projects
automatically.
Then something I learned is that
if you happen to have this kind of usage also,
which is cool because it's where copier shines actually,
(37:26):
I recommend
to make pre releases of your template
and apply them manually to some specific projects, or maybe if you are several
developers
involved specific template,
Do developers,
little group of people,
try your premises
because it will help. Sometimes
(37:48):
it's hard to predict
how a project will evolve
and how your template will behave.
Because,
well, this SmartDiff algorithm
makes
most of the work for you. But sometimes,
you will try let's say, you add a new file to your template,
and you didn't happen to
(38:09):
expect that that file was existing before
because the user
added it manually to a project
or maybe changed
somehow.
So by having some prereleases,
you can there there's a flag in Copier CLI,
which
is slash slash
pre prerelease
(38:30):
that tells Copier to consider pre releases when when fetching the last the the latest version of the template.
It helps you to test your template before it's released.
Apart from that,
app tests
app tests. You don't usually need to add a test that tests that the template
(38:51):
when the template has a file that is called
whatever and it has a content and you pass that
answer,
copier will render that content. Well, that's obvious.
But
so don't test what copier is doing.
Test why is it doing it. So in your tests,
create a new project. And let's say that your test is testing
(39:16):
that you install PreCallmate
configuration, which includes black. Okay.
Actually, I want to test that. Then in your tests, create
a new template, a new sub project from that template,
and then add a Python file, which is badly formatted.
Try to commit that
and see how pre commit performance
(39:39):
that. So you're seeing
the way
you the way you do things and not the how you do things. And it's important for testing, for your test to be because if it just says that the temp file renders
like this when I pass
these kind of input,
well, it's gonna be true all the time,
but maybe if the input is wrong or the file is wrong, you won't detect that.
(40:02):
For people who have existing projects that they want
to extract some of the structure and best practices from and apply it to other systems, is there any capability for
being able to use copier on a project that already exists to apply some of the boiler plate and overall structural best practices to the project?
(40:23):
Yeah. Copier takes a simple approach
when analyzing your project.
And it has actually 2 commands.
The first command is copy and the second 1 is update. So
when you're
calling copy, it will
just copy everything and forget about what those files contained before.
So let's say your project is already a Git project.
(40:46):
Well, you just
apply the new template, copier, copy, whatever,
and then copier will
do his stuff,
and then you will have to review
the files that copier modified.
Why would you do that if the template supports updates? Because,
like I said before,
(41:07):
it's not mandatory.
It's a template
choice.
If it supports updates,
it will write a file that by default is called
dotcopieranswers.yml,
but you can change that.
Actually, you have to change that if you want to apply several templates. That's how it works.
(41:28):
Let's say you just use 1 template. That's the name of the file. Then it will store
all your answers to the questionnaire,
plus
the URL
of your git repo,
plus the tag that you applied.
So
copier,
once you commit all those changes,
copier will consider
(41:49):
that
the resulting
of copying that template
in that version
with those answers
is your sub project right now.
So it will
later
apply an update.
This algorithm
will compare
how it should have been
(42:10):
and how it comes to be, and then later evolve. But if you just have several projects
and want to apply Copier over them,
Once they already exist,
it's completely support. All you have to do is just copier copy
the first time,
and then
later, just copier updates, as usual.
(42:33):
That first time, you will have to review not only what copier is doing, but there's no other choice.
And for people who are looking for a project templating tool or something to help manage the overall life cycle of their projects, What are the cases where a copier is the wrong choice?
I hope it's the good choice.
(42:53):
That's why I like it. But when it is the bad choice?
I said that we try to dry things
to don't repeat
yourself in your code. And
when you cannot
dry, then you have a drying machine. Okay. But if you can dry,
just dry your code. You don't use copier.
(43:15):
Better dry it. If some parts of your codes can be externalized in a library, then do that library. Don't just copy that everywhere.
And I actually do that
because it's faster and more and easier. But I have plans to me, right, to actually extend the library for this,
copier template project I have.
(43:36):
So I'm telling you not to do that, but it works.
So so maybe it's also not a bad choice.
As you continue working on Copier and continue using it, what do you have planned for the near to medium future of the project?
Copier 6 is gonna be a huge release.
Really, really huge. Until now,
(43:57):
let's say copier 12 was,
simple, fun
scaffolding project, just like any other, but easier.
Copy of 345
has been
the best templating engine in the world.
No. Not the best, but, you know, it has these killer features.
(44:18):
But Copier 6 is gonna be a complete refactor of Copier.
I am confident about it because we have very, very big
test
coverage.
So I'm refactoring it because Copier was built around
functions,
but functions were too dependent
(44:39):
among themselves. And at the end of the day, it became
hard to
detect
when is something happening, when not, or to have a relationships
among the whole
copying process.
So
it became hard to maintain in that regard, and I'm making a bigger factor to do it object oriented,
(45:04):
which is gonna be way easier to maintain in the future.
Right now, there are some alpha releases of Copra 6, but are very, very far from what Copier 6 is gonna
be. But if you use them, you will help me because it will detect bugs. So
do it, please.
And you will have
(45:25):
conditional questions. That's something that it's already in the last ask for.
Also, a more interactive
terminal user
interface,
which for instance, also supports multiline input.
That was not easy, but
it's working now. It has some bugs, but, well, it's
(45:47):
more things
that are gonna come in Copier 6 are
multiple answers for the same question.
That's something asked asked by the community,
And, also,
the self templating
support
to have a template that applies to itself, like I said before.
And I will take a chance to remove some options that
(46:11):
make no more sense that,
those kind of things you add,
but don't really have much sense. And 1 thing I would like to do also is to change Jinja defaults
because right now, Jinja is,
by default,
different of the official Jinja. For instance, it uses
brackets
(46:31):
instead of curly braces.
So I think that if there's somebody who comes to Copier for the first time,
and then he has to go to the Jinja website and for the first time
learn Jinja.
And maybe somebody who doesn't even
write Python.
Well, it can be challenging if apart from all of that,
(46:52):
also
Copier uses different placeholders
than official Jinja.
So possibly there's gonna be some changes there. Then maybe Jira templates will not work with Copier 6 if they worked with Copier 5. So another things that I will have to add is a version tech
(47:12):
that fails if your template is not ready for
Copier 6. I have to think about that a little bit more.
But right now, I am mostly involved in the refactor.
And once the refactor is settled,
then I will start adding these other things, which are gonna be very much easier
after the source code is reorganized.
(47:36):
And after copy of 6, then I think
there's gonna be not
some main more things to do. The project is very, very nice.
Just more
evolving of the tool, but it's gonna be more more or less what it should be.
Are there any other aspects of the copier project or the overall topic of project templating and scaffolding and life cycle management
(48:02):
that we didn't cover yet that you'd like to discuss before we close out the show?
There is 1 complex
case
in Copier,
which is answer migration.
It happened to me in my template
when I released, I think it was version 2.
You can answer to copier
in YAML format.
So an answer to a copier question can be anything
(48:25):
that YAML or JSON supports.
So you can
put a dict, an array,
a boolean, or whatever,
combination of them as an answer.
So let's say that you expect a specific format,
and then later
it needs to change.
It happened to me because the format I was asking
(48:49):
was not smart enough
for, different well, actually, if you want to include it,
it was because of traffic, which is an amazing inverse proxy,
which
for traffic
1
was
much simpler
to use and to set up. But for traffic 2, it's much more complex,
(49:12):
but actually also very much more powerful.
So the structure
we needed in our project to include the traffic configurations
was different because we started supporting traffic too. And then,
for instance, things like
supporting several
host names for a project
(49:32):
was not possible before. Now it was possible
that structure didn't match. So
Copier includes a migrations
feature
that
allows you to even
change
the answers
before migrating.
And this is very handy because
if you had your project in template version
(49:56):
1,
and then for template version 2, some question needs to change its structure.
You can automate that.
And then your
copier project, when updating to version 2, it's gonna take the answers from version 1. It's gonna convert them to the new format,
and then
use them for your version 2 template. Well, it's not copier who's gonna make it. It's you because you are the 1 who defined that structure.
(50:22):
But you have to program that in your template,
And that's,
a bit of a pain points, so
I advise to
to try to keep your questions
simple if you can.
For instance,
if you want
to ask the user, what systems do you want to use? Please input
(50:42):
a YAML list.
Maybe it's simpler for you to just say
2 booleans.
Do you want to use system a? Yes. No. Do you want to use system b? Yes. No.
So it's easier to evolve that kind of questions,
but it's not always possible. So commits across migrations also.
And another thing that I recommend
(51:04):
is that, well, when copier
updates
automatically
a project,
it can detect conflicts just like a git merge.
But copier will leave a dot reg file,
which includes the conflicts.
So if you trust Copier too much,
(51:25):
maybe
you can
say,
okay. Copy of that worked. Then the project is okay. But if there's a reg file,
possibly it's not okay. That's something you have to review by yourself. That's something that the Smart diff algorithm
was not able to solve.
So there's Precommit
(51:45):
hook,
which I think it's in the docs
copier,
as a recommendation to forbid your
git project to commit that reg files. Because
that way you make sure you are not committing
a failed
update. It's important.
And also a little problem that many users face
(52:09):
is that they expect Copier to work
a way that it doesn't work. I mean,
when these users update project,
they copy your
answers file. And then after it's been edited,
they commit that
and then they run copier update force, copier force update. Sorry. Then they're tricking copier. They're telling copier that
(52:32):
those answers
produce that project, but it wasn't true. The answer that's produced that project were the previous ones. So maybe copier
will succeed,
could happen,
but possibly it will fail also because you're tweaking this SmartBeef algorithm. So if you need to change,
an answer,
use
(52:53):
copier to do it. Do not edit copier answers directly.
This is not documented in the stable documentation. It's documented in the latest branch of the documentation.
So maybe you will not find this advice
yet, but just don't do it.
Just say you can pass the new answer to copier by the data flag in the CLI,
(53:19):
or you can just
answer the questionnaire again. The the last answer is really there, so you just have to change the 1 that changed.
But don't do it like that because
it will fail at some point and you will think
that it's a bug and it's not.
So
just let's copy or edit that file.
(53:41):
Well, for anybody who wants to get in touch with you and follow along with the work that you're doing or contribute to the work that you're doing in copier, I'll have you add your preferred contact information to the show notes.
And so with that, I'll move us into the picks. And this week, I'm going to choose having a good quality set of playing cards on hand because
it's a versatile
tool. You can use it for playing all kinds of different games, either by yourself or with other people. So definitely worth having 1 that is
(54:07):
good quality and can stand up to a lot of use rather than just something that's cheap because then they're not as fun to use and you're less likely to pick them up. So definitely recommend just having some good playing cards around. And so with that, I'll pass it to you, Jairo. Do you have any picks this week?
Well, I have some.
1 thing I tried recently was Mozilla Hubs. I don't know if you have tried it,
(54:28):
but it's very nice.
Mozilla Hubs
is an alternative to a video call. It's not a game. It seems like a game, but it's not a game. It's a video conferencing tool in virtual reality,
which helps
you stay in a three-dimensional
area
where you can hang out with your friends or workmates.
(54:52):
And it's very fun
because
it avoids
the Zoom burnout,
the video conference
burnout that your brain experiences. It's something
interesting also
could be an epic.
These days,
most people need to
stay away from their friends,
(55:13):
but using this tool, you can feel like you are in the same room. And
1 thing that seems stupid is like copier, has little features that make your life different.
And this little feature of,
hey, I'm with 10 friends, but I want to talk to 2 of them, then we can go to a corner of the room and talk. And
(55:39):
you don't have this
chicken house
noise
when everybody is speaking in the same video call. It's very nice.
Alright. Well, thank you very much for taking the time today to join me and discuss the work that you've been doing with Copier. It's definitely a very interesting project and a
well designed tool for being able to help manage the full life cycle of projects and the boilerplate occur that spans across multiple different projects. So thank you for all the time and effort you've put into that, and I hope you have a good rest of your day.
(56:09):
I just want to say thank you to Juan Pablo Escaletti for starting the project, to Ben Felder
for maintaining it during last year and also
this year and making me a comment in it just because he liked my work. So thank you.
Also, thank you to Technitiva
that has sponsored
the work
(56:29):
of copier
from version 3 to version
5.
But latest version, latest stable version
because
they just needed a good tool that didn't exist, and they decided to let me invest time on it.
Although from now on, it's
just
(56:50):
basic issue management, but I have to develop that in my free time right now. And also
thank you to Timo Seema and Mazzuccotelli
because he helps a lot, and he provided for us the basic skeleton for building
our documentation,
which looks really cool and it's all thanks to him. So thank you.
(57:14):
Thank you for listening. Don't forget to check out our other show, the Data Engineering Podcast at dataengineeringpodcast.com
for the latest on modern data management.
And visit the site at pythonpodcast.com
to subscribe to the show, sign up for the mailing list, and read the show
notes. And if you've learned something or tried out a project from the show, then tell us about it. Email host at podcastinit.com
(57:36):
with your story.
To help other people find the show, please leave a review on Itunes and tell your friends and coworkers.
Hello, and welcome to Podcast Dot in It, the podcast about Python and the people who make it great.
When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So take a look at our friends over at Linode.
With the launch of their managed Kubernetes platform, it's easy to get started with the next generation of deployment and scaling powered by the battle tested Linode platform,
(00:34):
including simple pricing, node balancers,
40 gigabit networking,
dedicated CPU and GPU instances, and worldwide data centers.
Go to python podcast.com/linode,
that's l I n o d e, today and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show.
(00:56):
Your host as usual is Tobias Macy. And today, I'm interviewing Jairo Yopis about Copier, a library for managing project templates. So Jairo, can you start by introducing yourself?
Hello, everybody. I'm Jairo Jopis.
I learned programming
like 10 to 13 years ago,
just because I needed
something
(01:17):
back there in where I was working. Started with Visual Basic, PHP, JavaScript, and then Python.
Right now I am full stack
developer,
at Technitiva since
2016, and also the security
infrastructure and developer tools
responsible.
And do you remember how you first got introduced to Python?
(01:39):
Not exactly. I think it must have been like 10 years ago or or something like that.
I landed by myself
just because I was
hitting
some maintainability problems in those PHP
projects I had, and I wanted something better.
And I dive into Python 3 dot net,
(02:01):
which is a nice website for learning Python. I liked it. I started
working with it, and
suddenly my other languages became more Pythonic, so I knew this was the language.
And, well, later,
I knew Odoo, which is
ERP or framework for ERP,
(02:22):
something like that,
where it has a lot of potential and started working on it. And right now, it's my main,
business,
my daily work.
And so somewhere along the line, you ended up getting involved with the Copier project. So can you start by just giving a bit of an overview about what that project is and how you got involved with it?
(02:44):
Copier is
this kind of tools are usually called template managers
or scaffolding manager or generator.
There's some tools similar.
However, since version 3,
which is the version I helped develop,
I would say it's more a project life cycle management tool,
(03:07):
which is somehow unique.
And I got involved,
because, well, I told you before I am working with Odoo, but Huddl has a very steep
learning curve.
To develop it, you need a lot of packages, not just
from Python, but from many things you cannot pip install, although
(03:28):
it's way more than that.
And its customer needs,
specific flavor of Hadoop, so it's very difficult.
I had to develop
1 system, which is called Dutba.
It's highly opinionated
image ready to put Odoo inside it, but without Odoo.
So
(03:49):
it needs,
templates
to
configure
how do you want to build Odo for this project.
Well, I needed
some templates tool,
but when I started this project, I mean, Tudo was the image.
It had also Duda Basket Folding, which was the template.
(04:11):
This was
2017.
There were no
project life cycle management tools available.
There were
project template tools, but this was not enough for our case.
So I just started with a git repo and merge, git merge and manual patch. And
fast forward to
2019,
(04:32):
git merge is just not enough.
All the template managers says are not enough, so
they all needed something better.
Copier was not enough. So
but it was the closest 1.
So by the end of 2019,
I started doing my first pull request to Copier, and then later I became a co maintainer.
(04:56):
The full life cycle management capabilities of Copier are definitely what I find interesting. And you mentioned that when you first came to it, it didn't quite have everything that you were looking for. So I'm wondering if you can give a bit of the background of
where the project came from and the state that it was in when you first found it and what you saw as the promise
to enable you to take it to where you needed it to be?
(05:18):
Well, I have to ask the other man trainer.
He is called Ben Feller, and
he is very nice.
And I have to ask give history to get some reserve to project because I'm new in it somehow.
This project was born in
2011
by Juan Pablo Scalletti,
who I never met. The project was called Voodoo
(05:40):
back then.
It seemed like it had a joke of resurrecting project skeletons.
Then in 2018,
we can see the first comment from Ben.
And last summer,
it was
Juan Pablo left the party and Ben Feller was the new maintainer. And then,
(06:01):
so do by the end of last year, I started contributing. And
in January, I became a command center.
This contributing ID since late 2019
to
around
April 1st,
sponsored by Technitivo.
But these days, Ben is very busy, and I'm mostly maintaining
(06:21):
it on my own in my free time.
Basically the state of the project was a simple project.
I like that.
Risk following managers,
well,
these tools,
there are not many of them, at least not many serious
ones.
The other ones, all of them
had very complex
(06:43):
mindsets
or workflows
or ways to deal with the template itself.
So this was a little simple project.
You were able to make a template very easily,
and I like that. I like simple things.
So I thought that if I had to invest a lot of time improving 1 tool, I wanted to improve a tool that was simple,
(07:08):
not something to be too complex
that was gonna be very hard to prove.
Back then, before I started, contributing these new features,
the thing I liked most about Copier was how simple it was.
I saw the the the source code of the project was simple, but also
(07:30):
the template
you produced
with it
were simple also.
In terms of the actual
uses for copier
and other project templating tools, I know that there are a number of different ways that people have employed them. You mentioned the need for
being able to set up an Odoo environment and all the different packages and configuration necessary to go along with that.
(07:55):
There are projects for being able to set out the skeleton of a Python repository.
Ruby on Rails has its own built in scaffolding engine to bootstrap a project. Django has 1. I'm wondering if you can just talk through some of the
common use cases that you've seen and that you've used these templating tools for, and then maybe talk a bit about how Copier compares to the overall ecosystem of the available options.
(08:21):
We are programmers,
and we love
to, to keep our code dry. What we call dry, don't repeat yourself.
That's very nice, but sometimes you just cannot
do that.
Recently, I started 1 very, very little library that I needed for specific need. It's called Versort.
(08:42):
It's for sorting versions.
It's very small. It was like an exercise
to see how much
these
undryable things
are in a project. This project is 144
lines of Python code,
plus
111
lines of Pytest code. So it's very, very small,
(09:05):
but the total amount of lines
it's
678,
almost 700 lines
for
around 150
lines of real code.
Where are those lines from?
It's dot GitHub slash worksflows
slash,
I don't know what the yaml.
(09:27):
It's the license. It's precomicconfid.jamal.
It's Travis CI or GitLab CI or pritzhererc.jammerorreadthedocs.jammerormypy.inioypides.iniomkdocs.
That's yml. Editorconfinflak8.
Configuration
for Versus
Code. Well,
(09:48):
every new IDE,
every new automation
tool that comes to the market
says the same to you. Hey. You just have to add 1 dot 5 to your project.
But finally, you end up with a 1, 000 of dot files,
and you really need them because those are great tools
that save time for you. But how do you manage all those configurations for your projects?
(10:14):
Well, this is a very specific aspect where you happen to need a tool
that handles all of these
repetitive things for you. So when you cannot keep your code dry, you need a drying machine and this is copier.
You also asked about how does it compare
(10:34):
with other tools? I will say that there are 2 major alternatives to Copier, at least otherwise the analysis I made before
investing time on developing
it. 1 of them
is cookiecutter,
which is also written in Python, very famous in the Python world.
(10:54):
And the other 1 is Yeoman.
Both of them are
very good at what they
do, but these are just templating tools.
They only serve to bootstrap a project and many others you also mentioned because I also
looked at many other
(11:15):
alternatives, although these 2 are the biggest ones. All of them have the same problem. All you do is bootstrap the project.
So, okay, I bootstrap my project. I create these, let's say, pre commit config dot yml file.
Okay. But then
after some time, that configuration
also evolves.
(11:36):
Maybe
some CI tool requires
new
syntax,
maybe
some tool you are using with Precommit
gets a new release with a bug fix you need and all of that. So you need to manage all of that.
I'm talking just about this use case, but any other use case also
requires
(11:58):
maintenance.
Well, this is what I call day 2 operations.
It's cool to start a project. Yeah.
Everybody can do that. You can even just copy a folder you have.
But
what happens the next day, and the next week, and the next year? And what happens if you require that your framework
(12:19):
has a specific
folder structure,
and you want to provide a template to your community, which was our case, we used to do the project.
This template needs to evolve with the project.
So what do you do? How can you help people evolve their projects?
This is a key feature of Copier,
(12:43):
that as far as I know, no other templating tool has.
And Copier
relies on Git
and uses, SmartDiff
algorithm
that compares
the template evolution, the template git evolution,
with
your project evolution.
(13:05):
And then somehow tries to apply diff
just like git would do
if you
have
both stories in your git project, which is not the case.
You just have to separate completely different stories because,
yes, you you you use the bootstrapping,
but then your project evolves
(13:25):
as it want it wants. You can even change some answers to some questions.
And then this SmartDiff algorithm
that I developed,
handles all of this
and lets you actually evolve your projects.
This is, I think, the best feature of Copier, but it has another 1 that also I didn't find
(13:50):
in any of the other tools,
which is the ability to apply several
templates
to a same project.
So let's say that you develop
several languages.
For instance, you develop,
let's say, JavaScript projects
and Python projects.
If you develop
(14:10):
JavaScript stuff, you're gonna need the NPM
configurations,
JavaScript configurations. I'm not a JavaScript developer, so I might be failing
in something
I say.
Well, you need your kind of tools. See, if you develop Rust, it's the cargo stuff.
And if you develop Python, you might need to set up the py or you might prefer a pyproject.
(14:37):
Yml or a PPM, whatever.
So you have separate
scenarios
depending on the language you're using,
but
many of them
use the same tools.
Let's say you use all of these to produce
Docker images.
Okay. Then the Dockerfile is is common for everyone, or maybe the Docker ignore a file,
(15:02):
or maybe the
GitHub workflows,
things like pre commit config
that make your code pretty no matter what kind of code are you writing.
So
there's a split of needs. You may need to apply the predefined tools,
but
also the Python specific tools in a project.
(15:24):
And in a different project, you need to apply the predefined tools and the CI
specific
files also, or maybe the JavaScript ones.
So Copier lets you
apply several templates to our same project,
and this combines with a previous point. So both templates
can be applied simultaneously to the same project and make that project evolve
(15:46):
also.
So these are 2 killer features. I think Copier is,
you might like more
something
of the other projects, but these 2 features are essential
distinguishing.
And there are a couple more points.
1 of them is that you don't have to write JSON, which I think it's a big feature
(16:08):
as compared with the other tools.
I hate writing JSON. Don't know what do you think, but I hate
it. You you write Yamo on the other hand in case you like it.
The other feature is that for publishing a template,
there are some
tools
such as Yeoman
or also I think it was something like mister Bob templates for Python,
(16:32):
where you need to produce
an actual package
to make a template.
Because for instance, Yeoman is not a template manager, it's more a template framework,
which helps you build a note project,
which is your template.
But
if I'm developing Python
(16:53):
and I'm a Python developer,
and I don't know how to do a a NPM project, or I don't like JavaScript.
Why do I have to learn JavaScript,
to learn NPM, to publish my templates
as an NPM package
just to help people
(17:14):
deal with Python stuff. It makes no sense.
But on the other hand,
I like Python. I know how pip works, but also maybe somebody comes from Node and wants to use Copier and they're in the opposite situation.
So Copier,
if you want to publish a template,
you can just use a git repo,
(17:35):
and it's just enough.
I have a blog post in my blog where I explain in the in-depth
why Copier think it's way better than our alternatives.
The specific thing that intrigued me about Copier and maybe wanna reach out and learn more about it is the ability for being able to apply changes to the underlying template to projects that were generated with it, which
(18:01):
most tools such as cookiecutter or Yeoman, as you said, you can generate a baseline scaffold.
But then as you work on other projects that use some of that same structure, you find improvements or enhancements, and then you bring that back into
the project template that you're working with. But there's no way to then reflect that back out to all the other projects that it was created with. And so you have to try and figure out doing it manually in a piecemeal fashion.
(18:26):
And so you end up with a lot of drift in terms of the best practices that you've applied. And so your original project doesn't get to benefit from the learnings that you found in project number 5 or whatever. And 1 of the things that I didn't realize in what you were just saying is the ability to combine multiple templates.
You can actually
rather than having to have 1 monolithic template that covers all the concerns that you need for a project or have
(18:51):
a bunch of different conditionals that it will generate project skeletons for based on the inputs that you provide.
The input prompts, you can instead have a modular approach where you have a template that says, this is going to handle all of my
tooling for my configuration. So it will put in my pyproject.toml.
It'll configure, flakate. It'll set up my py. And then this other 1 will set up the baseline for the Django app structure. And then this other thing will handle setting up all the configuration for my CICD, and then I can apply all those together,
(19:24):
or only 1 or 2 of them, rather than having to have everything in 1 project. I think that's a pretty amazing capability as well.
I think these are very
important features for the project.
I mean,
the ones that make you choose Copier over other projects.
It has others. For instance, even if even your templates can share
(19:45):
code among themselves. So you maybe have a question, Adi.
When I say questionnaire, I mean, you ask questions to the user so you can produce the sub project you want. Okay?
So if you have a questionnaire,
you can actually combine
several questionnaires in 1.
(20:06):
So if you have several templates,
you can include them
just skids modules and combine
in the main template
several parts.
This is something that is not perfect yet, but
it also helps. So it's very good, I just say.
And so digging more into the project itself, I'm wondering if you can talk through the actual
(20:32):
copier project and how it's implemented
and how the overall design and feature set has evolved over the past year that you've been working on it.
When I since I've been working in it, I think these 2 key features
I told you were new in version 3. When I came to Copier, it was versions
2
dot
(20:53):
something. So
it's these new features is something that I am involved with, so I'm happy with it.
But the project itself, the settlement
of the project as it was when I came in
and how it is now.
It is,
API
and CLI
(21:13):
project.
So main use for copier is CLI.
You use it in your terminal.
It should be comfortable
there.
But you can actually use it as a library if you need to expand anything.
It's written for Python
3.6
or later.
Recently, it supports 3.9 also in the latest version.
(21:37):
It uses,
poetry to manage the project, which is a pleasure. I love poetry.
And
also
it's completely
typed.
Well, with static typing, it uses my pie to check that.
If you plan to use Copier as an API tool, as a library,
(21:59):
let me tell you that please use just the copy
function in the main
module because
everything else is gonna change. We'll talk about that later.
So just
use copy, which is the makes most of the work.
Something that is very, very nice about copier
(22:20):
is
how it has a very big test suite
that helps us develop safely.
And I think that the most interesting part is how do you actually
make a template?
Because
most people, I think, don't really
care about how is the tool written, but they care more about how do I write my templates,
(22:45):
which is what affects them more
directly. Right?
So
you don't need to add any logic to your templates. You can just use copier
to copy a folder.
But if you need to add logic, you have Jinja too. You don't need to add any configuration to your templates.
But if you need it to just add a copy of the the demo
(23:08):
file, which also
can
can evolve from simple to complex to multiple
includes
easily. So
it's
as complex as you need.
You don't need your template to be a git repo, although
if you use it, it's better, and you have
(23:30):
other features such as updates.
You don't need to support updates in your project. So it's something that the template
explicitly supports.
And if you don't support updates, then CopyEdge has becomes scaffolding tool, just like the
other tools we also talked about.
But if you need those updates,
(23:51):
all you need is to
allow your template to write, copy your answers file, which tracks last answers from the project.
You need to
make it git repo, and Copier will use Git tags to evolve your project.
You also don't need to program
anything,
but if you need to program something for your template,
(24:13):
Copier uses super processes
where needed.
So if you're a template's
writer,
you don't need to know Python.
You can just use any binary you want. Copier will communicate with the binary for some tasks
just using the arguments you say to hit to tweet
and some environment variables. So it's very easy to block with
(24:38):
any tooling you want.
And I said it before also, you don't need to publish a package. If you want to publish a template, you can just publish a git repo, and you're done.
To your point of being able to call out to any arbitrary
command as part of the template generation
plays into the fact of it being language agnostic. And you called out earlier the issue of
(25:02):
if you like Yeoman, for instance, but you're not a JavaScript developer, you still have to work with the NPM tool chain and things like that. I'm I'm wondering if you've looked at any
tooling around Copier to be able to package it up as a single executable so that somebody
who's interested in using its capabilities but isn't a Python native can still be able to benefit from it, and if you have any of the challenges or issues that you've come across in that process.
(25:32):
Yes. Actually, that's the main reason why this is done like that. Well,
that's 1 reason. The other 1 is that the project have this feature already when I came.
So I like that. And since I had been
checking Yeoman, I just had, for example,
I compared that and I said, okay. I like this. I like that Copier is a templating tool for the world, not just for the Python world. It's much better.
(25:59):
So if I write Rust
and I want to let
my template users
just execute
tasks against the template
with a Rust library,
or if it is written in Go, or if it's JavaScript or whatever.
I don't want these people to care about
(26:20):
anything else than just pip install
well, better, pipx install copier.
And that's it. If they do that, they
know just copier docs. I mean, there's something they will have to know, which is Jinja.
But I think it's a very simple templating language, very comfortable. So that's
(26:42):
well, at the end, you have to to have some template language.
But apart from that, you don't need to write a single Python line to make your project your template
be able to use all Copier features.
And so digging a bit more into the process of building templates for Copier,
I'm curious if you have gone down the path of taking an existing template for something such as cookie cutter and converting it into something that is compatible with Copier or any of the other frameworks. And what are some of the design considerations
(27:16):
for template authors to be able to build something that will work well with Copier and all of its capabilities?
My use case
was a very complex 1.
I mean, my use case was
so complex that I could not use
cookiecutter
or human.
So
1 good thing about how Copier has developed
(27:38):
is that I used it. I mean, I don't work gain
too much money with Copier. I just
work with other things and I needed something that
did this work. So
if you take a look at a repo that is called Dutba copier template,
you can see it's a very, very complex template
(28:00):
and it helps to make copy of what it is. But
after
Copier 3 was released,
some people started liking it, started using it and started migrating cookie cutter templates to copier templates.
And I think this is
bit of a challenge because
(28:21):
copier is a new mindset.
You cannot have a template which is compatible with both.
You can have a template which is similar,
but it's not compatible,
a 100%.
So you might have to read carefully the documentation to make your templates
match copiers.
1 differentiation
(28:41):
is, for instance, in copier, you don't need that your template is a subfolder
of your project.
The project
and the template can be the same folder.
In Qutr, for instance,
you have a main folder,
which contains your readme, your license, your whatever, and then a folder inside that repo
(29:05):
that contains the actual template.
But in Copier, it doesn't
necessarily work like that.
Actually, it didn't support that, but later we added support because somebody
was asking for it because he came from cookie cutter.
But you don't actually need that because in copier,
(29:26):
for a file to be considered a template, a Jinja template,
is masked and in a specific suffix,
which by default is dot
t m p l.
So let's say your project contains
readme.md
and then a readme.md.tmpl.
(29:48):
Copier will ignore the first 1 and just take the second 1.
So if you want to have a readme for your template,
but also a different readme
generated from your template,
you can just have these 2 files together,
then you don't actually need to have your template inside a subfolder.
So it's just a matter of taste. How do you prefer to organize your your things?
(30:12):
But it's a mindset change that if you come from cookie cutter, you you need to do. Last week, I think it was, we had an issue.
Somebody was asking us to support having several templates
in the Sangit repo,
but this is not possible for us
because Copier uses Git tags
(30:32):
to evolve your project.
So if we were to do that, you would have several templates,
which will
evolve
separately,
but actually
all would be versioned with the same version.
You would have
some confusing
things. Things like,
(30:53):
template evolving,
having a new release, but with no change.
And so
1 Git repo
will be 1 template,
and this is by design.
So we'll be like that forever, just because
we need to support updates. And and because we designed the project with a specific
goal,
it requires a specific template.
(31:15):
In terms of people who are using Copier, what are the options for extensibility
or integration with it? And
particularly, you mentioned the Python API,
just some of the interesting capabilities or features that that exposes.
I think it's a matter of of comfort
because
let's say,
(31:36):
for instance,
I use
1 CI,
internal CI,
which applies
to our company projects,
the public template we have,
plus another
private template that we also have for this kind of projects. This is 1 example of how multiple templates per project help
(32:00):
because, for instance, the public templates
will help everybody to get up and running with
quite easy. But the private 1,
makes our specific
customizations,
our internal
CICD system work,
and other would for nobody else. So it's a thing we maintain
(32:21):
internally,
and every project has both of them. So 1 CI system
just
every night checks for a new release
in both templates.
And if there's a new release,
then it updates the project.
It runs the checks.
And
if there is any conflict or any problem, it will
(32:45):
just open a pull request.
But if there's no problem, it will also merge the pull request. And so the next morning you sit
in your desk
and your project is updated because you made a release
from the template.
So
if you just use command line based CI,
(33:06):
I think you will be more comfortable using
Copier CLI
or for your daily usage of Copier, it will be more comfortable also. But if you have a specific
CI tooling that you happen to have written in Python,
I think you will be more comfortable using Copier as a library.
(33:27):
But actually you can make the same thing with both approaches.
And in terms of the actual
uses for Copier, what are some of the most interesting or innovative or unexpected projects that you've seen built with it? Well, to be honest, I have not seen many
because
I don't track copier users.
We have in our readme 1 suggestion.
(33:49):
Please type your copier templates
as copier template.
There's a topic in GitHub
labeled copier template.
So we just expect people to tag their templates,
and then
every now and then
come around and see what they're doing.
I think the more the most interesting thing I have seen there
(34:12):
is there's 1
contributor we have, very active, actually. It's called he is called. Sorry.
He's called
hang on. I'll try to say it. Okay. Timothy Matsukotelli,
and he is the main tenant of MK Doc Strings. So it was a surprise for me to see
him involved in this project that he seems to love Copilot very much. And he has a template
(34:36):
to build poetry based Python projects.
Myself,
I started developing a template, which is actually, it's just a proof of concept.
It's under the copierarch
organization
in GitHub. You can see it. It's called Auto Pretty.
And my goal is to just bundle
(34:56):
all those kind of tools
do you usually need for any kind of project.
So I'm talking about linters,
formatters,
PreCommit,
Prettier, all these kind of tools that are very basic.
I happen to have everywhere.
And it's also a project that is trying
(35:17):
a new concept of a self
templating template.
So I want to be able to have a template
to prettify things,
but to actually be able
to apply these templates to the template itself.
So I can update my dot files for the template
from the very same template.
(35:40):
Well, it's not released yet. It's just there are some alpha, you can try it. And it's not working yet, but it is like a playground I have. And at some point, I will make a better release.
And in terms of your experience of working on and with Copier and using it for your own projects, what are some of the most interesting
(36:01):
challenging lessons that you've learned in the process?
I think I learned a lot of things.
To start, I learned,
lots of tools that I didn't know.
I might seem
a little bit stupid saying this in a pocket, but I never wrote Pytest test, and it's awesome. I
like it.
(36:21):
I started using Poetry. I liked very much. Also, this is the first project where I am deeply involved
that has static typing,
and it's very good. It helps a lot. So
that's something I learned.
Something I also learned is that sometimes you need
(36:41):
to test the template.
At first,
since I was the only 1 developing my template and using it,
nobody else
would use it.
Well, I just
started releasing versions and versions of the template without any problem when anything went wrong.
But later, when this template became a more critical part
(37:04):
of our toolkit
in the workplace,
and all my colleagues
started using it, and the CI system started updating projects
automatically.
Then something I learned is that
if you happen to have this kind of usage also,
which is cool because it's where copier shines actually,
(37:26):
I recommend
to make pre releases of your template
and apply them manually to some specific projects, or maybe if you are several
developers
involved specific template,
Do developers,
little group of people,
try your premises
because it will help. Sometimes
(37:48):
it's hard to predict
how a project will evolve
and how your template will behave.
Because,
well, this SmartDiff algorithm
makes
most of the work for you. But sometimes,
you will try let's say, you add a new file to your template,
and you didn't happen to
(38:09):
expect that that file was existing before
because the user
added it manually to a project
or maybe changed
somehow.
So by having some prereleases,
you can there there's a flag in Copier CLI,
which
is slash slash
pre prerelease
(38:30):
that tells Copier to consider pre releases when when fetching the last the the latest version of the template.
It helps you to test your template before it's released.
Apart from that,
app tests
app tests. You don't usually need to add a test that tests that the template
(38:51):
when the template has a file that is called
whatever and it has a content and you pass that
answer,
copier will render that content. Well, that's obvious.
But
so don't test what copier is doing.
Test why is it doing it. So in your tests,
create a new project. And let's say that your test is testing
(39:16):
that you install PreCallmate
configuration, which includes black. Okay.
Actually, I want to test that. Then in your tests, create
a new template, a new sub project from that template,
and then add a Python file, which is badly formatted.
Try to commit that
and see how pre commit performance
(39:39):
that. So you're seeing
the way
you the way you do things and not the how you do things. And it's important for testing, for your test to be because if it just says that the temp file renders
like this when I pass
these kind of input,
well, it's gonna be true all the time,
but maybe if the input is wrong or the file is wrong, you won't detect that.
(40:02):
For people who have existing projects that they want
to extract some of the structure and best practices from and apply it to other systems, is there any capability for
being able to use copier on a project that already exists to apply some of the boiler plate and overall structural best practices to the project?
(40:23):
Yeah. Copier takes a simple approach
when analyzing your project.
And it has actually 2 commands.
The first command is copy and the second 1 is update. So
when you're
calling copy, it will
just copy everything and forget about what those files contained before.
So let's say your project is already a Git project.
(40:46):
Well, you just
apply the new template, copier, copy, whatever,
and then copier will
do his stuff,
and then you will have to review
the files that copier modified.
Why would you do that if the template supports updates? Because,
like I said before,
(41:07):
it's not mandatory.
It's a template
choice.
If it supports updates,
it will write a file that by default is called
dotcopieranswers.yml,
but you can change that.
Actually, you have to change that if you want to apply several templates. That's how it works.
(41:28):
Let's say you just use 1 template. That's the name of the file. Then it will store
all your answers to the questionnaire,
plus
the URL
of your git repo,
plus the tag that you applied.
So
copier,
once you commit all those changes,
copier will consider
(41:49):
that
the resulting
of copying that template
in that version
with those answers
is your sub project right now.
So it will
later
apply an update.
This algorithm
will compare
how it should have been
(42:10):
and how it comes to be, and then later evolve. But if you just have several projects
and want to apply Copier over them,
Once they already exist,
it's completely support. All you have to do is just copier copy
the first time,
and then
later, just copier updates, as usual.
(42:33):
That first time, you will have to review not only what copier is doing, but there's no other choice.
And for people who are looking for a project templating tool or something to help manage the overall life cycle of their projects, What are the cases where a copier is the wrong choice?
I hope it's the good choice.
(42:53):
That's why I like it. But when it is the bad choice?
I said that we try to dry things
to don't repeat
yourself in your code. And
when you cannot
dry, then you have a drying machine. Okay. But if you can dry,
just dry your code. You don't use copier.
(43:15):
Better dry it. If some parts of your codes can be externalized in a library, then do that library. Don't just copy that everywhere.
And I actually do that
because it's faster and more and easier. But I have plans to me, right, to actually extend the library for this,
copier template project I have.
(43:36):
So I'm telling you not to do that, but it works.
So so maybe it's also not a bad choice.
As you continue working on Copier and continue using it, what do you have planned for the near to medium future of the project?
Copier 6 is gonna be a huge release.
Really, really huge. Until now,
(43:57):
let's say copier 12 was,
simple, fun
scaffolding project, just like any other, but easier.
Copy of 345
has been
the best templating engine in the world.
No. Not the best, but, you know, it has these killer features.
(44:18):
But Copier 6 is gonna be a complete refactor of Copier.
I am confident about it because we have very, very big
test
coverage.
So I'm refactoring it because Copier was built around
functions,
but functions were too dependent
(44:39):
among themselves. And at the end of the day, it became
hard to
detect
when is something happening, when not, or to have a relationships
among the whole
copying process.
So
it became hard to maintain in that regard, and I'm making a bigger factor to do it object oriented,
(45:04):
which is gonna be way easier to maintain in the future.
Right now, there are some alpha releases of Copra 6, but are very, very far from what Copier 6 is gonna
be. But if you use them, you will help me because it will detect bugs. So
do it, please.
And you will have
(45:25):
conditional questions. That's something that it's already in the last ask for.
Also, a more interactive
terminal user
interface,
which for instance, also supports multiline input.
That was not easy, but
it's working now. It has some bugs, but, well, it's
(45:47):
more things
that are gonna come in Copier 6 are
multiple answers for the same question.
That's something asked asked by the community,
And, also,
the self templating
support
to have a template that applies to itself, like I said before.
And I will take a chance to remove some options that
(46:11):
make no more sense that,
those kind of things you add,
but don't really have much sense. And 1 thing I would like to do also is to change Jinja defaults
because right now, Jinja is,
by default,
different of the official Jinja. For instance, it uses
brackets
(46:31):
instead of curly braces.
So I think that if there's somebody who comes to Copier for the first time,
and then he has to go to the Jinja website and for the first time
learn Jinja.
And maybe somebody who doesn't even
write Python.
Well, it can be challenging if apart from all of that,
(46:52):
also
Copier uses different placeholders
than official Jinja.
So possibly there's gonna be some changes there. Then maybe Jira templates will not work with Copier 6 if they worked with Copier 5. So another things that I will have to add is a version tech
(47:12):
that fails if your template is not ready for
Copier 6. I have to think about that a little bit more.
But right now, I am mostly involved in the refactor.
And once the refactor is settled,
then I will start adding these other things, which are gonna be very much easier
after the source code is reorganized.
(47:36):
And after copy of 6, then I think
there's gonna be not
some main more things to do. The project is very, very nice.
Just more
evolving of the tool, but it's gonna be more more or less what it should be.
Are there any other aspects of the copier project or the overall topic of project templating and scaffolding and life cycle management
(48:02):
that we didn't cover yet that you'd like to discuss before we close out the show?
There is 1 complex
case
in Copier,
which is answer migration.
It happened to me in my template
when I released, I think it was version 2.
You can answer to copier
in YAML format.
So an answer to a copier question can be anything
(48:25):
that YAML or JSON supports.
So you can
put a dict, an array,
a boolean, or whatever,
combination of them as an answer.
So let's say that you expect a specific format,
and then later
it needs to change.
It happened to me because the format I was asking
(48:49):
was not smart enough
for, different well, actually, if you want to include it,
it was because of traffic, which is an amazing inverse proxy,
which
for traffic
1
was
much simpler
to use and to set up. But for traffic 2, it's much more complex,
(49:12):
but actually also very much more powerful.
So the structure
we needed in our project to include the traffic configurations
was different because we started supporting traffic too. And then,
for instance, things like
supporting several
host names for a project
(49:32):
was not possible before. Now it was possible
that structure didn't match. So
Copier includes a migrations
feature
that
allows you to even
change
the answers
before migrating.
And this is very handy because
if you had your project in template version
(49:56):
1,
and then for template version 2, some question needs to change its structure.
You can automate that.
And then your
copier project, when updating to version 2, it's gonna take the answers from version 1. It's gonna convert them to the new format,
and then
use them for your version 2 template. Well, it's not copier who's gonna make it. It's you because you are the 1 who defined that structure.
(50:22):
But you have to program that in your template,
And that's,
a bit of a pain points, so
I advise to
to try to keep your questions
simple if you can.
For instance,
if you want
to ask the user, what systems do you want to use? Please input
(50:42):
a YAML list.
Maybe it's simpler for you to just say
2 booleans.
Do you want to use system a? Yes. No. Do you want to use system b? Yes. No.
So it's easier to evolve that kind of questions,
but it's not always possible. So commits across migrations also.
And another thing that I recommend
(51:04):
is that, well, when copier
updates
automatically
a project,
it can detect conflicts just like a git merge.
But copier will leave a dot reg file,
which includes the conflicts.
So if you trust Copier too much,
(51:25):
maybe
you can
say,
okay. Copy of that worked. Then the project is okay. But if there's a reg file,
possibly it's not okay. That's something you have to review by yourself. That's something that the Smart diff algorithm
was not able to solve.
So there's Precommit
(51:45):
hook,
which I think it's in the docs
copier,
as a recommendation to forbid your
git project to commit that reg files. Because
that way you make sure you are not committing
a failed
update. It's important.
And also a little problem that many users face
(52:09):
is that they expect Copier to work
a way that it doesn't work. I mean,
when these users update project,
they copy your
answers file. And then after it's been edited,
they commit that
and then they run copier update force, copier force update. Sorry. Then they're tricking copier. They're telling copier that
(52:32):
those answers
produce that project, but it wasn't true. The answer that's produced that project were the previous ones. So maybe copier
will succeed,
could happen,
but possibly it will fail also because you're tweaking this SmartBeef algorithm. So if you need to change,
an answer,
use
(52:53):
copier to do it. Do not edit copier answers directly.
This is not documented in the stable documentation. It's documented in the latest branch of the documentation.
So maybe you will not find this advice
yet, but just don't do it.
Just say you can pass the new answer to copier by the data flag in the CLI,
(53:19):
or you can just
answer the questionnaire again. The the last answer is really there, so you just have to change the 1 that changed.
But don't do it like that because
it will fail at some point and you will think
that it's a bug and it's not.
So
just let's copy or edit that file.
(53:41):
Well, for anybody who wants to get in touch with you and follow along with the work that you're doing or contribute to the work that you're doing in copier, I'll have you add your preferred contact information to the show notes.
And so with that, I'll move us into the picks. And this week, I'm going to choose having a good quality set of playing cards on hand because
it's a versatile
tool. You can use it for playing all kinds of different games, either by yourself or with other people. So definitely worth having 1 that is
(54:07):
good quality and can stand up to a lot of use rather than just something that's cheap because then they're not as fun to use and you're less likely to pick them up. So definitely recommend just having some good playing cards around. And so with that, I'll pass it to you, Jairo. Do you have any picks this week?
Well, I have some.
1 thing I tried recently was Mozilla Hubs. I don't know if you have tried it,
(54:28):
but it's very nice.
Mozilla Hubs
is an alternative to a video call. It's not a game. It seems like a game, but it's not a game. It's a video conferencing tool in virtual reality,
which helps
you stay in a three-dimensional
area
where you can hang out with your friends or workmates.
(54:52):
And it's very fun
because
it avoids
the Zoom burnout,
the video conference
burnout that your brain experiences. It's something
interesting also
could be an epic.
These days,
most people need to
stay away from their friends,
(55:13):
but using this tool, you can feel like you are in the same room. And
1 thing that seems stupid is like copier, has little features that make your life different.
And this little feature of,
hey, I'm with 10 friends, but I want to talk to 2 of them, then we can go to a corner of the room and talk. And
(55:39):
you don't have this
chicken house
noise
when everybody is speaking in the same video call. It's very nice.
Alright. Well, thank you very much for taking the time today to join me and discuss the work that you've been doing with Copier. It's definitely a very interesting project and a
well designed tool for being able to help manage the full life cycle of projects and the boilerplate occur that spans across multiple different projects. So thank you for all the time and effort you've put into that, and I hope you have a good rest of your day.
(56:09):
I just want to say thank you to Juan Pablo Escaletti for starting the project, to Ben Felder
for maintaining it during last year and also
this year and making me a comment in it just because he liked my work. So thank you.
Also, thank you to Technitiva
that has sponsored
the work
(56:29):
of copier
from version 3 to version
5.
But latest version, latest stable version
because
they just needed a good tool that didn't exist, and they decided to let me invest time on it.
Although from now on, it's
just
(56:50):
basic issue management, but I have to develop that in my free time right now. And also
thank you to Timo Seema and Mazzuccotelli
because he helps a lot, and he provided for us the basic skeleton for building
our documentation,
which looks really cool and it's all thanks to him. So thank you.
(57:14):
Thank you for listening. Don't forget to check out our other show, the Data Engineering Podcast at dataengineeringpodcast.com
for the latest on modern data management.
And visit the site at pythonpodcast.com
to subscribe to the show, sign up for the mailing list, and read the show
notes. And if you've learned something or tried out a project from the show, then tell us about it. Email host at podcastinit.com
(57:36):
with your story.
To help other people find the show, please leave a review on Itunes and tell your friends and coworkers.
Popular Podcasts
United States of Kennedy
United States of Kennedy is a podcast about our cultural fascination with the Kennedy dynasty. Every week, hosts Lyra Smith and George Civeris go into one aspect of the Kennedy story.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
Bookmarked by Reese's Book Club
Welcome to Bookmarked by Reese’s Book Club — the podcast where great stories, bold women, and irresistible conversations collide! Hosted by award-winning journalist Danielle Robay, each week new episodes balance thoughtful literary insight with the fervor of buzzy book trends, pop culture and more. Bookmarked brings together celebrities, tastemakers, influencers and authors from Reese's Book Club and beyond to share stories that transcend the page. Pull up a chair. You’re not just listening — you’re part of the conversation.