Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:07):
Welcome back to Adventures in DevOps, where every episode's a deep dive into a specifictopic with an expert guest.
Today's adventure focuses on writing documentation and feature flags.
As the expert, we've got someone with quite the unfamiliar title.
Previously, she's been a software developer, API integration engineer, and now is thedocumentation lead at Unleash, Melinda Fekete.
(00:28):
Hi Warren, thank you so much for having me.
Yeah, you know, and I'm really excited.
What I'll say is that we try to limit who shows up on the podcast based off of theirtitles.
And you let me in on a little secret before we started recording that you don't believe injob titles.
So what's that about?
Yeah, you know, um I work at a very small company.
So I did build the documentation website and I maintain it.
(00:51):
And I also do all of the technical content that is on there, but it's also, just a part ofmy role.
So I do other DevRel things like conferences and meetups and developer education and talksand workshops and a of marketing, a bit of this and that.
So what would you say is a good job title for that?
(01:12):
I don't know.
You know, I think this is where if you're at a small enough company and still like aroundthe startup phase, then there's like this idea of founding engineer.
But then like, what do you call the second founding engineer and the third foundingengineer?
And then I feel like you have this idea that you should start applying titles or roles.
I think what I would say is that often labels are helpful, but they're all wrong in a way.
(01:34):
I do want to share though, a long time ago when I was in the university, I had a lot ofprofessors who would say, we want engineers who can do at least one other thing.
That's the highest number of feedback that we get from the industry.
And this was, uh wow, almost 20 years ago.
And I didn't really understand it at the time.
And I think the longer I've been in my career, the more I've started to come to terms withthis idea that just doing one specific thing like your
(02:01):
engineering job ends as soon as the code gets deployed to production is just like not thatmost critical aspect.
Yeah, it's what they sometimes refer to as T-shaped, right?
So being really good in one kind of vertical area and going really deep and having thatexpertise, but also ah just dipping your toes in a couple of other things and trying
(02:21):
things out.
And I think it also helps with job satisfaction.
And if you just do the same thing every single day, it's going to get boring quitequickly.
So of course there's engineers who only like to code and that's all they like to do.
But even back when I was an engineer,
Um, if you had just told me to code all day and do nothing else, I would have probablyquit because I really need that variety of like, um, being involved in like, I don't know,
(02:48):
interviews.
So it's nice to have a couple of different responsibilities that you also care about andcan kind of experiment and try new things.
How did you make that shift into an area where you felt comfortable with that wasn't yourprimary remit when you started?
Like you didn't just like one day wake up and like, you know what?
I only want to write documentation from now on.
(03:11):
For me, when I saw the spark and um excitement for the first time with documentation is umI was working at this coffee roasting company.
I was integrating um with different APIs.
um We built this IoT espresso machine.
A um big part of my role was just trying to figure out these different integrations.
(03:34):
I was looking at a lot of API docs and some of them were terrible.
You really had to spend days and days trying to, with trial and error, figure out what theheck was going on.
And that really inspired me to, how can you do this better?
And started looking at some of the companies who I thought were doing really well um eversince.
(03:55):
I probably wouldn't look back.
think that it's been three or four years now that I mostly do documentation and some smallthings on the side, but I love it.
So I think there is this old joke where no engineer comes into a company and is lookingover the current stack and looks at all the source code and says, wow, this last guy just
(04:19):
wrote the most perfect code ever.
I don't need to change anything.
But I actually think that's true for documentation.
I don't remember looking at any portal and being like, wow, the docs for this softwareproduct, they are fantastic.
ah So you mentioned that there are some that you're like, we should do better and look tosome.
pinnacle ones out there that should be modeled for any software product.
(04:40):
What are those in your mind?
uh Unleashed does a pretty good job.
Make your own company, okay, because you're working on it.
So everyone's gonna have to go look at that after this episode now.
Yeah, I mean, mean, some of the big open source projects like GitLab, I think do anamazing job.
Actually the landscape has changed, I would say significantly over the last couple ofmonths even.
(05:02):
LLams, AI, they're very good at writing documentation, right?
So you can produce a lot of good quality content with very minimal input.
So the name of the game is kind of more been around like...
what's some of the experience you can build around it.
things like the AI search piece of the puzzle now is like you have to make it, you have tomake your documentation also usable for LLMs because about 50 % of all documentation
(05:30):
visitors for a site like ours are now AI tools.
So you have to find a sweet balance of like, like what's working for humans and what'sworking for LLMs as well.
it shift your job responsibility from historically having to write docs for humans to nowhaving to write docs for LLMs?
There's a couple of fun things you can use, like the tooling that I use, it allows you todistinguish between how you want to write for a human and what you want to expose to a
(06:02):
human and what you want to expose to an LLM.
So you have different code blocks and you can show some of it to humans, some of it toLLMs.
A lot of it is like there's a good overlap, but...
LLMs typically do well when a lot of the content is just plain markdown.
You strip it off any of the fancy tables and accordions and all the things that um you puta lot of work into making the content readable for humans.
(06:32):
But then you have to strip away a lot of those fancy UI features for LLMs and things like,I don't know if you've heard about LLMs.txt, which is um typically something you...
you would expose for all of your pages so that when LLMs come and look at your docs, it'sa very clean and simple, easy to understand structure for them.
(06:53):
Yeah, I mean, can imagine really realistic from this, there's like two paths down there.
There's exposing the docs that you have to the training processes that companies who arebuilding models are making so that that information can end up in LLMs just straight away
so that when your users or whomever perspective customers are querying an LLM or promptingit for information that it can actually return the results.
(07:18):
And then there's basically a more complex search.
aspect where at runtime being able to answer questions from the web that haven'thistorically been able to be answered.
Realistically, you may be gotten away with some high level JSON blocks at the top of someweb pages, but now LMS are directly consuming the data that is on uh individual article
(07:39):
pages and summarizing it or providing a useful answer.
And that has a second nature there.
And I think one of the problems is a lot of the tools out there don't do a great job of
removing the visual elements that have been added in some way.
Like if you just look at generated HTML or even markdown in some cases, it's verydifficult to get useful, optimized LLM based uh output for or input for LLMs.
(08:05):
uh What's the strategy there?
Do you just like write the docs twice?
I think a lot of the modern documentation platforms do it behind the scenes for you.
So the way they generate the LLMs.txt file, if they're doing it well, they'llautomatically strip some of those.
We write markdown with custom React components, which are the things like the tables andthe drop downs and the buttons and whatnot.
(08:29):
And so all of that gets stripped out uh from the LLMs.txt.
And I think that alone is already a big win.
And yeah, I'm experimenting with including
Uh, certain explanations, like if you strip out a table, but you still want to tell likethe LLM what, what that is all about.
Like you can probably explain in a different way.
(08:49):
And I've, I've just started experimenting with some of it and I'm, um, I have the dataaround, you know, what pages got viewed by what percent of humans versus LLM.
So I still need to do a lot of digging around.
Um, but there's, there's some useful features out there.
for sure.
So just for context, we've been using DocuSource for a while now and the LLM's dot textfunctionality plugin is atrocious in every way.
(09:18):
ah So it's always something that we're sort of looking at, especially offering a verytechnical product.
You can be sure that it's more likely to get picked up in some way.
And I think one of the biggest problems there is when
you have custom react components or view components or just any sort of thing that youwrote yourself and not just pure markdown getting embedded links to work has been a huge
(09:39):
struggle for us like if you're using a customer react component that has like somethingclickable in it to go to a different page like is whatever process you're using to write
your documentation is going to be smart enough to somehow pull that out and then listedappropriately.
Yeah, I'm not like extensively looked at it.
So I would say em we recently migrated off of DocuSaurus.
(09:59):
Exactly um why you, some of the reasons you mentioned about the plugins and entireecosystem was giving me nightmares.
But uh the tool we have now is called Fern and we started using it as of last Monday.
So it's very new for me and I'm
(10:20):
Still just like trying to learn and experiment and understand what is happening behind thescenes with things like the LLMs.txt conversion.
So ask me again in six months and we can have a chat about it.
Well, if you do the research and write it down, then we'll definitely want those stats.
So Fern's the one that does the SDK generation but also contains the docs portal, Was themotivating factor mostly on the doc side or the automatic SDK generation?
(10:47):
We don't use them for the SDK stuff.
Just the docs.
Yeah, I wanted a platform that was a little bit easier to maintain for me.
And I think they're kind of out of the box components are a bit more sexy than thedocusaurus ones.
um just overall, the team was super nice.
(11:08):
So loved working with them throughout kind of the evaluation process.
um
They, you know, the AI search capabilities and also this um other AI tooling like the LLMsthat TXT Generation was not something we had before.
So for me to just get all of that out of the box with as little involvement from my sideas possible was a win.
(11:31):
you know, doing the platform and the content and also these bunch of other things arounddeveloper relations um can be uh time consuming.
So I really want to dive into that a little bit because one of the things that has comeup, I'd say biggest learning for me was who am I writing these docs for?
(11:52):
And you're nodding your head, I'm sure you have some opinions here.
So my first question is like primarily when you're thinking about writing the docs, are wetalking about like end users like through Fern who are technical users?
Are they how-tos?
Are they guided tours?
Well, so I would say about 80 to 90 % of our audience is developers who are either justgetting started with feature management or maybe they've already got Unleash and they're
(12:22):
trying to figure out how to integrate their SDK and maybe 10 to 15 % are business decisionmakers who are looking to evaluate what platform to buy.
We categorize the content into three or four main types.
So we have kind of getting started content, which is very developer focused.
Then we have more of the tutorials and guides, which is more of a step-by-step kind ofhand holding, very detailed across all the different SDKs.
(12:52):
And then we have the API documentation, which is one of the largest category.
And then the SDK documentation and release notes.
So I would say those categories are pretty classic in terms of what you'll find on thetypical documentation site.
we typically monitor our usage around what SDKs our customers are using and try and focuson those languages when we do examples or uh guides and tutorials.
(13:20):
absolutely makes sense.
guess one of the things I sort of realized early on is that the message that I heard whenI was writing documentation in like Wikidocs was, like, who is your audience?
And I think all my English teachers from my entire academic career would always try to getthis point across.
And I'll say I never understood what that actually meant.
(13:43):
But I stumbled upon a few years ago...
uh There's a website dietaxis.fr, which I think really makes the point here.
And you brought up whether they're how-to guides or release notes.
But really the point is, who is this for?
Why am I even writing this?
I think developers are notoriously hard to track as well.
So sometimes you don't get all of the data you wish you had.
(14:05):
um But I do have information on what people are searching for in the search bar.
I get information on what people are asking in the AI search and um what percent of theiranswers are being answered correctly.
m Then I also get em feedback on every specific page.
(14:28):
Did this page help you or not help you?
Or feedback on specific code examples.
And also inside the product, we send out a survey to users on I don't know, I think threeto six month basis.
You get a survey request and then some of them fill it out.
And then there's questions around the documentation there.
(14:49):
So I try to rely on all of those different feedback points to figure out, this pageworking well?
And is it solving the problem that I
think it should be solving.
uh
I mean you're using the information to decide like what to write next or to change pages,to improve docs in certain areas, or is there something else there?
(15:10):
Yeah, I think one of the most useful ones actually is the AI search.
And the dashboard will give you an indication, like ah what are some of your content gapsor what are some of the questions that people struggle with and don't get an answer in
your documentation.
And that is one of the main ways I prioritize what to work on next besides the newfeatures and the developments from the engineering teams.
(15:36):
I have a question that's going to be controversial.
Internal oh documentation for your architecture, for your services, for other developers,does it live inside a Git repository and committed to source code?
Or does it go in some sort of open platform that enables anyone to uh update, edit, writeit as much as they want?
(16:00):
Well, we're an open source company, almost everything that we do is in the open sourcerepo.
And we also document our architectural decision records or anything like that because wehave quite a lot of contributors to also help with developing the SDKs.
And it's very useful for them to know, like, to understand the evolution of the product.
(16:22):
so it's either on the docs or um inside the open source repo.
The documentation is open source as well.
um We have a few things internally, around the cloud architecture and things like that, orlike run books, things like that.
But ah I would say not a ton.
We try to really talk about everything publicly and our customer success team and oursupport function very heavily relies on the documentation.
(16:53):
So we don't have a ton of things that would only live in Slack or only live in...
private site and I think that's quite good.
Yeah, think that one of the arguments is that there's a huge overhead to get documentationinto a Git repository, that the whole pull request review process and merging this is
(17:17):
non-trivial for someone without technical capabilities.
uh Whereas on the other side, it's, I want it to be accurate and I want it to be reviewedbefore it goes uh public or becomes the source of truth there.
So it sounds like you're more on the side of, like,
Let's have it in source control.
so my question is going to be, what do you do for the non-technical writers?
(17:40):
You'd be like, well, too bad.
Technical writers should always have a technical background and be able to use Git inorder to write documentation.
I think it is simpler than it initially sounds, like learning Git and working in Markdown.
I would say that if you understand the technology that you're working on well enough, youprobably are not going to have a lot of trouble learning DocSense code.
(18:02):
um But a lot of the platforms, the documentation platforms, like Friend, they also offer ano-code editor.
I personally have not used that a ton, but I have given access to
this editor to, for example, my boss who's not really got time to learn how to downloadthe repository and work.
(18:26):
I'm sure you could do it.
He's just not got the time.
So he's got access to this NoCo platform and I think it works like Notion and you can dragand drop elements around and fix just like a small typo and it will open a pull request on
your behalf.
So like the all the benefits of a WYSIWYG notion-like or confluence-like documentationportal but backed by uh some sort of Git repository that gets updated and is auditable and
(18:51):
trackable and reviewable straight away.
So I do think though you're in an area like where you're forced to have a disciplinedapproach to what it looks like both from a tool usage standpoint and documentation as far
as rollout goes because
The primary aspect of your business, feel like as well aligned with this sort of mentalityand that's feature flags.
(19:15):
Yes, that is true.
Yeah, what's been a little bit scary for me in this past year, like 2025 was uh kind of amassive year for these cloud outages, right?
So you probably remember the AWS one, the GCP one, the Cloudflare one.
And in my mind, some of these teams are world-class engineering teams who basically wrotethe books on reliability and we look up to them for their practices and to see
(19:45):
To see some of these things fall apart was a bit scary because they, I assume, neverworked at Google, but I assume they have world-class CI-CD pipelines and super
sophisticated setup.
And they're amazing at DevOps and getting code into production.
But even companies like that are not good at staying in control of that code once it is inproduction.
(20:07):
So that massive GCP outage that happened in, I think it was in June, it was because of asingle line of policy change in Google's IAM.
And they merged the code and it was live for a couple of weeks, I think, before itsuddenly got activated and then half of the internet was down.
I looked at the incident report in quite a lot of detail to see what was going on.
(20:29):
they were able to identify the root cause in about 10 minutes, which I think is prettygood.
And they prepared the rollback and redeployed within 40 minutes, also OK.
But like the...
whole outage still lasted about four hours due to all of this systematic recovery delaysand backlog clearing and stuff.
(20:50):
that's the real scary part that even if you have perfect DevOps and you're really good atgetting code into production, uh DevOps cannot bring your code back up fast.
this is where it comes.
I've always been a long time advocate for feature flags, but these stories from the pastyear really kind of reinforce that, that you really need that.
(21:11):
added runtime control where, you know, our rollback is seconds rather than like fourhours.
And if I remember correctly, both the cloud flare.
And the Google outage in the action items are kind of the summary of the incident.
say, we wish this was behind a feature flag.
So yeah, I think that it's a good one to look at.
And if you have time to read through those incident reports, I would recommend it becauseum I would guess that Google has built their feature flagging platform internally and so
(21:39):
has some of these bigger companies like Amazon.
But something is going on there.
And maybe it's the fact that they're treating some of these backend changes likeconfiguration updates or
Policy changes like it doesn't need a flag and a lot of times people think of flags aslike UI changes or kind of these more cosmetic things.
But we've seen that it is also very useful for some of these more behind the scenes hiddenchanges because they can also lead to real outages that are then difficult to roll.
(22:07):
You bring up a really interesting point there.
So historically, the outages at say like AWS, which is the one that I've been tracking isnot is usually due to very complex sets of
root causes, like not just one thing, like it was a race condition plus a number of otherthings and not the time they brought down the whole internet because some engineer
switched off like the S3 bucket database, ah which did happen.
(22:30):
the thing about the policy is that, ah and so I don't even account for like Azure or GCPoutages anymore because they always seem like simple things that happen.
They put uh multiple different uh availability zones in the same data center.
There's that one, and there was a flood, and so the region was offline.
But yeah, I think the Cloudflare one was definitely like, we treated uh changing a featureflag as not the same attention that we would have to merging a pull request.
(22:56):
And configuration changes are just as critical and can have widespread effects, especiallyif you have
your code behind a feature flag, then you should really be having the same attention oreven more because that's where the critical path is at that moment.
I mean, I think it's an interesting point that you bring up, especially these hyperscalerswhere you want to trust them that they are uh much more rigorous in reliability.
(23:20):
And as you pointed out, literally wrote the docs on, well, like Cloudflare talks aboutlike edge workers and GCP talks about like the SRE, you know, has the SRE book, whatever.
And AWS has like a whole
whole portal dedicated to high reliability stuff.
obviously they're down all the time.
yeah, there is a question there.
(23:42):
It's like, if they can't get it right, how is anyone supposed to help?
Well, we've definitely tried to build in the similar sort of capabilities into theplatform as you have on a pull request and GitHub, kind of diff and review of what's
changing in this feature flag.
Same way to add any number of required approvals.
And we really try to lock down our production environment so that at least two developersneed to approve a change to a feature flag in production.
(24:10):
And I think that helps.
But another thing that's helped us at Unleash is we love boring technology.
So we're one of those people who just, you know, we love try new things, but when it comesto the products that we're building for our customers, we really try to prioritize the
true tried and tested stuff.
you know, things like in feature flags, latency is a big thing, right?
(24:34):
So when you toggle a flag, you want um it to take effect as quickly as possible.
So architecturally, you can decide, do you want to do streaming or do you want to dopolling?
between your SDKs and the feature flag server.
so streaming is very sexy and instant updates immediately.
But we can see that when there's an outage, things take longer to propagate.
(24:55):
And we try to do an approach where, for example, we do polling in our SDKs.
You configure the polling interval, and it's very reliable.
But if you really care about instant latency, we do offer streaming.
But always fall back to polling.
Yeah, I can definitely see that there is a spectrum here, right, for companies in whythey're introducing the flags.
(25:18):
I think early on, they are maybe not really specific about what their goal is and thenthrow the same approach at every single instance.
Whereas it really does like anything, even if you have a provider backing you, you need tohave an attention to what is the right way to approach this.
Because on one side, ah I would say latency is good actually here.
(25:38):
You prefer to be slow.
to get the reliability.
uh But having real-time switches, I know is what the marketing department wants.
anyone who's done anything with high-reliability systems knows that the trade-off is uhcash invalidation or uh extra network requests, Very fast polling or streaming, have extra
(26:00):
connections up.
So you're paying the cost somewhere else, which isn't necessarily a good thing.
In one of the previous episodes, we were discussing
with the guests about how great it was that they could avoid even having faster than likeone second or one minute it being wrong.
Like it's okay to wait 60 seconds before this gets rolled out.
(26:20):
It's not that critical that everyone who comes to the website sees the updated version,you know, two seconds from now.
Like you don't need that level of precision because the trade-off is a high risk to yourproduct or you you're working in production.
And I think risk is a huge aspect here because the research from Dora, which we actuallytalked a lot about in the 2025 report episode, says that more untested code is actually
(26:46):
getting into production because of AI.
uh
more code, uh velocities up but like, ability is down.
Yeah, quality is quality is down.
The interesting thing is that people feel like they are being more productive quoteunquote, whatever that means.
uh But the actual quality metrics show us that solutions are getting worse for the endusers for customers for clients.
(27:11):
And so there is this question of if you're using feature flags, there can be a tendency tothrow the work over the wall and have it be relied on by just enabling the flag.
How do you get teams to be disciplined about uh making sure stuff is tested beforeactually deploying it to production, even though it's behind a flag?
actually what we do at Unleash is we do breakathons with every new feature.
(27:34):
So we enable the feature in production only for ourselves and then get together on aGoogle Meet call like all of us and like try and break it.
So spend like an hour together, try and like find all of the things that are wrong withit.
And it's very fun actually.
um Maybe not so much for the person who built the thing, but for everyone else it's veryfun.
(27:54):
So once you're happy with it internally and you start rolling out.
we're rolling it out to say five or 10%.
You can put in some automation in place that says the error rates are below this thresholdfor let's say 12 hours.
Then we can progress to the next stage, which may be only 10 % or it could be 50 % ormaybe it's a segment of your customers that you think or you know are more like
(28:18):
experimental and ready to try new things rather than the ones where you really need thatstability.
Then you kind of progress through these stages and you can go away from your
laptop, you can go to sleep or whatever, and you know that if those metrics spike, thenyour rollout can be paused automatically or go back to the previous stage or whatever you
kind of define.
So I think we're probably at a good point to move over to pics for the episode.
(28:42):
Nice.
So I'll ask you, Melinda, what did you bring for the audience today?
So I brought a game.
It's a game is called the wavelength.
Do you know it?
No, it's like a how do I describe it like a communication?
collaborative communication game and it's something that I've played with five-year-oldsand my friends and family and But also like my co-workers.
(29:05):
We actually love to play this game as like um We're a remote team and we have a team hourevery Friday and we always pick a game and this is something we played
recently and like had tons of fun with it.
The way it works is it's a board game as well, but there's a digital version, like an appversion that is quite good.
You get a spectrum and with two extremes at the end and it changes every round.
(29:26):
And let's say in one round, you get a scale, which is from good pizza topping to bad pizzatopping.
And you get a random point on the scale and only you see that point in the scale.
And so if it was like off center towards bad pizza topping, you have to come up with aclue to your team.
to help them identify where that point in the scale is.
So I would say maybe like pineapple and the team have to like.
(29:47):
Going straight for controversy right there.
She's like decided before the episode.
You know what?
I was gonna be pineapple on pizza.
That's gonna be my example
Just trying to help you with the YouTube comments, you know?
And so the team like debate and discuss what um like what pineapple must mean and I tryand identify that point in scale and then you score points based on that.
(30:08):
But it's so much fun.
So I really recommend it if you're looking for something to play with your team or at
Yeah, really embodying that Italian philosophy there.
ah Yeah.
So my pick is actually a television show this time.
It's uh called Bosch.
It's a LA detective procedural.
Have you heard of it?
ah I started watching in December and I've binged like all 10 plus seasons of it.
(30:34):
uh Because it's just so great.
I love it as well.
I love detective shows and like hospital dramas and stuff.
I don't do the hospital dramas, I think the main actor uh Titus Wolver is just absolutelyfantastic.
It reminded me a lot of Law and Order, which I watched a lot when I was younger.
(30:56):
And it's so much better.
Honestly, this may be one of the best procedurals I've ever seen.
uh And it does these nice skips during the show to get rid of downtime that you wouldotherwise have to deal with.
You never know what's going on.
You get dropped into the middle of a situation and it's like, I'm still trying to figureout is there going to be a crime or what's going on in these people's lives at this
(31:20):
moment?
it's like always getting into a new show, which I find they really captured well.
It doesn't feel like every season is just like a continuation of the one before it.
does feel like uh new um every time you watch it.
Yeah, plus one for that.
Go watch it.
Well, thank you so much, Melinda, for coming on today's episode.
It's been absolutely fantastic.
Feature flags and how to use them correctly and most importantly, uh what's next indocumentation.
(31:46):
And thanks to all the listeners and viewers for coming on for today's episode and I hopewe'll see everyone back next week.