September 17, 2026 • 38 mins
Share Episode
From your feedback, we know that on-call issues, especially those that arise at the database layer are always the most challenging to solve. This episode builds on our Grafana Observability episode as we go deeper into the challenge of data ownership in data lakes. Apache Druid creator, previous Yahoo Distinguished Engineer, Fellow at Splunk, and currently the Chief Architect as Imply, Eric Tschetter joins us to confirm that there are unsolved problems in data platforms, and it doesn't get better when we throw agents into the mix.
In the true DevOps mindset aligment, Eric reminds us that whoever is operating the system, knows what is going on, they know their service boundaries, and so it's their choice to make. And the sad reality is that at large enough companies, you will always have independent teams making independent decisions, it's never possible to fully align them, and realistically we get down to why it is even a mistake to try.
💡 Notable Links:
- ✨ Episode: Grafana — Observability
- ✨ Episode: AST can run anything
- Book: Lean Manufacturing — The Toyota Way
- Monkeys typing the Complete Works of Shakespeare
- Paper: Ironies of Automation (& AI)
- Jeff Bezos API Rant — 2011
- Book: The Formula, Science Success
- Show: Burn Notice
- Warren - The Agency
- Eric - 継続は力なり — keizoku wa chikara nari
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:07):
Welcome back to Adventures in DevOps.
Our listeners rarely can find the time to gift us feedback.
When they do, at the top of their list is the current issues dealing with on calls arisingfrom the database layer.
A few weeks ago, we brought on the Grafana X CTO and co-founder to discuss whatobservability looks like in 2026.
And now we're going to go even deeper.
(00:28):
Today's focus is on challenges with observability, even when you get it right, schemamanagement, slow queries, and their consequences.
To get us through this,
Our guest today was previously fellow at Splunk, distinguished engineer at Yahoo, and iscurrently the chief architect at Imply.
Eric Cheddar.
Welcome to the show.
Thanks for having me, Warren, and um I'm really excited to be here.
(00:49):
our main topic for today, which is of course observability.
It seems like that no matter the size of the company, investigating incidents always seemslike it's a going to be a core component.
Like I just don't see that going away.
Even S AI SRE products popping up left and right.
I think fundamentally most of the challenges that we'll continue to see are how should werealistically store our data to make it actionable?
(01:11):
Absolutely.
Absolutely.
Like, I mean, I believe that the introduction of AI and what it's doing, it's actually arepetition of manufacturing automation.
And because those robots can create so much more so fast, the supply chain and logisticshave had to scale massively in order to get the materials to the factories so that they
(01:35):
can keep up with things.
And I think that draws a parallel with the data platform.
Like every time I use Claude, almost everything Claude tells me is wrong to some degree.
And I there's some amount of r adjustment, poking, pushing, prodding that I have to do inorder to get stuff.
And so that job, even with an SRE or whatever coming in as an agent, there's still a humanwho needs to be able to validate what's going on and and double check things.
(02:02):
I feel like for the last 30 years or so, engineering leaders have been trying to figureout how to take the ideas of lean manufacturing and apply it to software engineering.
And the ideas of lean are like there's different waste from a theoretical standpoint thatexist in your organization.
And removing those wastes means that more of your time, resources are spent to deliveringvalue to the customer.
(02:23):
And there's a whole bunch of different ways, but I'm not obviously gonna go into them.
I'm sure having spent some time there, you are somewhat familiar with with that concept.
The thing is I think I want to pull out is this idea that the work that software engineersare doing or engineering in general is doing is similar to the factory floor.
(02:43):
Do you think that a lot of the software engineering that we're doing today is repetitiveactivities that we're can can just simply automate, or is that creative, dynamic work
where everything is a unique one off?
The the
Creation of the run book is creation of repetition.
And so everything that has a run book is absolutely repetitive.
(03:04):
I think now some people are trying to use LLMs to answer that question for them.
We'll s we'll see if that turns out to be successful.
no
From what I've seen, LLMs like coming up with what should be done, horrible.
Bad odd.
It is it it does it does not work.
They're horrible at coming up with ideas.
There is this theory, if random chance makes you successful more often than intentionalactivity, then might as well let the LM do it because there's a chance you'll be
(03:30):
successful anyway, or more successful than letting a human decide.
Still waiting on the monkey to type up Shakespeare's works.
I I the yeah, I I know what you're talking about.
And I think, you know, that's that's a pretty good um, you know, counterpoint there wherethe original study was like a hundred monkeys typing on a hundred keyboards.
Uh so obviously it's like we'll eventually produce the works of Shakespeare and everythingelse ever produced, and the result was just a lot of like A's and E's on the keyboard
(03:57):
being hit over and over again and then, you know, punctuation and that was it.
I think there was some like throwing feces involved.
Uh at
And I think that is a very apt analogy for what L LMs are doing.
There's a there's a lot of people out there who try to sell the agentic SRE or agentic uhSOC analyst.
And so the agent can actually look across your silos and figure out what's going on, andyou don't have to worry about the silos anymore because the agent figures it out.
(04:24):
Um, which is kind of true.
The thing that they're missing is that if the agent is the only thing that can look acrossthe silos, how does the human supervise it?
How does the human ever know?
That like, yes, what you're doing actually makes sense.
It instead what it turns into is, Agent, are you sure that what you said is right?
(04:45):
And the agent's like, Yes, I'm sure and you're like, Okay, I guess that's right then.
There's this great paper that I believe I was actually my pick in a previous episodecalled The Ironies of Automation.
When you automate something, there still needs to be a human supervisor of that thing.
And the question that the human supervisor will have is exactly the one that youidentified.
How do they know that the system is working?
(05:05):
And so you need to expose some data from that system enough so that a human can go in andinvestigate what the problem is.
The issue is that the human learns the job of doing the investigation of that system.
Not of how to do the system, but of the job.
And often those people historically had a job of hands-on doing the work.
So they did the work historically, then they learn how to evaluate and monitor the systemand fix it if the problem is broken.
(05:29):
So when we shift to an automated system, we take the hands-on work that someone was doingand translate it to having to do the investigative work and the debugging work.
And then if something goes wrong, also doing the original work that they did.
Over time, systems become more reliable.
So fewer and fewer people have hands on work.
Now all the humans left only have the investigative work and the debugging work and neverthe hands on work.
(05:54):
So even if they're able to identify the problems that've been created, they are unable tofix it because we just don't have those people anymore.
So by automating stuff, we make the humans' jobs incredibly more challenging to actuallysolve.
The the complexity of doing that job becomes greater.
So unless you have a real problem that requires increased scale.
(06:14):
Automating it only costs you money.
So you should think about before introducing agents to any system, you should always askthe question, could we still do this with humans?
Because if the answer is yes, then you shouldn't automate it because that's going to endup in a worse state.
And I think this is gonna bring me back to asking about, you know, what do we actuallymean by the data platform part and whether or not humans could still be building and
(06:36):
maintaining that?
I don't
think that that um completely goes away.
Well if if it goes away there's hopefully some new uh some some new future.
What's the sort of frame of reference here?
So if we have a log solution, we're throwing logs in it, obviously those are productionlogs.
but maybe you're also talking about infrastructure logs or business relevant metrics, uhet cetera.
(06:58):
I love the question and I love the personas that you've uh introduced.
It turns out some logs are business relevant.
Some logs are inf only infrastructure.
Some logs are both.
And sometimes you need to do some things to the logs in order to make them businessrelevant or make them other things.
And the the traditional set of data tools that have been available have created a bit of aspaghetti of um, okay, let's put all of our logs, for example, uh going concrete, let's
(07:26):
put all of our logs into Splunk.
But then the PM wants something, and so they're told, okay, use Splunk for that.
They use Splunk for it, and then they need to share it with someone else.
There ends up being this notion of, okay, let's teach the business analyst to use Splunk.
They're like, I don't the thing that's most interesting about logs is the structure of thelog and not necessarily how it's used.
(07:49):
Where traditionally we've believed that you have to take the data and move it to differentsystems because of how it's used.
But that's not actually true.
The actual thing is that a log is interesting in and of itself.
You can compress it with kind of domain specific technologies.
There's specific ways that people want to extract things out of logs.
There's specific ways people want to look at logs.
(08:09):
But you can take that log and make it SQL queryable.
You can take that log and make it exposed in other systems.
You should be able to have your log, use it from whatever system you want.
And it's the same for the agent as well.
Like the agent should also be able to access that same data.
You're definitely an optimist.
The fact that you think that the logs in these systems are valuable.
Right.
Because the number one thing that always comes to me is like, our logs are useless.
(08:32):
Like w this log doesn't tell us anything that we actually want it to know at this point.
we've seen as we've talked with different people and worked with stuff is especially inthe observability world, the useful lifespan of a log is much shorter than in the security
world.
I mean, I think that's definitely accurate.
You for sure often don't know what will be necessary later, but at the same time peopleoften risk averse.
(08:59):
I think that's just a true a true fact.
and that includes uh hoarding logs that are useless uh because they don't they don't knowwhat they're going to need in the future.
With conscious thought you or deliberate action you would know you know, which logs wouldactually be relevant.
The
The logs of people interacting with the site, the logs of people interacting with theonline banking, or the logs of people interacting with the e commerce site are useful for
(09:25):
the observability team because they use it to figure out, you know, what what causedoutages, what was going on, how many people were impacted, what uh what things happened.
You can you look for the logs in order to discover anomalous behavior that you can respondto.
The exact same data set that's coming in that ends up powering each of them.
(09:48):
It's the access logs from how people are interacting with the website.
It's event telemetry from the single page JavaScript application that's being fired backabout how the users are working on stuff.
That often turns into, well, we have to get this data in, but then we also have to makesure that this subset of this data gets ETL'd over to this other system so that these
(10:09):
people can look at this thing.
And we have to get this other thing over here so that these people can look at that thing.
Oh
Historically, at least in my teams and my organizations, it we always go with the theDevOps mindset.
If you build it, you run it, which also means you capture logs and you have to reviewwhat's there in case of an incident, you're responsible there.
So of course the logs that you're generating have to be meaningful to you.
(10:32):
Um and engineers obviously have a notorious track record of being hard users to please.
So
The platform that we utilize in any of those moments has always been the thing everyonecomplains about.
Uh they hate historically, they hate sumo logic, they hate Elasticsearch, they hateGrafana, they hate CloudWatch insights.
You know, you stick in something here, uh logs log z dot io or whatever, however youpronounce it.
(10:55):
I mean, the list is goes on and on of all these platforms that I've seen w some team trytry.
Because every single time I come in, I I see they're using something like we hate it.
I'm like, great, do some research.
Find a platform you love.
And then we'll try that one.
They do that and they hate that one too.
And the re the reason I I say this is because even if we were able to find something thatthey liked, there's still this problem of are they collecting the right information?
(11:19):
And even if they are, how do other teams get access to that in a way which they're notheld accountable for the data that shows up in those logs?
And the reason I I bring that up is because historically there's this challenge, right?
There is some production data.
which another team uh or someone else in a different role, as you pointed out, likeproduct managers or business analysts, say, you know, hey, we wanna be able to perform
(11:44):
this query on our data.
And historically they'll they'll use one of their giant data lake uh SQL back querysolutions to do this.
And the question is how does the data get from the production system into that other, I'llsay third party system?
Is there an act an ETL extract transformer load job that runs?
If it does, who owns that?
(12:04):
Which fields are extracted or whole databases extracted?
And it doesn't seem like there's a right answer here because if you extract everything andand another team owns that responsibility, that's just always constantly broken.
Like if an engineering team doesn't do it, then the the data changes all the time.
Uh someone complains, hey, my query broke.
(12:24):
The ETL job breaks down because it all of a sudden security says, Hey, we can't beexposing full access on the production databases to third party tools.
So that that stops working.
And even after all of that, someone says, Hey, we're not actually getting the data that weonly want in the first place.
It doesn't even exist.
You know, we're not recording it.
(12:45):
We're writing it down, right?
We wrote other things.
Just like in a lot of production systems, we may forgot to write audit trails, right?
You know, you don't always have everything up front.
And on the other side of the spectrum, you have engineering teams being forced to der pullhand picked fields.
from their production databases and sending them over to your tableaus, your Splunk's,whatever have you.
(13:09):
And then the data isn't there anyway.
But at least they're responsible for doing it.
And you can go to a team and say, Hey, we're not getting this field.
Can we add it in some way?
That
is a great question.
It's a question that business intelligence has fought with for a very long time.
It's data governance.
Uh you you in in the business intelligence world you end up with these with people whohave these UML diagrams of table structure and they're like, thou shalt generate things
(13:37):
that follow this structure and put things in and
But developers never j like developers are horrible at at like aligning with that.
They're like, I need this thing.
I'm gonna add this field.
I'm not gonna look up what it's supposed to be named based on these people in this otherteam.
And so the there's that.
So I don't um I don't know exactly how to solve the data governance problem.
(13:59):
I don't believe that trying to organize everybody and create structure out of it isactually ever gonna work.
Because especially as you expand in an organization, you're just never gonna get everybodyto agree.
I I think I I have seen that the most successful mechanism for the data transfer betweenuh production or uh engineering organization and their databases to other people to be
(14:23):
able to use it is when the engineering team is uh responsible and accountable for decidingwhat their interfaces are going to be and what data they're going to expose in the schema
is as if they're maintaining their API format in an open API specification, except theformat and the schema that they're maintaining is in
uh some third party or other service running, such as you know, uh an object store in thecloud somewhere or Snowflake or, you know, what have you.
(14:51):
And then the team says, yes, you know, we are we we are accountable.
We decide as part of the responsibility of our team that we're going to publish data tothis specific repository.
And here is the schema for that.
And anyone who wants to use it can use it.
And that's it.
End of story.
Because then you don't have a separation between
Someone who's trying to define or has expectations on that schema and the team that isbasically going to be held accountable for it.
(15:17):
I think putting the ability and responsibility uh to the team that will end up doing thisis the right thing.
You know, you push down the decision and ownership down to the team that has it, uh andnot expose the whole database.
If
you talk to anyone at a company, even at Yahoo size, and ask them what technologies areyour company using, the answer will be everything.
(15:40):
Like, are are you using this?
Yes, we are.
Somewhere.
I don't know where.
Somewhere we're we're probably using it.
That that's just such a common mode of operation in a large enterprise.
As much as you might want to force consolidation of technology, a new company is going tobe acquired.
And that company that you acquired had no idea about any of the things that you tried toenforce until after you acquired them.
(16:05):
And now the question is like, is it actually worth it to carry all of that over?
And I think all of this is just pointing to this notion of silos and independent decisionmaking.
I just saw every technology used in all the places.
Uh there were there was one team that um was
(16:25):
for telemetry and metrics, they were using one system for it.
my background is uh kind of with Apache Druid and stuff.
I started the that project and and built stuff and so we used the teams that I worked withwe used Apache Druid to do uh some advertising analytics and and things like that.
And there there were conversations sometimes with the telemetry folks around, you know, wecould use this for that.
(16:48):
I don't think they were wrong
They never really were like, Yes, we're we're gonna adopt this and run with it.
And I don't think they're wrong for that choice.
They're the ones operating the system.
They kind of know what's going on.
They're they're getting at it.
They have their own service boundaries, so they they get to make their choice.
We had other teams, you know, that were though for analytics it was heavy on Hive and onum Tez and Hive running on Tez and and all of that stuff.
(17:15):
This was back when uh Hadoop was still a challenger.
At large organizations, that you will always have independent teams making independentdecisions.
And I don't think it's ever possible to fully align them.
And so instead of doing that, you have to lean into what is something that gives us theability to just store the data as is and then connect the dots later.
(17:42):
When a a local team makes a decision and they're like, this is how we're going to exposeit almost from an open API.
perspective, right?
If it's a rest endpoint that they're using to expose it from, how does business analystuse rest?
They they don't unless they they can tell SQL to go and issue uh REST requests, right?
See, I was gonna say correctly, but I I mean I guess I guess incorrectly is also the uhviable answer that's in the multiple choice drop down there.
(18:11):
Yeah, like they they but they're they're just not going to.
They're gonna be like, I don't know how to I don't know how to how to interact with that.
So here here's a counter argument there.
Uh organizations that followed the rant that Bezos had set off, and now it's been I thinkover a decade, is that data analysts would be required to use the rest interfaces to
(18:32):
expose or get the data that they want from other teams.
So I I totally agree though, in most organizations, you don't have such a strong andspecific and maybe mature mindset about how teams will operate on top of
technology.
And I I feel like if some teams aren't as technologically savvy, then they are limited intheir capacity to solve particular business problems.
(18:55):
Or if the tools they're using because someone made a bad business decision and bought a 10year contract with a company to use a particular product that refuses to add in a OAuth to
integrated supported interface to or even REST API.
You know, maybe it's using WCF or SOAP or something horrific that reads and writes, youknow, is it flat
(19:16):
Text files on a on a physical machine somewhere, then you are limited by the real worldsituation there.
One one thing that does come up, and I think you sort of hit on this, is that it doesn'tmatter what the situation is, even if you prescribe the perfect tool for the job, someone
someone on the other end can always still mess it up.
(19:37):
Right.
Well they'll mess it up.
My team has defined this API.
So if you want to use it, you need to change what you're doing in order to align with mydecision.
Getting humans to change their behavior is, I believe, the most difficult thing to do inthe world.
You're more likely to get people to be like, okay, I don't need it than you are to getthem to actually change their behavior.
(20:00):
And so this leads me to the what if.
So what if
You could have your data in a place, and that data could be accessible via the tool ofyour local team's choice, and also accessible via the tool of that the other team's
choice, and also accessible via the tool of some other team you've never heard of'schoice.
(20:26):
But you have the data in that one place, you can own the definition of it, and it can beaccessible from all of the different places that people have.
That is, I believe, the Holy Grail.
Yeah, I know what you're talking about.
You're about to say it it's going to be give them direct access to the production databaseso they can read the files on the server directly and use whatever data they want there.
(20:49):
You you have the files up on cloud storage.
Yeah.
So if you use that as a way to disintermediate your production systems from the actualkind of analytics and and access of it, have this three layer cake where things have
decoupled to allow for it.
And the the primary thing that allowed for that decoupling is the SQL query language.
(21:14):
Where that's fundamentally the API that the visualization layers use and that the businessanalysts use.
And so when you talk about um the team should define the API that they're gonna support,the the thing is there's not just one API that you're gonna support.
And in the logs world, people want to access logs from SQL.
(21:34):
And the language is more belonging to the persona, where the data format itself, the log,
can be queried from a number of different languages and different personas preferdifferent languages because the the language is aligned to the job that they're trying to
do.
We have had a similar challenge in in our space when we're defining the, and I hate tocall them policies, we call them access records for one of our products.
(22:00):
And I mean, we provide like login and access control as a as a SaaS.
And customers come in and say, Hey, you know, can you support this?
Like, we love this particular format.
I'm like, or this format, or this format.
And uh realistically, it's all stored as JSON for us.
It uh I mean you can store it as some other DSLs, like they're
The every every domain specific language, DSL, is is the worst.
(22:23):
They're they're all literally the worst.
Uh they're almost like a programming language, but they promise that they're parsableuntil people find vulnerabilities with the the language itself and find ways to in inject
attacks in.
And they're all unique that you have to understand the domain in a lot of ways to actuallybe able to write one or write the the form to even do the query in the first place.
(22:44):
And those that are amateurs and have come from one world make it very difficult to
expose in different way.
But it's very easy for us to add in different content types to our API and expose theexact same data via different formats because all of them are constructible.
And the one thing I learned, I say, my last 20 years of engineering, and it wasn't thatearly on, maybe this is a bit of a regret, is that I always tried to avoid SQL uh when
(23:09):
making an API for querying particular data on a production database.
It it's the worst, you know, let's use something constructible, something nice.
that is can be defined in in JSON and passed over the API and can be validated.
But over time, what you find out is you just made another bad uh domain-specific language.
You just made a worse DSL somewhere.
(23:30):
And at this point, I'm just like, everything should be able to speak SQL because it has isessentially the thing that humanity has centered upon working correctly.
And it's portable from one system to another one.
And as long as you can provide that
specifically as your query language, then often it's easy to add these other things ontop.
And it sounds like for your product, you know, for your data pipeline, uh being able toport from the underlying data store to the to the bad terrible DSL that every single
(23:57):
different persona has said, you know, I want it in CQL in and SQL and whatever, whateverhave you, and also understand the semantics of that particular language, like the the
MySQL of the world versus the
MS SQL, the Microsoft SQL.
Like it's not exactly the same, right?
It it there are some fundamental differences there based on the data.
(24:18):
Getting the semantics right is definitely a key element of things.
So there's some people out there who are like observability, SQL for observability.
Everything should be SQL, all observability should be SQL.
And while I think that everything should support SQL, I actually think that the thediaspora of different query languages for logs is indicative of a different usage pattern
(24:45):
that
is actually extremely meaningful for working with logs that is not as interesting forworking with business data and like rigid column or structure data.
And so I I think that I I do actually think while right now there's tons of differentquery languages for logs, I do expect that to converge down on a subset of query languages
(25:10):
that uh will remain and that that will exist.
Changing human behavior is hard.
It instead of telling them that they they need to change what they're doing, if you canjust meet them where they are, um, you can see that's in metrics too for observability.
Like people use PromQL so much more than they use SQL for for metrics and telemetry.
Um really PromQL kind of has, I believe, has won as a de facto standard for metrics andtime series style things.
(25:37):
I'm I'm less aware of different languages in that space.
And you're totally right.
The smart thing to do is to convert whatever interface someone has into the best, mostgeneric solution that can be run on any single platform.
So it doesn't matter what DSL someone is using, if you can convert that to an abstract
syntax tree and AST, then you can directly execute that AST on whatever run engine you'vegot.
(26:01):
We actually had uh an episode on how uh a company is doing that specifically and I willplug that in the description.
But I I think this this may this solves a lot of problems, right?
You don't have as long as you can come up with a can a transform from that to an AST thatyou can run and you just run the AST, you've solved the problem.
And if you can't convert it to the AST, then you know that there's a feature in the deadDSL that
(26:23):
you aren't able to fully utilize.
And then the question is why.
I do think there's an interesting uh corollary here, realistically, whereas if you are ina different space, if you can't convert it to the same kind of ASTs that you're used to
running, are people thinking about that problem differently?
And I'll just use an example.
I think very early on I was full in on traces.
(26:45):
This is the idea of storing log information that is grouped together during the wholerequest and maybe across requests.
At a single moment.
So when you want to query something, you're looking at a single object with lots ofadditional data in it.
An example is like, let's say there's an error in production and you see a log messagewith that error.
Now maybe you have some sort of correlation ID.
(27:05):
And so you make your SQL statement, select all the rows where the correlation ID equalsthe value you know.
You're pulling multiple different logs and you're getting that data.
For us, that was always really annoying.
Especially when you have a whole list of correlation IDs that you want to pull stuff forand you want to join it up with.
So very early on, what we did was whenever there was a log message, well, during the wholeprogram execution, we would store a property bag of all the context of all the variables
(27:29):
of everything that was happening, and we shove it into the error message.
So when it gets logged, we have everything there in one place that we can actuallyvalidate and use.
And I think this has caught on recently.
I think uh in the last few years, Honeycomb was a huge proponent of what they were callingobservability two point zero.
And I think since then a lot of other companies have jumped on the bandwagon here of beinglike, hey, you know what?
Storing line by line stuff isn't necessarily the right answer.
(27:51):
The reason I bring this up is because that query sort of depends on the fact that we havedata stored in a particularly different way.
And so it wouldn't be straightforward to necessarily convert from the query language to anAST that could run on these different kinds of data.
So it's not just necessarily about the
transforming of the data language that we're using the the DSL to something that we canexecute, we also need to understand fundamentally how the data may be relevantly stored
(28:17):
for accessing, you know, things like indexes, the what's in a a wide table format or acolumnar format, like these things are potentially relevant, whether it's stored as a blob
and whatnot.
So there's there there are a lot of complexities here that any company would potentiallyhave to deal with.
And obviously, you know, some areas that you you've seen particularly.
I mean, obviously if it's just giving the consumer a DSL that matches their expectations.
(28:40):
You know, you could do a lot of hacks under the hood there.
but outside of that, you know, you really start to come into areas where there arechallenges because fundamentally the data has to be exposed in different ways.
Absolutely.
And and traces is the a great example.
Like I've been talking about logs a lot.
Interacting with traces and a language for interacting with traces will fundamentallyunderstand the notion of a trace versus a span.
(29:06):
But the actual like fundamentals of how is the data stored, how is it persisted, how doyou index it, and which query languages are supported is actually a function of the data
format itself.
And so a trace
requires different languages.
And same with metrics like PromQL.
You could convert PromQL into something that runs against logs.
(29:29):
the language believes it's working with time series objects.
And if you're reconstructing time series objects at query time, you're wasting a lot ofCPU.
Well, I Postgres, I think, would disagree with you because they've managed to just say,yes, we're always going to take all the data and store it in rows and columns, uh,
including your you know key value storage and and graph graph databases and vectors andwhatever and somehow make it make it work.
(29:50):
But and then there's i of course the alternative argument, which is that uh the relationaldatabase is always the fastest, no matter what.
Uh which which I've also seen is true.
Uh we we've sort of validated that graph databases are so slow that performing
Stupid not great queries in in SQL tend to be faster uh by making lots of extra queries,ten queries at the same time and stitching together the results, rather than using a graph
(30:15):
database, which is sort of uh unfortunate in a lot of ways.
But I I mean, you sort of echoed my point, which is that of it's not necessarily thelanguages which are the problem, but some of the semantics of them, but realistically how
the data is being stored.
So how do you deal with the fact that the data format is important?
relational database can store everything, yes.
And everything that's in a relational database can also be a log.
(30:37):
Um everything that's in a relational database you can also model as traces.
I don't know that there's actually a one size fits all.
This is the way that all data shall be modeled.
Back to the uh if you don't have the business problem then there's then there's no there'sno issue technological problem you have to solve.
(30:58):
It's always a great one.
Uh which is why pushing back is important, right?
if you tell someone that, it's gonna be really difficult to do that, they may go back andfind out that they don't need to do it after all.
Y yes, yes.
You know, uh, Eric, I think this may be a good moment to switch over to picks for theepisode.
So maybe you could share with the audience what you've brought.
(31:19):
Uh yeah, I spent a a bunch of time in Japan.
really enjoyed learning the language and stuff.
And and learning the language to me is I think computer science and all that, it's alsolanguage learning in in my mind.
I think of it as as similar.
But the the big thing is there's a a saying that I love uh from Japanese.
(31:39):
It's keizokwa chikaronari, which literally means persistence leads to strength or
Continuous improvement kind of leads to strength.
And it's this notion that if you do something five minutes a day, ten minutes a day, aftera year, you will have done 3,650 minutes.
(32:02):
After two years, after three years, you will have done it so repetitively that you willfind that you are now capable of doing it.
And and I think looking back on my life on everything, like there's a bunch of things thatI did and then stopped doing.
And maybe I was okay at it for a little time, but now and I can't do it at all.
But it's that notion of like, it's not so much about getting to this end state and beinglike, okay, it's done, declare victory and we've won.
(32:29):
It's about making sure that you're continuously investing and kind of driving towardssomething, whether that be large time commitments or small time commitments.
Uh I I think this notion and this philosophy, I just I I love it is that
The things that are really important, if you just continuously work on them, even if it'snot a lot, out outcomes will will come along with it.
(32:53):
The non research, the sort of feeling result was that it's about a thousand hours ofanything before you become a master.
And a five minutes a day would take you about 20 years to master anything.
Uh, but then you can look at it that way.
After 20 years, you can have mastered whatever it is that you want.
Uh, I think the research actually comes from a book called The Formula, The Science ofSuccess by Barabasi.
(33:16):
And it was where they evaluated which
in there, there was a lot of good research, but one of them was which Ol Olympic athletesbecome Olymp like successful, like actually get gold medals.
And the ones that get to that level are usually not the ones with raw talent or ability,but the ones who go through significant practice and basically perseverance through
(33:37):
through training to get there.
So not that you're born with it, but that you actually went and did it.
They weren't the best y younger on or when they started, but by the end that that is whenuh they were still able to achieve
I mean obviously there are some, you know, naturals that still take it, but by by andlarge, the most victorious ones are are the ones that just continued at it consistently.
(34:00):
couple it with the notion um every time I look in the past, time always went really fast.
Where when you look in the future, time like it if things seem much further out.
Even that five minutes a day, after three years, looking in the past, you will be able todo something.
Maybe you're not a master, you'll be able to do something and it will have taken like notime at all because it will have all gone extremely quickly.
(34:24):
I think especially in the Western world now, there's w it does feel like things arespeeding up rather than what has historically been recommended, which is slow down and do
things deliberately.
And uh I think maybe maybe AI is to blame for that.
Maybe cultural is to blame for that.
Uh we'll see.
That's not related to my pick.
Um my my pick uh is and only 'cause I just completely binged it was a television showcalled The Agency with Michael Fassbender.
(34:53):
Um
It has to do with the CIA operating out of London.
I mean, obviously, complete fiction.
and I don't know what it is.
I I I don't I don't have love for stuff that is centered in the in the US.
I I always I don't know what it is, but the CIA operating of London, you it's it honestlyis really, really good.
And I think it's produced by Showtime.
(35:13):
Okay.
Nice.
So I don't know if you're into uh spy dramas.
Yeah, I'm I'm trying I'm trying to remember that name.
There was uh there was a a show a while back about uh an ex spy based out of Yeah, BurnNotice, yes.
Uh
Burn Notice, uh absolutely fantastic.
Uh as well.
(35:35):
Yeah, it's it's definitely something you can also binge.
It's it's great.
I mean the nice thing about Burn Notice is that it was produced in a time before w showsgot canceled before they were over.
Michael Weston, uh not the actor, the the character.
I don't remember the actor's name.
Uh he's in a bunch of things, but uh also very good.
(35:55):
So
Yeah.
I d I don't know what it is.
Uh and you know what, there's sort of like it's not like such negative dark comedy or ordrama.
Like the dark the it's not as bad as dark drama.
I mean there's still stuff going on, but it doesn't feel like uh they just do it for thethe kick, so
(36:16):
Like one of the nice things about Burn Notice was that it it was very episodic, but therewas a little bit of a background thread running between them.
And so you could enjoy one episode and and get completion, get some sort of closure, butstill there was like the the allure or like what's going on behind the scenes, who's
actually doing it.
(36:37):
Is is the agency like you gotta watch the whole season to get the gear closure or is itone
Yeah, I think that unfortunately that's still where we're at.
Don't worry.
Uh the second screen experience is is coming soon and every show that shows up on astreaming service will all be episodic so that people don't have to pay attention anymore.
But the agency is still I think it's it's not totally epis um not episodic like a singlethread, but it uh you pretty you do need to sort of know what's going on from the previous
(37:06):
episode.
There still is a major thing there.
There's two seasons right now and they are
Sort of separated.
So you could watch only one and feel like it's complete.
Still, yeah, unfortunately, if you're just looking for single stuff, that's not it.
I will say that the nice part about Burn Notice was that they also sort of educated you onwhat is reasonable in Spycraft.
(37:28):
I don't know if that was accurate.
I mean, I think there are lots of videos on YouTube basically saying, you know, thesethings they show in shows, you know, aren't real.
I think Burn Notice does a better job.
And I think some of the things they do in the agency also, like if you're payingattention, uh really help to clue you in on like how things actually work in practice
rather than like today, rather than uh just doing stuff for the for the heck of it.
(37:51):
Okay.
So um thank you, Eric, for coming on and discussing observability with us and thechallenges of managing the database.
Yeah, thank you.
Thank you for having me.
It was great discussion.
And thanks to the audience for tuning in for this week and hopefully we'll see everyoneback again next week.
Welcome back to Adventures in DevOps.
Our listeners rarely can find the time to gift us feedback.
When they do, at the top of their list is the current issues dealing with on calls arisingfrom the database layer.
A few weeks ago, we brought on the Grafana X CTO and co-founder to discuss whatobservability looks like in 2026.
And now we're going to go even deeper.
(00:28):
Today's focus is on challenges with observability, even when you get it right, schemamanagement, slow queries, and their consequences.
To get us through this,
Our guest today was previously fellow at Splunk, distinguished engineer at Yahoo, and iscurrently the chief architect at Imply.
Eric Cheddar.
Welcome to the show.
Thanks for having me, Warren, and um I'm really excited to be here.
(00:49):
our main topic for today, which is of course observability.
It seems like that no matter the size of the company, investigating incidents always seemslike it's a going to be a core component.
Like I just don't see that going away.
Even S AI SRE products popping up left and right.
I think fundamentally most of the challenges that we'll continue to see are how should werealistically store our data to make it actionable?
(01:11):
Absolutely.
Absolutely.
Like, I mean, I believe that the introduction of AI and what it's doing, it's actually arepetition of manufacturing automation.
And because those robots can create so much more so fast, the supply chain and logisticshave had to scale massively in order to get the materials to the factories so that they
(01:35):
can keep up with things.
And I think that draws a parallel with the data platform.
Like every time I use Claude, almost everything Claude tells me is wrong to some degree.
And I there's some amount of r adjustment, poking, pushing, prodding that I have to do inorder to get stuff.
And so that job, even with an SRE or whatever coming in as an agent, there's still a humanwho needs to be able to validate what's going on and and double check things.
(02:02):
I feel like for the last 30 years or so, engineering leaders have been trying to figureout how to take the ideas of lean manufacturing and apply it to software engineering.
And the ideas of lean are like there's different waste from a theoretical standpoint thatexist in your organization.
And removing those wastes means that more of your time, resources are spent to deliveringvalue to the customer.
(02:23):
And there's a whole bunch of different ways, but I'm not obviously gonna go into them.
I'm sure having spent some time there, you are somewhat familiar with with that concept.
The thing is I think I want to pull out is this idea that the work that software engineersare doing or engineering in general is doing is similar to the factory floor.
(02:43):
Do you think that a lot of the software engineering that we're doing today is repetitiveactivities that we're can can just simply automate, or is that creative, dynamic work
where everything is a unique one off?
The the
Creation of the run book is creation of repetition.
And so everything that has a run book is absolutely repetitive.
(03:04):
I think now some people are trying to use LLMs to answer that question for them.
We'll s we'll see if that turns out to be successful.
no
From what I've seen, LLMs like coming up with what should be done, horrible.
Bad odd.
It is it it does it does not work.
They're horrible at coming up with ideas.
There is this theory, if random chance makes you successful more often than intentionalactivity, then might as well let the LM do it because there's a chance you'll be
(03:30):
successful anyway, or more successful than letting a human decide.
Still waiting on the monkey to type up Shakespeare's works.
I I the yeah, I I know what you're talking about.
And I think, you know, that's that's a pretty good um, you know, counterpoint there wherethe original study was like a hundred monkeys typing on a hundred keyboards.
Uh so obviously it's like we'll eventually produce the works of Shakespeare and everythingelse ever produced, and the result was just a lot of like A's and E's on the keyboard
(03:57):
being hit over and over again and then, you know, punctuation and that was it.
I think there was some like throwing feces involved.
Uh at
And I think that is a very apt analogy for what L LMs are doing.
There's a there's a lot of people out there who try to sell the agentic SRE or agentic uhSOC analyst.
And so the agent can actually look across your silos and figure out what's going on, andyou don't have to worry about the silos anymore because the agent figures it out.
(04:24):
Um, which is kind of true.
The thing that they're missing is that if the agent is the only thing that can look acrossthe silos, how does the human supervise it?
How does the human ever know?
That like, yes, what you're doing actually makes sense.
It instead what it turns into is, Agent, are you sure that what you said is right?
(04:45):
And the agent's like, Yes, I'm sure and you're like, Okay, I guess that's right then.
There's this great paper that I believe I was actually my pick in a previous episodecalled The Ironies of Automation.
When you automate something, there still needs to be a human supervisor of that thing.
And the question that the human supervisor will have is exactly the one that youidentified.
How do they know that the system is working?
(05:05):
And so you need to expose some data from that system enough so that a human can go in andinvestigate what the problem is.
The issue is that the human learns the job of doing the investigation of that system.
Not of how to do the system, but of the job.
And often those people historically had a job of hands-on doing the work.
So they did the work historically, then they learn how to evaluate and monitor the systemand fix it if the problem is broken.
(05:29):
So when we shift to an automated system, we take the hands-on work that someone was doingand translate it to having to do the investigative work and the debugging work.
And then if something goes wrong, also doing the original work that they did.
Over time, systems become more reliable.
So fewer and fewer people have hands on work.
Now all the humans left only have the investigative work and the debugging work and neverthe hands on work.
(05:54):
So even if they're able to identify the problems that've been created, they are unable tofix it because we just don't have those people anymore.
So by automating stuff, we make the humans' jobs incredibly more challenging to actuallysolve.
The the complexity of doing that job becomes greater.
So unless you have a real problem that requires increased scale.
(06:14):
Automating it only costs you money.
So you should think about before introducing agents to any system, you should always askthe question, could we still do this with humans?
Because if the answer is yes, then you shouldn't automate it because that's going to endup in a worse state.
And I think this is gonna bring me back to asking about, you know, what do we actuallymean by the data platform part and whether or not humans could still be building and
(06:36):
maintaining that?
I don't
think that that um completely goes away.
Well if if it goes away there's hopefully some new uh some some new future.
What's the sort of frame of reference here?
So if we have a log solution, we're throwing logs in it, obviously those are productionlogs.
but maybe you're also talking about infrastructure logs or business relevant metrics, uhet cetera.
(06:58):
I love the question and I love the personas that you've uh introduced.
It turns out some logs are business relevant.
Some logs are inf only infrastructure.
Some logs are both.
And sometimes you need to do some things to the logs in order to make them businessrelevant or make them other things.
And the the traditional set of data tools that have been available have created a bit of aspaghetti of um, okay, let's put all of our logs, for example, uh going concrete, let's
(07:26):
put all of our logs into Splunk.
But then the PM wants something, and so they're told, okay, use Splunk for that.
They use Splunk for it, and then they need to share it with someone else.
There ends up being this notion of, okay, let's teach the business analyst to use Splunk.
They're like, I don't the thing that's most interesting about logs is the structure of thelog and not necessarily how it's used.
(07:49):
Where traditionally we've believed that you have to take the data and move it to differentsystems because of how it's used.
But that's not actually true.
The actual thing is that a log is interesting in and of itself.
You can compress it with kind of domain specific technologies.
There's specific ways that people want to extract things out of logs.
There's specific ways people want to look at logs.
(08:09):
But you can take that log and make it SQL queryable.
You can take that log and make it exposed in other systems.
You should be able to have your log, use it from whatever system you want.
And it's the same for the agent as well.
Like the agent should also be able to access that same data.
You're definitely an optimist.
The fact that you think that the logs in these systems are valuable.
Right.
Because the number one thing that always comes to me is like, our logs are useless.
(08:32):
Like w this log doesn't tell us anything that we actually want it to know at this point.
we've seen as we've talked with different people and worked with stuff is especially inthe observability world, the useful lifespan of a log is much shorter than in the security
world.
I mean, I think that's definitely accurate.
You for sure often don't know what will be necessary later, but at the same time peopleoften risk averse.
(08:59):
I think that's just a true a true fact.
and that includes uh hoarding logs that are useless uh because they don't they don't knowwhat they're going to need in the future.
With conscious thought you or deliberate action you would know you know, which logs wouldactually be relevant.
The
The logs of people interacting with the site, the logs of people interacting with theonline banking, or the logs of people interacting with the e commerce site are useful for
(09:25):
the observability team because they use it to figure out, you know, what what causedoutages, what was going on, how many people were impacted, what uh what things happened.
You can you look for the logs in order to discover anomalous behavior that you can respondto.
The exact same data set that's coming in that ends up powering each of them.
(09:48):
It's the access logs from how people are interacting with the website.
It's event telemetry from the single page JavaScript application that's being fired backabout how the users are working on stuff.
That often turns into, well, we have to get this data in, but then we also have to makesure that this subset of this data gets ETL'd over to this other system so that these
(10:09):
people can look at this thing.
And we have to get this other thing over here so that these people can look at that thing.
Oh
Historically, at least in my teams and my organizations, it we always go with the theDevOps mindset.
If you build it, you run it, which also means you capture logs and you have to reviewwhat's there in case of an incident, you're responsible there.
So of course the logs that you're generating have to be meaningful to you.
(10:32):
Um and engineers obviously have a notorious track record of being hard users to please.
So
The platform that we utilize in any of those moments has always been the thing everyonecomplains about.
Uh they hate historically, they hate sumo logic, they hate Elasticsearch, they hateGrafana, they hate CloudWatch insights.
You know, you stick in something here, uh logs log z dot io or whatever, however youpronounce it.
(10:55):
I mean, the list is goes on and on of all these platforms that I've seen w some team trytry.
Because every single time I come in, I I see they're using something like we hate it.
I'm like, great, do some research.
Find a platform you love.
And then we'll try that one.
They do that and they hate that one too.
And the re the reason I I say this is because even if we were able to find something thatthey liked, there's still this problem of are they collecting the right information?
(11:19):
And even if they are, how do other teams get access to that in a way which they're notheld accountable for the data that shows up in those logs?
And the reason I I bring that up is because historically there's this challenge, right?
There is some production data.
which another team uh or someone else in a different role, as you pointed out, likeproduct managers or business analysts, say, you know, hey, we wanna be able to perform
(11:44):
this query on our data.
And historically they'll they'll use one of their giant data lake uh SQL back querysolutions to do this.
And the question is how does the data get from the production system into that other, I'llsay third party system?
Is there an act an ETL extract transformer load job that runs?
If it does, who owns that?
(12:04):
Which fields are extracted or whole databases extracted?
And it doesn't seem like there's a right answer here because if you extract everything andand another team owns that responsibility, that's just always constantly broken.
Like if an engineering team doesn't do it, then the the data changes all the time.
Uh someone complains, hey, my query broke.
(12:24):
The ETL job breaks down because it all of a sudden security says, Hey, we can't beexposing full access on the production databases to third party tools.
So that that stops working.
And even after all of that, someone says, Hey, we're not actually getting the data that weonly want in the first place.
It doesn't even exist.
You know, we're not recording it.
(12:45):
We're writing it down, right?
We wrote other things.
Just like in a lot of production systems, we may forgot to write audit trails, right?
You know, you don't always have everything up front.
And on the other side of the spectrum, you have engineering teams being forced to der pullhand picked fields.
from their production databases and sending them over to your tableaus, your Splunk's,whatever have you.
(13:09):
And then the data isn't there anyway.
But at least they're responsible for doing it.
And you can go to a team and say, Hey, we're not getting this field.
Can we add it in some way?
That
is a great question.
It's a question that business intelligence has fought with for a very long time.
It's data governance.
Uh you you in in the business intelligence world you end up with these with people whohave these UML diagrams of table structure and they're like, thou shalt generate things
(13:37):
that follow this structure and put things in and
But developers never j like developers are horrible at at like aligning with that.
They're like, I need this thing.
I'm gonna add this field.
I'm not gonna look up what it's supposed to be named based on these people in this otherteam.
And so the there's that.
So I don't um I don't know exactly how to solve the data governance problem.
(13:59):
I don't believe that trying to organize everybody and create structure out of it isactually ever gonna work.
Because especially as you expand in an organization, you're just never gonna get everybodyto agree.
I I think I I have seen that the most successful mechanism for the data transfer betweenuh production or uh engineering organization and their databases to other people to be
(14:23):
able to use it is when the engineering team is uh responsible and accountable for decidingwhat their interfaces are going to be and what data they're going to expose in the schema
is as if they're maintaining their API format in an open API specification, except theformat and the schema that they're maintaining is in
uh some third party or other service running, such as you know, uh an object store in thecloud somewhere or Snowflake or, you know, what have you.
(14:51):
And then the team says, yes, you know, we are we we are accountable.
We decide as part of the responsibility of our team that we're going to publish data tothis specific repository.
And here is the schema for that.
And anyone who wants to use it can use it.
And that's it.
End of story.
Because then you don't have a separation between
Someone who's trying to define or has expectations on that schema and the team that isbasically going to be held accountable for it.
(15:17):
I think putting the ability and responsibility uh to the team that will end up doing thisis the right thing.
You know, you push down the decision and ownership down to the team that has it, uh andnot expose the whole database.
If
you talk to anyone at a company, even at Yahoo size, and ask them what technologies areyour company using, the answer will be everything.
(15:40):
Like, are are you using this?
Yes, we are.
Somewhere.
I don't know where.
Somewhere we're we're probably using it.
That that's just such a common mode of operation in a large enterprise.
As much as you might want to force consolidation of technology, a new company is going tobe acquired.
And that company that you acquired had no idea about any of the things that you tried toenforce until after you acquired them.
(16:05):
And now the question is like, is it actually worth it to carry all of that over?
And I think all of this is just pointing to this notion of silos and independent decisionmaking.
I just saw every technology used in all the places.
Uh there were there was one team that um was
(16:25):
for telemetry and metrics, they were using one system for it.
my background is uh kind of with Apache Druid and stuff.
I started the that project and and built stuff and so we used the teams that I worked withwe used Apache Druid to do uh some advertising analytics and and things like that.
And there there were conversations sometimes with the telemetry folks around, you know, wecould use this for that.
(16:48):
I don't think they were wrong
They never really were like, Yes, we're we're gonna adopt this and run with it.
And I don't think they're wrong for that choice.
They're the ones operating the system.
They kind of know what's going on.
They're they're getting at it.
They have their own service boundaries, so they they get to make their choice.
We had other teams, you know, that were though for analytics it was heavy on Hive and onum Tez and Hive running on Tez and and all of that stuff.
(17:15):
This was back when uh Hadoop was still a challenger.
At large organizations, that you will always have independent teams making independentdecisions.
And I don't think it's ever possible to fully align them.
And so instead of doing that, you have to lean into what is something that gives us theability to just store the data as is and then connect the dots later.
(17:42):
When a a local team makes a decision and they're like, this is how we're going to exposeit almost from an open API.
perspective, right?
If it's a rest endpoint that they're using to expose it from, how does business analystuse rest?
They they don't unless they they can tell SQL to go and issue uh REST requests, right?
See, I was gonna say correctly, but I I mean I guess I guess incorrectly is also the uhviable answer that's in the multiple choice drop down there.
(18:11):
Yeah, like they they but they're they're just not going to.
They're gonna be like, I don't know how to I don't know how to how to interact with that.
So here here's a counter argument there.
Uh organizations that followed the rant that Bezos had set off, and now it's been I thinkover a decade, is that data analysts would be required to use the rest interfaces to
(18:32):
expose or get the data that they want from other teams.
So I I totally agree though, in most organizations, you don't have such a strong andspecific and maybe mature mindset about how teams will operate on top of
technology.
And I I feel like if some teams aren't as technologically savvy, then they are limited intheir capacity to solve particular business problems.
(18:55):
Or if the tools they're using because someone made a bad business decision and bought a 10year contract with a company to use a particular product that refuses to add in a OAuth to
integrated supported interface to or even REST API.
You know, maybe it's using WCF or SOAP or something horrific that reads and writes, youknow, is it flat
(19:16):
Text files on a on a physical machine somewhere, then you are limited by the real worldsituation there.
One one thing that does come up, and I think you sort of hit on this, is that it doesn'tmatter what the situation is, even if you prescribe the perfect tool for the job, someone
someone on the other end can always still mess it up.
(19:37):
Right.
Well they'll mess it up.
My team has defined this API.
So if you want to use it, you need to change what you're doing in order to align with mydecision.
Getting humans to change their behavior is, I believe, the most difficult thing to do inthe world.
You're more likely to get people to be like, okay, I don't need it than you are to getthem to actually change their behavior.
(20:00):
And so this leads me to the what if.
So what if
You could have your data in a place, and that data could be accessible via the tool ofyour local team's choice, and also accessible via the tool of that the other team's
choice, and also accessible via the tool of some other team you've never heard of'schoice.
(20:26):
But you have the data in that one place, you can own the definition of it, and it can beaccessible from all of the different places that people have.
That is, I believe, the Holy Grail.
Yeah, I know what you're talking about.
You're about to say it it's going to be give them direct access to the production databaseso they can read the files on the server directly and use whatever data they want there.
(20:49):
You you have the files up on cloud storage.
Yeah.
So if you use that as a way to disintermediate your production systems from the actualkind of analytics and and access of it, have this three layer cake where things have
decoupled to allow for it.
And the the primary thing that allowed for that decoupling is the SQL query language.
(21:14):
Where that's fundamentally the API that the visualization layers use and that the businessanalysts use.
And so when you talk about um the team should define the API that they're gonna support,the the thing is there's not just one API that you're gonna support.
And in the logs world, people want to access logs from SQL.
(21:34):
And the language is more belonging to the persona, where the data format itself, the log,
can be queried from a number of different languages and different personas preferdifferent languages because the the language is aligned to the job that they're trying to
do.
We have had a similar challenge in in our space when we're defining the, and I hate tocall them policies, we call them access records for one of our products.
(22:00):
And I mean, we provide like login and access control as a as a SaaS.
And customers come in and say, Hey, you know, can you support this?
Like, we love this particular format.
I'm like, or this format, or this format.
And uh realistically, it's all stored as JSON for us.
It uh I mean you can store it as some other DSLs, like they're
The every every domain specific language, DSL, is is the worst.
(22:23):
They're they're all literally the worst.
Uh they're almost like a programming language, but they promise that they're parsableuntil people find vulnerabilities with the the language itself and find ways to in inject
attacks in.
And they're all unique that you have to understand the domain in a lot of ways to actuallybe able to write one or write the the form to even do the query in the first place.
(22:44):
And those that are amateurs and have come from one world make it very difficult to
expose in different way.
But it's very easy for us to add in different content types to our API and expose theexact same data via different formats because all of them are constructible.
And the one thing I learned, I say, my last 20 years of engineering, and it wasn't thatearly on, maybe this is a bit of a regret, is that I always tried to avoid SQL uh when
(23:09):
making an API for querying particular data on a production database.
It it's the worst, you know, let's use something constructible, something nice.
that is can be defined in in JSON and passed over the API and can be validated.
But over time, what you find out is you just made another bad uh domain-specific language.
You just made a worse DSL somewhere.
(23:30):
And at this point, I'm just like, everything should be able to speak SQL because it has isessentially the thing that humanity has centered upon working correctly.
And it's portable from one system to another one.
And as long as you can provide that
specifically as your query language, then often it's easy to add these other things ontop.
And it sounds like for your product, you know, for your data pipeline, uh being able toport from the underlying data store to the to the bad terrible DSL that every single
(23:57):
different persona has said, you know, I want it in CQL in and SQL and whatever, whateverhave you, and also understand the semantics of that particular language, like the the
MySQL of the world versus the
MS SQL, the Microsoft SQL.
Like it's not exactly the same, right?
It it there are some fundamental differences there based on the data.
(24:18):
Getting the semantics right is definitely a key element of things.
So there's some people out there who are like observability, SQL for observability.
Everything should be SQL, all observability should be SQL.
And while I think that everything should support SQL, I actually think that the thediaspora of different query languages for logs is indicative of a different usage pattern
(24:45):
that
is actually extremely meaningful for working with logs that is not as interesting forworking with business data and like rigid column or structure data.
And so I I think that I I do actually think while right now there's tons of differentquery languages for logs, I do expect that to converge down on a subset of query languages
(25:10):
that uh will remain and that that will exist.
Changing human behavior is hard.
It instead of telling them that they they need to change what they're doing, if you canjust meet them where they are, um, you can see that's in metrics too for observability.
Like people use PromQL so much more than they use SQL for for metrics and telemetry.
Um really PromQL kind of has, I believe, has won as a de facto standard for metrics andtime series style things.
(25:37):
I'm I'm less aware of different languages in that space.
And you're totally right.
The smart thing to do is to convert whatever interface someone has into the best, mostgeneric solution that can be run on any single platform.
So it doesn't matter what DSL someone is using, if you can convert that to an abstract
syntax tree and AST, then you can directly execute that AST on whatever run engine you'vegot.
(26:01):
We actually had uh an episode on how uh a company is doing that specifically and I willplug that in the description.
But I I think this this may this solves a lot of problems, right?
You don't have as long as you can come up with a can a transform from that to an AST thatyou can run and you just run the AST, you've solved the problem.
And if you can't convert it to the AST, then you know that there's a feature in the deadDSL that
(26:23):
you aren't able to fully utilize.
And then the question is why.
I do think there's an interesting uh corollary here, realistically, whereas if you are ina different space, if you can't convert it to the same kind of ASTs that you're used to
running, are people thinking about that problem differently?
And I'll just use an example.
I think very early on I was full in on traces.
(26:45):
This is the idea of storing log information that is grouped together during the wholerequest and maybe across requests.
At a single moment.
So when you want to query something, you're looking at a single object with lots ofadditional data in it.
An example is like, let's say there's an error in production and you see a log messagewith that error.
Now maybe you have some sort of correlation ID.
(27:05):
And so you make your SQL statement, select all the rows where the correlation ID equalsthe value you know.
You're pulling multiple different logs and you're getting that data.
For us, that was always really annoying.
Especially when you have a whole list of correlation IDs that you want to pull stuff forand you want to join it up with.
So very early on, what we did was whenever there was a log message, well, during the wholeprogram execution, we would store a property bag of all the context of all the variables
(27:29):
of everything that was happening, and we shove it into the error message.
So when it gets logged, we have everything there in one place that we can actuallyvalidate and use.
And I think this has caught on recently.
I think uh in the last few years, Honeycomb was a huge proponent of what they were callingobservability two point zero.
And I think since then a lot of other companies have jumped on the bandwagon here of beinglike, hey, you know what?
Storing line by line stuff isn't necessarily the right answer.
(27:51):
The reason I bring this up is because that query sort of depends on the fact that we havedata stored in a particularly different way.
And so it wouldn't be straightforward to necessarily convert from the query language to anAST that could run on these different kinds of data.
So it's not just necessarily about the
transforming of the data language that we're using the the DSL to something that we canexecute, we also need to understand fundamentally how the data may be relevantly stored
(28:17):
for accessing, you know, things like indexes, the what's in a a wide table format or acolumnar format, like these things are potentially relevant, whether it's stored as a blob
and whatnot.
So there's there there are a lot of complexities here that any company would potentiallyhave to deal with.
And obviously, you know, some areas that you you've seen particularly.
I mean, obviously if it's just giving the consumer a DSL that matches their expectations.
(28:40):
You know, you could do a lot of hacks under the hood there.
but outside of that, you know, you really start to come into areas where there arechallenges because fundamentally the data has to be exposed in different ways.
Absolutely.
And and traces is the a great example.
Like I've been talking about logs a lot.
Interacting with traces and a language for interacting with traces will fundamentallyunderstand the notion of a trace versus a span.
(29:06):
But the actual like fundamentals of how is the data stored, how is it persisted, how doyou index it, and which query languages are supported is actually a function of the data
format itself.
And so a trace
requires different languages.
And same with metrics like PromQL.
You could convert PromQL into something that runs against logs.
(29:29):
the language believes it's working with time series objects.
And if you're reconstructing time series objects at query time, you're wasting a lot ofCPU.
Well, I Postgres, I think, would disagree with you because they've managed to just say,yes, we're always going to take all the data and store it in rows and columns, uh,
including your you know key value storage and and graph graph databases and vectors andwhatever and somehow make it make it work.
(29:50):
But and then there's i of course the alternative argument, which is that uh the relationaldatabase is always the fastest, no matter what.
Uh which which I've also seen is true.
Uh we we've sort of validated that graph databases are so slow that performing
Stupid not great queries in in SQL tend to be faster uh by making lots of extra queries,ten queries at the same time and stitching together the results, rather than using a graph
(30:15):
database, which is sort of uh unfortunate in a lot of ways.
But I I mean, you sort of echoed my point, which is that of it's not necessarily thelanguages which are the problem, but some of the semantics of them, but realistically how
the data is being stored.
So how do you deal with the fact that the data format is important?
relational database can store everything, yes.
And everything that's in a relational database can also be a log.
(30:37):
Um everything that's in a relational database you can also model as traces.
I don't know that there's actually a one size fits all.
This is the way that all data shall be modeled.
Back to the uh if you don't have the business problem then there's then there's no there'sno issue technological problem you have to solve.
(30:58):
It's always a great one.
Uh which is why pushing back is important, right?
if you tell someone that, it's gonna be really difficult to do that, they may go back andfind out that they don't need to do it after all.
Y yes, yes.
You know, uh, Eric, I think this may be a good moment to switch over to picks for theepisode.
So maybe you could share with the audience what you've brought.
(31:19):
Uh yeah, I spent a a bunch of time in Japan.
really enjoyed learning the language and stuff.
And and learning the language to me is I think computer science and all that, it's alsolanguage learning in in my mind.
I think of it as as similar.
But the the big thing is there's a a saying that I love uh from Japanese.
(31:39):
It's keizokwa chikaronari, which literally means persistence leads to strength or
Continuous improvement kind of leads to strength.
And it's this notion that if you do something five minutes a day, ten minutes a day, aftera year, you will have done 3,650 minutes.
(32:02):
After two years, after three years, you will have done it so repetitively that you willfind that you are now capable of doing it.
And and I think looking back on my life on everything, like there's a bunch of things thatI did and then stopped doing.
And maybe I was okay at it for a little time, but now and I can't do it at all.
But it's that notion of like, it's not so much about getting to this end state and beinglike, okay, it's done, declare victory and we've won.
(32:29):
It's about making sure that you're continuously investing and kind of driving towardssomething, whether that be large time commitments or small time commitments.
Uh I I think this notion and this philosophy, I just I I love it is that
The things that are really important, if you just continuously work on them, even if it'snot a lot, out outcomes will will come along with it.
(32:53):
The non research, the sort of feeling result was that it's about a thousand hours ofanything before you become a master.
And a five minutes a day would take you about 20 years to master anything.
Uh, but then you can look at it that way.
After 20 years, you can have mastered whatever it is that you want.
Uh, I think the research actually comes from a book called The Formula, The Science ofSuccess by Barabasi.
(33:16):
And it was where they evaluated which
in there, there was a lot of good research, but one of them was which Ol Olympic athletesbecome Olymp like successful, like actually get gold medals.
And the ones that get to that level are usually not the ones with raw talent or ability,but the ones who go through significant practice and basically perseverance through
(33:37):
through training to get there.
So not that you're born with it, but that you actually went and did it.
They weren't the best y younger on or when they started, but by the end that that is whenuh they were still able to achieve
I mean obviously there are some, you know, naturals that still take it, but by by andlarge, the most victorious ones are are the ones that just continued at it consistently.
(34:00):
couple it with the notion um every time I look in the past, time always went really fast.
Where when you look in the future, time like it if things seem much further out.
Even that five minutes a day, after three years, looking in the past, you will be able todo something.
Maybe you're not a master, you'll be able to do something and it will have taken like notime at all because it will have all gone extremely quickly.
(34:24):
I think especially in the Western world now, there's w it does feel like things arespeeding up rather than what has historically been recommended, which is slow down and do
things deliberately.
And uh I think maybe maybe AI is to blame for that.
Maybe cultural is to blame for that.
Uh we'll see.
That's not related to my pick.
Um my my pick uh is and only 'cause I just completely binged it was a television showcalled The Agency with Michael Fassbender.
(34:53):
Um
It has to do with the CIA operating out of London.
I mean, obviously, complete fiction.
and I don't know what it is.
I I I don't I don't have love for stuff that is centered in the in the US.
I I always I don't know what it is, but the CIA operating of London, you it's it honestlyis really, really good.
And I think it's produced by Showtime.
(35:13):
Okay.
Nice.
So I don't know if you're into uh spy dramas.
Yeah, I'm I'm trying I'm trying to remember that name.
There was uh there was a a show a while back about uh an ex spy based out of Yeah, BurnNotice, yes.
Uh
Burn Notice, uh absolutely fantastic.
Uh as well.
(35:35):
Yeah, it's it's definitely something you can also binge.
It's it's great.
I mean the nice thing about Burn Notice is that it was produced in a time before w showsgot canceled before they were over.
Michael Weston, uh not the actor, the the character.
I don't remember the actor's name.
Uh he's in a bunch of things, but uh also very good.
(35:55):
So
Yeah.
I d I don't know what it is.
Uh and you know what, there's sort of like it's not like such negative dark comedy or ordrama.
Like the dark the it's not as bad as dark drama.
I mean there's still stuff going on, but it doesn't feel like uh they just do it for thethe kick, so
(36:16):
Like one of the nice things about Burn Notice was that it it was very episodic, but therewas a little bit of a background thread running between them.
And so you could enjoy one episode and and get completion, get some sort of closure, butstill there was like the the allure or like what's going on behind the scenes, who's
actually doing it.
(36:37):
Is is the agency like you gotta watch the whole season to get the gear closure or is itone
Yeah, I think that unfortunately that's still where we're at.
Don't worry.
Uh the second screen experience is is coming soon and every show that shows up on astreaming service will all be episodic so that people don't have to pay attention anymore.
But the agency is still I think it's it's not totally epis um not episodic like a singlethread, but it uh you pretty you do need to sort of know what's going on from the previous
(37:06):
episode.
There still is a major thing there.
There's two seasons right now and they are
Sort of separated.
So you could watch only one and feel like it's complete.
Still, yeah, unfortunately, if you're just looking for single stuff, that's not it.
I will say that the nice part about Burn Notice was that they also sort of educated you onwhat is reasonable in Spycraft.
(37:28):
I don't know if that was accurate.
I mean, I think there are lots of videos on YouTube basically saying, you know, thesethings they show in shows, you know, aren't real.
I think Burn Notice does a better job.
And I think some of the things they do in the agency also, like if you're payingattention, uh really help to clue you in on like how things actually work in practice
rather than like today, rather than uh just doing stuff for the for the heck of it.
(37:51):
Okay.
So um thank you, Eric, for coming on and discussing observability with us and thechallenges of managing the database.
Yeah, thank you.
Thank you for having me.
It was great discussion.
And thanks to the audience for tuning in for this week and hopefully we'll see everyoneback again next week.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.