July 18, 2025 • 39 mins
Rachel Brindle, maintainer of Nimble and Quick, comes on to talk about what's new in Swift Testing and what the future of Swift Testing may hold.
Guest
Related Links
- What's new in Testing, 2025 Edition
- nalexn/ViewInspector: Runtime introspection and unit testing of SwiftUI views
- Quick/Nimble: A Matcher Framework for Swift and Objective-C
- Quick/Quick: The Swift (and Objective-C) testing framework.
- pointfreeco/swift-snapshot-testing: 📸 Delightful Swift snapshot testing.
- swift-evolution/proposals/testing/NNNN-polling-confirmations.md at younata/testing-polling-expectations · younata/swift-evolution
Related Episodes
- Live from CommunityKit WWDC 2025 with Matt Massicotte
- The Case of the Crimson Test Suite with Daniel Steinberg
- Empowering Accessibility with Via Fairchild
- It Depends with Brandon Williams
- Practical Year - Part 1 with Donny Wals
- SwiftUI Tips and Tricks with Craig Clayton
- Mobile System Design with Tjeerd in 't Veen
Social Media
LinkedIn - @leogdion
GitHub - @brightdigit
GitHub - @leogdion
TikTok - @brightdigit
Mastodon - @leogdion@c.im
Youtube - @brightdigit
BlueSky - @leogdion.bsky.social
Twitter Leo - @leogdion
Twitter BrightDigit - @brightdigit
Credits
Music from https://filmmusic.io
"Blippy Trance" by Kevin MacLeod (https://incompetech.com)
License: CC BY (http://creativecommons.org/licenses/by/4.0/)
- (00:00) - Comparing Swift Testing and XCTest
- (11:08) - SwiftUI Testing
- (17:37) - Advanced Testing Techniques and Tips
- (23:47) - Future of Swift Testing
- Edward Sanchez
- Steven Lipton
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:03):
Welcome to anotherepisode of Empower Apps.
I'm your host, Leo Dion.
Today I'm joined by Rachel Brindle.
Rachel, thank you so muchfor taking time to come on
Yeah,
before we get into Swift Testingand all the new stuff with testing.
I'll let you go aheadand introduce yourself.
Yeah, so I am Rachel.
I am a principal engineer at Autodesk.
(00:25):
And I have been involved withtesting, test-driven development and
such, basically my entire career.
A lot of fairly deep knowledge of likehow to go through and test things, as
well as mentoring others in learninghow to test code, how to write better
tests that aren't brittle and won'tbreak all the time and so forth.
(00:47):
Awesome.
So what year did Swift Testing come out?
That was, It shipped in Xcode16 last year, but it was open
sourced in September, 2023.
Okay.
So you did a lot of work with XCTesting.
You have a few libraries you actuallybuilt, for XCTesting, to help folks.
(01:07):
What did you think about the processas far as Swift Testing is concerned
and how we got to where we are now?
How we went from XCTestingto Swift Testing.
And what do you miss?
what are you happy that wegot rid of Things like that,
So Swift Testing is obviously, 'causeit's only like publicly, it's only
(01:29):
like a year and a half old, thoughI'm, I know for a fact that like,
it didn't just pop outta nowhere.
they've been working on it sinceat least macros were a thing.
Swift Testing, if it's stilllike no more than, it can't be
older than like three years.
It is.
Nowhere near as mature as XCTest is.
XCTest has been, was introducedin iOS seven, iOS eight era.
(01:55):
But even still it dates back to mucholder Objective C focused testing APIs
like syn testing kit and so forth.
And those date back allthe way to the nineties.
I was gonna ask about that.
'cause like XCTest, I would assume.
Like you said is from quite a while sinceObjective C. 'cause I can't even imagine
(02:16):
what they used for testing before that.
Yeah, it was, I don't think Applehad, so it was initially started as
some, third party, like predating.
It might even predatelike Apple buying next.
I don't that.
That I, I don't quiteknow the history there.
But it was, there was a send testingkit and like a bunch of like other
(02:40):
third party stuff is what people used.
At some point in the two thousands,apple started shipping or like including
send testing kit with, Xcode so thatyou didn't have to download that, And
they started to just refine that overtime and eventually they rewrote it
with XC test and, and shipped that.
Okay, so Swift tests, like the big thing Iwas gonna talk about is what you miss from
(03:08):
XCTesting, but then like a part of thatis just what we could do with Objective
C that we can't get away with in Swift.
So there is the open source XCTest.
it's Swift CoreLibs XCTest and thatis written entirely in Swift and
that enables that has been aroundsince Swift package manager was open
sourced back in like 2015 or so.
(03:30):
And that is not feature complete withlike the private XCTest from Apple.
But it does a lot of whatyou can do with XCTest.
XCTest is really much better for buildingtesting tools on top of, like you can
dynamically create and register testsat runtime and XC test, which is not a
thing you can do in Swift Testing at all.
(03:51):
I know that there are people onthe Swift Testing team interested
in enabling something like that,but it's just not there yet.
Though reading between the lines,they're definitely moving to enable
that feature in, Swift Testing.
And then there's other minorstuff like, performance tests.
The XCTest has the measure API, soyou can Figure out how long it takes
(04:15):
to run a bit of code so you can seelike, oh, do I have any performance
regressions in this area and such?
And you can kind of do that withSwift Testing, but for a bunch of
reasons, Swift Testing's, concurrenttest runner being a major issue there.
it's not great.
And of course there's not reallyany equivalent to the measure API.
(04:35):
So what did you really like aboutSwift Testing and where it's headed
right now compared to XCTesting?
So the thing that I really likeabout it first, the DSL of the
experience of actually writing testsis much nicer than with XCTest.
The macro or just being able tospecify like the attest macro and
(04:56):
then just like a bear function, isgreat, especially in Swift six two.
I think it's se 4 51 ads just likeyou can, I think it's bear function.
You can use a backtick and then put anarbitrary string or nearly arbitrary
string between back ticks and likethat's the name of your function.
So you can have spaces commasand punctuation in there.
(05:20):
that will render nicely as a full humanreadable string so you don't have to
do this weird thing where like you haveattest and then whatever human readable
string and then a non-human readable, butlike valid function identifier afterwards.
Okay, so now you could just makethe function whatever you want,
and put whatever characters you
yeah.
That's crazy.
That's useful for non-testing stuff,but mostly it's useful for testing.
(05:44):
That's like reading the proposal for that.
That's like the main motivation they cite.
Yeah.
Yeah.
So yeah, like the, going back to otherthings that I really like about, Swift
Testing I like how it's much more open.
Both like it's developedmostly in the open.
there is a testing workgroup, which I'm a member of.
(06:07):
And it's.
It, follows a form of theSwift evolution process.
Like testing proposals arein the Swift evolution repo.
But the proposals are formattedslightly differently and such so
I'm really liking that there's alot more community engagement in
the development of Swift Testing.
Yeah.
(06:27):
And I think that's the biggest advantageis the fact that it's open source and
that we can build a whole set of toolsaround that and help it facilitate its
growth, what point do you think it'dbe important for a team to migrate
from XCTesting to Swift Testing?
Really that depends on the team,there's a ton of tests that are
written right now that cannot beeasily migrated or like can't, it's
(06:50):
not gonna be an easy process to migrateit from XCTest to Swift Testing.
Do you have an example?
So like anything that accesses like, ormodify some like shared global state,
For like one specific test, youmight change it to some value.
And then another test wantsit to be some other value.
because Swift Testing runs everythingconcurrently in process, then
(07:12):
you're gonna run into issues ifthose two tests are ever actually
like running at the same time.
Should you be doing that in the first
I mean no,
Okay.
Yeah.
Yeah.
In a way it forces you to be cleaner.
Like you'll end up replacing that typeof thing with a task local or whatever.
And that also brings along all theniceties of like, you're now going
to force your tests to actuallylike build under the Swift six
(07:33):
language mode and all that such.
But there are plenty of tests that existright now that work perfectly fine in
the world with XCTest where it's only onetest is running at a time in the process.
That are going to be difficultto migrate to Swift Testing.
Okay.
And then the other benefit is it'llbe slightly easier to read stuff.
(07:55):
You'll get faster runtime of testsbecause they can execute in parallel but
you're gonna get slightly slower buildtimes because macros take a bit of time.
Even tool chain macros, they just add alittle bit more time to the compile time.
automation tests stillaren't a thing and so forth.
But yeah, if you have working XCtests, there's no reason to, unless
(08:19):
you really want to, there's no reasonto go rewrite them into Swift Testing.
Maybe.
So like that's the approachto that idea is new components
are written in Swift Testing.
But if it's an existing component thatalready has tests written for it, then
I'm just gonna keep it in XC test.
Do you think macros have aplace in filling the gap when
(08:39):
it comes to what's missing?
In Swift Testing?
Yes and no.
So well for I guess to correctlyanswer, yes, they have a gap.
But like
they are very painful to write andmaintain as I'm sure you don't need to
be reminded, I looked at how, the expectand require macros and Swift Testing
(09:00):
are implemented and like mad respect tothe people on the testing team who wrote
them, because I do not know if I could.
And there's also the other stuff oflike, until very recently you had to
rebuild sw, Swift, Swift syntax for like.
Third party macros and such, thatisn't really a thing now that we
have like the cast with Swift syntaxas of like 16 four or code 16 four.
(09:25):
But like there are, thereactually are places for macros.
Every so often someone will postin the Swift forums, the community
showcase about a library that usesmacros to generate test doubles.
Conforming to the protocol specified.
So a test double is just kind ofa stand in for the real thing.
Okay.
So you might have your type that hassome methods or whatever and you have
(09:51):
a protocol that defines that type.
You might then use one of these tools to.
Create a test double that automaticallyconforms to and implements all of
these methods such that you canthen either just directly pass it
in and it'll work in the thing.
Or you can modify behavioror you could even examine how
it was called and so forth.
(10:11):
that's really cool to see andthat saves a bunch of boilerplate.
Yeah, so that's one of the main examplesof using macros to fill in the gap and
do things that Apple probably isn't gonnado, at least not with Swift Testing.
At least not in the next decade.
Yeah, you had this like marketablemacro, which is really cool.
(10:31):
I just started looking at that, becauselike that's one of the things people
love about Objective C is everythingis dynamic so you can go in and swizzle
and do all sorts of crazy stuff, whereasSwift is super strict about that stuff.
Like for good reason, like, outsideof using tests to like mock out
behavior, we might get the into thisin a, in a little bit, but like this
(10:55):
is part of what made testing UI kita lot less painful was being able to
swizzle out a whole bunch of stuff.
Which you can't even do any ofthat with Swift test with Swift UI.
And like Swift UI is also not untestable.
Yeah.
Well, let's talk about that.
That's one of the biggest complaintsis how SwiftUI is testable.
What does that mean exactly?
(11:15):
Yeah, so you can write tests forSwift UI using Swift UI but you can
only write automation tests for it.
So that's like the XCUI test stuffthat, works by spinning up your app in
the simulator and then having anotherprocess that actually runs the tests that.
Uses IPC to interact with the appusing the accessibility system.
(11:40):
Yep.
and out of the box you can onlylike, just interact with the
UI and examine stuff in the ui.
You can't like go in and examine oh,did this insert something into, did
this actually insert something into thedatabase or is the UI just lying to me?
Out of the box, it's going to makelike every single network request
and so forth, which it's not really.
(12:01):
But there's a bunch of third party stuffto make, automation tests, or like to
fake out network requests and so forthand like enable you to better inspect
stuff going on at automation tests.
Yeah, well, like one thing I hear a lotabout is snapshot testing as a way, which
I think is from the point free team.
so there are several waysyou can do snapshot testing.
(12:23):
There's an, older, API from I think Uberwas the first who made it iOS snapshot
test case both of these, the 0.31 doesa little bit more, you have a little
bit more options to actually inspect it.
it works by just rendering the UI intoan image and then basically working
entirely as a regression test of like,you render the eye into an image.
(12:45):
You test that and you just assert that.
The image is the same as apreviously recorded image
Yep.
That can work either as like, likean XCUI automation test or as more
of a unit test, like in process test.
And that's fine.
Like it works for likepure regression stuff.
(13:06):
There's, it's very brittle.
Like I do not envy the people with,with snapshot tests, going into
who are probably not having a veryfun summer, having to rerecord all
of their stuff for like the liquidglass design, language change.
And especially like eachnew beta, it's gonna change.
(13:27):
But even going between like minorversions of the os, there, there
are like gonna be slight changesto how things render and so forth.
There's even like stuff of whether whenthe simulator runs on like an Intel
or like a, apple, silicon Mac you,it will render it slightly different.
So you have to do like some sortof like fuzzy image compare instead
(13:47):
of just doing a straight up,
You're right.
pixel by pixel compare.
Okay.
so like snapshot tests work,but they're extremely brittle.
When I say that Swift UI isn'ttestable, what I mean is that I
can't write a unit test for it.
One of the things that you can do withUI kit is you can directly call the
(14:10):
methods that Youi kit itself callswhen it's like when you tap for that
lead to a button callback being called.
So it's possible for me to write a testthat's just like, Hey, when this button is
tapped I write to the database and I canactually then inspect that the database
was written to, or that a test double thatrepresents the database was talked to.
(14:32):
and that's not somethingyou can do with Swift UI.
You can't simulate a button.
Press in test.
could you use something likean inspect and push the button?
Yes, there is a third partylibrary called a view Inspector.
I'll add it to your,show notes To its credit.
It does a really good job of allowingyou to inspect and test like, Swift
(14:55):
UI stuff, Swift UI views and so forth.
But it's incomplete.
there's a bunch of APIs from Apple thatit doesn't have good inspection for.
it uses the mirroring API tobe able to inspect things.
And then it has to know about specific.
Like sub components within to, or liketo go like inspect, oh, this was the, UI
(15:16):
button or the, sorry, the Swift UI button.
tap handler and such.
It doesn't automatically support newApple provided, Swift UI views by default.
It does work with like yourown third party views 'cause.
But it works.
It's also brittle, it's vulnerableto internal changes to in Swift UI of
(15:37):
That's exactly what I was gonna say
that happened last year.
So, yeah.
really what I would love to seeis first party support for that.
I know for a fact that every time, 'causeI bring it up every year and a WW TC lab
that both the Swift UI and testing teamsget that very frequently as a request.
(15:58):
But for various reasons theyhaven't been able to deliver that.
How would you picture something like that?
Like you just create a Swift toeye view and then some sort of
select, like how does web do it?
Right?
Web would just be like, oh, grab thebutton using a selector, and then push.
(16:18):
Yeah.
So that's exactly what I'm thinking.
Like, something that I've beenworking on as like a blog post is,
exploring what that might look like.
Through the lens of like, this iswhat we might try to do for UI kit and
then because we can do it for UI kit.
I like the approach, the interactingthrough the accessibility system
(16:39):
that automation tests take.
'Cause that already exists,that forces you to actually
add accessibility, support,
I was gonna say that.
And such.
I think that'd be areally good thing to do.
plus it's also fairly, like blackbox, like you could call in an
automation test, you can call a tapon an element that cannot be tapped.
It's non-interactiveand nothing will happen.
(17:02):
Like you won't get a compile error.
It's just that nothing will happenand your test will silently fail.
or maybe it won't fail dependingon what your next assert is
that's something that I think is.
Really powerful as a direction togo maybe not exactly mirroring the,
public API of automation tests.
But that's a great starting point oflet's interact through the accessibility
(17:23):
system to be able to like pull on,pull an element that should exist.
and then like be able to interactwith it, test or tap on it,
swipe on it, and so forth.
Yeah, that makes complete sense.
So I wanna jump back, talk a littlebit more about Swift Testing.
So one of the things I really likedabout it is the way that you can organize
(17:44):
your tests, without needing to docertain things with classes and stuff.
Kind of explain the whole suite testingthing and what are some things developers
should be thinking about when they writetheir tests and how they organize them.
Yeah, one of the really cool thingsabout Swift Testing is the fact that
you can nest sweets as much as you want.
(18:07):
you can have zero sweets, so just barefunctions that are with that test macro.
You can organize, you can havelike a single layer of sweets.
You can have as many as you want.
There is, you do kind of run into theissue where, if you have properties on
like the parent suite, a child suitecannot reference it because they're
(18:29):
structs and they can't know about,they don't, they're not getting like
an a reference to like an instanceof the parent struct and so forth.
So you can't you have that limitation.
I'm hoping that once apple.
Provides like more APIs for likedynamically adding tests and so
(18:49):
forth, dynamically that there'llbe community solutions to that.
Something like what the quickframework provides for XCTest.
Could you give like a use case of that?
so
a parent suite info or metadata.
Yeah, yeah.
So that, that's something like, Even justas simple as like having shared setups.
So you only have to create the objectbeing tested like in one place in the,
(19:13):
like in it for the, in like the parentor whatever, the outermost suite.
And then later on, like you might havelike a child suite that is just calling a
method on that object that's being tested.
That's.
One way to do that.
I also like to nest suites, or Iwould like to nest suites for like,
(19:34):
conditional and such, a child, onesuite for if this conditional is true.
Another for if it's false and such.
Because then you also get like the nicefo code, code voting capabilities well, I
don't really care about the positive caseright now, so I'm just gonna fold that
up and only look at the negative case.
Like what,
what I use a lot, and we'll get intotraits later, but I have to have, I'll
(19:59):
have a suite that I do a lot of fullstacks with stuff and so I'll have
something that's only supported on Appleplatform, so I end up having to do like
pound if can import or something like thatto kind of distinguish between the two
SOI, for instance.
And then I have to create a variableand then use that as a global variable
(20:20):
that says whether it's available or not.
which it's kind of a mess, but it works.
We'll talk about traits morebut back to like sweets.
What do you think, what aresome tips you have as far as how
you should organize your tests?
Yeah, I personally tend to just do onelayer of sweets just because I would
like to nest them for conditionals,but doesn't really work out well.
(20:42):
So I just do one layer and then eitherorganizing them with tags or using.
good old Mark comments.
Like I usually just do, I'mstill stuck in the way of like.
I'll have a test suite lined up witha type essentially and do it that way.
Is that, do you feel like that'sstill a good way to go about
(21:04):
that's how I do it.
I'll have like a suitethat's like, my object tests
Okay.
I'll
And it's good for like codecoverage too, because then you could
tell, okay, what is this testing
Yeah.
But you could also dothat, without the suite.
It does make it nicer in terms ofgetting the, test diamond, on the
left side of the editor because thenyou can just select everything there.
(21:27):
But if you didn't want that, Youcould just have them all on top of
the file and then have the file beyour top level organization for it,
Got it.
but then you don't get likeshared setup and so forth.
Right, exactly.
What are some other traitsthat people should be aware of?
I would recommend making like a customtrait that automatically does the, like,
oh, if Mac, if iOS, if Linux or whatever,
(21:51):
How hard is it to do that?
it's not that hard.
you can, create a type that conformsto, you'll probably want to do
both test trait and sweet trait.
and if you do conform to sweet trait,You should implement the is recursive
and then just have that be true isrecursive means that for a sweet
trait, it also applies that trait toevery single like child of it, all the
(22:14):
tests, all the sub suites and so forth.
Yeah, that makes total sense.
you can just have it say like, thisalso is a disabled trait, or like,
this marks the test as disabled.
A fun project for me
yeah, but it shouldn'ttake that long to do.
Okay.
There are the, comment in bugtraits, which just allow you to
associate like text information.
(22:34):
With a tester suite.
Okay.
There is the serialized trait whichforces the test covered by that suite
or the like parameterized test torun serially, so not concurrently,
Okay.
That's kind of weird.
'cause that only applies tolike those specific tests.
if you have like two like sibling teststhat both have like the serialized trait
(22:59):
and you run them, but like the parents,the containing suite isn't serialized.
If you run those, then like those twotests will actually run concurrently
or like in parallel with each other.
Should you be doing serialized tests?
it's a thing that exists.
Especially in a post Swift six world, like
(23:20):
I haven't run into like aspecific use case that needs it.
I am a hundred percent certainthat that use case exists,
otherwise it wouldn't exist.
right.
But like every time that I've usedthat I've reached for it, it turns
out that actually what I shouldhave just done is made whatever the
thing that was forcing the thingto be serial to actually make it.
(23:42):
Be thread safe, like changea global variable into a task
local or something like that
so I did want to cover your article,which I'll put in the show notes, but
what are some highlights about, what'snew this year that folks should really
start looking at using in their tests?
Okay, so the major new change is exittests, which doesn't apply to iOS.
(24:08):
on the server side team?
Yeah, service side, even Macdevelopment windows and such.
They allow you to verify that.
Code exits properly as you expect
code.
returns an exit code.
It throws
Okay.
or something like that.
So you can basically say, Hey, makesure the signal is the right integer.
(24:31):
Yeah, so you
standard error context.
What?
How do you check that?
so,
The result has STA standarderror content as a property.
the example I wrotecould have been better,
Now.
'cause I didn't understand whatresult was and I was like, I'm
looking at this, and I'm like, okay.
So result is basically some sort ofstructure, which has all that metadata
Yeah, so
(24:51):
okay.
it has the, if you tell it, ifyou tell your exit test to grab
the standard error, then it will.
Then that, standard errorcontent will be non nil.
If you specify with standard output,it will also be a non nil thing.
Those are just an array of, Ithink it's U eight but an array
of, yeah, it's array of U eight.
(25:12):
It's array of bites that if you wanna,you then have to like convert it to a
string using one of the methods there.
If you wanna do a string comparison.
there are like some weird beta bugsin there right now where it will
like, capture a lot more of thelike standard error than you need.
And that's just kind ofhow the, posik spawn, API
(25:35):
Oh, really?
So yeah, it's.
But it works really well.
I'm really impressed with how this works.
you can't do this with XC test at all.
Like Apple's never added support for this.
So if you wanted to verify thatyour function had a precondition or
such in it, and that preconditionworked as you expected, you had
(25:56):
to go way outta your way to like.
Construct something to do that, whichno one has ever done in their life.
Yeah
so it's really nice to have this.
that sounds like a server side thing.
Yeah, so really, I thinkI want it to come to iOS.
the sandbox in iOS is a thingand that's gonna be difficult.
(26:17):
what's the use case for iOS?
Because like if the, the app shouldn'tcrash, I mean, it should just never crash.
Right.
But you do have APIs whereit's like, Hey, I expect, for
example, array subscript, right?
So, or like subscript an array.
If you pass negative one then, or if youpass like an out of bounds index, then
fatal error,
a fatal error.
(26:38):
And like that.
Whether or not it should or, orshould return nil or do or like throw
an error or something that you canactually handle is another question.
But that sort of thing exists andwe should be able to like handle
that test case just to have likethat coverage that it's working.
I was about to ask that.
'cause a thousand years ago when Idid a, an episode with on this very
(26:59):
subject, I was asking like, how do youtest for fatal error or precondition
or assert or any of that stuff?
And the way we've had to doit is basically have a way to
like overload it essentially.
But we don't have anything like that.
And this, which what it sounds like isthis would work in that case, but then
at the same time it wouldn't work on iOS.
(27:21):
Is that correct?
Yeah, so the way that I would approachthis is I would, I would either just
not, or I would like put in like aninject, like a, an injectable using
like dependency, injecting like a shimthat just in actuality it calls, as
you were just saying, like it justcalls precondition and actuality.
And then in test we can examinethat it was called that.
(27:42):
For this I might either write, likeif it's a cross platform app, like
I wanna have it run on Mac OS andiOS then I would just have the.
Have it be covered under the testfor macOS with like a pound if or
like if os macOS stuff covering that.
(28:03):
There's a few things that arepromising that isn't in there yet.
The thing that I didn't write thatI'm most excited for is a thing
called, issue handling traits.
And these allow you tofilter or modify issues.
So like if you have a poundexpect that fails, like it.
Pound expect true, equal, equal false.
(28:25):
That's, guaranteed failure there.
Then it allows you to, specify a traitthat allows you to modify the issue.
You might wanna convertthe, severity of it.
you might wanna just entirely
Yeah.
Okay.
That's really powerful.
I don't think it's gonnahappen in Swiss six two.
But you think it'spossible in the next six
(28:48):
Six probably six three.
It's actively being discussed.
Yeah,
there's other stuff there that's,exciting, from what it enables, like,
range confirmations, which were mergedin, Swift six one and have been out
since Xcode 16, three earlier this year.
This is a thing that they view asnecessary for automation tests.
(29:10):
So range confirmations is basicallyjust, you expect the callback to be
called within some amount of range.
And the example given in the proposalis using an automation test of
like, oh, you have a button and.
You can expect it to be tapped,clicked once, but what if the user
(29:32):
actually interacts with the app atthat point, like a human operator on
the simulator or something like that.
Like in that case, you'll get a Furioustap event and it'll be or you'll
get like a tap event that the testwasn't expecting and then you'll get a
failure that you don't really care for.
There's also, in that same vein,attachments, were added, I'm just
(29:53):
jumping around in this article.
So is, is attachments likeany sort of like metadata
you want to attach to a test?
Yeah.
So this will be, this is alreadypretty useful to have as like.
I can attach like a generatedJSON payload in the test.
And then for debugging reasons, I want toinspect what this JSON was so that I can
(30:14):
more easily see like, oh, this is actuallyan issue with my production code, or this
is an issue with the test, or so forth.
But.
They're also really useful in automationtests where they're used for storing
screenshots of what the app was doing,at the time, or even videos and such.
I don't remember if the currentcode does this, but I know that
(30:35):
Xcode 26 supports recording video
Like failing automationtests, which is really cool.
Yeah.
That's a thing that is probablynot necessary, but makes the
experience of debugging automationtests much easier, especially when
these things are running on ci.
both of those are things that I'm reallyexcited for in terms of what they enable.
(30:58):
Let's talk about it.
Actually before we get into that,you wanna talk a little bit more
about specifically your proposalregarding polling confirmations?
Yeah.
one of the libraries that Imaintain is called Nimble.
Nimble is a library that I believe,Swift Testing took a lot of, or at least
the pound expect and pound require.
(31:20):
APIs took a lot of inspiration from,I haven't asked to confirm this
but that provides a, syntax for,it's also inspired by ARS Spec.
So, or Nibble is inspired by ARS Spec.
So like there's that too.
It provides just a syntax for that'sbetter than the XCT assert, in my opinion.
For comparing v. Asserting that somebehavior happened, something happened.
(31:47):
one of the features of Nimble is somethingthat I call pulling expectations,
which is like the form is, expectsomething to eventually something else.
So the idea of a pollingexpectation is that it will
run the thing that you expect.
the value being sent intoexpect is actually a closure.
So it'll run that closure many timespulling it continuously until it passes
(32:12):
the matcher, in this case, equal two.
So in this case, or in the example I gave,that would always fail, but if you gave
it like a method and then that methodmight change, how its return value is
over time then that might eventually pass.
This is something that likeI think is really necessary
for Swift Testing, to support.
You can.
(32:33):
Kind of do it as a third party, but asI found it actually implementing this
as a pitch there are problems with themore naive solution or approach to this.
So yeah, in the article I give anexample of like, all right, you call
some method on a background task.
And then in the test youwanna say like, I want to.
(32:56):
Confirm that this eventually, you know,the method being run in the background
that it eventually does its work.
And that's precisely what this is doing.
like in your example, it's like itkeeps calling raised dolphins until the
account gets up to one, Is that correct?
So what happens is that raiseddolphins is called once but.
(33:18):
As part of
Okay.
yeah, as part of, and as part of like theinternal implementation of it, it might
increment the number of dolphins on it.
And so then you're just checkinglike that is that eventually there
is at least one dolphin there?
I assume it would take some sortof time interval or like duration.
'cause yeah.
Yeah.
really interesting,
(33:39):
yeah.
That's been like a thing of, that'sthe, time interval thing of like
when to cut this off has been themost eye-opening thing about this
right?
Swift Testing submits every singleone of your tests to be run at once,
and it submits it all to the Swiftconcurrency shared thread pool.
(33:59):
So on your system, you might have thatshared thread pool, might have like
10 threads available, but if you havelike 300 or so tests, you know, to run,
Then it's gonna run those all and theymight all be running concurrently, or
at least all spend some amount of timeon that, shared thread pool before,
a wait causes the next one to run.
(34:21):
And so that prevented anissue with the timeout.
Or the desire to have this automaticallytimeout after like one second.
If you have 300, 3000, you know,10,000 or whatever tests to run on it.
And then your test gets run, youthen like await to as part of this.
(34:41):
There's no guarantee.
And in fact, the more tests you have,the more likely, and especially in
like resource constrained environmentslike ci there's no guarantee that
you'll actually get that callbackwithin that, like one second timeout.
So on like extremely large onextremely large test suites, this will.
Polling confirmations as I originallyimplemented it, I have fixed that now.
(35:04):
Will be increasingly flaky and unreliable.
Okay.
That makes sense.
Yeah, disappointing.
Yeah.
before we close out, a lot of peopleare curious about like CUI tests.
They're still very much tied to the oldObjective C APIs and things like that.
(35:24):
Where, what do you see as likethe future of that as far as
being more Swift friendly,
I mean, it's gotta be likeSwift Testing support.
There's a lot of, it is a lot ofthose APIs are very extremely typed.
So ideally there'll besome way to make them.
Not be as much there is.
(35:46):
I don't really see how, I guess there isthe, what is it, the dynamic callable or
like the dynamic property accessor from awhile ago that could be used to at least
make it feel more Swifty, even if itdoesn't actually provide any guarantees.
that might be it in terms of like makingit feel more like idiomatic Swift.
(36:06):
But like I think what they'remore interested in doing is
enabling, like using the automationframework with Swift Testing.
Okay.
Because one of the things that they did, Idon't remember whether it was 16 three or
16 four, but they pulled out or Xcode 16,three or four they pulled out the, audit.
They made a new XC automation framework,which is separate from XC test, which
(36:31):
is another one of those reading betweenthe lines of like, they're clearly
going in the direction of enablingautomation tests with Swift Testing.
What is the automation framework exactly?
It's all of the XCUI test APIs.
So it's like the XCUI element,so you can get an element by
the, accessibility identifier.
(36:53):
it's like the XCUI application andinteracting with it through there.
Is that used outside oftesting for like Mac OS apps.
No, it's not.
It's entirely meant foruse with automation tests.
Because I know like there'sapps that I'll always ask for,
like accessibility, access.
That is a different, that is also it's adifferent automation library from Apple.
(37:17):
Great.
so that is much more of likejust being able to interact
with the system through the.
Automation A PII believe.
I wouldn't be surprised if XEautomation is fundamentally using
like those same APIs under the hood,
Under the hood.
but I don't have any way to verify that.
Anything else you wanted to talkabout, especially with dub or
(37:41):
testing before we close out?
Yeah, The other stuff I'm reallyexcited for all of the, concurrency
improvements with Swift six two.
I've been diving into those andjust been really excited for that.
The defaulting to running everything orconstraining everything to the main actor.
As opposed to non isolation issomething that I'm excited to see.
(38:03):
Though it is right now very buggy.
But you know, it's also just happened.
It's early in the beta season.
We'll see how it turns out.
Well Rachel, thank you somuch for coming on the show.
it's great to have you onand talk about this stuff.
Where can people find you online?
yeah, I am.
my blog is at rachel brindle.com.
(38:25):
I am on, Mastodon at unata@hackderm.io.
And I. Instagram, though,that's mostly flying stuff now.
at Rachel Brindle.
Yeah.
Thank you again.
we'll put links to the proposal,your blog post, all that
stuff in the show notes below.
(38:46):
People can find me on Mastodonat Leo G. Dion at c Im I'm on
Blue Sky as well at Leo g Dion.
Take a look at me there.
Links are in the show notes.
If you really enjoyed thisepisode, please like and subscribe.
Early access is to those whojoined Patreon, so thank you
for all the Patreon supporters.
It's super helpful.
(39:06):
Thank you so much and I look forwardto talking to you in the next episode.
Bye everybody.
Welcome to anotherepisode of Empower Apps.
I'm your host, Leo Dion.
Today I'm joined by Rachel Brindle.
Rachel, thank you so muchfor taking time to come on
Yeah,
before we get into Swift Testingand all the new stuff with testing.
I'll let you go aheadand introduce yourself.
Yeah, so I am Rachel.
I am a principal engineer at Autodesk.
(00:25):
And I have been involved withtesting, test-driven development and
such, basically my entire career.
A lot of fairly deep knowledge of likehow to go through and test things, as
well as mentoring others in learninghow to test code, how to write better
tests that aren't brittle and won'tbreak all the time and so forth.
(00:47):
Awesome.
So what year did Swift Testing come out?
That was, It shipped in Xcode16 last year, but it was open
sourced in September, 2023.
Okay.
So you did a lot of work with XCTesting.
You have a few libraries you actuallybuilt, for XCTesting, to help folks.
(01:07):
What did you think about the processas far as Swift Testing is concerned
and how we got to where we are now?
How we went from XCTestingto Swift Testing.
And what do you miss?
what are you happy that wegot rid of Things like that,
So Swift Testing is obviously, 'causeit's only like publicly, it's only
(01:29):
like a year and a half old, thoughI'm, I know for a fact that like,
it didn't just pop outta nowhere.
they've been working on it sinceat least macros were a thing.
Swift Testing, if it's stilllike no more than, it can't be
older than like three years.
It is.
Nowhere near as mature as XCTest is.
XCTest has been, was introducedin iOS seven, iOS eight era.
(01:55):
But even still it dates back to mucholder Objective C focused testing APIs
like syn testing kit and so forth.
And those date back allthe way to the nineties.
I was gonna ask about that.
'cause like XCTest, I would assume.
Like you said is from quite a while sinceObjective C. 'cause I can't even imagine
(02:16):
what they used for testing before that.
Yeah, it was, I don't think Applehad, so it was initially started as
some, third party, like predating.
It might even predatelike Apple buying next.
I don't that.
That I, I don't quiteknow the history there.
But it was, there was a send testingkit and like a bunch of like other
(02:40):
third party stuff is what people used.
At some point in the two thousands,apple started shipping or like including
send testing kit with, Xcode so thatyou didn't have to download that, And
they started to just refine that overtime and eventually they rewrote it
with XC test and, and shipped that.
Okay, so Swift tests, like the big thing Iwas gonna talk about is what you miss from
(03:08):
XCTesting, but then like a part of thatis just what we could do with Objective
C that we can't get away with in Swift.
So there is the open source XCTest.
it's Swift CoreLibs XCTest and thatis written entirely in Swift and
that enables that has been aroundsince Swift package manager was open
sourced back in like 2015 or so.
(03:30):
And that is not feature complete withlike the private XCTest from Apple.
But it does a lot of whatyou can do with XCTest.
XCTest is really much better for buildingtesting tools on top of, like you can
dynamically create and register testsat runtime and XC test, which is not a
thing you can do in Swift Testing at all.
(03:51):
I know that there are people onthe Swift Testing team interested
in enabling something like that,but it's just not there yet.
Though reading between the lines,they're definitely moving to enable
that feature in, Swift Testing.
And then there's other minorstuff like, performance tests.
The XCTest has the measure API, soyou can Figure out how long it takes
(04:15):
to run a bit of code so you can seelike, oh, do I have any performance
regressions in this area and such?
And you can kind of do that withSwift Testing, but for a bunch of
reasons, Swift Testing's, concurrenttest runner being a major issue there.
it's not great.
And of course there's not reallyany equivalent to the measure API.
(04:35):
So what did you really like aboutSwift Testing and where it's headed
right now compared to XCTesting?
So the thing that I really likeabout it first, the DSL of the
experience of actually writing testsis much nicer than with XCTest.
The macro or just being able tospecify like the attest macro and
(04:56):
then just like a bear function, isgreat, especially in Swift six two.
I think it's se 4 51 ads just likeyou can, I think it's bear function.
You can use a backtick and then put anarbitrary string or nearly arbitrary
string between back ticks and likethat's the name of your function.
So you can have spaces commasand punctuation in there.
(05:20):
that will render nicely as a full humanreadable string so you don't have to
do this weird thing where like you haveattest and then whatever human readable
string and then a non-human readable, butlike valid function identifier afterwards.
Okay, so now you could just makethe function whatever you want,
and put whatever characters you
yeah.
That's crazy.
That's useful for non-testing stuff,but mostly it's useful for testing.
(05:44):
That's like reading the proposal for that.
That's like the main motivation they cite.
Yeah.
Yeah.
So yeah, like the, going back to otherthings that I really like about, Swift
Testing I like how it's much more open.
Both like it's developedmostly in the open.
there is a testing workgroup, which I'm a member of.
(06:07):
And it's.
It, follows a form of theSwift evolution process.
Like testing proposals arein the Swift evolution repo.
But the proposals are formattedslightly differently and such so
I'm really liking that there's alot more community engagement in
the development of Swift Testing.
Yeah.
(06:27):
And I think that's the biggest advantageis the fact that it's open source and
that we can build a whole set of toolsaround that and help it facilitate its
growth, what point do you think it'dbe important for a team to migrate
from XCTesting to Swift Testing?
Really that depends on the team,there's a ton of tests that are
written right now that cannot beeasily migrated or like can't, it's
(06:50):
not gonna be an easy process to migrateit from XCTest to Swift Testing.
Do you have an example?
So like anything that accesses like, ormodify some like shared global state,
For like one specific test, youmight change it to some value.
And then another test wantsit to be some other value.
because Swift Testing runs everythingconcurrently in process, then
(07:12):
you're gonna run into issues ifthose two tests are ever actually
like running at the same time.
Should you be doing that in the first
I mean no,
Okay.
Yeah.
Yeah.
In a way it forces you to be cleaner.
Like you'll end up replacing that typeof thing with a task local or whatever.
And that also brings along all theniceties of like, you're now going
to force your tests to actuallylike build under the Swift six
(07:33):
language mode and all that such.
But there are plenty of tests that existright now that work perfectly fine in
the world with XCTest where it's only onetest is running at a time in the process.
That are going to be difficultto migrate to Swift Testing.
Okay.
And then the other benefit is it'llbe slightly easier to read stuff.
(07:55):
You'll get faster runtime of testsbecause they can execute in parallel but
you're gonna get slightly slower buildtimes because macros take a bit of time.
Even tool chain macros, they just add alittle bit more time to the compile time.
automation tests stillaren't a thing and so forth.
But yeah, if you have working XCtests, there's no reason to, unless
(08:19):
you really want to, there's no reasonto go rewrite them into Swift Testing.
Maybe.
So like that's the approachto that idea is new components
are written in Swift Testing.
But if it's an existing component thatalready has tests written for it, then
I'm just gonna keep it in XC test.
Do you think macros have aplace in filling the gap when
(08:39):
it comes to what's missing?
In Swift Testing?
Yes and no.
So well for I guess to correctlyanswer, yes, they have a gap.
But like
they are very painful to write andmaintain as I'm sure you don't need to
be reminded, I looked at how, the expectand require macros and Swift Testing
(09:00):
are implemented and like mad respect tothe people on the testing team who wrote
them, because I do not know if I could.
And there's also the other stuff oflike, until very recently you had to
rebuild sw, Swift, Swift syntax for like.
Third party macros and such, thatisn't really a thing now that we
have like the cast with Swift syntaxas of like 16 four or code 16 four.
(09:25):
But like there are, thereactually are places for macros.
Every so often someone will postin the Swift forums, the community
showcase about a library that usesmacros to generate test doubles.
Conforming to the protocol specified.
So a test double is just kind ofa stand in for the real thing.
Okay.
So you might have your type that hassome methods or whatever and you have
(09:51):
a protocol that defines that type.
You might then use one of these tools to.
Create a test double that automaticallyconforms to and implements all of
these methods such that you canthen either just directly pass it
in and it'll work in the thing.
Or you can modify behavioror you could even examine how
it was called and so forth.
(10:11):
that's really cool to see andthat saves a bunch of boilerplate.
Yeah, so that's one of the main examplesof using macros to fill in the gap and
do things that Apple probably isn't gonnado, at least not with Swift Testing.
At least not in the next decade.
Yeah, you had this like marketablemacro, which is really cool.
(10:31):
I just started looking at that, becauselike that's one of the things people
love about Objective C is everythingis dynamic so you can go in and swizzle
and do all sorts of crazy stuff, whereasSwift is super strict about that stuff.
Like for good reason, like, outsideof using tests to like mock out
behavior, we might get the into thisin a, in a little bit, but like this
(10:55):
is part of what made testing UI kita lot less painful was being able to
swizzle out a whole bunch of stuff.
Which you can't even do any ofthat with Swift test with Swift UI.
And like Swift UI is also not untestable.
Yeah.
Well, let's talk about that.
That's one of the biggest complaintsis how SwiftUI is testable.
What does that mean exactly?
(11:15):
Yeah, so you can write tests forSwift UI using Swift UI but you can
only write automation tests for it.
So that's like the XCUI test stuffthat, works by spinning up your app in
the simulator and then having anotherprocess that actually runs the tests that.
Uses IPC to interact with the appusing the accessibility system.
(11:40):
Yep.
and out of the box you can onlylike, just interact with the
UI and examine stuff in the ui.
You can't like go in and examine oh,did this insert something into, did
this actually insert something into thedatabase or is the UI just lying to me?
Out of the box, it's going to makelike every single network request
and so forth, which it's not really.
(12:01):
But there's a bunch of third party stuffto make, automation tests, or like to
fake out network requests and so forthand like enable you to better inspect
stuff going on at automation tests.
Yeah, well, like one thing I hear a lotabout is snapshot testing as a way, which
I think is from the point free team.
so there are several waysyou can do snapshot testing.
(12:23):
There's an, older, API from I think Uberwas the first who made it iOS snapshot
test case both of these, the 0.31 doesa little bit more, you have a little
bit more options to actually inspect it.
it works by just rendering the UI intoan image and then basically working
entirely as a regression test of like,you render the eye into an image.
(12:45):
You test that and you just assert that.
The image is the same as apreviously recorded image
Yep.
That can work either as like, likean XCUI automation test or as more
of a unit test, like in process test.
And that's fine.
Like it works for likepure regression stuff.
(13:06):
There's, it's very brittle.
Like I do not envy the people with,with snapshot tests, going into
who are probably not having a veryfun summer, having to rerecord all
of their stuff for like the liquidglass design, language change.
And especially like eachnew beta, it's gonna change.
(13:27):
But even going between like minorversions of the os, there, there
are like gonna be slight changesto how things render and so forth.
There's even like stuff of whether whenthe simulator runs on like an Intel
or like a, apple, silicon Mac you,it will render it slightly different.
So you have to do like some sortof like fuzzy image compare instead
(13:47):
of just doing a straight up,
You're right.
pixel by pixel compare.
Okay.
so like snapshot tests work,but they're extremely brittle.
When I say that Swift UI isn'ttestable, what I mean is that I
can't write a unit test for it.
One of the things that you can do withUI kit is you can directly call the
(14:10):
methods that Youi kit itself callswhen it's like when you tap for that
lead to a button callback being called.
So it's possible for me to write a testthat's just like, Hey, when this button is
tapped I write to the database and I canactually then inspect that the database
was written to, or that a test double thatrepresents the database was talked to.
(14:32):
and that's not somethingyou can do with Swift UI.
You can't simulate a button.
Press in test.
could you use something likean inspect and push the button?
Yes, there is a third partylibrary called a view Inspector.
I'll add it to your,show notes To its credit.
It does a really good job of allowingyou to inspect and test like, Swift
(14:55):
UI stuff, Swift UI views and so forth.
But it's incomplete.
there's a bunch of APIs from Apple thatit doesn't have good inspection for.
it uses the mirroring API tobe able to inspect things.
And then it has to know about specific.
Like sub components within to, or liketo go like inspect, oh, this was the, UI
(15:16):
button or the, sorry, the Swift UI button.
tap handler and such.
It doesn't automatically support newApple provided, Swift UI views by default.
It does work with like yourown third party views 'cause.
But it works.
It's also brittle, it's vulnerableto internal changes to in Swift UI of
(15:37):
That's exactly what I was gonna say
that happened last year.
So, yeah.
really what I would love to seeis first party support for that.
I know for a fact that every time, 'causeI bring it up every year and a WW TC lab
that both the Swift UI and testing teamsget that very frequently as a request.
(15:58):
But for various reasons theyhaven't been able to deliver that.
How would you picture something like that?
Like you just create a Swift toeye view and then some sort of
select, like how does web do it?
Right?
Web would just be like, oh, grab thebutton using a selector, and then push.
(16:18):
Yeah.
So that's exactly what I'm thinking.
Like, something that I've beenworking on as like a blog post is,
exploring what that might look like.
Through the lens of like, this iswhat we might try to do for UI kit and
then because we can do it for UI kit.
I like the approach, the interactingthrough the accessibility system
(16:39):
that automation tests take.
'Cause that already exists,that forces you to actually
add accessibility, support,
I was gonna say that.
And such.
I think that'd be areally good thing to do.
plus it's also fairly, like blackbox, like you could call in an
automation test, you can call a tapon an element that cannot be tapped.
It's non-interactiveand nothing will happen.
(17:02):
Like you won't get a compile error.
It's just that nothing will happenand your test will silently fail.
or maybe it won't fail dependingon what your next assert is
that's something that I think is.
Really powerful as a direction togo maybe not exactly mirroring the,
public API of automation tests.
But that's a great starting point oflet's interact through the accessibility
(17:23):
system to be able to like pull on,pull an element that should exist.
and then like be able to interactwith it, test or tap on it,
swipe on it, and so forth.
Yeah, that makes complete sense.
So I wanna jump back, talk a littlebit more about Swift Testing.
So one of the things I really likedabout it is the way that you can organize
(17:44):
your tests, without needing to docertain things with classes and stuff.
Kind of explain the whole suite testingthing and what are some things developers
should be thinking about when they writetheir tests and how they organize them.
Yeah, one of the really cool thingsabout Swift Testing is the fact that
you can nest sweets as much as you want.
(18:07):
you can have zero sweets, so just barefunctions that are with that test macro.
You can organize, you can havelike a single layer of sweets.
You can have as many as you want.
There is, you do kind of run into theissue where, if you have properties on
like the parent suite, a child suitecannot reference it because they're
(18:29):
structs and they can't know about,they don't, they're not getting like
an a reference to like an instanceof the parent struct and so forth.
So you can't you have that limitation.
I'm hoping that once apple.
Provides like more APIs for likedynamically adding tests and so
(18:49):
forth, dynamically that there'llbe community solutions to that.
Something like what the quickframework provides for XCTest.
Could you give like a use case of that?
so
a parent suite info or metadata.
Yeah, yeah.
So that, that's something like, Even justas simple as like having shared setups.
So you only have to create the objectbeing tested like in one place in the,
(19:13):
like in it for the, in like the parentor whatever, the outermost suite.
And then later on, like you might havelike a child suite that is just calling a
method on that object that's being tested.
That's.
One way to do that.
I also like to nest suites, or Iwould like to nest suites for like,
(19:34):
conditional and such, a child, onesuite for if this conditional is true.
Another for if it's false and such.
Because then you also get like the nicefo code, code voting capabilities well, I
don't really care about the positive caseright now, so I'm just gonna fold that
up and only look at the negative case.
Like what,
what I use a lot, and we'll get intotraits later, but I have to have, I'll
(19:59):
have a suite that I do a lot of fullstacks with stuff and so I'll have
something that's only supported on Appleplatform, so I end up having to do like
pound if can import or something like thatto kind of distinguish between the two
SOI, for instance.
And then I have to create a variableand then use that as a global variable
(20:20):
that says whether it's available or not.
which it's kind of a mess, but it works.
We'll talk about traits morebut back to like sweets.
What do you think, what aresome tips you have as far as how
you should organize your tests?
Yeah, I personally tend to just do onelayer of sweets just because I would
like to nest them for conditionals,but doesn't really work out well.
(20:42):
So I just do one layer and then eitherorganizing them with tags or using.
good old Mark comments.
Like I usually just do, I'mstill stuck in the way of like.
I'll have a test suite lined up witha type essentially and do it that way.
Is that, do you feel like that'sstill a good way to go about
(21:04):
that's how I do it.
I'll have like a suitethat's like, my object tests
Okay.
I'll
And it's good for like codecoverage too, because then you could
tell, okay, what is this testing
Yeah.
But you could also dothat, without the suite.
It does make it nicer in terms ofgetting the, test diamond, on the
left side of the editor because thenyou can just select everything there.
(21:27):
But if you didn't want that, Youcould just have them all on top of
the file and then have the file beyour top level organization for it,
Got it.
but then you don't get likeshared setup and so forth.
Right, exactly.
What are some other traitsthat people should be aware of?
I would recommend making like a customtrait that automatically does the, like,
oh, if Mac, if iOS, if Linux or whatever,
(21:51):
How hard is it to do that?
it's not that hard.
you can, create a type that conformsto, you'll probably want to do
both test trait and sweet trait.
and if you do conform to sweet trait,You should implement the is recursive
and then just have that be true isrecursive means that for a sweet
trait, it also applies that trait toevery single like child of it, all the
(22:14):
tests, all the sub suites and so forth.
Yeah, that makes total sense.
you can just have it say like, thisalso is a disabled trait, or like,
this marks the test as disabled.
A fun project for me
yeah, but it shouldn'ttake that long to do.
Okay.
There are the, comment in bugtraits, which just allow you to
associate like text information.
(22:34):
With a tester suite.
Okay.
There is the serialized trait whichforces the test covered by that suite
or the like parameterized test torun serially, so not concurrently,
Okay.
That's kind of weird.
'cause that only applies tolike those specific tests.
if you have like two like sibling teststhat both have like the serialized trait
(22:59):
and you run them, but like the parents,the containing suite isn't serialized.
If you run those, then like those twotests will actually run concurrently
or like in parallel with each other.
Should you be doing serialized tests?
it's a thing that exists.
Especially in a post Swift six world, like
(23:20):
I haven't run into like aspecific use case that needs it.
I am a hundred percent certainthat that use case exists,
otherwise it wouldn't exist.
right.
But like every time that I've usedthat I've reached for it, it turns
out that actually what I shouldhave just done is made whatever the
thing that was forcing the thingto be serial to actually make it.
(23:42):
Be thread safe, like changea global variable into a task
local or something like that
so I did want to cover your article,which I'll put in the show notes, but
what are some highlights about, what'snew this year that folks should really
start looking at using in their tests?
Okay, so the major new change is exittests, which doesn't apply to iOS.
(24:08):
on the server side team?
Yeah, service side, even Macdevelopment windows and such.
They allow you to verify that.
Code exits properly as you expect
code.
returns an exit code.
It throws
Okay.
or something like that.
So you can basically say, Hey, makesure the signal is the right integer.
(24:31):
Yeah, so you
standard error context.
What?
How do you check that?
so,
The result has STA standarderror content as a property.
the example I wrotecould have been better,
Now.
'cause I didn't understand whatresult was and I was like, I'm
looking at this, and I'm like, okay.
So result is basically some sort ofstructure, which has all that metadata
Yeah, so
(24:51):
okay.
it has the, if you tell it, ifyou tell your exit test to grab
the standard error, then it will.
Then that, standard errorcontent will be non nil.
If you specify with standard output,it will also be a non nil thing.
Those are just an array of, Ithink it's U eight but an array
of, yeah, it's array of U eight.
(25:12):
It's array of bites that if you wanna,you then have to like convert it to a
string using one of the methods there.
If you wanna do a string comparison.
there are like some weird beta bugsin there right now where it will
like, capture a lot more of thelike standard error than you need.
And that's just kind ofhow the, posik spawn, API
(25:35):
Oh, really?
So yeah, it's.
But it works really well.
I'm really impressed with how this works.
you can't do this with XC test at all.
Like Apple's never added support for this.
So if you wanted to verify thatyour function had a precondition or
such in it, and that preconditionworked as you expected, you had
(25:56):
to go way outta your way to like.
Construct something to do that, whichno one has ever done in their life.
Yeah
so it's really nice to have this.
that sounds like a server side thing.
Yeah, so really, I thinkI want it to come to iOS.
the sandbox in iOS is a thingand that's gonna be difficult.
(26:17):
what's the use case for iOS?
Because like if the, the app shouldn'tcrash, I mean, it should just never crash.
Right.
But you do have APIs whereit's like, Hey, I expect, for
example, array subscript, right?
So, or like subscript an array.
If you pass negative one then, or if youpass like an out of bounds index, then
fatal error,
a fatal error.
(26:38):
And like that.
Whether or not it should or, orshould return nil or do or like throw
an error or something that you canactually handle is another question.
But that sort of thing exists andwe should be able to like handle
that test case just to have likethat coverage that it's working.
I was about to ask that.
'cause a thousand years ago when Idid a, an episode with on this very
(26:59):
subject, I was asking like, how do youtest for fatal error or precondition
or assert or any of that stuff?
And the way we've had to doit is basically have a way to
like overload it essentially.
But we don't have anything like that.
And this, which what it sounds like isthis would work in that case, but then
at the same time it wouldn't work on iOS.
(27:21):
Is that correct?
Yeah, so the way that I would approachthis is I would, I would either just
not, or I would like put in like aninject, like a, an injectable using
like dependency, injecting like a shimthat just in actuality it calls, as
you were just saying, like it justcalls precondition and actuality.
And then in test we can examinethat it was called that.
(27:42):
For this I might either write, likeif it's a cross platform app, like
I wanna have it run on Mac OS andiOS then I would just have the.
Have it be covered under the testfor macOS with like a pound if or
like if os macOS stuff covering that.
(28:03):
There's a few things that arepromising that isn't in there yet.
The thing that I didn't write thatI'm most excited for is a thing
called, issue handling traits.
And these allow you tofilter or modify issues.
So like if you have a poundexpect that fails, like it.
Pound expect true, equal, equal false.
(28:25):
That's, guaranteed failure there.
Then it allows you to, specify a traitthat allows you to modify the issue.
You might wanna convertthe, severity of it.
you might wanna just entirely
Yeah.
Okay.
That's really powerful.
I don't think it's gonnahappen in Swiss six two.
But you think it'spossible in the next six
(28:48):
Six probably six three.
It's actively being discussed.
Yeah,
there's other stuff there that's,exciting, from what it enables, like,
range confirmations, which were mergedin, Swift six one and have been out
since Xcode 16, three earlier this year.
This is a thing that they view asnecessary for automation tests.
(29:10):
So range confirmations is basicallyjust, you expect the callback to be
called within some amount of range.
And the example given in the proposalis using an automation test of
like, oh, you have a button and.
You can expect it to be tapped,clicked once, but what if the user
(29:32):
actually interacts with the app atthat point, like a human operator on
the simulator or something like that.
Like in that case, you'll get a Furioustap event and it'll be or you'll
get like a tap event that the testwasn't expecting and then you'll get a
failure that you don't really care for.
There's also, in that same vein,attachments, were added, I'm just
(29:53):
jumping around in this article.
So is, is attachments likeany sort of like metadata
you want to attach to a test?
Yeah.
So this will be, this is alreadypretty useful to have as like.
I can attach like a generatedJSON payload in the test.
And then for debugging reasons, I want toinspect what this JSON was so that I can
(30:14):
more easily see like, oh, this is actuallyan issue with my production code, or this
is an issue with the test, or so forth.
But.
They're also really useful in automationtests where they're used for storing
screenshots of what the app was doing,at the time, or even videos and such.
I don't remember if the currentcode does this, but I know that
(30:35):
Xcode 26 supports recording video
Like failing automationtests, which is really cool.
Yeah.
That's a thing that is probablynot necessary, but makes the
experience of debugging automationtests much easier, especially when
these things are running on ci.
both of those are things that I'm reallyexcited for in terms of what they enable.
(30:58):
Let's talk about it.
Actually before we get into that,you wanna talk a little bit more
about specifically your proposalregarding polling confirmations?
Yeah.
one of the libraries that Imaintain is called Nimble.
Nimble is a library that I believe,Swift Testing took a lot of, or at least
the pound expect and pound require.
(31:20):
APIs took a lot of inspiration from,I haven't asked to confirm this
but that provides a, syntax for,it's also inspired by ARS Spec.
So, or Nibble is inspired by ARS Spec.
So like there's that too.
It provides just a syntax for that'sbetter than the XCT assert, in my opinion.
For comparing v. Asserting that somebehavior happened, something happened.
(31:47):
one of the features of Nimble is somethingthat I call pulling expectations,
which is like the form is, expectsomething to eventually something else.
So the idea of a pollingexpectation is that it will
run the thing that you expect.
the value being sent intoexpect is actually a closure.
So it'll run that closure many timespulling it continuously until it passes
(32:12):
the matcher, in this case, equal two.
So in this case, or in the example I gave,that would always fail, but if you gave
it like a method and then that methodmight change, how its return value is
over time then that might eventually pass.
This is something that likeI think is really necessary
for Swift Testing, to support.
You can.
(32:33):
Kind of do it as a third party, but asI found it actually implementing this
as a pitch there are problems with themore naive solution or approach to this.
So yeah, in the article I give anexample of like, all right, you call
some method on a background task.
And then in the test youwanna say like, I want to.
(32:56):
Confirm that this eventually, you know,the method being run in the background
that it eventually does its work.
And that's precisely what this is doing.
like in your example, it's like itkeeps calling raised dolphins until the
account gets up to one, Is that correct?
So what happens is that raiseddolphins is called once but.
(33:18):
As part of
Okay.
yeah, as part of, and as part of like theinternal implementation of it, it might
increment the number of dolphins on it.
And so then you're just checkinglike that is that eventually there
is at least one dolphin there?
I assume it would take some sortof time interval or like duration.
'cause yeah.
Yeah.
really interesting,
(33:39):
yeah.
That's been like a thing of, that'sthe, time interval thing of like
when to cut this off has been themost eye-opening thing about this
right?
Swift Testing submits every singleone of your tests to be run at once,
and it submits it all to the Swiftconcurrency shared thread pool.
(33:59):
So on your system, you might have thatshared thread pool, might have like
10 threads available, but if you havelike 300 or so tests, you know, to run,
Then it's gonna run those all and theymight all be running concurrently, or
at least all spend some amount of timeon that, shared thread pool before,
a wait causes the next one to run.
(34:21):
And so that prevented anissue with the timeout.
Or the desire to have this automaticallytimeout after like one second.
If you have 300, 3000, you know,10,000 or whatever tests to run on it.
And then your test gets run, youthen like await to as part of this.
(34:41):
There's no guarantee.
And in fact, the more tests you have,the more likely, and especially in
like resource constrained environmentslike ci there's no guarantee that
you'll actually get that callbackwithin that, like one second timeout.
So on like extremely large onextremely large test suites, this will.
Polling confirmations as I originallyimplemented it, I have fixed that now.
(35:04):
Will be increasingly flaky and unreliable.
Okay.
That makes sense.
Yeah, disappointing.
Yeah.
before we close out, a lot of peopleare curious about like CUI tests.
They're still very much tied to the oldObjective C APIs and things like that.
(35:24):
Where, what do you see as likethe future of that as far as
being more Swift friendly,
I mean, it's gotta be likeSwift Testing support.
There's a lot of, it is a lot ofthose APIs are very extremely typed.
So ideally there'll besome way to make them.
Not be as much there is.
(35:46):
I don't really see how, I guess there isthe, what is it, the dynamic callable or
like the dynamic property accessor from awhile ago that could be used to at least
make it feel more Swifty, even if itdoesn't actually provide any guarantees.
that might be it in terms of like makingit feel more like idiomatic Swift.
(36:06):
But like I think what they'remore interested in doing is
enabling, like using the automationframework with Swift Testing.
Okay.
Because one of the things that they did, Idon't remember whether it was 16 three or
16 four, but they pulled out or Xcode 16,three or four they pulled out the, audit.
They made a new XC automation framework,which is separate from XC test, which
(36:31):
is another one of those reading betweenthe lines of like, they're clearly
going in the direction of enablingautomation tests with Swift Testing.
What is the automation framework exactly?
It's all of the XCUI test APIs.
So it's like the XCUI element,so you can get an element by
the, accessibility identifier.
(36:53):
it's like the XCUI application andinteracting with it through there.
Is that used outside oftesting for like Mac OS apps.
No, it's not.
It's entirely meant foruse with automation tests.
Because I know like there'sapps that I'll always ask for,
like accessibility, access.
That is a different, that is also it's adifferent automation library from Apple.
(37:17):
Great.
so that is much more of likejust being able to interact
with the system through the.
Automation A PII believe.
I wouldn't be surprised if XEautomation is fundamentally using
like those same APIs under the hood,
Under the hood.
but I don't have any way to verify that.
Anything else you wanted to talkabout, especially with dub or
(37:41):
testing before we close out?
Yeah, The other stuff I'm reallyexcited for all of the, concurrency
improvements with Swift six two.
I've been diving into those andjust been really excited for that.
The defaulting to running everything orconstraining everything to the main actor.
As opposed to non isolation issomething that I'm excited to see.
(38:03):
Though it is right now very buggy.
But you know, it's also just happened.
It's early in the beta season.
We'll see how it turns out.
Well Rachel, thank you somuch for coming on the show.
it's great to have you onand talk about this stuff.
Where can people find you online?
yeah, I am.
my blog is at rachel brindle.com.
(38:25):
I am on, Mastodon at unata@hackderm.io.
And I. Instagram, though,that's mostly flying stuff now.
at Rachel Brindle.
Yeah.
Thank you again.
we'll put links to the proposal,your blog post, all that
stuff in the show notes below.
(38:46):
People can find me on Mastodonat Leo G. Dion at c Im I'm on
Blue Sky as well at Leo g Dion.
Take a look at me there.
Links are in the show notes.
If you really enjoyed thisepisode, please like and subscribe.
Early access is to those whojoined Patreon, so thank you
for all the Patreon supporters.
It's super helpful.
(39:06):
Thank you so much and I look forwardto talking to you in the next episode.
Bye everybody.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Special Summer Offer: Exclusively on Apple Podcasts, try our Dateline Premium subscription completely free for one month! With Dateline Premium, you get every episode ad-free plus exclusive bonus content.