This tutorial series is basically showing how Selenium moves from “clicking elements” into real-world browser automation, where pages are messy, slow, and full of UI traps.🧭 1. Core Setup + Basic NavigationEverything starts with controlling the browser:
You can listen and download our episodes for free on more than 10 different platforms:
https://linktr.ee/cybercode_academy
- ChromeDriver setup
- Acts as the bridge between Python and Chrome
- driver.get(url)
- Opens a webpage inside the automated browser session
- locate input field
- clear existing text
- send new text using keyboard input
- submit or trigger search
- fastest and most stable
- common in forms (login, search, signup)
- uses class-based targeting
- flexible and widely used in real projects
- most powerful option
- works even when HTML is messy or missing IDs/classes
- exact link text
- partial link text
Useful for navigation between pages.
- wait until element appears
- wait until element becomes clickable
- pause execution until condition is met
- collect all elements
- loop through them
- match desired value
- click selection
- embedded pages inside pages
- Selenium cannot access them directly
- must switch context before interacting
- accept (OK)
- dismiss (Cancel)
- read alert text
- they load slowly
- they restructure DOM dynamically
- they interrupt workflows with modals and alerts
- Open browser
- Wait for page stability
- Find elements reliably (ID/CSS/XPath)
- Interact carefully (click/type/select)
- Handle interruptions (alerts, iframes, delays)
- Repeat across navigation flows
You can listen and download our episodes for free on more than 10 different platforms:
https://linktr.ee/cybercode_academy
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:00):
So imagine you've got this this digital assistant, and it
literally just sits at your computer, opens up a web browser,
and you does all your digital chores for you.
Speaker 2 (00:09):
Oh, that would be amazing.
Speaker 1 (00:11):
Right, I'm talking about like filling out those super tedious
multi page onboarding forms, or clicking through just endless confirmation buttons,
or even scouring databases for specific information, all at lightning
speed while you just go grab a coffee.
Speaker 2 (00:25):
Yeah. I mean, it sounds like science fiction or at least,
you know, a very expensive enterprise software solution that most
people wouldn't have access to.
Speaker 1 (00:32):
Exactly, But it's actually just standard web automation, it is.
Speaker 2 (00:35):
And the crazy thing about it is to pull this off,
you kind of have to completely unlearn how you view
the Internet. What do you mean, Well, to a piece
of automation code, the Internet has no colors, it has
no shiny buttons, no pretty layouts. To automate the web,
we have to look past the user interface.
Speaker 1 (00:54):
Entirely, right, You're interacting directly with the raw architecture underneath exactly.
Speaker 2 (00:58):
You're dealing with the skeleton of the site, not the
paint job.
Speaker 1 (01:01):
So today, for this deep dive, we are basically going
to act as the translators between human intention and machine execution.
We're going to walk you through step by step how
a script actually parses and manipulates the web page.
Speaker 2 (01:16):
Yeah, and we'll be talking specifically about using tools like
Selenium and uh something called Chrome Driver.
Speaker 1 (01:24):
Right, So let's start with the ignition key. Basically, to
have an invisible assistant click on anything, we need a
way for our Python code to talk to the browser
in the first place.
Speaker 2 (01:32):
Yeah, and that is exactly the role of the Chrome Driver.
I mean, think of it as an executable application that
acts as a.
Speaker 1 (01:38):
Bridge, okay, a bridge between what and what.
Speaker 2 (01:40):
Between your script and the browser. So your script sends
commands to the driver, and then the driver translates those
commands into native browser actions.
Speaker 1 (01:49):
Got it.
Speaker 2 (01:50):
When you instantiate this driver in your code, an actual
instance of Chrome boots up on your machine, but it's
entirely controlled by your script. You can literally sit there
and watch it move on on its own, which.
Speaker 1 (02:00):
Is always a little spooky the first time you see it.
Speaker 2 (02:02):
It really is.
Speaker 1 (02:03):
Now, one of the first things a developer typically does
after booting that instance is they pass an argument to
maximize the browser window right away.
Speaker 2 (02:11):
Yes, that is a crucial step, but on.
Speaker 1 (02:14):
The surface that seems I don't know, purely cosmetic, like
why would the script care how big the window is
if it's just reading the underlying code anyway?
Speaker 2 (02:23):
Well, functionally, it's all about responsive design. I mean, modern
web pages are built to change their structural layout based
on the size of the screen you're using.
Speaker 1 (02:31):
Oh right, Like sites look different on a phone versus
a desktop exactly.
Speaker 2 (02:36):
So if you load a complex web app in a tiny,
minimized window, the site's code might dynamically hide at a
navigation menu. It might shove it behind one of those
mobile style Hamburger icons.
Speaker 1 (02:48):
Oh wow, I see where this is going.
Speaker 2 (02:50):
Yeah, So if your script is hunting for a specific
settings button that has suddenly been hidden or removed from
the active.
Speaker 1 (02:57):
Layout, the automation just crashes, right, It.
Speaker 2 (02:59):
Throws an error because the button isn't there anymore. So
maximizing the window ensures the layout is predictable, It ensures
all the standard desktop elements are rendered and you know,
actually available to interact with.
Speaker 1 (03:12):
That makes total sense. So we've set the stage, we've
maximized the window, and we've used to command like a driver,
do get to navigate to a specific url.
Speaker 2 (03:21):
Yep, the page loads.
Speaker 1 (03:22):
Now to a human user, a search bar is obvious.
It's a white rectangle with a little magnifine glass next
to it.
Speaker 2 (03:28):
Right, you just click it.
Speaker 1 (03:28):
But since our script doesn't have eyes, how does it
actually pinpoint that specific interactive element out of like thousands
of lines of raw HTML code.
Speaker 2 (03:39):
It relies entirely on the Document object model, or the
DOM as most people call it the DOM. Okay, Yeah,
when a browser loads a web page, it takes the
raw HTML text and turns it into this giant inverted tree.
Speaker 1 (03:52):
Structure, like a family tree sort of.
Speaker 2 (03:54):
Yeah, the page itself is the trunk, and every container,
every paragraph, every button is a brand or a leaf
on that tree.
Speaker 1 (04:01):
Okay, I'm visualizing that.
Speaker 2 (04:02):
So the script navigates this tree. If you want to
find a specific element, you need a locator strategy, and
the absolute most reliable strategy is looking for a unique
identifier that the original developers left behind.
Speaker 1 (04:15):
It's kind of like giving our script the exact GPS
coordinates of a specific house in a sprawling city.
Speaker 2 (04:20):
That is a perfect analogy.
Speaker 1 (04:22):
Actually, so if the developers built the site, well, they
assign these unique attributes to elements.
Speaker 2 (04:27):
Yeah.
Speaker 1 (04:27):
Like if you use a browsers and spectool on a
well designed site's search bar, you might see an attribute
called ID, right, like i'd search field.
Speaker 2 (04:37):
Yes, and an ID is the gold standard for locators.
Why is it the gold standard because by definition in
standard HTML rules, an ID is supposed to be entirely
unique on a single web page. It is literally like
a social security number for a web element.
Speaker 1 (04:52):
Wow. Okay, so it's foolproof pretty much.
Speaker 2 (04:54):
If you command your script to findelement bide and pass
it that specific string that driver searches, the dom tree
instantly locks onto that specific node and completely ignores the
rest of the page.
Speaker 1 (05:05):
That's super efficient. But what if the developer didn't use
an ID, because we all know not every site is
coded perfectly.
Speaker 2 (05:13):
Oh absolutely, it happens all the time.
Speaker 1 (05:15):
Like sometimes you inspect an element and it just has
a name attribute, Like the main search box on a
coding tutorial site might just have the name attribute assigned
to the letter Q.
Speaker 2 (05:23):
Right. Q usually stands for a query and the fidelment
by name command works very similarly to ID. It's incredibly fast.
Speaker 1 (05:32):
But I hear a butt coming.
Speaker 2 (05:34):
But name attributes aren't strictly required to be unique.
Speaker 1 (05:38):
Ah, there it is.
Speaker 2 (05:39):
Yeah. So if a developer got sloppy and named both
the top search bar and say a bottom newsletter sign
up form Q.
Speaker 1 (05:47):
Then your script might grab the.
Speaker 2 (05:49):
Wrong one exactly, because the script generally just defaults to
the very first match and encounters as it scans the
document from top to bottom.
Speaker 1 (05:55):
Oh right, so it hits the first que, says found it,
and stops looking. Okay, so IDs and names are the
easy street addresses, But we also have to navigate around
the site. Say I want the script to click a
specific text link that just says my courses.
Speaker 2 (06:10):
Well, Selenium provides a findelement by link text method for
exactly this scenario. Oh nice, Yeah, you just passed the
text of the link to the script, and it hunts
for an anchor tag in the HTML that contains that
exact string.
Speaker 1 (06:23):
I imagine that demand's absolute precision, though.
Speaker 2 (06:25):
Oh it really does.
Speaker 1 (06:26):
It's like telling a taxi driver to take you to
the Grand Hotel and they flat out refuse to move
because the physical sign on the building actually says the
Grand Hotel Incorporated.
Speaker 2 (06:36):
Yes, the machine has zero intuition. It is executing a
literal string comparison.
Speaker 1 (06:43):
So if I tell the script to look for my courses,
but the web designer capitalized it to my courses, yeah,
in all caps.
Speaker 2 (06:50):
The script will just crash. It'll throw a no such
element exception, A single mismatch space, an invisible line break,
or different capitalization will break it.
Speaker 1 (06:59):
That sounds incredibly fragile.
Speaker 2 (07:00):
It can be. But to mitigate that fragility, you can
use a variation called partial link text.
Speaker 1 (07:06):
Oh, or you only need a piece of it exactly.
Speaker 2 (07:08):
You only match a substring. So if the link is
a dynamically generated product title like blue Sneaker size ten,
ID four four five nine, you can just target the
blue Sneakers portion.
Speaker 1 (07:18):
Okay, So that ensures the script finds it even if
the ID numbers or the sizes change tomorrow.
Speaker 2 (07:23):
Right. It gives you a lot more flexibility.
Speaker 1 (07:25):
Got it. So far we've relied on developers giving us
clean IDs, names or exact text links. But what if
we're automating some highly complex, deeply nested web app where
the elements don't have any of those clean attributes.
Speaker 2 (07:39):
Yeah, the tricky stuff.
Speaker 1 (07:41):
How do we find the house if there's literally no
address on the mailbox?
Speaker 2 (07:44):
This is where we shift from simple addresses to architectural blueprints.
When basic locators fail, we rely on advanced locators, specifically
CSS selectors and XPath Okay, let's.
Speaker 1 (07:55):
Start with CSS now. CSS is what dictates the visual
styling of a page, right like colors, padding, Yes, exactly,
and developers apply these styles by assigning elements to classes.
So if a button is red and round, it might
have a class called primary submit btn. We can actually
hijack those styling hooks for automation.
Speaker 2 (08:12):
You absolutely can. Using a CSS selector locator, you can
command the script to find an element based entirely on
how it was styled.
Speaker 1 (08:20):
That's so clever.
Speaker 2 (08:21):
Yeah, in your code, you use standard CSS syntax like
a dot to denote a class, so you'd tell the
driver to find dot primary dash submit dash btn.
Speaker 1 (08:30):
And is that fast?
Speaker 2 (08:31):
It is incredibly fast. The beauty of this is that
CSS selectors are heavily optimized by the browser itself.
Speaker 1 (08:37):
But CSS classes still rely on the developer leaving a
recognizable hook. Sure, what if there was absolutely nothing unique
about the element itself, no class, no ID. That's where
XPath comes in, right, Yes, because I've seen complex XPath
strings and honestly, they look like absolute gibberish. It's just
like slashes, brackets and random HTML tags.
Speaker 2 (08:57):
They do look intimidating. But XPath is the heavy artillery
of web automation. It stands for XML path language. Okay,
remember how we describe the dom as an inverted tree.
This path is a literal set of step by step
directions for navigating the branches of that tree. You can
use it to triangulate an element based entirely on its
relationship to other elements.
Speaker 1 (09:16):
How does that actually map out in the code though?
Speaker 2 (09:18):
Okay, imagine you want to click a specific input box,
but it has no ID, no name, and a generic class.
Speaker 1 (09:25):
Basically a ghost element.
Speaker 2 (09:27):
Right, But you notice it sits inside a larger container,
a divtag that does have an idea of login panel.
Speaker 1 (09:33):
Okay, so the parent container has a clear address exactly.
Speaker 2 (09:37):
Using XPath, you can write a path that says, scan
the document for a div with the ID log in panel.
Once you find it, step inside that container, find the
second paragraph, and give me the input box inside that paragraph. Wow,
it doesn't matter if the input box has no identifying
features of its own. You've located it purely based on
(09:57):
its surroundings.
Speaker 1 (09:58):
It's literally like saying, go to the house house on
the corner of Fifth and Main, walk to the kitchen
and grab the second drawer on the left. Yes, you
don't need a label in the drawer if you know
the exact path to reach it.
Speaker 2 (10:08):
That is the power of x path. It can navigate
down from parent to child, or even up from a
child to a parent, or sideways across sibling elements. It's
incredibly robust.
Speaker 1 (10:17):
All right, So we've mastered finding elements using the dom tree.
But just pointing at a search bar doesn't execute a search.
We have to interact with it, right.
Speaker 2 (10:26):
Once the driver locates the element, it stores it as
an object in your Python code, okay, and you can
then call methods on that object. So to type into
a field, you use the sen keys command. Is a
keys You pass it a string of texts, and the
driver basically simulates the electrical signals of a human physically
typing on a keyboard, sending those exact keystrokes directly into
(10:47):
the input node.
Speaker 1 (10:48):
Wait if it just sends keystrokes blindly, what happens if
the search bar already has pre populated text, Like sometimes
you visit a portal and the search box already says
search by employee name and light gray text. If I
just fire off send keys with the name John, does
it just appendit like? Does it search for search by
employee name John.
Speaker 2 (11:08):
It absolutely will do exactly that. Oh no, yeah, send
keys doesn't overwrite existing values, it appends to them. This
is a classic architectural trap.
Speaker 1 (11:17):
Because humans instinctively recognize placeholder text right we backspace it
or we highlight it before typing.
Speaker 2 (11:24):
Exactly, but the machine does not. Therefore, robust automation requires
explicit instruction. Before you ever send new keys to a
text input, you invoke the dot clear method yeap. This
commands the driver to wipe the input node clean of
any pre existing string values. You locate it, you clear it,
and only then do you send your keys.
Speaker 1 (11:46):
Man, it really emphasizes how much micrologic goes into our
daily browsing habits. Stuff we don't even think about.
Speaker 2 (11:53):
We take it totally for granted.
Speaker 1 (11:54):
So we've cleared the field and typed our query. To
actually run the search, we need to get enter. How
do we do that?
Speaker 2 (12:01):
You import a specific keys class in your script. This
class contains representations of all the non alphanumeric keyboard.
Speaker 1 (12:07):
Keys, like shift, alt enter all that.
Speaker 2 (12:09):
Exactly, So you just deppend keys, dot enter or keys
return to your sen keys command. The driver simulates that
specific keypress, the web server receives the form submission, and boom,
your automated search is running.
Speaker 1 (12:21):
That is so cool. Now, this all works beautifully in
a vacuum. But let's look at the real world, where
the Internet is messy, connections are slow, and web apps
are incredibly complex.
Speaker 2 (12:31):
Oh yes, the real world is a nightmare for automation, right.
Speaker 1 (12:35):
What happens when our script encounters, say, a multi step
sign up process with a really heavy loading screen.
Speaker 2 (12:42):
This is where we hit the fundamental friction of web automation.
It's the speed gap between the CPU and the network.
Speaker 1 (12:49):
The speed gap.
Speaker 2 (12:50):
Yeah, your Python script executes commands in fractions of a millisecond.
It tells the driver to click next. The browser sends
a request to the server, and the server takes say two,
two full seconds to respawn and draw the new page.
Speaker 1 (13:02):
And during those two seconds, our script is already trying
to execute the next line of code right yep, which
is filling out the first name field on a page
that literally doesn't exist yet exactly.
Speaker 2 (13:12):
The script checks the dom doesn't see the first name
element and instantly throws a fatal no such element exception.
It just crashes.
Speaker 1 (13:20):
Okay, So the obvious brute force fix would be to
just tell the script a pause, like use a sleep
command to freeze the code for five seconds after every
single click, just to be safe.
Speaker 2 (13:31):
You could do that, but static sleeps are a terrible
engineering practice.
Speaker 1 (13:34):
Really why well, think about it.
Speaker 2 (13:36):
If you tell the code to sleep for ten seconds
and the page loads in two seconds, you are wasting
eight seconds of compute time.
Speaker 1 (13:43):
Oh true.
Speaker 2 (13:44):
If you scale that script up to process a ten
thousand records, you've just added hours of dead air.
Speaker 1 (13:50):
That's highly inefficient.
Speaker 2 (13:51):
And conversely, if the server is having a slow day
and takes eleven seconds to load, your script still crashes anyway.
Speaker 1 (13:57):
Oh man, Yeah, so we need a dynamic way to
bridge the CPU speed and then network speed.
Speaker 2 (14:03):
Right, And the elegant solution here is something called an
explicit weight. It's often implemented as web driver weight.
Speaker 1 (14:09):
How does that differ from just sleeping.
Speaker 2 (14:11):
Instead of freezing the code entirely? You define a maximum
time out, say fifteen seconds, and you define a specific
condition like wait until the first name element is clickable.
Speaker 1 (14:21):
So how does that look under the hood? Is it
just checking constantly?
Speaker 2 (14:24):
It utilizes a polling mechanism. The driver hooks into the
process and checks the DOM tree every five hundred milliseconds.
Oh wow, it asks, is the element here yet?
Speaker 1 (14:34):
No?
Speaker 2 (14:35):
Half a second later, is it here yet? No? The
absolute millisecond. The server finishes rendering the element and attaches
it to the DOM. The driver detects it and then
he just goes instantly. It breaks the weight sequence and
executes the next command. It completely optimizes the script's efficiency.
Speaker 1 (14:52):
That is brilliant. It ties the execution pace strictly to
the application's actual state, not just arbitrary guessing. Exactly all right,
So our weight command worked. The page is loaded and
we're looking at a registration form. Normal text fields are easy.
Now we locate clear and send keys. But then we
hit the country of residence field and it's a drop
(15:12):
down menus.
Speaker 2 (15:13):
Oh, drop down menus. They reveal how deceptive graphical user
interfaces really are.
Speaker 1 (15:17):
What do you mean?
Speaker 2 (15:18):
As users? We just see a single seamless box that
we click and a nice list drops down. But the
script bypasses the UI completely. It looks at the HTML
document and sees a completely different structure.
Speaker 1 (15:28):
Right, Because HTML strictly defines how data is nested. A
dropdown isn't just one element, is it. It's a parent
container holding dozens of child elements.
Speaker 2 (15:38):
Precisely. It is typically built using a select tag for
the main container, and inside that container are individual option
tags for every single choice, one for Canada, one for
the UK, one for the United States.
Speaker 1 (15:52):
So you can't just type into it.
Speaker 2 (15:53):
No, you cannot just use send keys on the select
container because it's not a text input field.
Speaker 1 (15:59):
So we have to somehow tell the script to dig
inside that container, sift through all the option tags and
pick the right one. How do we do that?
Speaker 2 (16:08):
First you use a locator to isolate the main select
dropdown element. Once you have that object, you run a
command called finned elements plural with an s okay plural yes,
and you tell it to find every single child element
with the tag name option.
Speaker 1 (16:21):
Because it's plural, it doesn't just stop at the first
match like before. It grabs every single one of them.
Speaker 2 (16:26):
Exactly.
Speaker 1 (16:26):
What does it do with all that data?
Speaker 2 (16:28):
It extracts them from the dom and stores them in
your script's memory as a list like an array of
web element objects. So if there are one hundred and
ninety five countries in that dropdown, your script is now
holding an array of one hundred and ninety five distinct objects.
Speaker 1 (16:41):
Okay, so the script has a massive bucket full of
country options. How does it isolate just the United States
option so it can click it.
Speaker 2 (16:49):
You employ a standard programming loop, afore loop. You instruct
your code to iterate.
Speaker 1 (16:54):
Over that array, so it just checks some one by one.
Speaker 2 (16:57):
Basically, the script takes the first element, reads it's internal
text attribute, and asks a logical question, does this text
equal United States?
Speaker 1 (17:06):
And if it doesn't?
Speaker 2 (17:07):
If false, it discards it and checks the second element.
It loops through the entire array at the speed of
the processor, verifying the text of each option tag.
Speaker 1 (17:15):
In the moment it hits the true condition. The moment
the text matches exactly, we trigger the doc click method
on that specific object exactly.
Speaker 2 (17:22):
It scans the array, finds the match, executes the click,
and breaks the loop. The browser receives the click event
on that specific option note, and the drop down selection
updates on the screen.
Speaker 1 (17:32):
It is a wildly methodical process to achieve what takes
a human like a flick of the wrist.
Speaker 2 (17:38):
It really is.
Speaker 1 (17:39):
It really pulls back the curtain on how complex the
underlying web is.
Speaker 2 (17:43):
Ye.
Speaker 1 (17:44):
I mean we've mastered finding elements, typing, waiting for a
synchrodous network loads, and looping through complex data arrays. Our
script must be practically unstoppable.
Speaker 2 (17:55):
At this point you'd think so, wouldn't you. But modern
web architecture has a few security features and structural quarks
designed specifically to keep elements isolated. Uh oh, if you
aren't aware of them, your automation will hit an invisible wall.
Speaker 1 (18:08):
I think I can guess one of these traps. Sometimes
you inspect an element, you write the absolutely perfect XPath,
you run the script, and it still throws a nos
element exception. It's like the element is a complete ghost.
Speaker 2 (18:18):
Yes. When you encounter a ghost element like that, you
are almost certainly dealing with an iframe?
Speaker 1 (18:23):
An iframe, inline frame? What is the actual architectural purpose
of an iframe?
Speaker 2 (18:27):
Think of it as a strict security sandbox. An iframe
allows a developer to embed an entirely separate, independent HTML
document inside their main web page. Oh weird, Yeah, it
has its own isolated dom. You see this constantly with
third party content, like if a blog embeds a YouTube video.
That video player is living inside an iframe.
Speaker 1 (18:49):
Okay, that makes sense.
Speaker 2 (18:50):
Or if an e commerce site uses a third party
payment processor like Stripe to collect credit cards, those input
fields are hosted inside an iframe, so the main website's
code can't seek, at least scrape your credit card number.
Speaker 1 (19:02):
It's literally a web page operating inside a web page.
It's web page inception.
Speaker 2 (19:06):
And here is exactly why it breaks automation. When you
instantiate your driver, it operates entirely within the execution context
of the parent web pagees dom right, it is strictly
blind to the contents of any embedded iframes. It's like
standing in a hallway looking at a locked office door.
You know there is a room there, but you cannot
see the desks or the computers inside.
Speaker 1 (19:27):
So if I try to command the script to click
the play button on that embedded video, it scans the hallway,
doesn't see a play button, and crashes. Exactly, how do
we give the script the key to the office door.
Speaker 2 (19:37):
Then you must explicitly command the driver to switch its
execution context. You use a command like driver dot switch
toe dot frame, and you pass it the locator for
the iframe itself like its ID or its index number.
Speaker 1 (19:50):
Okay.
Speaker 2 (19:50):
Once that command executes, the driver effectively steps through the doorway.
It leaves the parent dom behind and maps out the
internal dom of the iframe. It can easily find and
click the play button.
Speaker 1 (20:02):
That makes perfect sense. But knowing how rigid this logic is,
there has to be a catch when we want to
go back to the main web page. Right.
Speaker 2 (20:08):
Oh, a massive catch. Once the driver steps into the iframe,
it becomes entirely blind to the hallway outside.
Speaker 1 (20:14):
I knew it.
Speaker 2 (20:15):
Yeah, if your script tries to go back to filling
out the main form without switching its context back, it
will crash. You must explicitly command the driver to switchtoe
dot default content to step back out into the parent dom.
Speaker 1 (20:27):
Wow.
Speaker 2 (20:28):
Context management is genuinely one of the hardest things to
debug in a complex automation script.
Speaker 1 (20:34):
Note to self, never leave the script locked inside the
iframe room. Now what about pop ups? And I don't
mean HTML modals that are built into the page. I
mean those native browser alert boxes that drop down from
the top of the screen asking are you sure you
want to leave this page? Oh?
Speaker 2 (20:54):
The ones that freeze everything.
Speaker 1 (20:56):
Yeah, they freeze the entire browser window until.
Speaker 2 (20:59):
You deal with them the browser, and they will completely
freeze your script too. Because those alerts are generated by
the browser itself or even the operating system, they are
not part of the HTML dom. Wait really, yeah, you
cannot inspect them. You cannot write an XPath for the
ok button on a native alert if they.
Speaker 1 (21:15):
Aren't in the dom. How on earth does Selenium interact
with them.
Speaker 2 (21:18):
Well, the driver has a dedicated interface just for handling
these native interruptions. When an alert fires, you instruct the
code to switch its focus to the alert manager. You
basically grab the currently active alert object into your code's memory.
Speaker 1 (21:33):
So you capture the alert itself as a programmable.
Speaker 2 (21:35):
Object precisely, and once captured, the interface exposes specific methods
for it. You can call dot accept, which tells the
browser to simulate clicking okay nice. You can call dot dismiss,
which simulates clicking cancel. You can even use dot text
to extract the message inside the alert box, just to
verify the servers and the right air code before dismissing it.
Speaker 1 (21:57):
This has been such a fascinating breakdown of how Autumn
actually functions under the hood. We started with just a
blank maximize browser window. We learn how the dom tree works,
how to navigate it using IDs and architectural xpaths.
Speaker 2 (22:10):
It's a lot to take in, it really is.
Speaker 1 (22:12):
We explore the necessity of clearing text before typing, using
explicit weights to outsmart network latency, looping through hidden arrays
and drop downs, and context switching into secure eyeframes.
Speaker 2 (22:23):
It really demystifies the magic of automation, doesn't it. We
take the intuitive, visual, human experience of browsing the web
and we strip it down to raw structure and precise
programmatic logic.
Speaker 1 (22:34):
We really do. To wrap things up, I actually have
a quick mental exercise for you listening to try later today.
Next time you are on your favorite web application, maybe
checking your bank balance or reading the news, right click
a button on the page, hit inspect and just look
at the raw HTML that pops up.
Speaker 2 (22:52):
That's a great exercise.
Speaker 1 (22:53):
Yeah, see if you can identify its tag name, its
glasses or an ID. Look at how deeply it is
nested inside other containers. Try to visualize how you would
write the directions for a script to find it. It
permanently changes the way you experience the Internet.
Speaker 2 (23:08):
It absolutely does. You start seeing the matrix behind the design.
Speaker 1 (23:11):
You really do. And as you're looking at that raw code,
I want to leave you with one final thought. Now
that you know how methodically a script parses a web page,
you know, bypassing the visual design entirely to read the
underlying data structures, hidden rays, and secure frames, how does
that change your perspective on the Internet's traffic.
Speaker 2 (23:30):
That's a deep question, right.
Speaker 1 (23:32):
If a simple automation script can read and extract data
from the matrix of a web page this easily, what
does that imply about the massive fleets of invisible bots
that are constantly crawling, scraping, interacting with the Internet right
alongside us, every single second of the day. Something to
think about.
So imagine you've got this this digital assistant, and it
literally just sits at your computer, opens up a web browser,
and you does all your digital chores for you.
Speaker 2 (00:09):
Oh, that would be amazing.
Speaker 1 (00:11):
Right, I'm talking about like filling out those super tedious
multi page onboarding forms, or clicking through just endless confirmation buttons,
or even scouring databases for specific information, all at lightning
speed while you just go grab a coffee.
Speaker 2 (00:25):
Yeah. I mean, it sounds like science fiction or at least,
you know, a very expensive enterprise software solution that most
people wouldn't have access to.
Speaker 1 (00:32):
Exactly, But it's actually just standard web automation, it is.
Speaker 2 (00:35):
And the crazy thing about it is to pull this off,
you kind of have to completely unlearn how you view
the Internet. What do you mean, Well, to a piece
of automation code, the Internet has no colors, it has
no shiny buttons, no pretty layouts. To automate the web,
we have to look past the user interface.
Speaker 1 (00:54):
Entirely, right, You're interacting directly with the raw architecture underneath exactly.
Speaker 2 (00:58):
You're dealing with the skeleton of the site, not the
paint job.
Speaker 1 (01:01):
So today, for this deep dive, we are basically going
to act as the translators between human intention and machine execution.
We're going to walk you through step by step how
a script actually parses and manipulates the web page.
Speaker 2 (01:16):
Yeah, and we'll be talking specifically about using tools like
Selenium and uh something called Chrome Driver.
Speaker 1 (01:24):
Right, So let's start with the ignition key. Basically, to
have an invisible assistant click on anything, we need a
way for our Python code to talk to the browser
in the first place.
Speaker 2 (01:32):
Yeah, and that is exactly the role of the Chrome Driver.
I mean, think of it as an executable application that
acts as a.
Speaker 1 (01:38):
Bridge, okay, a bridge between what and what.
Speaker 2 (01:40):
Between your script and the browser. So your script sends
commands to the driver, and then the driver translates those
commands into native browser actions.
Speaker 1 (01:49):
Got it.
Speaker 2 (01:50):
When you instantiate this driver in your code, an actual
instance of Chrome boots up on your machine, but it's
entirely controlled by your script. You can literally sit there
and watch it move on on its own, which.
Speaker 1 (02:00):
Is always a little spooky the first time you see it.
Speaker 2 (02:02):
It really is.
Speaker 1 (02:03):
Now, one of the first things a developer typically does
after booting that instance is they pass an argument to
maximize the browser window right away.
Speaker 2 (02:11):
Yes, that is a crucial step, but on.
Speaker 1 (02:14):
The surface that seems I don't know, purely cosmetic, like
why would the script care how big the window is
if it's just reading the underlying code anyway?
Speaker 2 (02:23):
Well, functionally, it's all about responsive design. I mean, modern
web pages are built to change their structural layout based
on the size of the screen you're using.
Speaker 1 (02:31):
Oh right, Like sites look different on a phone versus
a desktop exactly.
Speaker 2 (02:36):
So if you load a complex web app in a tiny,
minimized window, the site's code might dynamically hide at a
navigation menu. It might shove it behind one of those
mobile style Hamburger icons.
Speaker 1 (02:48):
Oh wow, I see where this is going.
Speaker 2 (02:50):
Yeah, So if your script is hunting for a specific
settings button that has suddenly been hidden or removed from
the active.
Speaker 1 (02:57):
Layout, the automation just crashes, right, It.
Speaker 2 (02:59):
Throws an error because the button isn't there anymore. So
maximizing the window ensures the layout is predictable, It ensures
all the standard desktop elements are rendered and you know,
actually available to interact with.
Speaker 1 (03:12):
That makes total sense. So we've set the stage, we've
maximized the window, and we've used to command like a driver,
do get to navigate to a specific url.
Speaker 2 (03:21):
Yep, the page loads.
Speaker 1 (03:22):
Now to a human user, a search bar is obvious.
It's a white rectangle with a little magnifine glass next
to it.
Speaker 2 (03:28):
Right, you just click it.
Speaker 1 (03:28):
But since our script doesn't have eyes, how does it
actually pinpoint that specific interactive element out of like thousands
of lines of raw HTML code.
Speaker 2 (03:39):
It relies entirely on the Document object model, or the
DOM as most people call it the DOM. Okay, Yeah,
when a browser loads a web page, it takes the
raw HTML text and turns it into this giant inverted tree.
Speaker 1 (03:52):
Structure, like a family tree sort of.
Speaker 2 (03:54):
Yeah, the page itself is the trunk, and every container,
every paragraph, every button is a brand or a leaf
on that tree.
Speaker 1 (04:01):
Okay, I'm visualizing that.
Speaker 2 (04:02):
So the script navigates this tree. If you want to
find a specific element, you need a locator strategy, and
the absolute most reliable strategy is looking for a unique
identifier that the original developers left behind.
Speaker 1 (04:15):
It's kind of like giving our script the exact GPS
coordinates of a specific house in a sprawling city.
Speaker 2 (04:20):
That is a perfect analogy.
Speaker 1 (04:22):
Actually, so if the developers built the site, well, they
assign these unique attributes to elements.
Speaker 2 (04:27):
Yeah.
Speaker 1 (04:27):
Like if you use a browsers and spectool on a
well designed site's search bar, you might see an attribute
called ID, right, like i'd search field.
Speaker 2 (04:37):
Yes, and an ID is the gold standard for locators.
Why is it the gold standard because by definition in
standard HTML rules, an ID is supposed to be entirely
unique on a single web page. It is literally like
a social security number for a web element.
Speaker 1 (04:52):
Wow. Okay, so it's foolproof pretty much.
Speaker 2 (04:54):
If you command your script to findelement bide and pass
it that specific string that driver searches, the dom tree
instantly locks onto that specific node and completely ignores the
rest of the page.
Speaker 1 (05:05):
That's super efficient. But what if the developer didn't use
an ID, because we all know not every site is
coded perfectly.
Speaker 2 (05:13):
Oh absolutely, it happens all the time.
Speaker 1 (05:15):
Like sometimes you inspect an element and it just has
a name attribute, Like the main search box on a
coding tutorial site might just have the name attribute assigned
to the letter Q.
Speaker 2 (05:23):
Right. Q usually stands for a query and the fidelment
by name command works very similarly to ID. It's incredibly fast.
Speaker 1 (05:32):
But I hear a butt coming.
Speaker 2 (05:34):
But name attributes aren't strictly required to be unique.
Speaker 1 (05:38):
Ah, there it is.
Speaker 2 (05:39):
Yeah. So if a developer got sloppy and named both
the top search bar and say a bottom newsletter sign
up form Q.
Speaker 1 (05:47):
Then your script might grab the.
Speaker 2 (05:49):
Wrong one exactly, because the script generally just defaults to
the very first match and encounters as it scans the
document from top to bottom.
Speaker 1 (05:55):
Oh right, so it hits the first que, says found it,
and stops looking. Okay, so IDs and names are the
easy street addresses, But we also have to navigate around
the site. Say I want the script to click a
specific text link that just says my courses.
Speaker 2 (06:10):
Well, Selenium provides a findelement by link text method for
exactly this scenario. Oh nice, Yeah, you just passed the
text of the link to the script, and it hunts
for an anchor tag in the HTML that contains that
exact string.
Speaker 1 (06:23):
I imagine that demand's absolute precision, though.
Speaker 2 (06:25):
Oh it really does.
Speaker 1 (06:26):
It's like telling a taxi driver to take you to
the Grand Hotel and they flat out refuse to move
because the physical sign on the building actually says the
Grand Hotel Incorporated.
Speaker 2 (06:36):
Yes, the machine has zero intuition. It is executing a
literal string comparison.
Speaker 1 (06:43):
So if I tell the script to look for my courses,
but the web designer capitalized it to my courses, yeah,
in all caps.
Speaker 2 (06:50):
The script will just crash. It'll throw a no such
element exception, A single mismatch space, an invisible line break,
or different capitalization will break it.
Speaker 1 (06:59):
That sounds incredibly fragile.
Speaker 2 (07:00):
It can be. But to mitigate that fragility, you can
use a variation called partial link text.
Speaker 1 (07:06):
Oh, or you only need a piece of it exactly.
Speaker 2 (07:08):
You only match a substring. So if the link is
a dynamically generated product title like blue Sneaker size ten,
ID four four five nine, you can just target the
blue Sneakers portion.
Speaker 1 (07:18):
Okay, So that ensures the script finds it even if
the ID numbers or the sizes change tomorrow.
Speaker 2 (07:23):
Right. It gives you a lot more flexibility.
Speaker 1 (07:25):
Got it. So far we've relied on developers giving us
clean IDs, names or exact text links. But what if
we're automating some highly complex, deeply nested web app where
the elements don't have any of those clean attributes.
Speaker 2 (07:39):
Yeah, the tricky stuff.
Speaker 1 (07:41):
How do we find the house if there's literally no
address on the mailbox?
Speaker 2 (07:44):
This is where we shift from simple addresses to architectural blueprints.
When basic locators fail, we rely on advanced locators, specifically
CSS selectors and XPath Okay, let's.
Speaker 1 (07:55):
Start with CSS now. CSS is what dictates the visual
styling of a page, right like colors, padding, Yes, exactly,
and developers apply these styles by assigning elements to classes.
So if a button is red and round, it might
have a class called primary submit btn. We can actually
hijack those styling hooks for automation.
Speaker 2 (08:12):
You absolutely can. Using a CSS selector locator, you can
command the script to find an element based entirely on
how it was styled.
Speaker 1 (08:20):
That's so clever.
Speaker 2 (08:21):
Yeah, in your code, you use standard CSS syntax like
a dot to denote a class, so you'd tell the
driver to find dot primary dash submit dash btn.
Speaker 1 (08:30):
And is that fast?
Speaker 2 (08:31):
It is incredibly fast. The beauty of this is that
CSS selectors are heavily optimized by the browser itself.
Speaker 1 (08:37):
But CSS classes still rely on the developer leaving a
recognizable hook. Sure, what if there was absolutely nothing unique
about the element itself, no class, no ID. That's where
XPath comes in, right, Yes, because I've seen complex XPath
strings and honestly, they look like absolute gibberish. It's just
like slashes, brackets and random HTML tags.
Speaker 2 (08:57):
They do look intimidating. But XPath is the heavy artillery
of web automation. It stands for XML path language. Okay,
remember how we describe the dom as an inverted tree.
This path is a literal set of step by step
directions for navigating the branches of that tree. You can
use it to triangulate an element based entirely on its
relationship to other elements.
Speaker 1 (09:16):
How does that actually map out in the code though?
Speaker 2 (09:18):
Okay, imagine you want to click a specific input box,
but it has no ID, no name, and a generic class.
Speaker 1 (09:25):
Basically a ghost element.
Speaker 2 (09:27):
Right, But you notice it sits inside a larger container,
a divtag that does have an idea of login panel.
Speaker 1 (09:33):
Okay, so the parent container has a clear address exactly.
Speaker 2 (09:37):
Using XPath, you can write a path that says, scan
the document for a div with the ID log in panel.
Once you find it, step inside that container, find the
second paragraph, and give me the input box inside that paragraph. Wow,
it doesn't matter if the input box has no identifying
features of its own. You've located it purely based on
(09:57):
its surroundings.
Speaker 1 (09:58):
It's literally like saying, go to the house house on
the corner of Fifth and Main, walk to the kitchen
and grab the second drawer on the left. Yes, you
don't need a label in the drawer if you know
the exact path to reach it.
Speaker 2 (10:08):
That is the power of x path. It can navigate
down from parent to child, or even up from a
child to a parent, or sideways across sibling elements. It's
incredibly robust.
Speaker 1 (10:17):
All right, So we've mastered finding elements using the dom tree.
But just pointing at a search bar doesn't execute a search.
We have to interact with it, right.
Speaker 2 (10:26):
Once the driver locates the element, it stores it as
an object in your Python code, okay, and you can
then call methods on that object. So to type into
a field, you use the sen keys command. Is a
keys You pass it a string of texts, and the
driver basically simulates the electrical signals of a human physically
typing on a keyboard, sending those exact keystrokes directly into
(10:47):
the input node.
Speaker 1 (10:48):
Wait if it just sends keystrokes blindly, what happens if
the search bar already has pre populated text, Like sometimes
you visit a portal and the search box already says
search by employee name and light gray text. If I
just fire off send keys with the name John, does
it just appendit like? Does it search for search by
employee name John.
Speaker 2 (11:08):
It absolutely will do exactly that. Oh no, yeah, send
keys doesn't overwrite existing values, it appends to them. This
is a classic architectural trap.
Speaker 1 (11:17):
Because humans instinctively recognize placeholder text right we backspace it
or we highlight it before typing.
Speaker 2 (11:24):
Exactly, but the machine does not. Therefore, robust automation requires
explicit instruction. Before you ever send new keys to a
text input, you invoke the dot clear method yeap. This
commands the driver to wipe the input node clean of
any pre existing string values. You locate it, you clear it,
and only then do you send your keys.
Speaker 1 (11:46):
Man, it really emphasizes how much micrologic goes into our
daily browsing habits. Stuff we don't even think about.
Speaker 2 (11:53):
We take it totally for granted.
Speaker 1 (11:54):
So we've cleared the field and typed our query. To
actually run the search, we need to get enter. How
do we do that?
Speaker 2 (12:01):
You import a specific keys class in your script. This
class contains representations of all the non alphanumeric keyboard.
Speaker 1 (12:07):
Keys, like shift, alt enter all that.
Speaker 2 (12:09):
Exactly, So you just deppend keys, dot enter or keys
return to your sen keys command. The driver simulates that
specific keypress, the web server receives the form submission, and boom,
your automated search is running.
Speaker 1 (12:21):
That is so cool. Now, this all works beautifully in
a vacuum. But let's look at the real world, where
the Internet is messy, connections are slow, and web apps
are incredibly complex.
Speaker 2 (12:31):
Oh yes, the real world is a nightmare for automation, right.
Speaker 1 (12:35):
What happens when our script encounters, say, a multi step
sign up process with a really heavy loading screen.
Speaker 2 (12:42):
This is where we hit the fundamental friction of web automation.
It's the speed gap between the CPU and the network.
Speaker 1 (12:49):
The speed gap.
Speaker 2 (12:50):
Yeah, your Python script executes commands in fractions of a millisecond.
It tells the driver to click next. The browser sends
a request to the server, and the server takes say two,
two full seconds to respawn and draw the new page.
Speaker 1 (13:02):
And during those two seconds, our script is already trying
to execute the next line of code right yep, which
is filling out the first name field on a page
that literally doesn't exist yet exactly.
Speaker 2 (13:12):
The script checks the dom doesn't see the first name
element and instantly throws a fatal no such element exception.
It just crashes.
Speaker 1 (13:20):
Okay, So the obvious brute force fix would be to
just tell the script a pause, like use a sleep
command to freeze the code for five seconds after every
single click, just to be safe.
Speaker 2 (13:31):
You could do that, but static sleeps are a terrible
engineering practice.
Speaker 1 (13:34):
Really why well, think about it.
Speaker 2 (13:36):
If you tell the code to sleep for ten seconds
and the page loads in two seconds, you are wasting
eight seconds of compute time.
Speaker 1 (13:43):
Oh true.
Speaker 2 (13:44):
If you scale that script up to process a ten
thousand records, you've just added hours of dead air.
Speaker 1 (13:50):
That's highly inefficient.
Speaker 2 (13:51):
And conversely, if the server is having a slow day
and takes eleven seconds to load, your script still crashes anyway.
Speaker 1 (13:57):
Oh man, Yeah, so we need a dynamic way to
bridge the CPU speed and then network speed.
Speaker 2 (14:03):
Right, And the elegant solution here is something called an
explicit weight. It's often implemented as web driver weight.
Speaker 1 (14:09):
How does that differ from just sleeping.
Speaker 2 (14:11):
Instead of freezing the code entirely? You define a maximum
time out, say fifteen seconds, and you define a specific
condition like wait until the first name element is clickable.
Speaker 1 (14:21):
So how does that look under the hood? Is it
just checking constantly?
Speaker 2 (14:24):
It utilizes a polling mechanism. The driver hooks into the
process and checks the DOM tree every five hundred milliseconds.
Oh wow, it asks, is the element here yet?
Speaker 1 (14:34):
No?
Speaker 2 (14:35):
Half a second later, is it here yet? No? The
absolute millisecond. The server finishes rendering the element and attaches
it to the DOM. The driver detects it and then
he just goes instantly. It breaks the weight sequence and
executes the next command. It completely optimizes the script's efficiency.
Speaker 1 (14:52):
That is brilliant. It ties the execution pace strictly to
the application's actual state, not just arbitrary guessing. Exactly all right,
So our weight command worked. The page is loaded and
we're looking at a registration form. Normal text fields are easy.
Now we locate clear and send keys. But then we
hit the country of residence field and it's a drop
(15:12):
down menus.
Speaker 2 (15:13):
Oh, drop down menus. They reveal how deceptive graphical user
interfaces really are.
Speaker 1 (15:17):
What do you mean?
Speaker 2 (15:18):
As users? We just see a single seamless box that
we click and a nice list drops down. But the
script bypasses the UI completely. It looks at the HTML
document and sees a completely different structure.
Speaker 1 (15:28):
Right, Because HTML strictly defines how data is nested. A
dropdown isn't just one element, is it. It's a parent
container holding dozens of child elements.
Speaker 2 (15:38):
Precisely. It is typically built using a select tag for
the main container, and inside that container are individual option
tags for every single choice, one for Canada, one for
the UK, one for the United States.
Speaker 1 (15:52):
So you can't just type into it.
Speaker 2 (15:53):
No, you cannot just use send keys on the select
container because it's not a text input field.
Speaker 1 (15:59):
So we have to somehow tell the script to dig
inside that container, sift through all the option tags and
pick the right one. How do we do that?
Speaker 2 (16:08):
First you use a locator to isolate the main select
dropdown element. Once you have that object, you run a
command called finned elements plural with an s okay plural yes,
and you tell it to find every single child element
with the tag name option.
Speaker 1 (16:21):
Because it's plural, it doesn't just stop at the first
match like before. It grabs every single one of them.
Speaker 2 (16:26):
Exactly.
Speaker 1 (16:26):
What does it do with all that data?
Speaker 2 (16:28):
It extracts them from the dom and stores them in
your script's memory as a list like an array of
web element objects. So if there are one hundred and
ninety five countries in that dropdown, your script is now
holding an array of one hundred and ninety five distinct objects.
Speaker 1 (16:41):
Okay, so the script has a massive bucket full of
country options. How does it isolate just the United States
option so it can click it.
Speaker 2 (16:49):
You employ a standard programming loop, afore loop. You instruct
your code to iterate.
Speaker 1 (16:54):
Over that array, so it just checks some one by one.
Speaker 2 (16:57):
Basically, the script takes the first element, reads it's internal
text attribute, and asks a logical question, does this text
equal United States?
Speaker 1 (17:06):
And if it doesn't?
Speaker 2 (17:07):
If false, it discards it and checks the second element.
It loops through the entire array at the speed of
the processor, verifying the text of each option tag.
Speaker 1 (17:15):
In the moment it hits the true condition. The moment
the text matches exactly, we trigger the doc click method
on that specific object exactly.
Speaker 2 (17:22):
It scans the array, finds the match, executes the click,
and breaks the loop. The browser receives the click event
on that specific option note, and the drop down selection
updates on the screen.
Speaker 1 (17:32):
It is a wildly methodical process to achieve what takes
a human like a flick of the wrist.
Speaker 2 (17:38):
It really is.
Speaker 1 (17:39):
It really pulls back the curtain on how complex the
underlying web is.
Speaker 2 (17:43):
Ye.
Speaker 1 (17:44):
I mean we've mastered finding elements, typing, waiting for a
synchrodous network loads, and looping through complex data arrays. Our
script must be practically unstoppable.
Speaker 2 (17:55):
At this point you'd think so, wouldn't you. But modern
web architecture has a few security features and structural quarks
designed specifically to keep elements isolated. Uh oh, if you
aren't aware of them, your automation will hit an invisible wall.
Speaker 1 (18:08):
I think I can guess one of these traps. Sometimes
you inspect an element, you write the absolutely perfect XPath,
you run the script, and it still throws a nos
element exception. It's like the element is a complete ghost.
Speaker 2 (18:18):
Yes. When you encounter a ghost element like that, you
are almost certainly dealing with an iframe?
Speaker 1 (18:23):
An iframe, inline frame? What is the actual architectural purpose
of an iframe?
Speaker 2 (18:27):
Think of it as a strict security sandbox. An iframe
allows a developer to embed an entirely separate, independent HTML
document inside their main web page. Oh weird, Yeah, it
has its own isolated dom. You see this constantly with
third party content, like if a blog embeds a YouTube video.
That video player is living inside an iframe.
Speaker 1 (18:49):
Okay, that makes sense.
Speaker 2 (18:50):
Or if an e commerce site uses a third party
payment processor like Stripe to collect credit cards, those input
fields are hosted inside an iframe, so the main website's
code can't seek, at least scrape your credit card number.
Speaker 1 (19:02):
It's literally a web page operating inside a web page.
It's web page inception.
Speaker 2 (19:06):
And here is exactly why it breaks automation. When you
instantiate your driver, it operates entirely within the execution context
of the parent web pagees dom right, it is strictly
blind to the contents of any embedded iframes. It's like
standing in a hallway looking at a locked office door.
You know there is a room there, but you cannot
see the desks or the computers inside.
Speaker 1 (19:27):
So if I try to command the script to click
the play button on that embedded video, it scans the hallway,
doesn't see a play button, and crashes. Exactly, how do
we give the script the key to the office door.
Speaker 2 (19:37):
Then you must explicitly command the driver to switch its
execution context. You use a command like driver dot switch
toe dot frame, and you pass it the locator for
the iframe itself like its ID or its index number.
Speaker 1 (19:50):
Okay.
Speaker 2 (19:50):
Once that command executes, the driver effectively steps through the doorway.
It leaves the parent dom behind and maps out the
internal dom of the iframe. It can easily find and
click the play button.
Speaker 1 (20:02):
That makes perfect sense. But knowing how rigid this logic is,
there has to be a catch when we want to
go back to the main web page. Right.
Speaker 2 (20:08):
Oh, a massive catch. Once the driver steps into the iframe,
it becomes entirely blind to the hallway outside.
Speaker 1 (20:14):
I knew it.
Speaker 2 (20:15):
Yeah, if your script tries to go back to filling
out the main form without switching its context back, it
will crash. You must explicitly command the driver to switchtoe
dot default content to step back out into the parent dom.
Speaker 1 (20:27):
Wow.
Speaker 2 (20:28):
Context management is genuinely one of the hardest things to
debug in a complex automation script.
Speaker 1 (20:34):
Note to self, never leave the script locked inside the
iframe room. Now what about pop ups? And I don't
mean HTML modals that are built into the page. I
mean those native browser alert boxes that drop down from
the top of the screen asking are you sure you
want to leave this page? Oh?
Speaker 2 (20:54):
The ones that freeze everything.
Speaker 1 (20:56):
Yeah, they freeze the entire browser window until.
Speaker 2 (20:59):
You deal with them the browser, and they will completely
freeze your script too. Because those alerts are generated by
the browser itself or even the operating system, they are
not part of the HTML dom. Wait really, yeah, you
cannot inspect them. You cannot write an XPath for the
ok button on a native alert if they.
Speaker 1 (21:15):
Aren't in the dom. How on earth does Selenium interact
with them.
Speaker 2 (21:18):
Well, the driver has a dedicated interface just for handling
these native interruptions. When an alert fires, you instruct the
code to switch its focus to the alert manager. You
basically grab the currently active alert object into your code's memory.
Speaker 1 (21:33):
So you capture the alert itself as a programmable.
Speaker 2 (21:35):
Object precisely, and once captured, the interface exposes specific methods
for it. You can call dot accept, which tells the
browser to simulate clicking okay nice. You can call dot dismiss,
which simulates clicking cancel. You can even use dot text
to extract the message inside the alert box, just to
verify the servers and the right air code before dismissing it.
Speaker 1 (21:57):
This has been such a fascinating breakdown of how Autumn
actually functions under the hood. We started with just a
blank maximize browser window. We learn how the dom tree works,
how to navigate it using IDs and architectural xpaths.
Speaker 2 (22:10):
It's a lot to take in, it really is.
Speaker 1 (22:12):
We explore the necessity of clearing text before typing, using
explicit weights to outsmart network latency, looping through hidden arrays
and drop downs, and context switching into secure eyeframes.
Speaker 2 (22:23):
It really demystifies the magic of automation, doesn't it. We
take the intuitive, visual, human experience of browsing the web
and we strip it down to raw structure and precise
programmatic logic.
Speaker 1 (22:34):
We really do. To wrap things up, I actually have
a quick mental exercise for you listening to try later today.
Next time you are on your favorite web application, maybe
checking your bank balance or reading the news, right click
a button on the page, hit inspect and just look
at the raw HTML that pops up.
Speaker 2 (22:52):
That's a great exercise.
Speaker 1 (22:53):
Yeah, see if you can identify its tag name, its
glasses or an ID. Look at how deeply it is
nested inside other containers. Try to visualize how you would
write the directions for a script to find it. It
permanently changes the way you experience the Internet.
Speaker 2 (23:08):
It absolutely does. You start seeing the matrix behind the design.
Speaker 1 (23:11):
You really do. And as you're looking at that raw code,
I want to leave you with one final thought. Now
that you know how methodically a script parses a web page,
you know, bypassing the visual design entirely to read the
underlying data structures, hidden rays, and secure frames, how does
that change your perspective on the Internet's traffic.
Speaker 2 (23:30):
That's a deep question, right.
Speaker 1 (23:32):
If a simple automation script can read and extract data
from the matrix of a web page this easily, what
does that imply about the massive fleets of invisible bots
that are constantly crawling, scraping, interacting with the Internet right
alongside us, every single second of the day. Something to
think about.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Betrayal Weekly
Betrayal Weekly is back for a new season. Every Thursday, Betrayal Weekly shares first-hand accounts of broken trust, shocking deceptions, and the trail of destruction they leave behind. Hosted by Andrea Gunning, this weekly ongoing series digs into real-life stories of betrayal and the aftermath. From stories of double lives to dark discoveries, these are cautionary tales and accounts of resilience against all odds. From the producers of the critically acclaimed Betrayal series, Betrayal Weekly drops new episodes every Thursday. If you would like to share your story, you can reach out to the Betrayal Team by emailing them at betrayalpod@gmail.com and follow us on Instagram at @betrayalpod and @glasspodcasts. Please join our Substack for additional exclusive content, curated book recommendations, and community discussions. Sign up FREE by clicking this link Beyond Betrayal Substack. Join our community dedicated to truth, resilience, and healing. Your voice matters! Be a part of our Betrayal journey on Substack.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com