July 25, 2022 • 51 mins
Get ready for a doozy. This episode tries to explain how storage works on Android with a focus on Google's on-again, off-again approach to scoped storage and how apps navigate it all. Hopefully you don't come out of this more confused than before.
We talk with Raymond Lai, part of the team behind Amaze File Manager.
- 03:01 - How did storage access used to work in the early days of Android? What is the difference between internal and external storage?
- 05:00 - Why is external storage mounted using a virtual file system? Where can apps store files, and what is the "Android" folder used for? What was the problem with mounting external storage as VFAT?
- 10:00 - Why did Android 4.4 switch from VFAT to using FUSE to emulate FAT32? Is FUSE still being used? What is SDCardFS?
- 12:07 - What is Scoped Storage? How did it affect external storage access?
- 13:12 - What is the MediaStore API? How does Scoped Storage protect users?
- 17:02 - What is the Storage Access Framework, and why was it so controversial? How did it affect apps?
- 24:00 - How did Scoped Storage break file access on some Android TV devices?
- 26:54 - How did Google respond to developer feedback on Scoped Storage before Android 10's release?
- 27:43 - How did Scoped Storage change in Android 11? What is the "all files access" permission and what are its limitations? How did file managers get around these limitations?
- 31:55 - Why did Android return to FUSE? What was the limitation with SDCardFS? How did Google improve FUSE, and what problems does it still have?
- 38:44 - What is FUSE passthrough in Android 12? How does it improve performance?
- 40:22 - How will Android 13 change file access on Android?
Android Bytes is hosted by Mishaal Rahman, Senior Technical Editor, and David Ruddock, Editor in Chief, of Esper.
Esper enables next-gen device management for company-owned and managed tablets, kiosks, smart phones, IoT edge devices, and more.
For more about Esper:
- Esper Blog
- Mobile Device Management (MDM) Guide
- Android MDM Guide
- iOS MDM Guide
- MDM Solutions
- MDM APIs & SDK
New from the Esper blog:
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
David (00:01):
Hello, and welcome to Android bites powered by Asper I'm, David
Rudick and each week I'm joinedby my cohost Michelle ramen.
As we dive deep into the technicalworld of Android, and we're going to
be talking about something that keepseverybody awake at night file access
and file systems on Android, which isafter reading the outline for this show.
(00:22):
I am more confused than when we started.
So I'm really excited to learn hereand I'll let Michelle introduce our.
Mishaal (00:29):
thank you, David.
And I apologize in advance to anyonewho's gonna walk away more confused
than before they came into this show.
So, uh, before we dive into the,uh, confusing aspects, let me
just offer a brief opportunity forRaymond to introduce themselves.
So welcome to the show, Raymond,and tell our listeners what you do.
What do you work on?
Raymond (00:47):
Hello everyone.
I'm Raymond from Hong Kong.
And currently I'm work on,uh, Android farm manager
project called ma farm manager.
And I joined it from 2018 when I needa farm manager that can dosh copy.
So after back and forthsome good requests, the team
(01:09):
invited me to join the project.
Currently my focus in the MAs projectis mostly on integrations about
different protocols, the FTP server,and the recent one I'm working, been
working on and have been asked by.
The users many time is the far FTP client
Mishaal (01:30):
support.
Well, thank you for joining us.
And for those of you who haven't usedthe app before it's available on Google
plays, the maze file manager app.
And the reason I wanted to invite adeveloper who's worked on file managers
is because the topic of today's discussionis scope storage, or rather how scope
storage changed Android file access.
(01:51):
And it's, it's a very complicated subject.
Um, one that I still.
To this day, I have to verify that Iknow what I'm talking about by looking
it up on the inter developer docs.
Every single time I wannawrite about scope storage.
I have to go back and makesure did I get this bit right?
And a lot of times it's I, you, youmiss something minor detail, but the
reason scope storage is confusing isnot anything particularly to do with the
(02:14):
concept itself, but rather how Googleapproached the scope storage model.
It's because that when they introducescope storage with Android, The way
file access worked then continued tochange with every Android release since.
So with Android 11, it gotsignificantly changed with Android 12.
There was some behind the scene changes.
(02:34):
And with Android 13, there are somerather significant, but not a significant
changes, but still like if you're an,a developer, who's trying to figure
out, okay, I want to access files.
Where do I store them?
How do I access them?
It's gotten a lot more complicated thanit used to be in the earlier days of.
And because file access intersectswith so many different other aspects
of Android, including the Linux kernel,the underlying file systems, the
(02:55):
way the storage ship is partitioned.
I think it's an interesting topic andsomething that's worth discussing.
So before we actually talk aboutscope storage, I think it's important.
We actually go way, way, way back.
I mean, the early days of Android, likeover a decade ago to talk about SD cards.
So in the very early days of Android,I'm talking like in the nexus one era.
(03:17):
A lot of phones or most phones ratherdepended on having removable SD
cards to hold media files as wellas additional apps in their data.
Since at the time internal storageships had rather mediocre capacities.
So during this time Google referredto the SD card as Android's external
storage, which obviously made sensebecause SD cards were removable and
they weren't built into the device.
(03:37):
When Google shifted away fromSD cards with the nexus S though
everything then had to be stored ontothe devices, built in storage ship,
including all the media files and allthe app data, et cetera, that were
previously stored on the external SD.
However, in order to maintaincompatibility apps that were expecting
files to be stored on the physicalexternal SD card Mount point would instead
(03:58):
be transparently directed to the newslash data slash media Mount point to this
day, you'll see a lot of ADB commands,ADB tutorials that refer to SD slash
SD card when such a thing doesn't existon most Android phones instead slash SD
card is just symbolic link to slash dataslash media because of this legacy of
Android used to rely on physical SD card.
(04:19):
So the reason I'm talking about SDcards and stuff, that's over a decade
old is because to this day, Androidstill refers to slash data slash media
as external storage, even though nineoutta 10 times, that's actually storage.
That's internal to the device.
The way that Android sees storageis that you have the underlying
user data partition on phones,which is mounted at slash data it's.
(04:41):
So the data partition is writeable media.
It's where all your app data, allyour removal files are stored.
It's generally split into two parts.
You have everything outside ofslash data slash media, which
Android considers internal storage.
Then you have everything insideof slash data slash media.
That's considered external storage.
And the reason those two are separatedis because the internal storage and the
(05:04):
external storage at slash data slashmedia are actually mounted differently.
So even though the underlyingdata partition may be formatted
in F two Fs or X four file systemsslash data slash media is actually
mounted with a virtual file system.
There are many reasons to do thismost importantly, being that.
So you don't have to unmount theentire phones data partition whenever
(05:26):
you wanna hook it up to a PC, butthere are also some privacy and
security benefits to having this.
So prior to Android, 4.4 kit cat,this virtual SD card partition is
what soft often referred to as wasmounted using the VF a T file system.
But this posed someproblems in the early days.
This is because the slash data slashmedia Mount point or the external
(05:49):
storage is what I'll refer to itpretty much from now on is not only
where you'll find the shared downloadsfolder, your pictures, music, movies,
and other sensitive media folders.
But it's also home to the Android folder.
And the Android folder is where apps canread or write whatever files they want
within their app specific directories.
So like slash Android slash data slashAndroid slash B for, uh, game files.
(06:12):
So just another broad step backapps could store files in one
of three primary locations.
There are app specific directoryin the internal storage I E
and slash data slash data.
There are app specific directoriesin the external storage.
Many times you see it asslash SD card slash Android.
When in actuality it's slash data slashmedia slash the user ID slash Android
(06:35):
or any other directory in externalstorage that they have access to.
So app specific directories inthe internal storage are rather
limited in space intentionally by.
So apps that need to store lotsof large files or just lots of
files in general, generally pickssomewhere in the external storage.
So an app like telegram, forexample, has a ton of cashed
images and documents and videos.
(06:56):
So it stores all of that in its appspecific external storage directory.
The problem with the way this wasimplemented in Android prior to
4.4, is that the V F a T file systemdidn't support the kind of permission
management that Google wanted to.
I.
The issue was that apps that wanted toread and write files into their own app.
Specific external storage directorieshad to request the read external
(07:19):
storage and write external storagepermissions in order to do so.
But requesting those permissionsalso gave apps broad access to the
entirety of the external storage,just by needing the permissions to
access their own specific directories.
They also had access to the entirety.
The external storage.
That was a problem because this wasbasically start of apps, having too
(07:40):
much access to external storage, whichalso housed other shared media files.
So that was a lot before we moveon to how Android or how Google
started to address this problem.
I wanted to ask about your thoughtsand experiences dealing with file
management in the early days of Android.
Like, what were your thoughts on SD cars?
Things like apps to SD mass storage, et.
Raymond (08:01):
First of all.
I think Google's decision to use weat is external storage because of
compatibility concern, because about10 years ago, many users are still on
using windows or Mac silver, littlewho are using Linux as their desktop.
So using we as the file system, they,they, they can Mount on, they can
(08:25):
do this without any special drivers.
So that they can just plug into theUSB port and they can access the files
inside the Android phone directly.
And then before that they of keep cats.
Mishaal (08:41):
I think we mostly use
Raymond (08:44):
the farm manager that the Android phone bundled in.
So perhaps there are many different kindsof implementations from different vendors,
but generally you can see the Androidphone as simply as, uh, speech fund drive.
If you can, if you sayotherwise, very simple,
David (09:05):
that's an interesting analogy.
And I had not thought of itthat way, but that makes a lot
of sense actually, intuitively.
Mishaal (09:11):
Yeah.
So as Raymond mentioned, using V Fa T allowed for broad compatibility,
because the way PC access worked wasthat Android would actually unmount the
virtual SD card partition, and then itwould Mount it onto the connected device.
Obviously this led to some problems earlyon, because if you wanted to use your
phone while it's connected to the PC,all of a sudden, a lot of apps will be
(09:32):
missing media files or missing some data.
So instead Android then shifted tousing the media transfer protocol
or MTP to handle a file transfer.
And this made it so that itdidn't matter what file system
was used on the Android device.
It would still be supported by the PC.
So be something connected to the PC.
(09:53):
And the reason you can look thatup how MTP works, but that's not
really important for this, but thatwas one of the ways that Google.
Was able to shift away from VF a T inAndroid, 4.4, starting from Android
4.4 and beyond, instead of mounting thevirtual SD card partition as VFA, Google
decided to use Lennox's file system anduser space feature or fuse for short.
(10:17):
To emulate F a T 32.
So the reason Google decided toswitch to emulating F a T 32, is
that it allowed Android to use Ffat's directory style permissions.
This made it possible for Android tolet apps access their own app specific
external storage directories withoutneeding those read external storage
and write external storage permiss.
(10:39):
Eventually Google did what Googledoes best and deprecated using
fuse switched to SD card Fest forvarious reasons, which is a Samsung
developed kernel file system wrapper.
And that, uh, was done in Android 8.0 toimprove performance and fixed some issues.
But then they reverted back to fuse inAndroid 11 after making some improvements.
And in order to better supportsome scope, storages, new feature.
(11:01):
But we'll get into that bit later.
It's a bit of Android file system history.
That is not super important tounderstand the nitty gritty of right now.
But what's important to take awayfrom this is that by switching
from the V a T style system tousing fuse, to emulate F a T 32.
The end result is that fewer apps neededto hold the read external storage and
(11:22):
write external storage permissions.
And so that means that fewer appshad the ability to clutter your
external storage with folders orSnoop on your private media files.
However, this didn't solve everythingbecause there's still far too many
apps having broad file access.
And over the years, Googletried to reduce the number.
The most significant thing theytried was Android 6.0, which
introduced runtime remission.
(11:44):
Read external storage and writeexternal storage now had to be manual
granted by the user at run time.
So like there would be a dialogue askingthe user, do you want to grant these
permissions instead of granting themautomatically at install time, but this
still did not do enough to limit thenumber of apps that had broad file access,
something big needed to change to dothis, which leads us to scope storage.
(12:07):
Scope storage for those of you don'tknow, was introduced with Android 10 and
it's described as a it's a new model forthe way file access works in Android.
It revamped the way fi apps could accessfiles located on the external storage.
The biggest change that scope storagemade was to the right external storage
and read external storage permissions.
So for apps that target Android10, right, external storage
(12:30):
effectively became deprecated.
And folded into read external storage.
So whenever you grant an appaccess to read external storage,
it implicitly has the ability toalso write files or modify files.
Read external storage on the otherhand was significantly reduced in
scope granting an app that permissionno longer gave it broad access to
the entirety of external storage.
Instead, the permission only gave accessto media files that were contributed to
(12:53):
media store collections by other apps.
I'm I'm gonna get themedia store in a bit.
So basically apps that contribute mediafiles to media store collections can do so
without needing any permissions to do so.
But in order to read or modify filesthat other apps contributed, they need to
hold the read external storage permiss.
So you're an app developer.
Raymond who's worked on file manager.
Can you tell us a bit about themedia store API for listeners who
(13:16):
may not be familiar with it about
Raymond (13:18):
the media store API is to carry API to carry different
kinds of media files on the devicecategorized by different video types,
like image, audio documents, cetera,
Mishaal (13:34):
et C.
Right.
So like Android has likean index of media files.
If you've ever opened like thefiles app, you have like the images
filter, it shows you images fromvarious filters on your device.
That's basically the user facing accessto the media store and apps can add their
own image files to that media store.
(13:54):
And then other apps can access thoseimage files in the media store.
But in order to access images thatother apps contributed, they need
the read external storage permiss.
So that's what end enter 10 and
David (14:06):
just provide some context on this.
I'm sure we can get into this later on aswell, but there were really good reasons.
Google did this.
Number one is just that applicationsin general, the number of files,
they would have broad access tothe ability to read and write.
There are obviously nefarious thingsthat they could do with that permission.
If a user did not understandwhat kind of permission they were
(14:28):
granting that application and users.
When an application, especially asks forfile access, I think are very likely to
say yes, because that is something thatfrom the windows and Mac O S paradigms,
um, is very common applications, alwaysneed file access because they need
to see directories access, you know,documents, whatever the case may be.
So users are very accustom to justbeing like, yeah, it's an app.
(14:51):
It needs access files.
I.
Files, but, and the way the Androidfile system works, which is say
something I don't understand very well.
I do not know enough to even saywhat granting full file system access
really would let an app do per se,compared to something like windows.
And so Google was trying to define,okay, what are the very safe.
File types.
(15:12):
We can let any app, as longas the user says, okay.
Access and obviously images, video,audio, because those are things that
people share constantly and reallydo want to move between applications,
whether it's touching up a photo anda photo editing app, or it's sharing
something and putting a filter on it onInstagram, because you already edited
somewhere else in another application.
(15:33):
People want that traceabilityfor their personal media files.
And that's what Google I thinkwas really trying to enable.
Mishaal (15:39):
Right.
And just to clarify for anyone who maybe concerned, when we say like broad
external storage access, externalstorage, remember is only files
contained within slash data slash media.
So it's only the folder.
It's only the files that are createdor stuff you download to your device.
You can delete the entirety of slash dataslash media and your phone is still the.
(16:02):
Because it's all files createdafter you install your apps
and you set up your device.
None of it's critical tosystem operation, all of that
stored in different partitions.
So it's not like windows or somethingwhere you grant an app administrative
access, it can literally nuke your device.
Um, I'm unless you're how true that isactually on anymore in modern windows, or
it's also not the case that you're givingan app like root access that could truly
(16:24):
Rick havoc, the main concern with givingan app broad external storage access is.
They can potentially readsensitive media files that you
don't want them to access to.
So like for example, if a game wants tostore files on a folder of its choice,
like they create a folder on the externalstorage in order to do that, they
needed that external storage permission.
(16:44):
But at the same time, by giving 'emthat permission, they all of a sudden
had the ability to read your photosor to like to look at your photos.
That's not something you want togive every game access to and you
who knows what they might be doing.
So that's the reason why Googlewanted to crack down on this.
Just there's just too much potentialof like a privacy invasion.
But on the other hand, there are appsthat legitimately do need broad, external
(17:05):
storage access, including file managerslike a maze, like the one that Raymond
works on, but because their scope ofread external storage access was reduced.
These apps need an alternativein order to access the same list
of directors they had before.
Fortunately an alternative alreadyexisted in Android in the form of
the storage access framework, whichis actually something that Google
introduced way back in Android, 4.4.
(17:26):
So the storage access framework orSA for short makes it possible for
apps to request the user, to grantaccess to certain directories.
So what would happen is the filemanager like maze would invoke SA
using an intent and that will thenpull up the system provided documents.
Picker also known as the files app.
If, if you look it up in many places,The user would then use the files app
(17:46):
to browse through the external storageand choose which directory to grant
the app access to which on Android 10,they could even choose to give an app
access to the entirety of the externalstorage by just choosing the root folder.
This sounded fine on paperusing SA to access files and
directories, except for one problem.
Accessing files via SFA.
API was slow, really, really slow.
(18:09):
So slow.
In fact, that benchmarks done bydevelopers showed it was orders of
magnitude slower than using traditionalJava file API or native file IO commands.
Something that might take like, uh,mere seconds to maybe scan for music
files on your external storage wouldtake maybe two minutes to do now.
Like ludicrously slower developershad raised alarms about SA F of four,
(18:32):
because it's actually been requiredto use since Android six point.
No, if you wanted to access files onexternal USB flash tribes, but with
Android 10, it became front and centerbecause it became required to use for apps
targeting Android 10 in order to actually.
Access or read many files becauseenter 10 didn't support, direct
file path access via the native Javafile API or native file IO commands.
(18:56):
So, Raymond, I think this is a bitwhere you probably know much more than
we do so many people may have heardabout the controversies around skip
storage, but they may not be awareof how exactly negatively affected
certain types of apps or use cases.
Can you tell us what exactly madefile access via SA significantly
worse than the previous method?
The
Raymond (19:14):
extra effort there to obtain the info stream stream of a particular
file on the storage device, wherever yousay, being an SD card or attached that
USB from drive or something, there isan extra effort required to obtain the
stream wheel the end, we the system.
(19:35):
So I think that's themain reason that made
Mishaal (19:41):
powers through the SA significant.
So apart from file managers likea maze, what are some other kinds
of apps that were also impacted byscope storage and SA and Android 10?
I think
Raymond (19:52):
some game key, uh, console AORs would be affected.
Actually, I didn't try myself becausethe last device I used to play games
is still running on Android act.
But I think as of Android 10, some, aembryos like retro art, Or dolphin would
(20:12):
maybe affected because the extra requireon assessing the files and the storage
and because the users have to put thefiles in a way that Google, like, so the
AMBS can smoothly assess those files.
David (20:29):
So I, I think, and I'll steal your lines from the outline here, Michelle,
because I think this is an interestinguse case and it's one that I understand.
So the emulator case is really interestingbecause essentially Google has said, all
right, well, if you want to use SAAF andyou wanna store stuff out here, you can
use that framework like it's accessible.
But if you're using something like a, weemulator that requires many, many, many,
(20:53):
many, many files to load a RO into memory.
It's going to be so slow and forreasons that are not apparently entirely
documented, apparently there areexcessive IPC calls from the document
via API when using SA basically justthat's what slows it down so much.
And so the solution that this weemulate dev had to do was basically.
(21:13):
Everything gets stored inthe app directory instead.
And that means save states.
It means your ROS, probably your biosfiles, all of it needs to go there,
uh, because that's the only way theycan guarantee it'll be performant.
And that means if you uninstall the app.
All of that goes away unless the developerhas hit that flag that says, Hey,
there's sensitive personal data here.
(21:35):
Do you want to keep itwhen you uninstall the app?
But Michelle, uh, I'm sure you canexplain more about this whole idea.
The Google play doesn't respectthat flag, which to me is nuts.
I didn't know
Mishaal (21:43):
this.
Yeah.
So there's actually a manifestflag called has fragile user data.
If you set that to true, then wheneveryou go to Android settings and try to
uninstall an app, you'll see a tick boxthat says keep so and so megabytes of.
So when you tick that box, the appspecific files in their external
storage directory doesn't get deleted.
So you can reinstall the appat a later time and all those
(22:05):
files will still be there.
The only problem is Google play.
Doesn't read that the appactually has that flag.
So if you try to uninstall an appvia Google, play it, doesn't ask
you if you wanna keep that data.
So you have to actually go through Androidsettings to uninstall it, to be prompted,
to keep the data, not through Google play.
So, yeah, it's aparticularly tricky problem.
And there because there's justAndrew, such a massive platform.
(22:28):
There are so many different appsand so many different reasons.
And so many different ways apps haveto access files on external storage.
Of course like Google's goalis good overall, but like
the way it impacts certain.
Types of apps is going to be different.
And interestingly, scopestorage also impacts other
Android platforms differently.
So from what I've heard, AndroidTV behaves very interestingly
(22:51):
in terms of scope storage.
So Raymond, do you know anything aboutthe way scope storage impacts Android TV?
Can you tell us a bit about that?
Raymond (22:59):
Actually, I didn't have too much experience on this.
I chatted my two boss at home just tonightand it's indeed running on Android 10.
It is a new report TV report.
So haven't tried much about foulassess restrictions, but on the other
hand, I can point out on my Androidphone, which is funny on Android 12.
(23:21):
I used to use video land player to playvideo files while I'm on the go and.
Power management is alreadyaffected by the new storages model
because of the scope of storage.
And I want to delete some mediafiles after watching it, but as of
today, video land players, storagesmodel is still rely on assessing
(23:45):
the roots of that storage device.
Hence making it impossible to delete,delete media file, like before,
like, um, Android and I, I think.
Before my upload, my draw on to
Mishaal (23:59):
Android trial.
So just to return tothat Android TV example.
So from what I've read, it's avery weird quirk of some Android
TV devices is that they don'tproperly respond to the SAAF intent.
So from what I said before isthat apps can send an intent
to SA to open the files.
That the user can then go and choosewhich directories to give access to
(24:21):
the app to the problem is that a lotof Android TV devices don't have the
documents UI app or the files app, orthey don't, haven't set it up properly.
So what happens is that app says, Hey,I want to have access to this directory.
Then they try to send an intent.
And then the app doesn'texist on the enter TV device.
So you effectively, literally cannotdo anything with that app or like
(24:42):
for a dolphin example, dolphin askedthe user, where did you store your.
You try to open thefolder, you can't do it.
So you literally cannot choose where tostore your games or you'd have to manually
move it to its app specific directory.
So it's a very weird situation.
And from what I read, it's justbecause Google didn't mandate it
on Android TV, even though SAAFlike the proper implementation is
(25:03):
mandated on Android for handhelds.
For some reason, it wasn'tmandated on Android for TV.
And, uh, I have no idea why, but thereis several bug reports about this, but
it does work on shield TV at least.
Yeah.
And
Raymond (25:15):
actually some reports about bar manager also from devices that
doesn't have documents, UI install.
Most are because theyare using custom realms.
David (25:27):
Yeah.
And I mean, that just is a, Michelle is agood example of how Google has to consider
platforms that may or may not get asmuch attention as mainline Android does.
And the effects that certainrequirements will have on them.
Android.
TV's a great example because it SPER.
We deal with a lot of Android TV devices,and it is a wild west in terms of what
manufacturers choose to implement and notimplement and allow you to even access
(25:51):
Google, apparently just does not enforcethings as strictly or care as much.
And so when you have thatkind of situation, yeah.
You're gonna have unintended consequencesand I'm sure Google knew that like
going in, I'm sure they could seelike, well, this is gonna break
Mishaal (26:06):
that.
So another interesting story whilewe're on the topic of Android TV and
paying attention to other platforms.
When Nvidia rolled out the AndroidTV, 11 update to the shield TV.
So on the initial release, theTV did have the documents UI.
So you could open the filesapp and be prompted to choose
which directory to access.
(26:27):
However, the documents UIdidn't provide a way to actually
navigate the app using the.
So even though it opened the app andasked you to choose a directory or
create a directory, you could do neither.
So you instead after like ADB into it andthen use your mouse, or like send inputs
that was fixed with a subsequent release,but it just goes to show, you know, these
(26:47):
kinds of things aren't tested as equitablyon all platforms and all devices.
But fortunately, you know,that's why with Android.
Google actually did respond to feedback.
There's a lot of backlash fromdevelopers who were worried about
the changes of about the amount ofdevelopment work they'd have to.
To update their apps toprepare for scope storage.
(27:09):
So Google actually backtrackedon some scope, storage
related changes in Android 10.
The first thing they backtracked onwas their plan to split, read external
storage into three separate permissions,which they later did in Android 13.
They also made it so Android 10wouldn't enforce scope storage at all
for apps that target Android nine.
And they also made it possible forapps targeting Android 10 to just
(27:30):
opt out of script storage entirelyby setting a specific flag in the.
So even though scope storage wastechnically introduced in Android
10, it didn't really go into effectat all until the next release.
So with Android 11 scope storageis really in effect now, or when
Android 11 was released with Android11, Google made a huge amount of
(27:51):
changes to address developer feedback.
One of the most important things theydid was introduce the new permission
called manage external storage.
Which is shown to the useras quote, all files access.
Uh, and I'm, I'm puttingthat in quotes for a reason.
I'll explain in a second.
So all files access or manageexternal storage permission grants,
the app access to quote all fileson external storage, except for
(28:14):
files under the Android directory.
So it's everything exceptfor the Android direct.
At the same time, they also updatedstorage access tray mix for that apps,
couldn't be given access to the routedirectory of the external storage.
In other words, they couldn't just broadlyask for access to all of external storage.
They couldn't ask for the downloaddirectory and they couldn't ask
(28:35):
for the Android directory either.
So this means that the most practicalway for apps like file managers
to gain access to most of externalstorage would be through the new
managed external storage permiss.
So this is something you'reprobably familiar with.
Raymond.
Can you tell our listeners how Googlewas able to introduce, manage external
storage without negating the effortsthat they went through to reduce
(28:58):
the scope of read external storage?
Like how exactly is manageexternal storage protected?
Raymond (29:03):
I think you are probably talking about the Google plays
approval on using this manageexternal storage flag in manifests.
So they required some apps who need toaccess to all the files on the storage
to raise this rack in the manifest andsend it to Google, play for reveal.
(29:27):
And once they got their approval, youcan publish it to Google play with
this flag, enable and devices runningon, running on enjoy 11 or onwards.
When they seen this flag, they willask user, if you want to allow the
app to you to look at or accessall the files on the storage.
(29:49):
So this is how Google play.
Mishaal (29:52):
So yeah, there's only certain types of apps that Google play will
allow to request this permission.
And you have to go through a permission,declaration, forum and whatnot.
So not just any app can requestit and be granted access to.
So even, so even though this, thispermission is shown to the user
as all files access, as I mention.
(30:12):
It doesn't actually give youaccess to all files on the external
storage apps with its permission.
Aren't allowed to access filesunder the Android directory.
However, amazed as well as many other filemanagers discovered a workaround to access
sub directories under the Android folder.
Anyways.
So I wanted to ask you, Raymond,can you describe for our listeners,
how did this loophole work?
Like how did a maze get accessto Android slash data and
(30:36):
Android slash B and Android 11?
Raymond (30:39):
Okay.
About some time user report, such problem.
And we also saw some app which actuallycan access these to directories and we
Googled and Googled until we find somedevelopers talking about using SA and
by declaring the intent to assess thisparticular path directly in the intent.
(31:02):
When is the intent sent to the documents?
UI, uh, documents, UI will ask you.
If you want access this Android data orAndroid be directory after you press yes.
Amaz or your code can access thesetools directories directly using SA
Mishaal (31:21):
right.
So to summarize basically Android made it.
So you can't ask for access to theAndroid folder in its entirety.
Developers figured out that you canjust ask users to grant access to
its sub directories individually.
So even though you can't accessall of Android, you can access all
of Android slash data and all ofAndroid slash B individually, which
(31:43):
effectively makes the restrictionkind of useless for now until Android
13, which will get on a little bit.
Moving on to the other changesthat Android 11 introduced
in terms of scope storage.
As I mentioned before, Android 11 wasthe first release to actually start
enforcing scope storage on apps.
Whereas apps targeting Android 10,but running on Android 11 could still
opt out using the manifest attributethat I talked about before apps
(32:06):
targeting Android 11 could not opt out.
So no matter what Android 11 isthe release where skip storage
is really starting to be.
And because of Google plays, shiftingtarget API level requirement, eventually
all new apps and apps that are beingupdated would have to target Android 11.
So scope storage is the lawof the land and Android.
Now, by this point in time,Fortunately Google did make some
(32:30):
other concessions to make scriptstories more relatable in Andrew 11.
It restored direct file systemaccess through the Java file
API and native file IO commands.
This was beneficial because there werea lot of Java libraries that relied on
these commands and it would, it wouldbe a huge hassle to have to update
those libraries in on top of your app.
I actually didn't know this before, butwhile I was researching the reason Android
(32:52):
10 didn't support, direct file systemaccess through the Java file API and
native file IO commands wasn't becauseof any intentional restriction by Google,
but it was because it would've requiredsignificant effort in intercepting kernel
calls, whenever they were developingscope storage, they weren't able to
implement enforce scope, storage, andimplement the Nel call interceptions.
(33:13):
So that's why Andrew 10 didn'tsupport direct file path access.
With Android 11, when they switched backto the fuse implementation for emulating
F ad 32 on the external storage, theyalso tuned it for better performance
to bring it on par with SD card Fs.
But one of the other things they did,one of the other reasons they switched
back to fuse is that it enabledbetter support for a script storage
(33:34):
feature that I didn't mention beforeand Android 10 Google added a new
permission called access media location.
This is a permission that appshave to hold in order to access.
Unredacted XF metadata from photos.
So for example, if an app wantedto access the GPS tag in a photo,
they'd have to hold the newaccess media location permission.
(33:54):
The problem is that in Android 10direct storage access through SD
card Fs didn't support this feature.
So like if an app were to use directfile system command, In order to access
files, they'd still be able to getaccess to unredacted excess metadata.
So that's probably partially why it wasn'tallowed in Android 10 and why Android 11.
(34:17):
It was restored because fusedoes support this feature.
So it is able to gate access toexcess metadata that apps hold.
And it's able to check that actuallyactually actually have the permission.
That's.
There's a bit of, a little complicatedunder the hood behind the scenes
kind of business going on withhow Google actually implements the
(34:37):
higher level entered features basedon lower level file system changes.
And it seems really confusing thatGoogle keeps switching back and forth
between these file systems and partitionslike every release, but there's a
reason for every change they make.
David (34:51):
and just, uh, on the subject of X's metadata and why that is something
that Google would get hung up on.
You could use that totrack somebody's location.
Like it'd be very easy using photosfrom the camera roll would be trivial.
Mishaal (35:04):
Yeah.
So I switching away from the SD cardFs, which was all inker and not having
to intercept those kernel calls.
They were able to support this feature bymoving back to fuse, which was all user.
So yeah, enough of thecomplicated under the hood talk.
Now let's just go back to the actualdiscussion scope storage, actually, LA
(35:24):
there is one more bit I'm gonna talk aboutlater, but before we move on to what was
introduced in Android 12, I wanted to askGraham, what are your thoughts overall on
the state of file access since Android 11?
Do you think Google really didaddress most of the issues that
were raised by developers whenthey first introduced skip storage?
Or are there still some lingeringissues maybe around perform.
I think
Raymond (35:43):
the most important complaint is that I can no longer use Java file
API as I did in back in the kit days.
I still miss that because I, I can stillfreely navigate to the direct as I like.
But that's my problem because Istill use the concept of desktop
(36:06):
computer into an Android device andignoring the those privacy concerns.
Uh, this that's what I can think of,but I think Google is tried to close
the loopholes, the chaos since the kitcat days, because of so many different
types of kinds of APIs, frameworks,they Google asked the ramps to use.
Mishaal (36:32):
Yeah.
And one of the other problems stillwith storage access on Android.
The overhead introduced by views itself.
So this is actually one of the reasonswhy Google originally shifted away
from views to SD card Fs was to reduceoverhead and file IO operations, but they
did address some of those performanceconcerns when they brought back views.
(36:55):
Because they, uh, actually tuned itfor better performance, but there's
still some performance overhead.
The reason why this performanceoverhead is because every single
time there's a reader write request.
All these requests have to be forwardedback and forth between the fuse Damon
and the fuse driver instead of an app.
Directly making calls that getsent to the underlying file system.
So like, there's this middleman that's checking each command
(37:18):
and passing each command.
So that introduces some overhead oncommands that increases the amount
of time it takes to traverse fileson certain directory, et cetera.
And this is especiallyfelt when you're trying to.
Do bulk file operations.
So like you're trying to mass deleteor mass move or mass copy, a whole
bunch of files, especially smallfiles, like a ton of small files.
(37:39):
You're really gonna feel thatperformance degradation with fuse.
But some of the optimizations that Googledid to reduce the impact that fuse has
is that any file operations that aredone within the external app specific
directories, AKA within Android slashdata, Android slash OPB bypass fuse,
so that all those commands are sentdirectly to the underlying file system.
(38:01):
So those are prettymuch native performance.
And then supposedly from what Iread, there was also an update to
the media provider mainline module.
So for those of you who know mediaprovider is one of the modular system
components that contains the fuseDamon that contains a lot of the
Android framework, APIs and classesfor actually managing file access.
And October, 2020 update, supposedlybrought improvements to bulk file
(38:23):
operation performance for apps that heldthe manage external storage permission,
but still there are still performanceoverheads that are unavoidable.
And the only way to avoid them isto really just rewrite how your
app works instead of like doingoperations that would invoke fuse.
You can use, uh, bulk APIs and contentprovider or use media provider APIs.
I.
(38:44):
there is one thing that changedAndroid 12, that that should
improve the performance offile operations through fuse.
And it's actually calledfuse pass through.
So it's a feature that relies onspecific support from the kernel and
the framework specific kernel versions.
I think Android 12 5.1and above 5.10 and above.
(39:05):
And then the framework that comes throughan update to the media provider module.
So what happens with fuse pass throughis that when the fuse Damon is handling a
file open request, and it determines thatthe requesting app has full access to the
requested file, then the system can thensubsequently forward all read and write
request directly to the lower file system.
So, as I mentioned before,what would normally happen?
(39:27):
Would that each write, read andwrite request is forwarded from
the fuse driver to the fuse Damon,which introduces a lot of overhead.
So by bypassing that forwarding,it improves performance for reading
and writing large files in theory,because once you open that large
file, you can continue to read andwrite to it without having to go
through the middle man of the fuse.
(39:47):
this, of course won't make a lot ofdifference for reading and writing lots of
small files because you still have to readand open it and then do a small read and
write command and then open another one.
And, you know, they'll still add up,but if you're reading and writing
data to a large file on the externalshared storage directory, then views
pass through should make a difference.
It's probably something you haven'theard of before, cuz it's not something
(40:09):
really Google announced, but it'ssomething that's there in the Android
docs and it's configured in the.
But something you may have heard iswhat Google introduced in Android
13 and what I alluded to before.
So in Android 13, they're actuallysplitting up the read external storage
permission into three new permissionscalled read media audio, read
(40:29):
media video, and read media images.
So what happens is that apps thatare targeting Andrew 13, have to
request these individual permissionsto access the media files of each.
That are owned by other apps.
So for example, an app targeting Android13 that wants to read or modify image
files, it doesn't own would need torequest the read media images, permiss.
(40:51):
Similarly, if they wanted to modifyan audio file, it didn't own they'd
need the read media audio permission.
If you were tracking Android APIs acrossthe years, you might have noticed that
these APIs were actually planned orthese permissions were actually planned
to be introduced in Android 10, but theywere removed in Android 10 beta three,
because Google wanted to give developersmore time to adapt their apps for script
(41:12):
storage, but we're several years out.
So Google feels comfortable inreintroducing this change into Android.
One of the other things thatAndroid 13 did is that it closed
the loophole that file managerslike a maze used to gain access to
the sub directories under Android.
So you can no longer use storageaccess framework to request for
access to slash Android slash dataor slash Android slash B anymore.
(41:36):
I wanted to ask you Raymond,for what are your thoughts on
Android 13 closing this loophole?
And if you found a workaroundthat you can tell us,
Raymond (41:45):
Right now I, I, or our team haven't came up with a
solution on solving this problem.
And personally, as a mobile game directorin my previous job, actually, I will
welcome the decision to close this localbecause hackers can no longer use the
Android phone for long to modify the game.
(42:09):
Actually our team is so an amazedfor which teaches people how to
hack the Android games by using MAsand open JDK on an Android device.
On the negative side, it will meanmore work and more work around, even
more additions, negoti, uh, traditions,negotiation to Google because we want our.
(42:34):
Direct via assess API back, but on thepositive side, I think if you think in
an LS way, LS from the ground up hadrestricted to almost all files except
designated media types, like photos,like new sales videos, except the apple
requests use, use their own proprietary.
(42:58):
But perhaps some Android, maybe we canstill use TP or something like, yeah.
MTP is on the USB side.
That means you still plug on the computer.
You used to plug in the computerto access the whole Android device
as like accessing USB fund drive.
(43:20):
But on device, on the go, you areseeing the Android cloud system
as like a iPhone or something.
You only look at your music, yourphotos, your videos, this on the positive
side, but there's still one complain.
I personally this, uh, I still miss theability to play music in the folder.
(43:42):
I used to organize music in as photosand rather than scattered all them
into one folder and screw it by andfour drag bank to form a pay list.
I just, I still want to justorganize it photo nicely and
just throw into music player.
And there you go.
Yeah, that's all mythat's one of my thoughts.
David (44:04):
I mean, I think we can all relate to that too.
I'm sure we all grew up usingeither windows or some other
traditional file management systemand that's just how our brains work.
It it's how we think about computersand how they organize data.
Mishaal (44:17):
I wanted to ask you a question, David, actually hearing
about all the changes that have comeso far with Android in relation to
file access and file management andall the restrictions that have come.
Do you think Android will continueto offer relatively open access to
the underlying external file system?
Or do you think will become moreand more like iOS where you don't
(44:39):
really have a sense of what actuallyhow the, the structure is actually.
David (44:44):
Yeah, iOS is so totally abstracted.
That way apps can createfolders in this protected media
space, essentially what it is.
It's the same thing asAndroids media collections.
I'm sure in principle, I'm sure itworks differently, but if apps can't
see anything outside that very,very narrowly scoped storage area.
(45:05):
I've run into this so many times onmy iPad where I'm just like, I want
to see the files, this app I know hasaccess to, but for whatever reason,
I can't, because probably those twodevelopers would need to cooperate
and say, Hey, I can't see your stuff.
Why can't you see it?
And they would have to figure out okay.
Um, what do I need to change myapplication to see like Google photos, for
(45:28):
example, one of the strange limitationson Google photos, you can only put one
Google photo in one iMessage at a time.
You can't do multiple Google photos,probably because iOS doesn't support
that for third party gallery styleapplications in the file picker.
So you can see the problems, this approachintroduces because it inherently limits
what you can do with the files too.
(45:50):
And explaining that to users.
Like, you're never going to get themto understand the nuances, why one
application can do one thing and whyone application can't do another.
And on iOS, that is somethingI run into all the time.
But like even me being fairly casualfor an Android enthusiast, this is not
the kind of stuff most people run into.
(46:10):
And I think that is where Google.
Kind of wants to take, at least theAndroid phone experience is to a place
where it's very hard for people to break.
That's probably the most important thing.
They don't want people in thesedirectories because the chance that they
could cause an app to misbehave and theneither crash or lose their data, God
(46:30):
forbid there's something they can mess up.
So yeah, I do see Googlecontinuing to move this direction.
And the sad truth is that there's not alot of incentive for them to preserve.
This more traditionalfile structure access.
I mean, can you imagine how manydevelopers out there, especially
for things like mass market mobilegames, get emails from people trying
(46:52):
to like mess around with directorystructure and they break something.
I'm sure it happens pretty frequently.
And not to say that thatis Google's problem.
Like, you know, users go doing thingsthat break devices all the time,
but at the end of the day, Google'sconcern is of a massive ecosystem.
And so they're.
What's best for their partners in thatecosystem, their partners are Samsung.
(47:14):
Show me all of thesemajor smartphone vendors.
And they may say, Hey, we'regetting a lot of customer support
issues around vis people are reallyhaving trouble using file apps.
How do we make this simpler?
And so Google has to digest that feedback.
And as much as we like to think thatthe Android team is above the fray, in
terms of deciding how the platform works.
(47:36):
I don't think they totally are.
They have to consider these things.
And they also have to consider thefact that there are hundreds of
millions of people out there whouse this and who they would like
to use an Android phone one day.
And the more they can make itfeel seamless in terms of that
experience, the easier it is toget somebody to switch platforms.
So there's a lot of stuffthey have to think about.
(47:58):
That's like, I mean, totally beyondthe engineering level there, but.
Yeah, I'm sure they're considering this.
And I would say that the directionwe've seen Android go that is they're
going toward a more restrictive
Mishaal (48:09):
model.
Oh.
And for those of you at home whodidn't immediately realize what
David meant when he said thishealed up an iPhone to show us.
So, uh, that's all the questions thatI actually had for this discussion.
I mean, I know we went through anabsolute ton of information and I
hope this is starting to make sense.
(48:29):
And I haven't actually confused youmore about going through all of this.
I mean, trust me, it took me a lotto actually digest and put all this
information down and there's stillstuff that I could talk about, like more
information, but you gotta stop somewhere.
And I think a one hour markis a pretty good place to.
David (48:46):
And if this is an interesting topic to you, the entire subject of how storage
works on Android and how applicationsinteract with the operating system.
That's a great question for a companylike Asper to answer because we deal
with apps on Android devices, especiallyunusual ones of varying OS distro,
or excuse me, varying OS versions,underlying platform versions all the time.
(49:06):
So if you're wonder.
Hey, can I have an app that likebehaves nicely on Android 12,
but also works on my Android ninedevices, which I already know it does.
We can help you figure some of this out.
We can help you understandhow the platform has changed.
What kind of devices you may need tocreate the sort of experience you want.
And when it comes to things like AndroidTV, well, how do you get around the
issue of not having access to thefiles app on an Android TV device?
(49:30):
That is also one of those stickyquestions that you can come to us.
And we can see if there's a solutionthere because we deal with very
strange Android devices all the time.
We are not a first customizing firmware.
We have our own distributioncalled foundation available
for arm and X 86 devices.
And this is the kind of thing that ifyou need an Android device to behave
(49:51):
in a particular way, because it has avery specific task associated with it.
That's what ER's here for.
Check us out@esper.io and Michelle,I'll let you go ahead and, uh,
Mishaal (50:01):
do the outro.
Thanks, David.
And, uh, thank you again,Raymond for joining us.
So if people want to follow you inyour work, where can they find you?
Oh, you can find me
Raymond (50:11):
at hab.com/trans love because I love trans music.
And you can also find me on our apps,telegram channel ma file manager,
which you can find it at the ma
Mishaal (50:25):
file manager app.
Yeah.
And if you're looking for amaterial themed open source file
manager app check out maze filemanager on the Google play store.
And if you're looking for the contentproduced by the editorial team at
Esper, check us out at blog, doper.io,where you'll find all of our content.
Not just our blog articles, not just ournewsletter, but also episodes of this
(50:47):
podcast, which I'd say thank you againfor joining us and listening through an
hour long discussion of the nerdiest finallike topic I could think of right now.
Um, hopefully this really helped you out.
It helped me out in piecing this together.
So thank you for joining us onanother episode of Android bites.
Hello, and welcome to Android bites powered by Asper I'm, David
Rudick and each week I'm joinedby my cohost Michelle ramen.
As we dive deep into the technicalworld of Android, and we're going to
be talking about something that keepseverybody awake at night file access
and file systems on Android, which isafter reading the outline for this show.
(00:22):
I am more confused than when we started.
So I'm really excited to learn hereand I'll let Michelle introduce our.
Mishaal (00:29):
thank you, David.
And I apologize in advance to anyonewho's gonna walk away more confused
than before they came into this show.
So, uh, before we dive into the,uh, confusing aspects, let me
just offer a brief opportunity forRaymond to introduce themselves.
So welcome to the show, Raymond,and tell our listeners what you do.
What do you work on?
Raymond (00:47):
Hello everyone.
I'm Raymond from Hong Kong.
And currently I'm work on,uh, Android farm manager
project called ma farm manager.
And I joined it from 2018 when I needa farm manager that can dosh copy.
So after back and forthsome good requests, the team
(01:09):
invited me to join the project.
Currently my focus in the MAs projectis mostly on integrations about
different protocols, the FTP server,and the recent one I'm working, been
working on and have been asked by.
The users many time is the far FTP client
Mishaal (01:30):
support.
Well, thank you for joining us.
And for those of you who haven't usedthe app before it's available on Google
plays, the maze file manager app.
And the reason I wanted to invite adeveloper who's worked on file managers
is because the topic of today's discussionis scope storage, or rather how scope
storage changed Android file access.
(01:51):
And it's, it's a very complicated subject.
Um, one that I still.
To this day, I have to verify that Iknow what I'm talking about by looking
it up on the inter developer docs.
Every single time I wannawrite about scope storage.
I have to go back and makesure did I get this bit right?
And a lot of times it's I, you, youmiss something minor detail, but the
reason scope storage is confusing isnot anything particularly to do with the
(02:14):
concept itself, but rather how Googleapproached the scope storage model.
It's because that when they introducescope storage with Android, The way
file access worked then continued tochange with every Android release since.
So with Android 11, it gotsignificantly changed with Android 12.
There was some behind the scene changes.
(02:34):
And with Android 13, there are somerather significant, but not a significant
changes, but still like if you're an,a developer, who's trying to figure
out, okay, I want to access files.
Where do I store them?
How do I access them?
It's gotten a lot more complicated thanit used to be in the earlier days of.
And because file access intersectswith so many different other aspects
of Android, including the Linux kernel,the underlying file systems, the
(02:55):
way the storage ship is partitioned.
I think it's an interesting topic andsomething that's worth discussing.
So before we actually talk aboutscope storage, I think it's important.
We actually go way, way, way back.
I mean, the early days of Android, likeover a decade ago to talk about SD cards.
So in the very early days of Android,I'm talking like in the nexus one era.
(03:17):
A lot of phones or most phones ratherdepended on having removable SD
cards to hold media files as wellas additional apps in their data.
Since at the time internal storageships had rather mediocre capacities.
So during this time Google referredto the SD card as Android's external
storage, which obviously made sensebecause SD cards were removable and
they weren't built into the device.
(03:37):
When Google shifted away fromSD cards with the nexus S though
everything then had to be stored ontothe devices, built in storage ship,
including all the media files and allthe app data, et cetera, that were
previously stored on the external SD.
However, in order to maintaincompatibility apps that were expecting
files to be stored on the physicalexternal SD card Mount point would instead
(03:58):
be transparently directed to the newslash data slash media Mount point to this
day, you'll see a lot of ADB commands,ADB tutorials that refer to SD slash
SD card when such a thing doesn't existon most Android phones instead slash SD
card is just symbolic link to slash dataslash media because of this legacy of
Android used to rely on physical SD card.
(04:19):
So the reason I'm talking about SDcards and stuff, that's over a decade
old is because to this day, Androidstill refers to slash data slash media
as external storage, even though nineoutta 10 times, that's actually storage.
That's internal to the device.
The way that Android sees storageis that you have the underlying
user data partition on phones,which is mounted at slash data it's.
(04:41):
So the data partition is writeable media.
It's where all your app data, allyour removal files are stored.
It's generally split into two parts.
You have everything outside ofslash data slash media, which
Android considers internal storage.
Then you have everything insideof slash data slash media.
That's considered external storage.
And the reason those two are separatedis because the internal storage and the
(05:04):
external storage at slash data slashmedia are actually mounted differently.
So even though the underlyingdata partition may be formatted
in F two Fs or X four file systemsslash data slash media is actually
mounted with a virtual file system.
There are many reasons to do thismost importantly, being that.
So you don't have to unmount theentire phones data partition whenever
(05:26):
you wanna hook it up to a PC, butthere are also some privacy and
security benefits to having this.
So prior to Android, 4.4 kit cat,this virtual SD card partition is
what soft often referred to as wasmounted using the VF a T file system.
But this posed someproblems in the early days.
This is because the slash data slashmedia Mount point or the external
(05:49):
storage is what I'll refer to itpretty much from now on is not only
where you'll find the shared downloadsfolder, your pictures, music, movies,
and other sensitive media folders.
But it's also home to the Android folder.
And the Android folder is where apps canread or write whatever files they want
within their app specific directories.
So like slash Android slash data slashAndroid slash B for, uh, game files.
(06:12):
So just another broad step backapps could store files in one
of three primary locations.
There are app specific directoryin the internal storage I E
and slash data slash data.
There are app specific directoriesin the external storage.
Many times you see it asslash SD card slash Android.
When in actuality it's slash data slashmedia slash the user ID slash Android
(06:35):
or any other directory in externalstorage that they have access to.
So app specific directories inthe internal storage are rather
limited in space intentionally by.
So apps that need to store lotsof large files or just lots of
files in general, generally pickssomewhere in the external storage.
So an app like telegram, forexample, has a ton of cashed
images and documents and videos.
(06:56):
So it stores all of that in its appspecific external storage directory.
The problem with the way this wasimplemented in Android prior to
4.4, is that the V F a T file systemdidn't support the kind of permission
management that Google wanted to.
I.
The issue was that apps that wanted toread and write files into their own app.
Specific external storage directorieshad to request the read external
(07:19):
storage and write external storagepermissions in order to do so.
But requesting those permissionsalso gave apps broad access to the
entirety of the external storage,just by needing the permissions to
access their own specific directories.
They also had access to the entirety.
The external storage.
That was a problem because this wasbasically start of apps, having too
(07:40):
much access to external storage, whichalso housed other shared media files.
So that was a lot before we moveon to how Android or how Google
started to address this problem.
I wanted to ask about your thoughtsand experiences dealing with file
management in the early days of Android.
Like, what were your thoughts on SD cars?
Things like apps to SD mass storage, et.
Raymond (08:01):
First of all.
I think Google's decision to use weat is external storage because of
compatibility concern, because about10 years ago, many users are still on
using windows or Mac silver, littlewho are using Linux as their desktop.
So using we as the file system, they,they, they can Mount on, they can
(08:25):
do this without any special drivers.
So that they can just plug into theUSB port and they can access the files
inside the Android phone directly.
And then before that they of keep cats.
Mishaal (08:41):
I think we mostly use
Raymond (08:44):
the farm manager that the Android phone bundled in.
So perhaps there are many different kindsof implementations from different vendors,
but generally you can see the Androidphone as simply as, uh, speech fund drive.
If you can, if you sayotherwise, very simple,
David (09:05):
that's an interesting analogy.
And I had not thought of itthat way, but that makes a lot
of sense actually, intuitively.
Mishaal (09:11):
Yeah.
So as Raymond mentioned, using V Fa T allowed for broad compatibility,
because the way PC access worked wasthat Android would actually unmount the
virtual SD card partition, and then itwould Mount it onto the connected device.
Obviously this led to some problems earlyon, because if you wanted to use your
phone while it's connected to the PC,all of a sudden, a lot of apps will be
(09:32):
missing media files or missing some data.
So instead Android then shifted tousing the media transfer protocol
or MTP to handle a file transfer.
And this made it so that itdidn't matter what file system
was used on the Android device.
It would still be supported by the PC.
So be something connected to the PC.
(09:53):
And the reason you can look thatup how MTP works, but that's not
really important for this, but thatwas one of the ways that Google.
Was able to shift away from VF a T inAndroid, 4.4, starting from Android
4.4 and beyond, instead of mounting thevirtual SD card partition as VFA, Google
decided to use Lennox's file system anduser space feature or fuse for short.
(10:17):
To emulate F a T 32.
So the reason Google decided toswitch to emulating F a T 32, is
that it allowed Android to use Ffat's directory style permissions.
This made it possible for Android tolet apps access their own app specific
external storage directories withoutneeding those read external storage
and write external storage permiss.
(10:39):
Eventually Google did what Googledoes best and deprecated using
fuse switched to SD card Fest forvarious reasons, which is a Samsung
developed kernel file system wrapper.
And that, uh, was done in Android 8.0 toimprove performance and fixed some issues.
But then they reverted back to fuse inAndroid 11 after making some improvements.
And in order to better supportsome scope, storages, new feature.
(11:01):
But we'll get into that bit later.
It's a bit of Android file system history.
That is not super important tounderstand the nitty gritty of right now.
But what's important to take awayfrom this is that by switching
from the V a T style system tousing fuse, to emulate F a T 32.
The end result is that fewer apps neededto hold the read external storage and
(11:22):
write external storage permissions.
And so that means that fewer appshad the ability to clutter your
external storage with folders orSnoop on your private media files.
However, this didn't solve everythingbecause there's still far too many
apps having broad file access.
And over the years, Googletried to reduce the number.
The most significant thing theytried was Android 6.0, which
introduced runtime remission.
(11:44):
Read external storage and writeexternal storage now had to be manual
granted by the user at run time.
So like there would be a dialogue askingthe user, do you want to grant these
permissions instead of granting themautomatically at install time, but this
still did not do enough to limit thenumber of apps that had broad file access,
something big needed to change to dothis, which leads us to scope storage.
(12:07):
Scope storage for those of you don'tknow, was introduced with Android 10 and
it's described as a it's a new model forthe way file access works in Android.
It revamped the way fi apps could accessfiles located on the external storage.
The biggest change that scope storagemade was to the right external storage
and read external storage permissions.
So for apps that target Android10, right, external storage
(12:30):
effectively became deprecated.
And folded into read external storage.
So whenever you grant an appaccess to read external storage,
it implicitly has the ability toalso write files or modify files.
Read external storage on the otherhand was significantly reduced in
scope granting an app that permissionno longer gave it broad access to
the entirety of external storage.
Instead, the permission only gave accessto media files that were contributed to
(12:53):
media store collections by other apps.
I'm I'm gonna get themedia store in a bit.
So basically apps that contribute mediafiles to media store collections can do so
without needing any permissions to do so.
But in order to read or modify filesthat other apps contributed, they need to
hold the read external storage permiss.
So you're an app developer.
Raymond who's worked on file manager.
Can you tell us a bit about themedia store API for listeners who
(13:16):
may not be familiar with it about
Raymond (13:18):
the media store API is to carry API to carry different
kinds of media files on the devicecategorized by different video types,
like image, audio documents, cetera,
Mishaal (13:34):
et C.
Right.
So like Android has likean index of media files.
If you've ever opened like thefiles app, you have like the images
filter, it shows you images fromvarious filters on your device.
That's basically the user facing accessto the media store and apps can add their
own image files to that media store.
(13:54):
And then other apps can access thoseimage files in the media store.
But in order to access images thatother apps contributed, they need
the read external storage permiss.
So that's what end enter 10 and
David (14:06):
just provide some context on this.
I'm sure we can get into this later on aswell, but there were really good reasons.
Google did this.
Number one is just that applicationsin general, the number of files,
they would have broad access tothe ability to read and write.
There are obviously nefarious thingsthat they could do with that permission.
If a user did not understandwhat kind of permission they were
(14:28):
granting that application and users.
When an application, especially asks forfile access, I think are very likely to
say yes, because that is something thatfrom the windows and Mac O S paradigms,
um, is very common applications, alwaysneed file access because they need
to see directories access, you know,documents, whatever the case may be.
So users are very accustom to justbeing like, yeah, it's an app.
(14:51):
It needs access files.
I.
Files, but, and the way the Androidfile system works, which is say
something I don't understand very well.
I do not know enough to even saywhat granting full file system access
really would let an app do per se,compared to something like windows.
And so Google was trying to define,okay, what are the very safe.
File types.
(15:12):
We can let any app, as longas the user says, okay.
Access and obviously images, video,audio, because those are things that
people share constantly and reallydo want to move between applications,
whether it's touching up a photo anda photo editing app, or it's sharing
something and putting a filter on it onInstagram, because you already edited
somewhere else in another application.
(15:33):
People want that traceabilityfor their personal media files.
And that's what Google I thinkwas really trying to enable.
Mishaal (15:39):
Right.
And just to clarify for anyone who maybe concerned, when we say like broad
external storage access, externalstorage, remember is only files
contained within slash data slash media.
So it's only the folder.
It's only the files that are createdor stuff you download to your device.
You can delete the entirety of slash dataslash media and your phone is still the.
(16:02):
Because it's all files createdafter you install your apps
and you set up your device.
None of it's critical tosystem operation, all of that
stored in different partitions.
So it's not like windows or somethingwhere you grant an app administrative
access, it can literally nuke your device.
Um, I'm unless you're how true that isactually on anymore in modern windows, or
it's also not the case that you're givingan app like root access that could truly
(16:24):
Rick havoc, the main concern with givingan app broad external storage access is.
They can potentially readsensitive media files that you
don't want them to access to.
So like for example, if a game wants tostore files on a folder of its choice,
like they create a folder on the externalstorage in order to do that, they
needed that external storage permission.
(16:44):
But at the same time, by giving 'emthat permission, they all of a sudden
had the ability to read your photosor to like to look at your photos.
That's not something you want togive every game access to and you
who knows what they might be doing.
So that's the reason why Googlewanted to crack down on this.
Just there's just too much potentialof like a privacy invasion.
But on the other hand, there are appsthat legitimately do need broad, external
(17:05):
storage access, including file managerslike a maze, like the one that Raymond
works on, but because their scope ofread external storage access was reduced.
These apps need an alternativein order to access the same list
of directors they had before.
Fortunately an alternative alreadyexisted in Android in the form of
the storage access framework, whichis actually something that Google
introduced way back in Android, 4.4.
(17:26):
So the storage access framework orSA for short makes it possible for
apps to request the user, to grantaccess to certain directories.
So what would happen is the filemanager like maze would invoke SA
using an intent and that will thenpull up the system provided documents.
Picker also known as the files app.
If, if you look it up in many places,The user would then use the files app
(17:46):
to browse through the external storageand choose which directory to grant
the app access to which on Android 10,they could even choose to give an app
access to the entirety of the externalstorage by just choosing the root folder.
This sounded fine on paperusing SA to access files and
directories, except for one problem.
Accessing files via SFA.
API was slow, really, really slow.
(18:09):
So slow.
In fact, that benchmarks done bydevelopers showed it was orders of
magnitude slower than using traditionalJava file API or native file IO commands.
Something that might take like, uh,mere seconds to maybe scan for music
files on your external storage wouldtake maybe two minutes to do now.
Like ludicrously slower developershad raised alarms about SA F of four,
(18:32):
because it's actually been requiredto use since Android six point.
No, if you wanted to access files onexternal USB flash tribes, but with
Android 10, it became front and centerbecause it became required to use for apps
targeting Android 10 in order to actually.
Access or read many files becauseenter 10 didn't support, direct
file path access via the native Javafile API or native file IO commands.
(18:56):
So, Raymond, I think this is a bitwhere you probably know much more than
we do so many people may have heardabout the controversies around skip
storage, but they may not be awareof how exactly negatively affected
certain types of apps or use cases.
Can you tell us what exactly madefile access via SA significantly
worse than the previous method?
The
Raymond (19:14):
extra effort there to obtain the info stream stream of a particular
file on the storage device, wherever yousay, being an SD card or attached that
USB from drive or something, there isan extra effort required to obtain the
stream wheel the end, we the system.
(19:35):
So I think that's themain reason that made
Mishaal (19:41):
powers through the SA significant.
So apart from file managers likea maze, what are some other kinds
of apps that were also impacted byscope storage and SA and Android 10?
I think
Raymond (19:52):
some game key, uh, console AORs would be affected.
Actually, I didn't try myself becausethe last device I used to play games
is still running on Android act.
But I think as of Android 10, some, aembryos like retro art, Or dolphin would
(20:12):
maybe affected because the extra requireon assessing the files and the storage
and because the users have to put thefiles in a way that Google, like, so the
AMBS can smoothly assess those files.
David (20:29):
So I, I think, and I'll steal your lines from the outline here, Michelle,
because I think this is an interestinguse case and it's one that I understand.
So the emulator case is really interestingbecause essentially Google has said, all
right, well, if you want to use SAAF andyou wanna store stuff out here, you can
use that framework like it's accessible.
But if you're using something like a, weemulator that requires many, many, many,
(20:53):
many, many files to load a RO into memory.
It's going to be so slow and forreasons that are not apparently entirely
documented, apparently there areexcessive IPC calls from the document
via API when using SA basically justthat's what slows it down so much.
And so the solution that this weemulate dev had to do was basically.
(21:13):
Everything gets stored inthe app directory instead.
And that means save states.
It means your ROS, probably your biosfiles, all of it needs to go there,
uh, because that's the only way theycan guarantee it'll be performant.
And that means if you uninstall the app.
All of that goes away unless the developerhas hit that flag that says, Hey,
there's sensitive personal data here.
(21:35):
Do you want to keep itwhen you uninstall the app?
But Michelle, uh, I'm sure you canexplain more about this whole idea.
The Google play doesn't respectthat flag, which to me is nuts.
I didn't know
Mishaal (21:43):
this.
Yeah.
So there's actually a manifestflag called has fragile user data.
If you set that to true, then wheneveryou go to Android settings and try to
uninstall an app, you'll see a tick boxthat says keep so and so megabytes of.
So when you tick that box, the appspecific files in their external
storage directory doesn't get deleted.
So you can reinstall the appat a later time and all those
(22:05):
files will still be there.
The only problem is Google play.
Doesn't read that the appactually has that flag.
So if you try to uninstall an appvia Google, play it, doesn't ask
you if you wanna keep that data.
So you have to actually go through Androidsettings to uninstall it, to be prompted,
to keep the data, not through Google play.
So, yeah, it's aparticularly tricky problem.
And there because there's justAndrew, such a massive platform.
(22:28):
There are so many different appsand so many different reasons.
And so many different ways apps haveto access files on external storage.
Of course like Google's goalis good overall, but like
the way it impacts certain.
Types of apps is going to be different.
And interestingly, scopestorage also impacts other
Android platforms differently.
So from what I've heard, AndroidTV behaves very interestingly
(22:51):
in terms of scope storage.
So Raymond, do you know anything aboutthe way scope storage impacts Android TV?
Can you tell us a bit about that?
Raymond (22:59):
Actually, I didn't have too much experience on this.
I chatted my two boss at home just tonightand it's indeed running on Android 10.
It is a new report TV report.
So haven't tried much about foulassess restrictions, but on the other
hand, I can point out on my Androidphone, which is funny on Android 12.
(23:21):
I used to use video land player to playvideo files while I'm on the go and.
Power management is alreadyaffected by the new storages model
because of the scope of storage.
And I want to delete some mediafiles after watching it, but as of
today, video land players, storagesmodel is still rely on assessing
(23:45):
the roots of that storage device.
Hence making it impossible to delete,delete media file, like before,
like, um, Android and I, I think.
Before my upload, my draw on to
Mishaal (23:59):
Android trial.
So just to return tothat Android TV example.
So from what I've read, it's avery weird quirk of some Android
TV devices is that they don'tproperly respond to the SAAF intent.
So from what I said before isthat apps can send an intent
to SA to open the files.
That the user can then go and choosewhich directories to give access to
(24:21):
the app to the problem is that a lotof Android TV devices don't have the
documents UI app or the files app, orthey don't, haven't set it up properly.
So what happens is that app says, Hey,I want to have access to this directory.
Then they try to send an intent.
And then the app doesn'texist on the enter TV device.
So you effectively, literally cannotdo anything with that app or like
(24:42):
for a dolphin example, dolphin askedthe user, where did you store your.
You try to open thefolder, you can't do it.
So you literally cannot choose where tostore your games or you'd have to manually
move it to its app specific directory.
So it's a very weird situation.
And from what I read, it's justbecause Google didn't mandate it
on Android TV, even though SAAFlike the proper implementation is
(25:03):
mandated on Android for handhelds.
For some reason, it wasn'tmandated on Android for TV.
And, uh, I have no idea why, but thereis several bug reports about this, but
it does work on shield TV at least.
Yeah.
And
Raymond (25:15):
actually some reports about bar manager also from devices that
doesn't have documents, UI install.
Most are because theyare using custom realms.
David (25:27):
Yeah.
And I mean, that just is a, Michelle is agood example of how Google has to consider
platforms that may or may not get asmuch attention as mainline Android does.
And the effects that certainrequirements will have on them.
Android.
TV's a great example because it SPER.
We deal with a lot of Android TV devices,and it is a wild west in terms of what
manufacturers choose to implement and notimplement and allow you to even access
(25:51):
Google, apparently just does not enforcethings as strictly or care as much.
And so when you have thatkind of situation, yeah.
You're gonna have unintended consequencesand I'm sure Google knew that like
going in, I'm sure they could seelike, well, this is gonna break
Mishaal (26:06):
that.
So another interesting story whilewe're on the topic of Android TV and
paying attention to other platforms.
When Nvidia rolled out the AndroidTV, 11 update to the shield TV.
So on the initial release, theTV did have the documents UI.
So you could open the filesapp and be prompted to choose
which directory to access.
(26:27):
However, the documents UIdidn't provide a way to actually
navigate the app using the.
So even though it opened the app andasked you to choose a directory or
create a directory, you could do neither.
So you instead after like ADB into it andthen use your mouse, or like send inputs
that was fixed with a subsequent release,but it just goes to show, you know, these
(26:47):
kinds of things aren't tested as equitablyon all platforms and all devices.
But fortunately, you know,that's why with Android.
Google actually did respond to feedback.
There's a lot of backlash fromdevelopers who were worried about
the changes of about the amount ofdevelopment work they'd have to.
To update their apps toprepare for scope storage.
(27:09):
So Google actually backtrackedon some scope, storage
related changes in Android 10.
The first thing they backtracked onwas their plan to split, read external
storage into three separate permissions,which they later did in Android 13.
They also made it so Android 10wouldn't enforce scope storage at all
for apps that target Android nine.
And they also made it possible forapps targeting Android 10 to just
(27:30):
opt out of script storage entirelyby setting a specific flag in the.
So even though scope storage wastechnically introduced in Android
10, it didn't really go into effectat all until the next release.
So with Android 11 scope storageis really in effect now, or when
Android 11 was released with Android11, Google made a huge amount of
(27:51):
changes to address developer feedback.
One of the most important things theydid was introduce the new permission
called manage external storage.
Which is shown to the useras quote, all files access.
Uh, and I'm, I'm puttingthat in quotes for a reason.
I'll explain in a second.
So all files access or manageexternal storage permission grants,
the app access to quote all fileson external storage, except for
(28:14):
files under the Android directory.
So it's everything exceptfor the Android direct.
At the same time, they also updatedstorage access tray mix for that apps,
couldn't be given access to the routedirectory of the external storage.
In other words, they couldn't just broadlyask for access to all of external storage.
They couldn't ask for the downloaddirectory and they couldn't ask
(28:35):
for the Android directory either.
So this means that the most practicalway for apps like file managers
to gain access to most of externalstorage would be through the new
managed external storage permiss.
So this is something you'reprobably familiar with.
Raymond.
Can you tell our listeners how Googlewas able to introduce, manage external
storage without negating the effortsthat they went through to reduce
(28:58):
the scope of read external storage?
Like how exactly is manageexternal storage protected?
Raymond (29:03):
I think you are probably talking about the Google plays
approval on using this manageexternal storage flag in manifests.
So they required some apps who need toaccess to all the files on the storage
to raise this rack in the manifest andsend it to Google, play for reveal.
(29:27):
And once they got their approval, youcan publish it to Google play with
this flag, enable and devices runningon, running on enjoy 11 or onwards.
When they seen this flag, they willask user, if you want to allow the
app to you to look at or accessall the files on the storage.
(29:49):
So this is how Google play.
Mishaal (29:52):
So yeah, there's only certain types of apps that Google play will
allow to request this permission.
And you have to go through a permission,declaration, forum and whatnot.
So not just any app can requestit and be granted access to.
So even, so even though this, thispermission is shown to the user
as all files access, as I mention.
(30:12):
It doesn't actually give youaccess to all files on the external
storage apps with its permission.
Aren't allowed to access filesunder the Android directory.
However, amazed as well as many other filemanagers discovered a workaround to access
sub directories under the Android folder.
Anyways.
So I wanted to ask you, Raymond,can you describe for our listeners,
how did this loophole work?
Like how did a maze get accessto Android slash data and
(30:36):
Android slash B and Android 11?
Raymond (30:39):
Okay.
About some time user report, such problem.
And we also saw some app which actuallycan access these to directories and we
Googled and Googled until we find somedevelopers talking about using SA and
by declaring the intent to assess thisparticular path directly in the intent.
(31:02):
When is the intent sent to the documents?
UI, uh, documents, UI will ask you.
If you want access this Android data orAndroid be directory after you press yes.
Amaz or your code can access thesetools directories directly using SA
Mishaal (31:21):
right.
So to summarize basically Android made it.
So you can't ask for access to theAndroid folder in its entirety.
Developers figured out that you canjust ask users to grant access to
its sub directories individually.
So even though you can't accessall of Android, you can access all
of Android slash data and all ofAndroid slash B individually, which
(31:43):
effectively makes the restrictionkind of useless for now until Android
13, which will get on a little bit.
Moving on to the other changesthat Android 11 introduced
in terms of scope storage.
As I mentioned before, Android 11 wasthe first release to actually start
enforcing scope storage on apps.
Whereas apps targeting Android 10,but running on Android 11 could still
opt out using the manifest attributethat I talked about before apps
(32:06):
targeting Android 11 could not opt out.
So no matter what Android 11 isthe release where skip storage
is really starting to be.
And because of Google plays, shiftingtarget API level requirement, eventually
all new apps and apps that are beingupdated would have to target Android 11.
So scope storage is the lawof the land and Android.
Now, by this point in time,Fortunately Google did make some
(32:30):
other concessions to make scriptstories more relatable in Andrew 11.
It restored direct file systemaccess through the Java file
API and native file IO commands.
This was beneficial because there werea lot of Java libraries that relied on
these commands and it would, it wouldbe a huge hassle to have to update
those libraries in on top of your app.
I actually didn't know this before, butwhile I was researching the reason Android
(32:52):
10 didn't support, direct file systemaccess through the Java file API and
native file IO commands wasn't becauseof any intentional restriction by Google,
but it was because it would've requiredsignificant effort in intercepting kernel
calls, whenever they were developingscope storage, they weren't able to
implement enforce scope, storage, andimplement the Nel call interceptions.
(33:13):
So that's why Andrew 10 didn'tsupport direct file path access.
With Android 11, when they switched backto the fuse implementation for emulating
F ad 32 on the external storage, theyalso tuned it for better performance
to bring it on par with SD card Fs.
But one of the other things they did,one of the other reasons they switched
back to fuse is that it enabledbetter support for a script storage
(33:34):
feature that I didn't mention beforeand Android 10 Google added a new
permission called access media location.
This is a permission that appshave to hold in order to access.
Unredacted XF metadata from photos.
So for example, if an app wantedto access the GPS tag in a photo,
they'd have to hold the newaccess media location permission.
(33:54):
The problem is that in Android 10direct storage access through SD
card Fs didn't support this feature.
So like if an app were to use directfile system command, In order to access
files, they'd still be able to getaccess to unredacted excess metadata.
So that's probably partially why it wasn'tallowed in Android 10 and why Android 11.
(34:17):
It was restored because fusedoes support this feature.
So it is able to gate access toexcess metadata that apps hold.
And it's able to check that actuallyactually actually have the permission.
That's.
There's a bit of, a little complicatedunder the hood behind the scenes
kind of business going on withhow Google actually implements the
(34:37):
higher level entered features basedon lower level file system changes.
And it seems really confusing thatGoogle keeps switching back and forth
between these file systems and partitionslike every release, but there's a
reason for every change they make.
David (34:51):
and just, uh, on the subject of X's metadata and why that is something
that Google would get hung up on.
You could use that totrack somebody's location.
Like it'd be very easy using photosfrom the camera roll would be trivial.
Mishaal (35:04):
Yeah.
So I switching away from the SD cardFs, which was all inker and not having
to intercept those kernel calls.
They were able to support this feature bymoving back to fuse, which was all user.
So yeah, enough of thecomplicated under the hood talk.
Now let's just go back to the actualdiscussion scope storage, actually, LA
(35:24):
there is one more bit I'm gonna talk aboutlater, but before we move on to what was
introduced in Android 12, I wanted to askGraham, what are your thoughts overall on
the state of file access since Android 11?
Do you think Google really didaddress most of the issues that
were raised by developers whenthey first introduced skip storage?
Or are there still some lingeringissues maybe around perform.
I think
Raymond (35:43):
the most important complaint is that I can no longer use Java file
API as I did in back in the kit days.
I still miss that because I, I can stillfreely navigate to the direct as I like.
But that's my problem because Istill use the concept of desktop
(36:06):
computer into an Android device andignoring the those privacy concerns.
Uh, this that's what I can think of,but I think Google is tried to close
the loopholes, the chaos since the kitcat days, because of so many different
types of kinds of APIs, frameworks,they Google asked the ramps to use.
Mishaal (36:32):
Yeah.
And one of the other problems stillwith storage access on Android.
The overhead introduced by views itself.
So this is actually one of the reasonswhy Google originally shifted away
from views to SD card Fs was to reduceoverhead and file IO operations, but they
did address some of those performanceconcerns when they brought back views.
(36:55):
Because they, uh, actually tuned itfor better performance, but there's
still some performance overhead.
The reason why this performanceoverhead is because every single
time there's a reader write request.
All these requests have to be forwardedback and forth between the fuse Damon
and the fuse driver instead of an app.
Directly making calls that getsent to the underlying file system.
So like, there's this middleman that's checking each command
(37:18):
and passing each command.
So that introduces some overhead oncommands that increases the amount
of time it takes to traverse fileson certain directory, et cetera.
And this is especiallyfelt when you're trying to.
Do bulk file operations.
So like you're trying to mass deleteor mass move or mass copy, a whole
bunch of files, especially smallfiles, like a ton of small files.
(37:39):
You're really gonna feel thatperformance degradation with fuse.
But some of the optimizations that Googledid to reduce the impact that fuse has
is that any file operations that aredone within the external app specific
directories, AKA within Android slashdata, Android slash OPB bypass fuse,
so that all those commands are sentdirectly to the underlying file system.
(38:01):
So those are prettymuch native performance.
And then supposedly from what Iread, there was also an update to
the media provider mainline module.
So for those of you who know mediaprovider is one of the modular system
components that contains the fuseDamon that contains a lot of the
Android framework, APIs and classesfor actually managing file access.
And October, 2020 update, supposedlybrought improvements to bulk file
(38:23):
operation performance for apps that heldthe manage external storage permission,
but still there are still performanceoverheads that are unavoidable.
And the only way to avoid them isto really just rewrite how your
app works instead of like doingoperations that would invoke fuse.
You can use, uh, bulk APIs and contentprovider or use media provider APIs.
I.
(38:44):
there is one thing that changedAndroid 12, that that should
improve the performance offile operations through fuse.
And it's actually calledfuse pass through.
So it's a feature that relies onspecific support from the kernel and
the framework specific kernel versions.
I think Android 12 5.1and above 5.10 and above.
(39:05):
And then the framework that comes throughan update to the media provider module.
So what happens with fuse pass throughis that when the fuse Damon is handling a
file open request, and it determines thatthe requesting app has full access to the
requested file, then the system can thensubsequently forward all read and write
request directly to the lower file system.
So, as I mentioned before,what would normally happen?
(39:27):
Would that each write, read andwrite request is forwarded from
the fuse driver to the fuse Damon,which introduces a lot of overhead.
So by bypassing that forwarding,it improves performance for reading
and writing large files in theory,because once you open that large
file, you can continue to read andwrite to it without having to go
through the middle man of the fuse.
(39:47):
this, of course won't make a lot ofdifference for reading and writing lots of
small files because you still have to readand open it and then do a small read and
write command and then open another one.
And, you know, they'll still add up,but if you're reading and writing
data to a large file on the externalshared storage directory, then views
pass through should make a difference.
It's probably something you haven'theard of before, cuz it's not something
(40:09):
really Google announced, but it'ssomething that's there in the Android
docs and it's configured in the.
But something you may have heard iswhat Google introduced in Android
13 and what I alluded to before.
So in Android 13, they're actuallysplitting up the read external storage
permission into three new permissionscalled read media audio, read
(40:29):
media video, and read media images.
So what happens is that apps thatare targeting Andrew 13, have to
request these individual permissionsto access the media files of each.
That are owned by other apps.
So for example, an app targeting Android13 that wants to read or modify image
files, it doesn't own would need torequest the read media images, permiss.
(40:51):
Similarly, if they wanted to modifyan audio file, it didn't own they'd
need the read media audio permission.
If you were tracking Android APIs acrossthe years, you might have noticed that
these APIs were actually planned orthese permissions were actually planned
to be introduced in Android 10, but theywere removed in Android 10 beta three,
because Google wanted to give developersmore time to adapt their apps for script
(41:12):
storage, but we're several years out.
So Google feels comfortable inreintroducing this change into Android.
One of the other things thatAndroid 13 did is that it closed
the loophole that file managerslike a maze used to gain access to
the sub directories under Android.
So you can no longer use storageaccess framework to request for
access to slash Android slash dataor slash Android slash B anymore.
(41:36):
I wanted to ask you Raymond,for what are your thoughts on
Android 13 closing this loophole?
And if you found a workaroundthat you can tell us,
Raymond (41:45):
Right now I, I, or our team haven't came up with a
solution on solving this problem.
And personally, as a mobile game directorin my previous job, actually, I will
welcome the decision to close this localbecause hackers can no longer use the
Android phone for long to modify the game.
(42:09):
Actually our team is so an amazedfor which teaches people how to
hack the Android games by using MAsand open JDK on an Android device.
On the negative side, it will meanmore work and more work around, even
more additions, negoti, uh, traditions,negotiation to Google because we want our.
(42:34):
Direct via assess API back, but on thepositive side, I think if you think in
an LS way, LS from the ground up hadrestricted to almost all files except
designated media types, like photos,like new sales videos, except the apple
requests use, use their own proprietary.
(42:58):
But perhaps some Android, maybe we canstill use TP or something like, yeah.
MTP is on the USB side.
That means you still plug on the computer.
You used to plug in the computerto access the whole Android device
as like accessing USB fund drive.
(43:20):
But on device, on the go, you areseeing the Android cloud system
as like a iPhone or something.
You only look at your music, yourphotos, your videos, this on the positive
side, but there's still one complain.
I personally this, uh, I still miss theability to play music in the folder.
(43:42):
I used to organize music in as photosand rather than scattered all them
into one folder and screw it by andfour drag bank to form a pay list.
I just, I still want to justorganize it photo nicely and
just throw into music player.
And there you go.
Yeah, that's all mythat's one of my thoughts.
David (44:04):
I mean, I think we can all relate to that too.
I'm sure we all grew up usingeither windows or some other
traditional file management systemand that's just how our brains work.
It it's how we think about computersand how they organize data.
Mishaal (44:17):
I wanted to ask you a question, David, actually hearing
about all the changes that have comeso far with Android in relation to
file access and file management andall the restrictions that have come.
Do you think Android will continueto offer relatively open access to
the underlying external file system?
Or do you think will become moreand more like iOS where you don't
(44:39):
really have a sense of what actuallyhow the, the structure is actually.
David (44:44):
Yeah, iOS is so totally abstracted.
That way apps can createfolders in this protected media
space, essentially what it is.
It's the same thing asAndroids media collections.
I'm sure in principle, I'm sure itworks differently, but if apps can't
see anything outside that very,very narrowly scoped storage area.
(45:05):
I've run into this so many times onmy iPad where I'm just like, I want
to see the files, this app I know hasaccess to, but for whatever reason,
I can't, because probably those twodevelopers would need to cooperate
and say, Hey, I can't see your stuff.
Why can't you see it?
And they would have to figure out okay.
Um, what do I need to change myapplication to see like Google photos, for
(45:28):
example, one of the strange limitationson Google photos, you can only put one
Google photo in one iMessage at a time.
You can't do multiple Google photos,probably because iOS doesn't support
that for third party gallery styleapplications in the file picker.
So you can see the problems, this approachintroduces because it inherently limits
what you can do with the files too.
(45:50):
And explaining that to users.
Like, you're never going to get themto understand the nuances, why one
application can do one thing and whyone application can't do another.
And on iOS, that is somethingI run into all the time.
But like even me being fairly casualfor an Android enthusiast, this is not
the kind of stuff most people run into.
(46:10):
And I think that is where Google.
Kind of wants to take, at least theAndroid phone experience is to a place
where it's very hard for people to break.
That's probably the most important thing.
They don't want people in thesedirectories because the chance that they
could cause an app to misbehave and theneither crash or lose their data, God
(46:30):
forbid there's something they can mess up.
So yeah, I do see Googlecontinuing to move this direction.
And the sad truth is that there's not alot of incentive for them to preserve.
This more traditionalfile structure access.
I mean, can you imagine how manydevelopers out there, especially
for things like mass market mobilegames, get emails from people trying
(46:52):
to like mess around with directorystructure and they break something.
I'm sure it happens pretty frequently.
And not to say that thatis Google's problem.
Like, you know, users go doing thingsthat break devices all the time,
but at the end of the day, Google'sconcern is of a massive ecosystem.
And so they're.
What's best for their partners in thatecosystem, their partners are Samsung.
(47:14):
Show me all of thesemajor smartphone vendors.
And they may say, Hey, we'regetting a lot of customer support
issues around vis people are reallyhaving trouble using file apps.
How do we make this simpler?
And so Google has to digest that feedback.
And as much as we like to think thatthe Android team is above the fray, in
terms of deciding how the platform works.
(47:36):
I don't think they totally are.
They have to consider these things.
And they also have to consider thefact that there are hundreds of
millions of people out there whouse this and who they would like
to use an Android phone one day.
And the more they can make itfeel seamless in terms of that
experience, the easier it is toget somebody to switch platforms.
So there's a lot of stuffthey have to think about.
(47:58):
That's like, I mean, totally beyondthe engineering level there, but.
Yeah, I'm sure they're considering this.
And I would say that the directionwe've seen Android go that is they're
going toward a more restrictive
Mishaal (48:09):
model.
Oh.
And for those of you at home whodidn't immediately realize what
David meant when he said thishealed up an iPhone to show us.
So, uh, that's all the questions thatI actually had for this discussion.
I mean, I know we went through anabsolute ton of information and I
hope this is starting to make sense.
(48:29):
And I haven't actually confused youmore about going through all of this.
I mean, trust me, it took me a lotto actually digest and put all this
information down and there's stillstuff that I could talk about, like more
information, but you gotta stop somewhere.
And I think a one hour markis a pretty good place to.
David (48:46):
And if this is an interesting topic to you, the entire subject of how storage
works on Android and how applicationsinteract with the operating system.
That's a great question for a companylike Asper to answer because we deal
with apps on Android devices, especiallyunusual ones of varying OS distro,
or excuse me, varying OS versions,underlying platform versions all the time.
(49:06):
So if you're wonder.
Hey, can I have an app that likebehaves nicely on Android 12,
but also works on my Android ninedevices, which I already know it does.
We can help you figure some of this out.
We can help you understandhow the platform has changed.
What kind of devices you may need tocreate the sort of experience you want.
And when it comes to things like AndroidTV, well, how do you get around the
issue of not having access to thefiles app on an Android TV device?
(49:30):
That is also one of those stickyquestions that you can come to us.
And we can see if there's a solutionthere because we deal with very
strange Android devices all the time.
We are not a first customizing firmware.
We have our own distributioncalled foundation available
for arm and X 86 devices.
And this is the kind of thing that ifyou need an Android device to behave
(49:51):
in a particular way, because it has avery specific task associated with it.
That's what ER's here for.
Check us out@esper.io and Michelle,I'll let you go ahead and, uh,
Mishaal (50:01):
do the outro.
Thanks, David.
And, uh, thank you again,Raymond for joining us.
So if people want to follow you inyour work, where can they find you?
Oh, you can find me
Raymond (50:11):
at hab.com/trans love because I love trans music.
And you can also find me on our apps,telegram channel ma file manager,
which you can find it at the ma
Mishaal (50:25):
file manager app.
Yeah.
And if you're looking for amaterial themed open source file
manager app check out maze filemanager on the Google play store.
And if you're looking for the contentproduced by the editorial team at
Esper, check us out at blog, doper.io,where you'll find all of our content.
Not just our blog articles, not just ournewsletter, but also episodes of this
(50:47):
podcast, which I'd say thank you againfor joining us and listening through an
hour long discussion of the nerdiest finallike topic I could think of right now.
Um, hopefully this really helped you out.
It helped me out in piecing this together.
So thank you for joining us onanother episode of Android bites.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!