Deleting data-blocks, "unmaking" assets, loading a .blend without UI, etc shouldn't cause the Asset Browser to reference invalid data.
It currently references the preview and asset-data of asset data-blocks. This is fragile.
We already store the ID.session_uuid and could use that to sanity check the references before access.
Besides this memory issue, we could just make sure the file list is refreshed for relevant notifiers while Current File is shown. If the list still ends up being outdated then, there at least won't be a crash/undefined behavior.
Also see T82162.