How do Newtonsoft.Json and System.Text.Json cause unloadability issues?
Newtonsoft.Json and System.Text.Json are libraries that achieve one goal: to provide an easy way to parse JSON files by serializing and deserializing them. In modern .NET frameworks, such as .NET 10.0, AssemblyLoadContext was introduced to provide you a way to load and unload assemblies cooperatively, and it was first released with .NET Core 1.0.
As a result, it’s frequently used in cases where loading and unloading assemblies are required, such as the plugin system in a C# application. However, there is a massive drawback when it comes to unloading assemblies that internally use Newtonsoft.Json and System.Text.Json. When an assembly uses one of the two libraries and their serialization functions, and when unloading such assembly is requested by AssemblyLoadContext.Unload(), even if garbage collection is forced, the assembly will not unload.
We will use two small Nitrocid addons as examples: Nitrocid.Extras.Chemistry and Nitrocid.Extras.Mods. While the Chemistry addon becomes impossible to unload once the element command is used, the Mods addon also becomes impossible to unload once it registers its own settings instance to the settings manager. We will first take a look at Nitrocid.Extras.Mods.
Newtonsoft.Json – Nitrocid.Extras.Mods
The Tips addon includes a settings class that is defined like this:
using Newtonsoft.Json; using Nitrocid.Base.Kernel.Configuration; using Nitrocid.Base.Kernel.Configuration.Instances; using Nitrocid.Base.Kernel.Configuration.Settings; using Nitrocid.Base.Kernel.Exceptions; using Nitrocid.Base.Languages; using Nitrocid.Base.Misc.Reflection.Internal; namespace Nitrocid.Extras.Mods.Settings { /// <summary> /// Configuration instance for Mods /// </summary> public partial class ModsConfig : BaseKernelConfig { /// <inheritdoc/> [JsonIgnore] public override SettingsEntry[] SettingsEntries { get { var dataStream = ResourcesManager.GetData("ModsSettings.json", ResourcesType.Misc, typeof(ModsConfig).Assembly) ?? throw new KernelException(KernelExceptionType.Config, LanguageTools.GetLocalized("NKS_MODS_SETTINGS_EXCEPTION_ENTRIESFAILED")); string dataString = ResourcesManager.ConvertToString(dataStream); return ConfigTools.GetSettingsEntries(dataString); } } } }
using Nitrocid.Base.Kernel.Configuration.Instances; namespace Nitrocid.Extras.Mods.Settings { /// <summary> /// Configuration instance for Mods /// </summary> public partial class ModsConfig : BaseKernelConfig { /// <summary> /// Automatically start the kernel modifications on boot. /// </summary> public bool StartKernelMods { get; set; } /// <summary> /// Allow untrusted mods /// </summary> public bool AllowUntrustedMods { get; set; } /// <summary> /// Write the filenames of the mods that will not run on startup. When you're finished, write "q". Write a minus sign next to the path to remove an existing mod. /// </summary> public string BlacklistedModsString { get; set; } = ""; /// <summary> /// Show the mod commands count on help /// </summary> public bool ShowModCommandsCount { get; set; } = true; } }
Internally, Nitrocid uses Newtonsoft.Json to read the configuration entries from the user configuration files found under the following paths:
- Windows:
%LOCALAPPDATA%\KS\ - Linux:
~/.config/ks/
So, in this case, the tips settings are stored in:
- Windows:
%LOCALAPPDATA%\KS\ModsConfig.json - Linux:
~/.config/ks/ModsConfig.json
When the current main branch of Nitrocid shuts down, it tells all the addon load context classes that the addon subsystem manages to unload all the addons’ load contexts when the addon finally unloads itself, as in the following code:
[MethodImpl(MethodImplOptions.NoInlining)] internal static void UnloadAddons() { Dictionary<string, string> errors = []; for (int addonIdx = addons.Count - 1; addonIdx >= 0; addonIdx--) { var addonInfo = addons[addonIdx]; var addonInstance = addonInfo.Addon; var alc = addonInfo.alc; try { using var context = alc.EnterContextualReflection(); addonInstance.StopAddon(); } catch (Exception ex) { DebugWriter.WriteDebug(DebugLevel.E, "Failed to stop addon {0}. {1}", vars: [addonInfo.AddonName, ex.Message]); DebugWriter.WriteDebugStackTrace(ex); errors.Add(addonInfo.AddonName, ex is KernelException kex ? kex.OriginalExceptionMessage : ex.Message); } finally { // Unload the assembly on garbage collection alc.Unload(); addons.RemoveAt(addonIdx); } } // Unload all addon assemblies GC.Collect(); GC.WaitForPendingFinalizers(); GC.Collect(); if (errors.Count != 0) throw new KernelException(KernelExceptionType.AddonManagement, LanguageTools.GetLocalized("NKS_KERNEL_EXTENSIONS_ADDONS_EXCEPTION_STOPFAILED") + $"\n - {string.Join("\n - ", errors.Select((kvp) => $"{kvp.Key}: {kvp.Value}"))}"); }
When looking at this code, one thinks that all addon assemblies must unload once we call the garbage collector to force disposal of all load contexts. When Nitrocid shut down, we’ve placed the breakpoint at the stage where the RPC is supposed to stop, as seen here.
When looking at the output window, we have seen that some of the addons got unloaded successfully. However, all addons that used the kernel configuration subsystem to read their own configuration that the kernel knows about didn’t get unloaded, including the Mods addon, as per the output window:
'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons/Extras.Calculators/Nitrocid.Extras.Calculators.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons/Extras.UnitConv/Nitrocid.Extras.UnitConv.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons/Extras.ColorConvert/Nitrocid.Extras.ColorConvert.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons/Extras.Dictionary/Nitrocid.Extras.Dictionary.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons/Extras.Caffeine/Nitrocid.Extras.Caffeine.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons/ThemePacks/Nitrocid.ThemePacks.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons/Extras.ThemeStudio/Nitrocid.Extras.ThemeStudio.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons/Extras.Hashes/Nitrocid.Extras.Hashes.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons/Extras.BeepSynth/Nitrocid.Extras.BeepSynth.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons.Essentials/Extras.Images/Nitrocid.Extras.Images.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons/Extras.Docking/Nitrocid.Extras.Docking.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:\Users\aptiv\Aptivi\Source\Public\Nitrocid\public\Nitrocid\KSBuild\net10.0\Addons.Essentials\Extras.Images.Icons\Terminaux.Images.Icons.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons.Essentials/Extras.Images.Icons/Nitrocid.Extras.Images.Icons.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons/Extras.Pastebin/Nitrocid.Extras.Pastebin.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons/Extras.Chemistry/Nitrocid.Extras.Chemistry.dll' 'Nitrocid.exe' (CoreCLR: clrhost): Unloaded 'C:/Users/aptiv/Aptivi/Source/Public/Nitrocid/public/Nitrocid/KSBuild/net10.0/Addons/Extras.Notes/Nitrocid.Extras.Notes.dll'
That’s 15 addons out of 31 addons! Now, let’s use Visual Studio to examine the situation. We are now at the RemoteProcedure.StopRPC() line, and the we’ve seen that 16 addons didn’t get unloaded, when they’re supposed to. Looking at the Diagnostic tools > Memory Usage and taking a snapshot of the memory, we’ve seen an object type of Newtonsoft.Json.Serialization.JsonProperty in the object type view of the snapshot. The count was 1715, the size was 376608, and the inclusive size was 5918488.
Digging deeper revealed something interesting.
This revealed that Newtonsoft.Json has been caching properties as soon as the Nitrocid kernel processes and reads the configuration, as per:
if (ConfigTools.IsCustomSettingBuiltin(typeName)) baseConfigurations[typeName] = (BaseKernelConfig?)JsonConvert.DeserializeObject(jsonContents, type.GetType()) ?? throw new KernelException(KernelExceptionType.Config, LanguageTools.GetLocalized("NKS_KERNEL_CONFIGURATION_EXCEPTION_BASECONFIGDESERIALIZE"), typeName); else customConfigurations[typeName] = (BaseKernelConfig?)JsonConvert.DeserializeObject(jsonContents, type.GetType()) ?? throw new KernelException(KernelExceptionType.Config, LanguageTools.GetLocalized("NKS_KERNEL_CONFIGURATION_EXCEPTION_CUSTOMCONFIGDESERIALIZE"), typeName);
The default contract resolver contains caches to contracts for each type, which, in turn, caches the properties as seen here:
This improves performance of future JSON operations, but causes unloading to fail due to them being strong references.
System.Text.Json – Nitrocid.Extras.Chemistry
When it comes to System.Text.Json, we’ve run another Nitrocid session and ran the element Br command to let ChemiStar load all elements before we shut the kernel down to tell all assemblies to unload. This caused the Chemistry addon to no longer unload. We’ve used the memory snapshot function and searched for ChemiStar-related classes, and got the JSON type info and property info instances that are related to how caching works.
As you can see, we have JSON serialization options that caches the types and the properties for performance reasons. However, those, again, cause unloading issues.
In general, mods and addons that use any of Newtonsoft.Json and System.Text.Json’s serialization functions will no longer be able to be unloaded once Nitrocid shuts down. You can find more information in one of the GitHub issues listed here:
#Net #Net100 #dotnet #news #NewtonsoftJson #SystemTextJson #Tech #Technology #update
