Nathanael Anderson

DevOps, SecOps, Author, Speaker, Father, and generally a nice guy. I do mobile, desktop, cloud and XR programming. Working on

[Nathanael Anderson @CongoCart]

For those people who follow me because of my Asset Manager Studio project.

I Just released v1.0.0, something like 30 releases to get to this point -- this is a BIG milestone.

Not only is downloading of assets massively faster, but now it it is completely cross platform.

[Nathanael Anderson @CongoCart]

@BloodPawWolf @fab @UnrealEngine @quixeltools @Leartes @BundleFoundry support in the next version... ;-)

[Nathanael Anderson @CongoCart]

The next version of Asset Manager Studio now can check Cosmos bundles! 😉

[Nathanael Anderson @CongoCart]

@BloodPawWolf @fab @UnrealEngine @quixeltools @Leartes Sweet, guess I need to add support to AMS for it. 😉

[Nathanael Anderson @CongoCart]

The rust and c++ app loads everything basically at startup. This is of course the simplest way to build an app, load everything at one time and display it all... It works well when you have a small list of items. It starts breaking down quickly as the item count grows.

The rust app also loads a lot more data than the C++ version as it has a decently complete view of each item when you click on it vs the C++ app. The c++ version just shows the very basics and that is it. So when the rust app load its ~3,000 items, it keeps a lot more in memory.

In mine, AMS, I not only use a virtual list for everything, and while I do load all 20k worth of asset info at startup (& a ton of other data) -- I load ONLY the fields that is needed to generate each entry and a few extra fields like developer, category and description that I allow the user to search/filter/sort upon. When you click on an item, then I load the extra 30ish fields of data for display in the detail view JUST for that item, so this keeps my memory usage a lot lower as those 30+ fields are never using memory for the other 20k worth of items. 😉

One thing that I have been seriously thinking about revamping since AMS now is dealing with > 20k of items -- is moving the search/sort/filter to the db layer as then I can eliminate loading the description (which actually tends to be rather large and wastes a lot of memory per item). This was a tradeoff I made early on when my program was also only dealing with about ~2,000 items. But it is something that now is bothering me as the majority of the chunk of my 600mb of memory AMS is using is because of loading all these extra fields that are solely used for filtering...

[Nathanael Anderson @CongoCart]

@borger17730488 @0xglitchbyte Well, my app is Asset Manager Studio. It handles all things in the Unreal Engine eco-system.

[Nathanael Anderson @CongoCart]

@msftsecresponse So how come your still suing him? https://t.co/Qqs4sL7qsX

[Nightmare Eclipse @ChaoticEclipse0]

Everyone except me ? We are in fact still in court over this.

[Nathanael Anderson @CongoCart]

@pavandavuluri @surface Erps, pasted the wrong link. Here is the direct link... https://t.co/Qqs4sL7qsX

[Nightmare Eclipse @ChaoticEclipse0]

Everyone except me ? We are in fact still in court over this.

[Nathanael Anderson @CongoCart]

@pavandavuluri @surface Well considering your own lawyers are suing a researching that exposed his own research on issues in windows because your team failed to address them -- I think it would be better to avoid any Windows products, who knows what might cause MS to sue them...

https://t.co/bwO3Whllb4

[Nathanael Anderson @CongoCart]

@cyb3rops -ish. It sounds like they are still suing him, so its like a very soft walk backish...
https://t.co/Qqs4sL7qsX

[Nightmare Eclipse @ChaoticEclipse0]

Everyone except me ? We are in fact still in court over this.

[Nathanael Anderson @CongoCart]

@cyannick I've played with it, but perhaps instead of either git or perforce you would like @nafonsopt Ark VCS, it is pretty awesome and cross platform.

[Nathanael Anderson @CongoCart]

I'm certainly glad it has worked for you! All I can do is share my story where it didn't work for me (including the btrfs scrubbing) when btrfs was faced with a hardware corruption issue.

ZFS worked perfectly with the exact same hardware issue over a longer time with zero data loss. I even was able to swap out each of the three "bad" drives one at a time in the array with a new drives, once I discovered it was a hardware issue, which tends to be an array killer when dealing with bad hardware... I actually just replaced all the drives, one at a time -- because I didn't trust that model despite all the drives being from different batches, getting 3 that had the same issue was too big of a coincidence for me to let them stay for my data...

Funnily enough, one of them I turned into the boot drive at a later point and it is still working great -- so it ended up not being "all" drives from that model line, just some... 🤷‍♂️

[Nathanael Anderson @CongoCart]

Yeah, I know -- I was actually really surprised, at how poor both of their memory usage was. Both the C++ and Rust ones are using load everything, the only reason the C++ is not as bad at memory is because it only shows very very very basic info about each item, so it only has the title and thumbnail and id, it doesn't show any details ever. The rust app actually does a really good job at showing all sorts of details about each item (almost as much as mine does) -- but as such it loads and retains massively more data into memory at startup.

I basically load in, Title, developer, id, thumbnail, description, categories and about 10 other fields for filter/search-ability. As all sort/filtering/searching is purely against the loaded data, which makes it tremendously fast to find & sort things for the user. When you click it, then It loads about 30+ additional fields of data on only that single item they clicked to show for the detailed view, that data is replaced when you click on another item, so memory stays leaner while using the app as it never has all the data for an item until you click it. I am now considering moving the filter/sort to my db layer as 22k of what I am loading is now wasting (imho) a lot of memory. Things like the description, developer and categories aren't even shown until you click the item in detail view so it is a lot of "waste" purely for behind the scenes filtering/searching. It wasn't bad when it was only 3k of items, as my program was only using about 300mb for everything (150mb-200mb is cost just using a webview) -- I could justify the 100-150mb extra, but now that it is 400-450mb of memory, that makes me unhappy...

[Nathanael Anderson @CongoCart]

Yep, that was what I thought too. It reported no issues, and seemed like an awesome FS. It wasn't until I started getting some weird results that I did a deep dive into the system logs and found that a large chunk of the files at rest on the array was corrupt w/o any new writes to those areas... (BTRFS still reported no issues!) I was happy with BTRFS until I wasn't. 🤣

The one thing I do know is under the exact same hardware issue on 2 drives, BTRFS was basically a total loss of all data. ZFS had _0_ bytes of loss over an even longer period of time running with the same HW corruption events across 3 drives. (an additional drive started doing the same thing, so ZFS dealt with more drives doing the same nasty data corruption fault for an even longer period of time!).

I think the reason why is because ZFS detected the corruption way sooner do to it checksums for everything. When I first installed ZFS and read about it, I thought wow, they are paranoid about data being changed... I think because of the paranoia, that when it detected a sector on drive A was "changed" underneath it, it used the other drives to rebuild it behind the scenes and returned the data while resaving the broken drive's data cluster.

[Nathanael Anderson @CongoCart]

Oh, I don't disagree -- most developers using Electron don't care and as such most Electron, Tauri, and Wails apps suck big time.

You just said you hadn't seen a single app, I love being the outlier. 🤣 I also would NOT be surprised if my electron app is the fastest and most performative electron app in existence that actually does something, since I actually benchmark startup on every launch. 😉

[Nathanael Anderson @CongoCart]

As I said the DEVELOPERs matters more than the underlying language.

The charts do matter because they do show that it is developer that matters more than the language. Litterally, you can swap each of the apps and do mostly the same things. (My app can do more, so you really can't swap it and do the same things. 😉). But you could swap out the C++ app for the Rust app and do the exact same thing or vise-versa, and then swap out either of them for my app and do the EXACT same things (plus more. 😉).

Since all of our apps are free, the rust team DID choose the easiest path forward. You guess correctly, they don't recycle anything AND they load literally everything into memory including the kitchen sink at launch and as such each additional item adds overhead making 3k worth of items use 15gb of memory.

My app, does recycle and it only loads what is needed, when it is needed. But the only reason it is so, is because I DO care about performance and as such I spent the extra time creating the architecture that allowed my app to be low memory and instant response at all times, I could have also chosen the "easy" route that both the C++ and Rust app took and loaded everything at startup...

[Nathanael Anderson @CongoCart]

Well I can tell you MY Electron app doesn't do that. It uses about 600mb of memory for 22k items loaded into memory, and has very little cpu usage (zero when in the background).

My only two competitors in my market, one written in Rust (no browser, 100% native controls) use 15.8GB, and the other one is C/C++ w/CEF uses 8.3GB. They both only handling 3k of items. 😉

[Nathanael Anderson @CongoCart]

No, I ran tests a couple years back (someone else just recently had the same issue). Flutter was the odd man out in the tests because I really needed a Webview for my project, but because I've done a ton of Mobile Flutter work I decided just for the fun of it to see how flutter would handle these two basic apps. Here is the specific blog entry on Flutter https://t.co/T3qVz8Sj2y

[Nathanael Anderson @CongoCart]

@BskiMike22802 Question for delicate electronics -- could a Faraday cage or even Faraday bags work to insulate it from Miyake event...