Self-funded Startup | Building proprietary data sources to transform internet signal to operational intelligence, tailored to human & AI ingestion requirements.
It's been a tough couple of months due to some family health issues starting back in June - but things are still progressing with the @dendrite_tech project. This is a quick recap on recent development, current state and a realistic projection for go-live.
I'm hoping to avoid any furhter delays, but we should be looking at September - October for launch 🙏 It's honesly infuriating how close things are. https://t.co/rgzXcJdMh0
The last major pre-launch round of data propagation should be complete in the next several hours. After that, I just have to make sure that the web app is ingesting / visualizing the data the right way - and it's sharing time!
Get ready for an annoying volume of screenshots with some videos sprinkled in there 😂.
I've been working on API's, connectors and my data delivery models today, and I have to say: I have a new found respect for the API engineering & security 🤣.
I can worry about a lot of this in the background while I'm making videos on a completed/connected graphical data explorer, but 🫡 to @HackingDave for setting the bar with NightBeacon having a 5 minute integration time. I've onboarded a lot of systems and data sources over the years and I'm going to try to get a close as possible to 5 minutes - but that's pushing out of ludicrous into plaid speed. IYKYK
Thought of this while having a convo with someone at or from DEF CON and accidentally pulled an all nighter to push over some final dev hurdles - I've been redesigning my core engine (correlations) for the last week, starting when the architecture started to feel less scalable during testing against a full prod-scale data set.
So there are now more than 700 branch types (links + meta + confidence = 1 type) between the primary ~20 data sources, which represents a ~15% increase from the previous model that produced 3.2 Tn correlations. The changes should result in higher fidelity joins and discoverable data, further expedited analysis for security teams, and deeper temporal reasoning ability for agents consuming derivative products...I hope.
Going to hit the gym and clear my head while running this next test. I know that I've been saying this but I really hope to build a video soon - this last thing has been holding it back.💪
Running a major transform (final step in pipeline) on the last the last 80TB block of data before shipping it to the cloud - which will might take a while for both lol. Once landed, I just have to ensure the full data category is queryable by the graphical explorer.
There's a solid chance that I might be able to wrap up the core tech this week, focus on demo videos and scaling the pipelines, building new connectors and post-launch roadmap!
@TG01_Actual As of this morning, there are roughly 51.361 Bn datapoints (key : value pair equivalent) in the Tor side of the Dark Web data collection alone. Each of these are correlated to other data points within this and the other 4 data sets.
I've had my head down in the weeds with other data sources but I ran a high level query against the @dendrite_tech Tor Dark Web database today (doesn't include I2P) and it's a pretty big WTF moment:
> 11.9M Enriched Service (site/blog/forum/market/chat) records - usually between 250 - 375 data points, includes sub-pages | mirrors & redirects not included
> 39M~ Tor users w/>100k mapped to the surface web due to poor OPSEC
> 149.7M identified URL's in que (includes sub-pages)
> 1.2Bn deep web & surface web references
....and I don't want to say how many unique crypto addresses located but only because I'm planning a post on this soon.
Where things start getting weird is when the C2 discovery & enumeration and Surface/Deep web pipelines find resources not referenced anywhere else on the dark web. More on this topic to come....
Wait 24 hours > Sample Output Data > Fail Data > Troubleshoot > CI/CD Pipeline > Repeat
I'm definitely my own toughest critic, but I really just want people to be blown away with the data quality when it's available on the data markets. And Yes: that's in scope 🤣
Check out our latest knowledge-share post by @TG01_Actual where we break down the functional differences between "training" & "grounding" AI Agents, our recommendations, and likely operational outcomes of errors in training strategies for security workflows.
https://t.co/fWafWiymAu
Our latest post by @TG01_Actual provides a high level overview of what dark web intelligence actually looks like operationally and explores the structural limitations of the monitoring product category. We also dive into what a relational, cross-domain picture of the dark web enables that keyword alerts never will.
https://t.co/0XstLGBgQY
@TG01_Actual Feed. Me.
Joking aside: This (and more) is needed to support a new data source that we're hoping to bring online in the coming weeks - a multi-chain crypto data explorer. Blog article coming soon!
Signals collection & processing will always be difficult - but the intel and data products it produces are worth it (in our opinion). Not a task for the light hearted or serverless.
The challenges in the opening are legitimate, just not as difficult as they're being made out to be. I'm leaning against what their solution means for training data for RNN/CNN - I obviously took a dif route. Thoughts @ImposeCost@j2k3k@jamieantisocial?
https://t.co/kHbjvx2LAL
I didn't want to say anything during, but I just completed a 7 day fast with only water, salt and 2 cups/day of coffee. There are a lot of things that I realized I need to do differently from now on.
Despite approaching 8 million web records from the .onion side of Void Runner, it's identified 98 million additional sites & pages worth crawling. Once fully enriched, each record holds an average of ~255 data fields - many reaching >500.
It really just depends on the problem we're trying to solve, but we anticipate a lot more Go coming up for short-lived internal processes. Go's main advantage over Rust is its development & compile speeds - possibly I/O performance, which makes it superior for cloud based apps and processes. Where Rust pulls ahead is for apps running (a) high concurrency ops (b) performing complex extract & transforms on large/raw data volumes over (c) long durations. The security/safety claims are also real.
In testing, we found unoptimized Rust outperformed Go at about 45 minutes of runtime in the A+B+C type scenarios shown above. Anything shorter and/or with fewer constraints around scaling & security will likely be built in Go.
@TG01_Actual@lucasMnts Based on our combined total prod and NPE code volumes, we're currently at 8.1% Go and 91.8% Rust. This will likely change to the favor of Go as we build more container apps for the public cloud, and short sprint processes.
Going to a rough start to the week but after setting up some GCP infra, I'm hoping to record & edit a walkthrough video of the Dendrite CLI. Really solid recommendation to check out @RiversidedotFM, thx @ZackKorman
@deepdives787117@TG01_Actual There will soon (tonight) be multiple themes to choose from for the dark and light modes as well. For the dark themes, we're adding crimson, embers and a some sort of blue or purple.
A sneak peek at the web portal, where users will be able to explore Dendrite and 3rd party data sources in graphical view, participate in forums, encrypted chat, manage org users & API keys.
Video walkthroughs by @TG01_Actual are in the works and for release soon.