Team Opsec

LayerZero Sybil Hunting Team

[Team Opsec @Team_Opsec]

All of Team Opsec's sybil reports were denied.

However, thanks to the efforts of @PrimordialAA and @LayerZero_Core all of our reports have since been accepted in full.

Many thanks to the LayerZero team, we're pleased to have solidified ourselves as a top bounty hunter.

[Team Opsec @Team_Opsec]

@SybilHorror6 Damn, the zigzag report has been receiving negative feedback and complaints for the entire duration of the program. I'm baffled how it's been included, but none of ours have 💀

[Team Opsec @Team_Opsec]

@PrimordialAA Although no Ruslan wallets are in the sybil list🤣

[Team Opsec @Team_Opsec]

@PrimordialAA Amazing job. There are 2,160,738 sybil wallets in this list, this is a phenomenal figure, I think the community bounty hunting program was well worth the overheads.

[Team Opsec @Team_Opsec]

@PrimordialAA Ruslan strikes again, all his wallets (apart from Rus0003.eth) are not included in the sybil list, and will receive an airdrop reward.

[Team Opsec @Team_Opsec]

@PrimordialAA We're a little disappointed that none of our reports were included, even our updated reduced list of undeniable clusters (https://t.co/9CpUMnJ1QL)

¯\_(ツ)_/¯

It is what it is, hopefully we get our bond back!

[Team Opsec @Team_Opsec]

GM @PrimordialAA & @LayerZero_Labs,

We at Team Opsec authored 3 great reports for you. This letter is regarding our report:

Multichain Clustering: https://t.co/BrXFHDK4L6

TL;DR
The Updated Reduced List (67,289 => 29,957 wallets) & $1,000 Reward Public Challenge for our report can be accessed at: https://t.co/48Od19XBXa

Long Version
Our report marked 67,289 unique wallets, clustered into 1,933 groups of size 20+ across two categories: Scroll & Multi-chain. We used a conservative cutoff score for the “silhouette score” quality metric of 0.3+ for either category. This was chosen to allow yourselves to qualify clusters to your own desired level of precision & clarity. We expected your team to be comfortable removing clusters they were unhappy with, accepting ones they found conclusive.

Upon seeing Brian Pellegrino’s latest tweets ( https://t.co/EzvFtlAUDb and https://t.co/LWbyqYxwFF) we are concerned that resources are scarce, and realised our report may be too demanding for you to process. We did not anticipate that there would be very little scope for your team to filter clusters. The worst outcome for everyone is for our report(s) to be discarded entirely due to issues with cluster clarity, despite our report consisting almost entirely of sybil operators. We want to clear ambiguity in the cluster assignments. You will no longer have to work to filter clusters.

The issue: Some clusters formed from a single grouping assignment (only scroll cluster, not multi-chain cluster) and towards the lower end of the silhouette score range (0.3-0.5) were oftentimes composed of multiple similar but distinctly different sybil operations. This commonly occured when popular sybilling tools such as NFTCopilot and Lootbot were used by different users, or when sybil farms had implemented minor behavioural variations and sub-clusters to avoid detection. These mixed operations, whilst all still very clearly being “sybil farms”, may not ALL easily be concluded as being operated by one entity.

In a high pressure, tight deadline situation such as the one you find yourselves in, we would hate for you to accidentally assume these mixed clusters are made up of legitimate users. This mistake is easily made due to activity not being identical across the entire cluster, but instead being segmented; this is unusual to see compared to other bounty hunter reports. Note: From reviewing over 6,000 wallets across 125 of our clusters we identified 9 legitimately independent users that existed at the lower end of single-grouping silhouette score clusters. This was within our margin of error, all these users are comfortably removed with these stricter criteria.

A similar report (https://t.co/mroccoJxc2) has been held in high regard by Brian. A refined list of sybil wallets for this report seems to have been accepted by LayerZero. They had similar concerns about LayerZero being unable to process their report internally (https://t.co/YUT9tCqKK6).

To follow by example we are issuing an extreme list containing only the most clear-cut and uniform clusters from our original report, they are extremely easy to process. Note: Of the wallets we have omitted, the VAST (99%+) majority are operated by sybil farms, but require more time to identify & interpret accurately due to the complexities of clustering assignments. Please, only use this new list as a last resort if you are unable to allocate resources into verifying our original findings. This report aims to help you out in a pinch, as a more favourable alternative to discarding our entire report.

To produce our extreme list we identified the cut-off point where mixed clusters were appearing. We added a comfortable margin, then removed all instances of wallets below. The criteria are:
1) Wallets with only a Scroll grouping: Silhouette score req. of 0.635+
2) Wallets with only a Multichain grouping: Silhouette score req. of 0.58+
3) Wallets with both a Scroll & Multichain grouping: Mean score of 0.345+
From the 125 cluster samples we analysed, the above scores eliminated 100% of mixed sybil clusters. In short, all clusters are now easily identifiable as operated by a single entity.

This new set of stricter criteria, along with requiring 5+ LayerZero transactions and then reclustering for groups of 20+, reduced our report from 67,289 (63.5% dual clustered) to 29,957 (97.4% dual clustered) unique wallets.

All 29,957 unique wallets included in this adjusted report are now in groups that are immediately noticeable as being by the same operator.

We have updated our public challenge; any who identify a cluster containing a small number of organic users receive a $200 reward. The reward criteria has been drastically relaxed based on feedback from Twitter user @artemis_rsch, and in-line with the quality improvement of our report.

We would like to welcome all to attempt this challenge, include @SybilHorror6.

Letter: https://t.co/JowBLreKbl

[Team Opsec @Team_Opsec]

@PrimordialAA To view the most successful reports & how many sybil wallets came from each report, you can view our publicly available Google Sheet: https://t.co/cx6RpcGkbZ

[Team Opsec @Team_Opsec]

@PrimordialAA Airdrops should reward long-term & durable users. If they do not seem durable in whatever metric (such as haven't used it for over a year) then it would be logical to penalise it.

[Team Opsec @Team_Opsec]

@PrimordialAA 🫡 happy to be in the trenches alongside you, fighting the good fight

[Team Opsec @Team_Opsec]

@CryptoRacoon12 @LayerZero_Labs @PrimordialAA This was something we have a passion for doing anyway, we were going to do this for as a side-gig. It was a coincidence that LayerZero announced their community bounty hunting program, but we were poised for it.

[Team Opsec @Team_Opsec]

@LayerZero_Labs & @PrimordialAA, we've loved working our asses off for the past 6 weeks to bring our sybil hunting methodologies to life. They've been in development for over 6 months (see picture), putting them to the test has helped us verify that they work and helped us to improve them greatly.

Regardless of the results of the bounty hunting program we thank you for the opportunity and wish you all the best with the token launch, and to the future success of LayerZero. We admire the effort you have put into innovating & running every aspect of this community bounty hunting program.

That said, after careful consideration, Team Opsec will not be releasing updated reduced lists for reports 21739 and 22624.

21739: We acknowledge that, as humans, there is a possibility to make mistakes; we may have been unable to remove a small number of organic users before submission. This was not "trying to sneak in high value users" (as some have been accused of), but due to an extremely tight deadline & FCFS rule we were trying to meet. We expect(ed) LayerZero Labs to have the tools to filter out individual occurrences of organic users. We feel that an updated reduced list does not make sense here, we would be removing less than 0.5% of the wallets. We believe this is a more than reasonable false-positive rate for L0 to process.

22624: We implemented an automated methodology inspired by the manual work done for 21739. The theory behind the methodology was largely successful - the vast majority of reported wallets belong to sybil operations. The metrics we provided alongside LayerZero's suite of tools, should be ample to pass judgement for each cluster. We believe this approach was fair, and hope effort was put in to making it work. At this point in time, with no dialogue from the team, we think it would not be a good use of time to create a list that would likely never be looked at.

To keep our reports readable we could not to share 90%+ of the work that actually went in to the start to finish process: 20+ dune dashboards, 100's of programs of varying languages, 500+ excel sheets... all summarised into the concise reports you see before you. We do ultimately hope the output from our work was useful to if not you, someone.

We'll be putting the sybil detection folder to sleep for now, and look forward to future sybil hunting programs.

[Team Opsec @Team_Opsec]

@NavitasAstaris @SybilHorror6 @PrimordialAA @LayerZero_Labs This address has no activity, please create one on-chain action so we know it's not a dead address.

[Team Opsec @Team_Opsec]

GM @PrimordialAA & @LayerZero_Labs,

We at Team Opsec authored 3 great reports for you. This letter is regarding our report:

Multichain Clustering: https://t.co/BrXFHDK4L6

TL;DR
The Updated Reduced List (67,289 => 29,957 wallets) & $1,000 Reward Public Challenge for our report can be accessed at: https://t.co/48Od19XBXa

Long Version
Our report marked 67,289 unique wallets, clustered into 1,933 groups of size 20+ across two categories: Scroll & Multi-chain. We used a conservative cutoff score for the “silhouette score” quality metric of 0.3+ for either category. This was chosen to allow yourselves to qualify clusters to your own desired level of precision & clarity. We expected your team to be comfortable removing clusters they were unhappy with, accepting ones they found conclusive.

Upon seeing Brian Pellegrino’s latest tweets ( https://t.co/EzvFtlAUDb and https://t.co/LWbyqYxwFF) we are concerned that resources are scarce, and realised our report may be too demanding for you to process. We did not anticipate that there would be very little scope for your team to filter clusters. The worst outcome for everyone is for our report(s) to be discarded entirely due to issues with cluster clarity, despite our report consisting almost entirely of sybil operators. We want to clear ambiguity in the cluster assignments. You will no longer have to work to filter clusters.

The issue: Some clusters formed from a single grouping assignment (only scroll cluster, not multi-chain cluster) and towards the lower end of the silhouette score range (0.3-0.5) were oftentimes composed of multiple similar but distinctly different sybil operations. This commonly occured when popular sybilling tools such as NFTCopilot and Lootbot were used by different users, or when sybil farms had implemented minor behavioural variations and sub-clusters to avoid detection. These mixed operations, whilst all still very clearly being “sybil farms”, may not ALL easily be concluded as being operated by one entity.

In a high pressure, tight deadline situation such as the one you find yourselves in, we would hate for you to accidentally assume these mixed clusters are made up of legitimate users. This mistake is easily made due to activity not being identical across the entire cluster, but instead being segmented; this is unusual to see compared to other bounty hunter reports. Note: From reviewing over 6,000 wallets across 125 of our clusters we identified 9 legitimately independent users that existed at the lower end of single-grouping silhouette score clusters. This was within our margin of error, all these users are comfortably removed with these stricter criteria.

A similar report (https://t.co/mroccoJxc2) has been held in high regard by Brian. A refined list of sybil wallets for this report seems to have been accepted by LayerZero. They had similar concerns about LayerZero being unable to process their report internally (https://t.co/YUT9tCqKK6).

To follow by example we are issuing an extreme list containing only the most clear-cut and uniform clusters from our original report, they are extremely easy to process. Note: Of the wallets we have omitted, the VAST (99%+) majority are operated by sybil farms, but require more time to identify & interpret accurately due to the complexities of clustering assignments. Please, only use this new list as a last resort if you are unable to allocate resources into verifying our original findings. This report aims to help you out in a pinch, as a more favourable alternative to discarding our entire report.

To produce our extreme list we identified the cut-off point where mixed clusters were appearing. We added a comfortable margin, then removed all instances of wallets below. The criteria are:
1) Wallets with only a Scroll grouping: Silhouette score req. of 0.635+
2) Wallets with only a Multichain grouping: Silhouette score req. of 0.58+
3) Wallets with both a Scroll & Multichain grouping: Mean score of 0.345+
From the 125 cluster samples we analysed, the above scores eliminated 100% of mixed sybil clusters. In short, all clusters are now easily identifiable as operated by a single entity.

This new set of stricter criteria, along with requiring 5+ LayerZero transactions and then reclustering for groups of 20+, reduced our report from 67,289 (63.5% dual clustered) to 29,957 (97.4% dual clustered) unique wallets.

All 29,957 unique wallets included in this adjusted report are now in groups that are immediately noticeable as being by the same operator.

We have updated our public challenge; any who identify a cluster containing a small number of organic users receive a $200 reward. The reward criteria has been drastically relaxed based on feedback from Twitter user @artemis_rsch, and in-line with the quality improvement of our report.

We would like to welcome all to attempt this challenge, include @SybilHorror6.

Letter: https://t.co/JowBLreKbl

[Bryan Pellegrino (臭企鹅) @PrimordialAA]

@SybilHorror6 Beautiful, would love to read it. You can check their updated reduced list here as well

https://t.co/693pVfps2w

[Bryan Pellegrino (臭企鹅) @PrimordialAA]

Ok so I see a lot more anticipation for the upcoming Sybil list or "The List" as people have started to reference it as. Let me explain what this list is and what it is not -- and what we hope will come from it long term.

We started with an enormous amount of data, way more than we ever expected.

469 bonders putting up 0.5 ETH total ($1.6m bonded value)

3,800 threads created by the bonders

22,000 Clusters within the threads to be reviewed

In order to make this manageable there are a few filters we set in place to be able to quickly evaluate quality of reports and whether they were worth reviewing in depth.

First -- discard any clusters with < 20 addresses

Second -- based on our placeholder eligibility, if <30% of the wallets in a given cluster were eligible we would put it to the back of the queue for review if there was enough time

Third -- if any report flagged any of the top ~250 addresses it was quickly checked (and usually) discarded. This prevented people from cheekily sneaking in high value addresses into their clusters.

From there it's been a pure slog, as everyone has seen from my own feed it's just been a battle. Hundreds of different methodologies, thousands of clusters, often time very hard to pin down unknown contracts or qualify statistical methods -- but it's a battle that had to be fought and I'm super happy about the end result here (will write more on this later).

So the end result

This report is going to be a large scale (20+) and industrial sybil list and how it relates to @LayerZero_Labs. We think the end result will have caught a massive amount of sybil, returned a massive amount back to real users, and be an amazing piece of research groundwork laid for the community to pick up the torch.

What it WON'T be:

By the nature of time and us not evaluating accounts that were unlikely to be eligible this means there is a ton of sybil that is NOT included in our list. Amazing reports that caught or may have caught sybil clusters but since there would be no reward for them since they had no eligibility they were discarded.

This also means some low volume sybil accounts that are not included in the list who won't qualify for the core LZ distribution but will end up qualifying for some small amounts of RFPs. The issue with evaluating these is some of them have 1, 2, or 5 tx on LayerZero and when you're getting to that point the risk of false positives is much MUCH higher and we chose to err on the side protecting real users as much as possible and minimizing false positive risk. It's a tradeoff to make but we think the most practical one given the end goal.

What we hope happens from here --

The @commondotxyz forum now has a permanent log of all 3,800 threads and 22,000 clusters. This is an INCREDIBLE resource and probably one of the greatest all time wide endeavours in combatting sybil. Our focus was on LZ related sybil but tons of the remaining may have high degrees of connection external to LZ with barely any LZ usage. We very much hope the effort continues after our portion is done and the torch is picked up and a final list is put together from the community. We would even be happy to issue a grant for a group to continue this work as it seems like an incredible resource to be wasted.

I have absolutely blown away by the amazing research conducted by so many incredible analysts and sleuths. I have endless appreciation for all of you and hope everyone knows we have absolutely killed ourselves in pursuit of paying proper respect to the work you have done and getting through as many reports as possible (team was all here until 4am reviewing reports last night and back in first thing in the morning to continue reviewing).

My thanks to all of you, we will have the final results available for public review this weekend and look forward to feedback and anything missed

🫡

[Team Opsec @Team_Opsec]

@PrimordialAA Amazing! When will sybil hunters find out which of their reports were accepted/partially accepted/declined?

[Team Opsec @Team_Opsec]

@PrimordialAA gm

Let us know if you want more data to look through ( ͡° ͜ʖ ͡°)

[Team Opsec @Team_Opsec]

@SybilHorror6 @PrimordialAA @LayerZero_Labs I agree. We have changed to this format, it's now much easier to see that clusters frequently have matching groups across both multi &amp; scroll (unsurprisingly).

Hopefully this gets reviewed @PrimordialAA

[Team Opsec @Team_Opsec]

Hi Navitas, our initial bounty was to find >50% organic wallets in a cluster (see pic).

You pointed out cluster 1,677 with 24 wallets, which contained: 2 organic wallets, 3x cluster, 7x cluster, 12x cluster. This is very far from the reward criteria.

That said, please provide a reward address & we will issue you a reward anyway for helping us identify one of the core problems of our report.

[Team Opsec @Team_Opsec]

@SybilHorror6 @PrimordialAA @LayerZero_Labs Hi @SybilHorror6,

There are two cluster assignments for 97.4% of wallets. It is likely they they will both be identical (I.E. 50 wallets with multi cluster 50, the same wallets with Scroll Cluster 258).

I would recommend prioritising multi_cluster, if only using one.

[Team Opsec @Team_Opsec]

@PrimordialAA Hi Brian, we have issued an updated reduced list (67,289 =&gt; 29,957) for report 18167.

It's painful to sacrifice so much of our findings, but we understand your need for easily verifiable clusters.

Please take a moment to review.

https://t.co/9CpUMnIu1d

[Team Opsec @Team_Opsec]

GM @PrimordialAA & @LayerZero_Labs,

We at Team Opsec authored 3 great reports for you. This letter is regarding our report:

Multichain Clustering: https://t.co/BrXFHDK4L6

TL;DR
The Updated Reduced List (67,289 => 29,957 wallets) & $1,000 Reward Public Challenge for our report can be accessed at: https://t.co/48Od19XBXa

Long Version
Our report marked 67,289 unique wallets, clustered into 1,933 groups of size 20+ across two categories: Scroll & Multi-chain. We used a conservative cutoff score for the “silhouette score” quality metric of 0.3+ for either category. This was chosen to allow yourselves to qualify clusters to your own desired level of precision & clarity. We expected your team to be comfortable removing clusters they were unhappy with, accepting ones they found conclusive.

Upon seeing Brian Pellegrino’s latest tweets ( https://t.co/EzvFtlAUDb and https://t.co/LWbyqYxwFF) we are concerned that resources are scarce, and realised our report may be too demanding for you to process. We did not anticipate that there would be very little scope for your team to filter clusters. The worst outcome for everyone is for our report(s) to be discarded entirely due to issues with cluster clarity, despite our report consisting almost entirely of sybil operators. We want to clear ambiguity in the cluster assignments. You will no longer have to work to filter clusters.

The issue: Some clusters formed from a single grouping assignment (only scroll cluster, not multi-chain cluster) and towards the lower end of the silhouette score range (0.3-0.5) were oftentimes composed of multiple similar but distinctly different sybil operations. This commonly occured when popular sybilling tools such as NFTCopilot and Lootbot were used by different users, or when sybil farms had implemented minor behavioural variations and sub-clusters to avoid detection. These mixed operations, whilst all still very clearly being “sybil farms”, may not ALL easily be concluded as being operated by one entity.

In a high pressure, tight deadline situation such as the one you find yourselves in, we would hate for you to accidentally assume these mixed clusters are made up of legitimate users. This mistake is easily made due to activity not being identical across the entire cluster, but instead being segmented; this is unusual to see compared to other bounty hunter reports. Note: From reviewing over 6,000 wallets across 125 of our clusters we identified 9 legitimately independent users that existed at the lower end of single-grouping silhouette score clusters. This was within our margin of error, all these users are comfortably removed with these stricter criteria.

A similar report (https://t.co/mroccoJxc2) has been held in high regard by Brian. A refined list of sybil wallets for this report seems to have been accepted by LayerZero. They had similar concerns about LayerZero being unable to process their report internally (https://t.co/YUT9tCqKK6).

To follow by example we are issuing an extreme list containing only the most clear-cut and uniform clusters from our original report, they are extremely easy to process. Note: Of the wallets we have omitted, the VAST (99%+) majority are operated by sybil farms, but require more time to identify & interpret accurately due to the complexities of clustering assignments. Please, only use this new list as a last resort if you are unable to allocate resources into verifying our original findings. This report aims to help you out in a pinch, as a more favourable alternative to discarding our entire report.

To produce our extreme list we identified the cut-off point where mixed clusters were appearing. We added a comfortable margin, then removed all instances of wallets below. The criteria are:
1) Wallets with only a Scroll grouping: Silhouette score req. of 0.635+
2) Wallets with only a Multichain grouping: Silhouette score req. of 0.58+
3) Wallets with both a Scroll & Multichain grouping: Mean score of 0.345+
From the 125 cluster samples we analysed, the above scores eliminated 100% of mixed sybil clusters. In short, all clusters are now easily identifiable as operated by a single entity.

This new set of stricter criteria, along with requiring 5+ LayerZero transactions and then reclustering for groups of 20+, reduced our report from 67,289 (63.5% dual clustered) to 29,957 (97.4% dual clustered) unique wallets.

All 29,957 unique wallets included in this adjusted report are now in groups that are immediately noticeable as being by the same operator.

We have updated our public challenge; any who identify a cluster containing a small number of organic users receive a $200 reward. The reward criteria has been drastically relaxed based on feedback from Twitter user @artemis_rsch, and in-line with the quality improvement of our report.

We would like to welcome all to attempt this challenge, include @SybilHorror6.

Letter: https://t.co/JowBLreKbl

[Team Opsec @Team_Opsec]

To verify our adjusted clusters we assessed the lowest quality clusters in our data. These were found by sorting the "multi_silhouette_score" column in ascending order, and picking out the 10 most commonly occurring groups from the bottom 5,000 wallets.

These are our findings:
1) Multichain 903 - 0/327 - one cluster
2) Multichain 182 - 0/246 - one cluster
3) Multichain 130 - 0/377 - one cluster (possibly same as 182)
4) Multichain 339 - 0/189 - one cluster
5) Multichain 522 - 10/620 wallets are in two different clusters (2x5). The two groups of 5 are extremely correlated (0.858+ and 0.933+). Pictures below.
6) Multichain 179 - 0/59 - one cluster
7) Multichain 110 - 0/297 - one cluster
8) Multichain 369 - 0/51 - one cluster
9) Multichain 488 - 0/79 - one cluster
10) Multichain 89 - 0/87 - one cluster

The multichain 522 “false positives” are sybil, but from different operations.

2322/2332 wallets are clearly identifiable as being in the correct, single operator cluster (99.6% rate of sybil operation crossover). This really is a remarkable rate for a report of this size, especially as this metric is from the absolute bottom of our report. We did not cherry pick the groups, you can verify our findings and analyse a random set of groups yourself.

We hope you will engage in dialogue to help us get our reports over the finishing line, we've put an immense amount of work into these, and these sybil operators do not deserve to receive an airdrop.

[Team Opsec @Team_Opsec]

@PrimordialAA @LayerZero_Labs We welcome everybody to have a go at reviewing the raw data yourselves. There are 130 columns and 29,958 rows - it's a lot. It's also absolutely conclusive.

You can access the raw data (23,358 KB) here: https://t.co/SOvpwYw0Pj