HaToan

[Dongdong She @DongdongShe]

Ep5. Rebuttal MLFuzz

Thanks Irina’s response. We never heard back from you and @AndreasZeller since last month when we sent the last email to ask if you guys were willing to write an errata of MLFuzz to acknowledge the bugs and wrong conclusion. So I am happy to communicate with you in the public channel about this issue and clarify the misleading conclusions in your paper MLFuzz in front of the fuzzing community.

Our first email pointed out 4 bugs in MLFuzz and we showed that if you fixed the 4 bugs you can successfully reproduce our results. We also provide a fixed version of your code and preliminary results on 4 FuzzBench programs. Your first response confirmed 3 bugs but refused to acknowledge the most severe one – an error in training data collection. For any ML model, garbage in, garbage out. If you manipulate the training data distribution, you can cook any arbitrary poor results for an ML model.

Why are you reluctant to fix the training data collection error? Instead, you insist on running NEUZZ with the WRONG training data and cooking invalid results even though we already notified you of this issue. We suspect maybe that’s the only way to keep reproducing your wrong experiment results and avoid acknowledging your error in MLFuzz.

Your research conduct raised a serious issue about how to properly reproduce fuzzing performance in the Fuzzing community. Devil’s advice: blindly, deliberately or stealthily run it with WRONG settings or patch it with a few bugs and claim its performance does not hold? Only an ill-configured fuzzer is a good baseline fuzzer.

We think a fair and scientific way to reproduce/revisit a fuzzer should ensure running a fuzzer properly as the original paper did, rather than free-style wrong settings and bug injections.

The fact is you guys wrote buggy code (you confirmed in the email) and cooked invalid results and wrong conclusions published in a top-tier conference @FSEconf 2023. We wrote a rebuttal to point out 4 fatal bugs in your code and wrong conclusions.

A responsible and professional response should directly address our questions about the 4 fatal bugs and wrong conclusions. But your response discussed the inconsistent performance number issue of NEUZZ (due to a different metric choice), the benchmark, seed corpus, IID issue of MLFuzz. They are research questions about NEUZZ and MLFuzz, but they are not the topic of this post: MLFuzz rebuttal.

They can only shift the audience's attention but cannot fix the bugs and errors in MLFuzz. I promise I will address every question in your response in a separate post on X, but not in this one. Stay tuned! @is_eqv @moyix @thorstenholz @mboehme_

[quarkslab @quarkslab]

Fast and Curious: Emulating Renesas RH850 System-on-Chip using Unicorn Engine
Brought to you by @virtualabs and @Phil_BARR3TT to make your automotive vulnerability research easier

https://t.co/hKF2iIHLIg

[TrendAI Zero Day Initiative @thezdi]

@hoangnx99, @KmaHaToan, @QuangHV99 from @vcslab achieves a 2-bug successful attack at #P2OToronto. #Pwn2Own

[Công Thành Nguyễn @ExLuck99]

SOHO entry bugs get killed during registration.
Am I sad? Yes
Can I do anything? No
😂😂😂😂😂😂
https://t.co/EdXykyqKH7

[HaToan @KmaHaToan]

@BSidesCbr Who Can be description chall for me? I am excited

[HaToan @KmaHaToan]

@lazyproc @attifyme big investments

[HaToan @KmaHaToan]

@PraveenSoni65 @C3iCenter @nullcon @aseemjakhar @antriksh_s @pratik_ghumade @DSCI_Connect @sandeepkshukla @ninja_pandit_ How to I can play game ScadaCTF 1rd,2,3 edition

[HaToan @KmaHaToan]

@yash_omer @domenuk @sgniwx @BlackHatEvents .

[HaToan @KmaHaToan]

@Noiq15 @it4sec Do you give me to your chall CTF. I want to analyze it. Thank you