Anatoliy Kuznetsov

BitMagic Library for search systems, bioinformatics, logical inference, succinct data structures, compression.
SIMD, HPC, data parallel programming.

[Anatoliy Kuznetsov @bitmagicio]

BitMagic: overview of succinct and compression algorithms for data science. Bit-vectors, Elias-Gamma, BInary Interpolative Coding, Rank-Select compression, Bit-Planes compression, XOR compression.
https://t.co/Fe8nBXh65V

[Anatoliy Kuznetsov @bitmagicio]

@srchvrs We may witness a new religion where a word of an Agent cannot be questioned.

[Anatoliy Kuznetsov @bitmagicio]

@miracache I am also unpleasantly surprised at how bad Apple keyboards are for the price. My magic keyboard came crooked (fixed myself) and now is all worn out (way worse than on the picture)...

[Anatoliy Kuznetsov @bitmagicio]

@geofflangdale I actually find normal that in debugging and troubleshooting agent starts meandering around. It surely needs additional guidance about structure, hypothesis formulation, where/how to collect relevant data(debug prints). How to isolate test case.
I start helping it. not yelling

[Anatoliy Kuznetsov @bitmagicio]

@thorstenball I doubt that Unit tests will die.
I think tests in general from unit to metamorphic randomized are getting even more important, because AI capability to write code in bulk yet missing details in the large context (sometimes even not realized by the prompt operator himself).

[Anatoliy Kuznetsov @bitmagicio]

BitMagic 9.3.2 released.
Memory buffer override issue fixed.
https://t.co/WQmDYMaPBI

[Anatoliy Kuznetsov @bitmagicio]

@srchvrs yes, cloud... Cloud with tons of moving parts and non-deterministic side-channels and neighbors can be really tough for reliability engineering (people write books about).

[Anatoliy Kuznetsov @bitmagicio]

@srchvrs Absolutely!
Randomized stress testing running for hours if not days days should augment unit testing (it was true well before AI). Of cause it will taper the pace (unless tests can be brute forced for parallel run).

[Anatoliy Kuznetsov @bitmagicio]

@lemire Yes, fancy RB-tree fragments heap like crazy and not CPU cache friendly, performance oriented crowd uses variants of associative arrays forever.

[Anatoliy Kuznetsov @bitmagicio]

@BillTheRadioGuy @details_with_ai Absolutely true, because AI also in fact works better on a higher level (close to human lang - easier for it).

[Anatoliy Kuznetsov @bitmagicio]

@details_with_ai I agree on many points, but just want to comment:
1. AI can and will forget to free memory in a complex app with lots of ownership transfer, where C++ smart ptrs are very reliable. (Yes, AI can catch that with ASAN)
2. C vs Py on the Cloud app. C++ obviously will on par with C

[Anatoliy Kuznetsov @bitmagicio]

@MansiCodez DDoc would go first.
Why? It can (very often) be generated from clean code.

[Anatoliy Kuznetsov @bitmagicio]

BitMagic 9.3.1 is out!
Keep large datasets compressed on SSD.
Selectively retrieve only the values you need
Stream serialization without full BLOBs in RAM
Persistent indexes for faster access
Compressed data is becoming directly queryable data.
https://t.co/aYExkMXlSa

[Anatoliy Kuznetsov @bitmagicio]

@venkateshdotdev Often Denormalization but for how much?
Need to stage a few benchmarking experiments to tell, otherwise it may be misleading. Pulling all dictionaries into one table can be a disaster.

[Anatoliy Kuznetsov @bitmagicio]

@ehud42 @lauriewired Yes! And programmable computers were invented not to pay electrical engineers for fixed program computers.

[Anatoliy Kuznetsov @bitmagicio]

@paulbohm I have no real opinion, but based on my experience IQ testing is a trainable skill, you can improve your scores a lot if you study it.
Since IQ testing became less popular, maybe (maybe) people just train less to show high IQ scores.

[Anatoliy Kuznetsov @bitmagicio]

@lemire It seems the goal of university education in the past was to develop a well rounded multidisciplinary educated person with focus on creativity (ingeniator - who invents, clever, see Jules Verne and his books).
Distinctly different from being just trained in coding and algos.

[Anatoliy Kuznetsov @bitmagicio]

@halvarflake BitMagic test takes at min level 1hr+ on Apple M3 (fairly fast hw), medium level validation 6hrs+ multi-core (same M3). Release ARM+x86 in all SIMD variants, compiler variants - close to 1 week (many-core). Any cut corners - getting regressions.. Thinking how to opt.process.

[Anatoliy Kuznetsov @bitmagicio]

@lemire Totally agree. Tests are easier to create but cannot skip it, since AI makes code experiments so much easier, role of comprehensive testing based on methodology and discipline is important. Each leap in coding needs to be cemented in well designed tests.

[Anatoliy Kuznetsov @bitmagicio]

@robertgraham @halvarflake @alexinexxx Same story, MSVC was the best, most convenient IDE (ok, after Turbo C) and now i use XCode the most.
But. But. Unix/Linux world had a dev.gem which i still kind of miss: DDD (Data Display Debugger).

[Anatoliy Kuznetsov @bitmagicio]

@hunterxtwt Coding is easier but should not be reckless.
Coding tests and debugging harness is also easier (by a lot) as a result debugging is also easier on condition that coding is not reckless and accounts for debugging.
Top performance optimization is still hard.