Chunking (τ, ן)

SN40 (τ, ן)

[Chunking (τ, ן) @chunking_subnet]

Patch 2.4.2 - Commit Reveal
PR: https://t.co/DfM6rogYHe

Key Changes:
- Enable CR3
- Chunks now properly end on sentence boundary, with NLTK tokenizer as ground truth
- When miners tie in a round, alpha now decreases linearly instead of exponentially

[Chunking (τ, ן) @chunking_subnet]

Release 2.4.0 - Enhanced Competition & Variety
PR: https://t.co/lF1NDWxb4f

Key Changes:
- New 'Loss Alpha' system accelerates rank advancement for consistently better miners
- Expanded chunk size variety: 2000-4000 chars with varying probabilities
- Mixed prompt types: 80% complex interwoven, 20% clear separation
- Added rank protection for winning miners

Loss Alpha Multipliers:
Group 0: 2x
Group 1: 1.25x
Group 2: 1.0625x
Group 3: 1.015625x

As we surpass industry standards, this system ensures better miners can advance faster despite increasingly gradual improvements.

[Chunking (τ, ן) @chunking_subnet]

Latest Chunking Subnet Benchmarks
The Power of Open, Incentivized Contest
Completed Dec 9, 2024

We have now surpassed the industry leaders @AI21Labs and @UnstructuredIO in intermediate contexts, and have widened our lead in low context!

🔗 Explore the interactive benchmark here: https://t.co/BHgaZLxf3G

Key findings:
💠 The subnet outperformed industry leaders at low (50 tokens) and intermediate (150) context
💠 The subnet widened its lead in low context to +25.99% on recall and +3.92% on precision
💠 The subnet is now winning on recall +8.93% and precision +7.66%
💠 The top miners did significantly better than the top miner of the last benchmark!

Considerations
🔸 This benchmark follows the same methodology as the previous benchmark, apart from the following:
🔸 New questions were generated using the process outlined by @trychroma here: https://t.co/kJfape62J7
🔸 The dataset now includes documents from openwebtext and wikitext
🔸 In the time since the last benchmark, AI21 released a new chunker

[Chunking (τ, ן) @chunking_subnet]

Release 2.2.0 - Enhanced Query Generation

PR: https://t.co/M8Ye6GQFMt

Key Changes:
- Improved system, initial generation, and continuation prompts
- Synthetic queries now weave together topics from each of three given articles much more, often even within the same sentence

Explore examples between the old and new generation: https://t.co/Tqk9VY82Dc

[Chunking (τ, ן) @chunking_subnet]

Patch 2.1.2

PR: https://t.co/3kMBbjEmMf

Key Changes:
- When creating a unique hash for all chunks returned by a miner, whitespace is now ignored.
- Extra checks to make sure nltk tokenizers are downloaded

[Chunking (τ, ן) @chunking_subnet]

Hotfix 2.1.1

PR: https://t.co/nN8Ts88KX3

Key Changes
- Improved Validator Task API concurrency.
- Improved W&B tracking to address the copy-source feature on https://t.co/7Y3kyDnla9, with additional classifications for rounds.

Validators
- Make sure to restrict access to the Task API to only allow in trusted traffic
- We recommend a cron restart of 1 hour

Miners
- Ensure your miner can handle concurrent requests, as we'll likely be sending in a significant amount of requests as part of the commercialization benchmark

[Chunking (τ, ן) @chunking_subnet]

Release 2.1.0 - Task API & https://t.co/xk22bV1GMy Network

PR: https://t.co/mVHcTq2I42

Key Changes:
- New integrated task API enabling concurrent organic & synthetic queries
- Sidecar API server for validators to handle client requests
- Removed sleep time between synthetic tournament rounds
- Optimized validator performance with async functions

For Commercialization Benchmark:
- Enable with --enable_task_api
- Whitelist https://t.co/xk22bV1GMy (IP in Discord)

Remember, the commercialization benchmark starts tomorrow and runs until November 1st.

1 TAO base reward, 50 TAO reward pool based on % queries answered.

See full implementation details and setup guide: https://t.co/wMX9fOc4co

[Chunking (τ, ן) @chunking_subnet]

Patch 1.4.1 - Ranking System Improvements

See PR: https://t.co/d7fR9zW7Yv

This patch addresses two key issues in the current ranking system that made it challenging for newer miners to advance.

Issue 1: Identical Chunks, Inconsistent Rewards
- OpenAI’s Embedding models are non-deterministic, meaning the same text can generate slightly different embeddings with each submission.
- As a result, miners submitting identical chunks could receive slightly different scores, leading to unequal rewards for the same work.

Resolution:
- Now, if multiple miners submit the same chunk, validators will evaluate the chunk only once, ensuring consistent rewards for all miners submitting identical work.

Issue 2: Two Groups, One Ranking.
- Many miners are assigned to two groups: one where they can improve their rank and another that could drag their rank down based on the performance of other miners.
- Because miner ranks are calculated using a weighted moving average, some miners may experience rank drops despite performing well in their lower group.

Example: If a miner ranks 2.7, they could be in both Group 1 (miners ranked 1-4) and Group 2 (ranked 3-8). Even if they consistently rank 1st in Group 2, their overall rank could still fall because 3 is higher than their initial rank of 2.7.

Resolution:
The starting rank value of each group will now be based on the average of two values:
- The rank at the overlapping position in the adjacent higher group.
- The last rank of the second adjacent higher group.

From this derived starting point, ranks will increment by 1 for each subsequent position, creating a more balanced ranking system.

For a more detailed explanation of the issues, solutions, and the logic behind these changes, please refer to our documentation here: https://t.co/2ZPsUXvZy0

[Chunking (τ, ן) @chunking_subnet]

Testnet Release 1.4.0

PR: https://t.co/9rYoUh00hq

Changelog (diff to mainnet):
1. Synthetic query upgrade: gpt-4o-mini generates unique 15k-60k char articles from Wikipedia
2. Weight hanging fix: Only waits for inclusion, not finalization
3. Reward function fix: Addresses tokenizer punctuation edge-cases

Upcoming:
- Lucid anti-relay system on testnet post-merge
- Mainnet merge: Wed 16:00 UTC
- Lucid mainnet: Oct 16th

New: You can now copy source text from dashboard rounds at https://t.co/3xFhFI09CT

Lucid docs: https://t.co/YxAd5J2qcZ

[Chunking (τ, ן) @chunking_subnet]

Patch 1.3.1

View the PR here: https://t.co/uIDTMSDd7h

Changelog:

1. Incentives are no longer cutoff after the first 5 ranks.
- The first seven ranks will continue to follow the function (1/2)^N
- The remaining incentive (1/128) is divided linearly (with a nearly zero, but non-zero slope) according to rank for ranks 8-200

2. The min burn parameter will rise to 100,000,000 or 0.1 TAO.

3. API framework is merged into main.

4. Adds custom verify function to the Default Miner, to handle "Nonce too old" error causing miners to refuse some validator queries.

This adds a new option
--neuron.synapse_verify_allowed_delta
so miners can choose when to deem validator synapses as stale. The default is now 10 seconds.

Of course, verification can always be disabled completely using
--neuron.disable_verification,
though this is not recommended.

[Chunking (τ, ן) @chunking_subnet]

Hotfix 1.2.3

This hotfix address an exploit which allowed miners to bypass restrictions on data integrity. The PR can be found here: https://t.co/gdaWxjsITq

Note that the dashboard will lag for a few hours as the wandb API catches up. If you need to see the latest info immediately, please use our wandb page: https://t.co/zJfBDdtpow

Validators not using auto-update, please update as soon as possible with

git checkout main
git pull
pip3 install -e .
bash https://t.co/vOeBzKPwiA

[Chunking (τ, ן) @chunking_subnet]

Patch 1.2.1

Corrects group size pattern from 4, 6, 8, 10 to 2, 4, 6, 8, 10

View PR here: https://t.co/m06mynbEeR

New Groups by Rank:
[[0, 1], [1, 2, 3, 4], [3, 4, 5, 6, 7, 8], [6, 7, 8, 9, 10, 11, 12, 13], [10, 11, 12, 13, 14, 15, 16, 17, 18, 19], [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26], [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34], [28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43], [36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53], [45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64], [55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76], [66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89], [78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103], [91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118], [105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134], [120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151], [136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169], [153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199]]

[Chunking (τ, ן) @chunking_subnet]

Release 1.2.0 Live on Mainnet

This is a breaking change for Validators with auto-update without a WandB key.

See the full details on our PR: https://t.co/8EOZ9gIyAA

Changelog:
1. Groups now having varying alphas, with lower ranked groups more sensitive to change and the higher ranked groups less sensitive to change.

2. Group sizes are no longer fixed, decreasing in size as ranks rise. Group sizes follow the pattern 4, 6, 8, 10 with overlap between each group increasing by 1 each time.

3. Fixed "all rankings" in WandB logs

4. Increased default num_embeddings to 150

5. WandB API key is now required in the start script

For Validators, please update at your earliest convenience.

Note that a Weights & Biases API Key is now required for the validator start script. Please set this variable in your .env.