Lightfeed

Extract and Maintain Web Data at Scale.

Effortlessly turn entire websites into clean, structured data. Build searchable, up-to-date and AI-ready data sources.

[Lightfeed @lightfeed_ai]

We've also added new features to our platform:

• Value History Tracking - instantly see what changed between extractions
• List/Detail Extraction Modes - guide the LLM for more accurate results
• Email Notifications - choose exactly which updates matter to you

If you're working with web data extraction or LLMs, we'd love to hear your thoughts. What challenges are you facing with web data extraction?

#WebData #MachineLearning #OpenSource #DataExtraction

[Lightfeed @lightfeed_ai]

🎉 We've open-sourced lightfeed-extract, our LLM-powered web data extraction library that's processed over 10M records in production!

💻 We built lightfeed-extract to solve real challenges: invalid JSON and broken links in LLM outputs. It provides clean HTML conversion, JSON sanitization, and automatic URL validation—all battle-tested in production.

Check it out: https://t.co/isX25xa31r

#OpenSource #DataExtraction #LLM #WebScraping

[Lightfeed @lightfeed_ai]

Here is the full blog post: https://t.co/kOgZsCTRPx

[Lightfeed @lightfeed_ai]

We're thrilled to launch our new Lightfeed API along with official TypeScript and Python SDKs, now available on GitHub: https://t.co/pB1691TSUu

⚡️ Instant data retrieval through simple API calls: eliminating the wait times and costs associated with traditional crawl and scrape operations - your data is pre-extracted and ready for search

💡 Intelligent extraction, no manual maintenance: our AI-powered extraction can search connected pages, reason and extract data from context. It adapts to website changes automatically, no need to manually maintain scraping scripts.

✅ Clean, consistent, and up-to-date data from any list of websites

#WebData #DataIntegration #ETL #DataPipelines

[Lightfeed @lightfeed_ai]

We're excited to announce Lightfeed Extraction, our solution for web data extraction and management at scale.

⚡️ Extraction with Just a Prompt: Getting structured data from websites using simple natural language prompts - no coding required. Just describe what you need, and let AI models handle the extraction.

🔎 Managed Database: Your extracted data is automatically organized in a dedicated database that stays fresh with regular updates. Query, filter, and access your data instantly without building infrastructure.

Together, these capabilities address the traditional challenges of web data extraction - eliminating manual scrapers, ongoing maintenance, and fragile workflows.

Learn more in our documentation: https://t.co/L1t3YPiFUr

#WebData #DataExtraction #AITools #DataEngineering #ETL

[Lightfeed @lightfeed_ai]

We also supported analyzing PDFs, more robust scrapers and more readable workflow emails.

Read more in our December change log here https://t.co/acRtzbe4jq

[Lightfeed @lightfeed_ai]

⚡️We just launched AI workflows - to automate search and analysis on websites at scale.

Run powerful AI hybrid search and use LLM agents to analyze thousands of websites every hour.

Here is a 3min walkthrough 👇

[Lightfeed @lightfeed_ai]

You can now track user or company activity on LinkedIn and Google search results.

[Lightfeed @lightfeed_ai]

Adding websites is now 10x easier and customizable, and you can directly ask AI what to extract.

[Lightfeed @lightfeed_ai]

We also released API-based pricing, onboarding and dozen of other new features.

All that and more on our first changelog here
https://t.co/Xin6qA4q4v

[Lightfeed @lightfeed_ai]

Introducing knowledge base in Lightfeed!

It is your custom database to extract website data at scale. We've been improving it for the last two months - making it able to extract any public website at the frequency you choose.

[Lightfeed @lightfeed_ai]

We also improved data visibility into dashboard by adding filters, table view and export to CSV.

[Lightfeed @lightfeed_ai]

New Docs and step-by-step tutorial📣

💡 How to robustly extract websites at scale using LLM
🔎 Create a search alert on contents you need using embeddings

We use LLM to understand semantic meaning in order to extract, index and search.

https://t.co/hhVB9tKUeL

[Lightfeed @lightfeed_ai]

🏗️No-code workflow editor is out - this makes it straightforward to configure a workflow - which is a sequence of runnable nodes:

Trigger workflow to run by new contents indexed. Use semantic search to query by LLM embeddings. Send results to email or publish to RSS.

[Lightfeed @lightfeed_ai]

We just released a new UI for dashboard!

On the left - you still have the workflow results (here I have several search workflows). What is new is on the right - you can see the latest contents from websites that got indexed into your knowledge base.

[Lightfeed @lightfeed_ai]

We just launched RSS export to https://t.co/d6PvZmwlaC

- so that workflow results can be fetched from any RSS readers
- or alternatively use it like an API and convert xml to JSON in your client

#lightfeed #knowledgeAPI #llm

[Andrew Zhong @roadatoz]

@CollisionHQ is a blast! First time with @lightfeed_ai and what I can say:

- Super friendly to early startups. Lots of events for showcasing
- Nice pool of startups and SMBs for lead gen
- Great to connect with likely-minded founders

Excited that it is coming to Vancouver 2025 🏔️