Turn the web into a database: An alternative to web crawling/scraping - Mixnode News Blog mixnode.com/blog/posts/turn-…
1
2
I’m too lazy to check – how does it work? Does it use Google’s index?
1
They just run their own crawler AFAIK
1
“Just”
1
I find the database part way more impressive, if they achieve fast response times. The indexing? Not exactly easy but manageable and more a question of how many ec2 nodes you can pay for?
1
Replying to @simkoelsch @phaus
Seems to me that in general,he index *is* the way to make a DB fast. If they’re not using Google‘s, I don’t see how they’d compete. If they do, it seems to be “just” syntax

Oct 8, 2018 · 7:39 AM UTC

1
Replying to @stilkov @simkoelsch
IMHO they don‘t want to have the whole index from the beginning, but on demand. I really like the Idea of database queries. Might also be nice for intranet applications. And for sure, there is currently more than one index, besides of google. E. g. of the @internetarchive
1