Loklak - A Distributed Crawler and Data Harvester for Overcoming Rate Limits
Modern social networks have become sources for vast quantities of data. Having access to such big data can be very useful for various researchers and data scientists. In this paper we describe Loklak, an open source distributed peer to peer crawler and scraper for supporting such research on platforms like Twitter, Weibo and other social networks. Social networks such as Twitter and Weibo pose various limitations to the user on the rate at which one could freely collect such data for research. Our crawler enables researchers to continuously collect data while overcoming the barriers of authentication and rate limits imposed to provide a repository of open data as a service.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
BUbiNG: Massive Crawling for the Masses
Although web crawlers have been around for twenty years by now, there is virtually no freely available, opensource crawling software that guarantees high throughput, overcomes the limits of single-machine systems and at …
URL ordering policies for distributed crawlers: a review
With the increase in size of web, the information is also spreading at large scale. Search Engines are the medium to access this information. Crawler is the module of search engine which is responsible for download the w…
Analysis of a Two-degree-of-freedom Beam for Rotational Piezoelectric Energy Harvesting
This study introduces a two-degree-of-freedom piezoelectric energy harvester designed to harness rotational motion as an energy source. The harvester is built using a cut-out beam, which enables the first two resonant fr…
Performance of energy harvesters with parameter mismatch
This study explores the impact of parameter mismatch on the stability of cross-well motion in energy harvesters, using a basin stability metric. Energy harvesters, essential for converting ambient energy into electricity…
Wireless Power Transmission on Martian Surface for Zero-Energy Devices
Exploration of the Red Planet is essential on the way through both human colonization and establishing a habitat on the planet. Due to the high costs of space missions, the use of distributed sensor networks has been inv…