feat: add FileDownloadCrawler#2043
Conversation
Mantisus
force-pushed
the
file-downloaw-crawler
branch
from
4669391 to
db4ded0
Compare
Pijukatel
left a comment
Pijukatel
left a comment
There was a problem hiding this comment.
For large files, it would be great to have something like https://pypi.org/project/smart-open/ , but our storages lacks interface to support such chunked uploads
|
|
||
| ## FileDownloadCrawler | ||
|
|
||
| The <ApiLink to="class/FileDownloadCrawler">`FileDownloadCrawler`</ApiLink> downloads files instead of scraping pages. It accepts any content type without parsing and gives the request handler direct access to the response body. By default the whole file is buffered in memory. For large files, construct the crawler with `stream=True` and consume the body in chunks via <ApiLink to="class/HttpResponse#read_stream">`read_stream()`</ApiLink>. For usage, see the [Download files](../examples/file-download) example. |
There was a problem hiding this comment.
For large files, I was wondering if we could benefit from apify/apify-client-python#927
Can you explore a little bit how to utilize this from the Crawlee level?
There was a problem hiding this comment.
For large files, I was wondering if we could benefit from apify/apify-client-python#927
The compression added in the PR (the Content-Encoding header) is a transport-level optimization that only affects how data moves over the network between the client and the server. It's fully transparent to the user, who never sees the compressed bytes directly. Similarly, when the server sends compressed data based on the client's Accept-Encoding header, decompression occurs automatically at the HTTP client level (impit, curl_cffi, httpx).
So we already get all the benefits from this.
Can you explore a little bit how to utilize this from the Crawlee level?
If we wanted to add compression at the Crawlee level, it would need to be a storage-level feature instead. It would compress data before it's written to the storage backend (files, Redis, SQL, etc.). Since that changes the stored format, it should be explicit and opt-in rather than automatic. For example, as a literal argument to KeyValueStore.set_value.
3 participants
Description
FileDownloadCrawlerwith support for streaming the response body in the request handler.Issues
FileDownloadcrawler #875Testing
FileDownloadCrawler