fetchfile


License

License

MIT
GroupId

GroupId

com.gaborpihaj
ArtifactId

ArtifactId

fetchfile_2.13
Last Version

Last Version

0.2.0
Release Date

Release Date

Type

Type

jar
Description

Description

fetchfile
fetchfile
Project URL

Project URL

https://github.com/voidcontext/fetch-file
Project Organization

Project Organization

com.gaborpihaj
Source Code Management

Source Code Management

https://github.com/voidcontext/fetch-file

Download fetchfile_2.13

How to add to project

<!-- https://jarcasting.com/artifacts/com.gaborpihaj/fetchfile_2.13/ -->
<dependency>
    <groupId>com.gaborpihaj</groupId>
    <artifactId>fetchfile_2.13</artifactId>
    <version>0.2.0</version>
</dependency>
// https://jarcasting.com/artifacts/com.gaborpihaj/fetchfile_2.13/
implementation 'com.gaborpihaj:fetchfile_2.13:0.2.0'
// https://jarcasting.com/artifacts/com.gaborpihaj/fetchfile_2.13/
implementation ("com.gaborpihaj:fetchfile_2.13:0.2.0")
'com.gaborpihaj:fetchfile_2.13:jar:0.2.0'
<dependency org="com.gaborpihaj" name="fetchfile_2.13" rev="0.2.0">
  <artifact name="fetchfile_2.13" type="jar" />
</dependency>
@Grapes(
@Grab(group='com.gaborpihaj', module='fetchfile_2.13', version='0.2.0')
)
libraryDependencies += "com.gaborpihaj" % "fetchfile_2.13" % "0.2.0"
[com.gaborpihaj/fetchfile_2.13 "0.2.0"]

Dependencies

compile (4)

Group / Artifact Type Version
org.scala-lang : scala-library jar 2.13.1
org.typelevel : cats-effect_2.13 jar 2.1.2
co.fs2 : fs2-core_2.13 jar 2.2.2
co.fs2 : fs2-io_2.13 jar 2.2.2

test (3)

Group / Artifact Type Version
org.scalatest : scalatest_2.13 jar 3.1.1
org.scalacheck : scalacheck_2.13 jar 1.14.1
org.scalatestplus : scalatestplus-scalacheck_2.13 jar 3.1.0.0-RC2

Project Modules

There are no modules declared in this project.

fetch-file Build Status Latest Version

Simple library to download a potentially large file over a potentially slow network, while the progress is printed to the stdout.

asciicast

Getting started

Currently only Scala 2.13.x is supported.

   libraryDependencies += "com.gaborpihaj" %% "fetchfile" % "0.3.0"

Usage

This example downloads the content of the pre-generated test file into a file in /tmp.

import cats.effect._
import cats.effect.concurrent.Ref
import cats.instances.list._
import vdx.fetchfile.Pipes._
import vdx.fetchfile._

import scala.io.Source

import java.io.{File, FileOutputStream}
import java.net.URL

object Main extends IOApp {
  def run(args: List[String]): IO[ExitCode] = {

    implicit val clock: MonotonicClock = MonotonicClock.system

    val shaSum = Source.fromFile("docker/static-files/100MB.bin.sha256").mkString.trim()
    val gzippedeShaSum = Source.fromFile("docker/static-files/100MB.bin.gz.sha256").mkString.trim()

    val outFile = new File("/tmp/100MB.bin")

    Blocker[IO].use { blocker =>
      implicit val backend: HttpClient[IO] = HttpURLConnectionClient[IO](blocker, 1024 * 16)

      val downloader = Downloader[IO](blocker, Progress.consoleProgress[IO])

      for {
        ref <- Ref.of[IO, List[Byte]](List.empty)
        _ <- downloader.fetch(
          new URL("http://localhost:8088/100MB.bin.gz"),
          Resource.fromAutoCloseable(IO.delay(new FileOutputStream(outFile))),
          pipes = List(
            collectSHA256(ref), // Collect SHA before gunzip
            fs2.compress.gunzip[IO](32 * 1024) // Unzip compressed stream
          ),
          last = ensureSHA256(shaSum) // Ensure gunzipped SHA is correct
        )
        collecetedSha <- ref.get.map(_.map("%02x".format(_)).mkString)
      } yield {
        println(s"Collected SHA: $collecetedSha, expected gzipped SHA: $gzippedeShaSum")
        ExitCode.Success
      }
    }
  }
}

See fulle example in examples.

Development

Some of the tests are relying on a HTTP webserver so that files can be downloaded through HTTP. The repository contains a predefined container for this. The test file needs to be generated before building the container.

  $ sh docker/static-files/pre_build.sh
  $ docker-compose up
  $ sbt test

Versions

Version
0.2.0
0.1.1