JarCasting
JarCasting JarCasting

BigMap

In some data processing tasks we need to use huge maps or sets that are bigger than available JVM heap space or they are loading too slow to standard Java or Scala Maps. We use TSV format (text file with tab separated columns) for persist this kind of Maps or Sets. Some columns are used as a key and rest of columns as a value. Idea of this library is simple. We can prepare these maps once (sort by key), store it to file and then use it as memory mapped file. Searching key in sorted file has log(n) complexity. If more processes uses the same memory mapped file, it exists in memory just once (on Linux). This file can be loaded lazy by OS.

License

License
GroupId

GroupId

com.avast
ArtifactId

ArtifactId

big-map_2.11
Last Version

Last Version

1.1
Release Date

Release Date
Type

Type

jar
Description

Description

BigMap
In some data processing tasks we need to use huge maps or sets that are bigger than available JVM heap space or they are loading too slow to standard Java or Scala Maps. We use TSV format (text file with tab separated columns) for persist this kind of Maps or Sets. Some columns are used as a key and rest of columns as a value. Idea of this library is simple. We can prepare these maps once (sort by key), store it to file and then use it as memory mapped file. Searching key in sorted file has log(n) complexity. If more processes uses the same memory mapped file, it exists in memory just once (on Linux). This file can be loaded lazy by OS.
Project URL

Project URL

https://github.com/avast/BigMap
Project Organization

Project Organization

AVAST Software

Download big-map_2.11

Filename Size
big-map_2.11-1.1.pom
big-map_2.11-1.1.jar 96 KB
big-map_2.11-1.1-sources.jar 12 KB
big-map_2.11-1.1-javadoc.jar 32 KB
Browse

How to add to project

<!-- https://jarcasting.com/artifacts/com.avast/big-map_2.11/ -->
<dependency>
    <groupId>com.avast</groupId>
    <artifactId>big-map_2.11</artifactId>
    <version>1.1</version>
</dependency>
// https://jarcasting.com/artifacts/com.avast/big-map_2.11/
implementation 'com.avast:big-map_2.11:1.1'
// https://jarcasting.com/artifacts/com.avast/big-map_2.11/
implementation ("com.avast:big-map_2.11:1.1")
'com.avast:big-map_2.11:jar:1.1'
<dependency org="com.avast" name="big-map_2.11" rev="1.1">
  <artifact name="big-map_2.11" type="jar" />
</dependency>
@Grapes(
@Grab(group='com.avast', module='big-map_2.11', version='1.1')
)
libraryDependencies += "com.avast" % "big-map_2.11" % "1.1"
[com.avast/big-map_2.11 "1.1"]

Dependencies

compile (5)

Group / Artifact Type Version
org.scala-lang : scala-library jar 2.11.6
com.github.scopt : scopt_2.11 jar 3.3.0
org.slf4j : slf4j-api jar 1.7.21
ch.qos.logback : logback-classic jar 1.1.7
commons-io : commons-io jar 2.5

Project Modules

There are no modules declared in this project.
com.avast

Avast

https://avast.github.io

Versions

Version
1.1