Alternative Big Data libraries for Python
Updated :
Dec. 7, 2023
vaex
Github stargazers
8073
Github forks
592
Commits
3636
Code contributors Contributors
64
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
Created
Sept. 27, 2014
Updated
July 21, 2023
License
mit
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
516
Homepage
BigDL
Github stargazers
4578
Github forks
1
Commits
20858
Code contributors Contributors
186
Accelerate LLM with low-bit (FP4 / INT4 / FP8 / INT8) optimizations using bigdl-llm
Created
Aug. 29, 2016
Updated
Dec. 7, 2023
License
apache-2.0
Github repo
Primary Language, based on Github DataLanguage
Jupyter
Issues
896
root
Github stargazers
2297
Github forks
1
Commits
79865
Code contributors Contributors
373
The official repository for ROOT: analyzing, storing and visualizing big data, scientifically
Created
June 27, 2013
Updated
Dec. 7, 2023
License
other
Github repo
Primary Language, based on Github DataLanguage
C++
Issues
884
Homepage
data-structures-algorithms-python
Github stargazers
1014
Github forks
1394
Commits
86
Code contributors Contributors
6
This tutorial playlist covers data structures and algorithms in python. Every tutorial has theory behind data structure or an algorithm, BIG O Complexity analysis and exercises that you can practice on.
Created
Sept. 29, 2020
Updated
Nov. 14, 2022
Github repo
Primary Language, based on Github DataLanguage
Jupyter
Issues
55
tuplex
Github stargazers
809
Github forks
45
Commits
68
Code contributors Contributors
7
Tuplex is a parallel big data processing framework that runs data science pipelines written in Python at the speed of compiled code. Tuplex has similar Python APIs to Apache Spark or Dask, but rather than invoking the Python interpreter, Tuplex generates optimized LLVM bytecode for the given pipeline and input data set.
Created
June 30, 2021
Updated
Oct. 28, 2023
License
apache-2.0
Github repo
Primary Language, based on Github DataLanguage
C++
Issues
31
oio-sds
Github stargazers
624
Github forks
97
Commits
7241
Code contributors Contributors
97
High Performance Software-Defined Object Storage for Big Data and AI, that supports Amazon S3 and Openstack Swift
Created
March 13, 2015
Updated
Dec. 5, 2023
License
other
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
28
Homepage
eland
Github stargazers
574
Github forks
93
Commits
511
Code contributors Contributors
30
Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch
Created
June 11, 2019
Updated
Nov. 30, 2023
License
apache-2.0
Github repo
Type
Module/library
Primary Language, based on Github DataLanguage
Python
Issues
86
python-bigquery-pandas
Github stargazers
399
Github forks
155
Commits
376
Code contributors Contributors
48
Google BigQuery connector for pandas
Created
Feb. 8, 2017
Updated
Dec. 7, 2023
License
bsd-3-clause
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
44
hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference
Github stargazers
376
Github forks
139
Commits
236
Code contributors Contributors
5
Real Time Big Data / IoT Machine Learning (Model Training and Inference) with HiveMQ (MQTT), TensorFlow IO and Apache Kafka - no additional data store like S3, HDFS or Spark required
Created
May 8, 2019
Updated
Nov. 5, 2020
License
apache-2.0
Github repo
Type
Module/library
Primary Language, based on Github DataLanguage
Jupyter
Issues
8
transbigdata
Github stargazers
374
Github forks
102
Commits
670
Code contributors Contributors
8
A Python package develop for transportation spatio-temporal big data processing, analysis and visualization.
Created
Oct. 21, 2021
Updated
Oct. 28, 2023
License
bsd-3-clause
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
8
arvados
Github stargazers
355
Github forks
111
Commits
30285
Code contributors Contributors
81
An open source platform for managing and analyzing biomedical big data
Created
April 11, 2013
Updated
Dec. 7, 2023
License
other
Github repo
Primary Language, based on Github DataLanguage
Go
Issues
13
Homepage
Herbie
Github stargazers
305
Github forks
57
Commits
942
Code contributors Contributors
12
Download numerical weather prediction datasets (HRRR, RAP, GFS, NAM, NBM, etc.) from NOMADS, NODD partners (Amazon, Google, Microsoft), ECMWF open data, and the University of Utah Pando Archive System.
Created
June 26, 2020
Updated
Nov. 7, 2023
License
mit
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
52
lithops
Github stargazers
287
Github forks
90
Commits
3942
Code contributors Contributors
38
A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
Created
April 23, 2018
Updated
Dec. 6, 2023
License
apache-2.0
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
6
PythonDataScienceFullThrottle
Github stargazers
241
Github forks
236
Commits
180
Code contributors Contributors
2
Downloads for my Safari Online Learning live training course Python Data Science Full Throttle: Introductory Artificial Intelligence (AI), Big Data and Cloud Case Studies
Created
July 18, 2019
Updated
Nov. 14, 2023
Github repo
Type
Resource
Primary Language, based on Github DataLanguage
Jupyter
Issues
4
gimel
Github stargazers
239
Github forks
96
Commits
167
Code contributors Contributors
23
Big Data Processing Framework - Unified Data API or SQL on Any Storage
Created
April 4, 2018
Updated
Nov. 8, 2022
License
apache-2.0
Github repo
Primary Language, based on Github DataLanguage
Scala
Issues
11
Homepage
bigdata-playground
Github stargazers
200
Github forks
74
Commits
465
Code contributors Contributors
4
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
Created
Dec. 12, 2017
Updated
Feb. 1, 2019
License
apache-2.0
Github repo
Type
App
Primary Language, based on Github DataLanguage
TypeScript
Issues
7
WallStreetBets_BigDataAnalysis
Github stargazers
159
Github forks
40
Commits
68
Code contributors Contributors
2
Research project aimed to classify the best stock research posts from r/WallStreetBets for you. 😏
Created
March 15, 2021
Updated
May 16, 2021
Github repo
Primary Language, based on Github DataLanguage
Jupyter
Issues
3
ai-flow
Github stargazers
158
Github forks
36
Commits
869
Code contributors Contributors
12
AI Flow is an open source framework that bridges big data and artificial intelligence.
Created
Oct. 14, 2021
Updated
Oct. 9, 2022
License
apache-2.0
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
12
.config
Github stargazers
144
Github forks
80
Commits
1
Code contributors Contributors
1
# # Automatically generated file; DO NOT EDIT. # OpenWrt Configuration # CONFIG_MODULES=y CONFIG_HAVE_DOT_CONFIG=y # CONFIG_TARGET_sunxi is not set # CONFIG_TARGET_apm821xx is not set # CONFIG_TARGET_ath25 is not set CONFIG_TARGET_ar71xx=y # CONFIG_TARGET_ath79 is not set # CONFIG_TARGET_bcm27xx is not set # CONFIG_TARGET_bcm53xx is not set # CONFIG_TARGET_b
Created
June 23, 2020
Updated
June 23, 2020
License
mit
Github repo
Primary Language, based on Github DataLanguage
Shell
Issues
33
kubernetes-bigquery-python
Read-only repository, archived by owner Archived
Github stargazers
130
Github forks
88
Commits
41
Code contributors Contributors
7
Example Kubernetes app that shows how to build a 'pipeline' to stream data into BigQuery. Uses Redis or Google Cloud PubSub
Created
Dec. 17, 2014
Updated
Oct. 20, 2020
License
apache-2.0
Github repo
Type
App
Primary Language, based on Github DataLanguage
Python
Issues
5