Alternative Big Data libraries for Python
Updated :
April 19, 2024
vaex
Github stargazers
8173
Github forks
588
Commits
3636
Code contributors Contributors
64
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
Created
Sept. 27, 2014
Updated
July 21, 2023
License
mit
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
528
Homepage
BigDL
Github stargazers
5899
Github forks
1
Commits
21025
Code contributors Contributors
97
Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, Baichuan, Mixtral, Gemma, etc.) on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max). A PyTorch LLM library that seamlessly integrates with llama.cpp, HuggingFace, LangChain, LlamaIndex, DeepSpeed, vLLM, FastChat, ModelScope, etc.
Created
Aug. 29, 2016
Updated
April 19, 2024
License
apache-2.0
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
1014
root
Github stargazers
2414
Github forks
1
Commits
80008
Code contributors Contributors
394
The official repository for ROOT: analyzing, storing and visualizing big data, scientifically
Created
June 27, 2013
Updated
April 19, 2024
License
other
Github repo
Primary Language, based on Github DataLanguage
C++
Issues
828
Homepage
data-structures-algorithms-python
Github stargazers
1111
Github forks
1423
Commits
86
Code contributors Contributors
6
This tutorial playlist covers data structures and algorithms in python. Every tutorial has theory behind data structure or an algorithm, BIG O Complexity analysis and exercises that you can practice on.
Created
Sept. 29, 2020
Updated
Nov. 14, 2022
Github repo
Primary Language, based on Github DataLanguage
Jupyter
Issues
59
tuplex
Github stargazers
813
Github forks
46
Commits
72
Code contributors Contributors
7
Tuplex is a parallel big data processing framework that runs data science pipelines written in Python at the speed of compiled code. Tuplex has similar Python APIs to Apache Spark or Dask, but rather than invoking the Python interpreter, Tuplex generates optimized LLVM bytecode for the given pipeline and input data set.
Created
June 30, 2021
Updated
Dec. 22, 2023
License
apache-2.0
Github repo
Primary Language, based on Github DataLanguage
C++
Issues
28
oio-sds
Github stargazers
643
Github forks
97
Commits
7256
Code contributors Contributors
99
High Performance Software-Defined Object Storage for Big Data and AI, that supports Amazon S3 and Openstack Swift
Created
March 13, 2015
Updated
April 19, 2024
License
other
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
29
Homepage
eland
Github stargazers
608
Github forks
95
Commits
513
Code contributors Contributors
33
Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch
Created
June 11, 2019
Updated
April 8, 2024
License
apache-2.0
Github repo
Type
Module/library
Primary Language, based on Github DataLanguage
Python
Issues
98
python-bigquery-pandas
Github stargazers
419
Github forks
118
Commits
383
Code contributors Contributors
49
Google BigQuery connector for pandas
Created
Feb. 8, 2017
Updated
April 12, 2024
License
bsd-3-clause
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
36
transbigdata
Github stargazers
416
Github forks
108
Commits
670
Code contributors Contributors
8
A Python package develop for transportation spatio-temporal big data processing, analysis and visualization.
Created
Oct. 21, 2021
Updated
Oct. 28, 2023
License
bsd-3-clause
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
10
hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference
Github stargazers
388
Github forks
138
Commits
236
Code contributors Contributors
5
Real Time Big Data / IoT Machine Learning (Model Training and Inference) with HiveMQ (MQTT), TensorFlow IO and Apache Kafka - no additional data store like S3, HDFS or Spark required
Created
May 8, 2019
Updated
Nov. 5, 2020
License
apache-2.0
Github repo
Type
Module/library
Primary Language, based on Github DataLanguage
Jupyter
Issues
8
Herbie
Github stargazers
374
Github forks
66
Commits
993
Code contributors Contributors
17
Download numerical weather prediction datasets (HRRR, RAP, GFS, IFS, etc.) from NOMADS, NODD partners (Amazon, Google, Microsoft), ECMWF open data, and the University of Utah Pando Archive System.
Created
June 26, 2020
Updated
April 17, 2024
License
mit
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
60
arvados
Github stargazers
362
Github forks
111
Commits
30473
Code contributors Contributors
81
An open source platform for managing and analyzing biomedical big data
Created
April 11, 2013
Updated
April 17, 2024
License
other
Github repo
Primary Language, based on Github DataLanguage
Go
Issues
8
Homepage
lithops
Github stargazers
305
Github forks
97
Commits
3968
Code contributors Contributors
41
A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
Created
April 23, 2018
Updated
April 18, 2024
License
apache-2.0
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
5
PythonDataScienceFullThrottle
Github stargazers
247
Github forks
210
Commits
180
Code contributors Contributors
2
Downloads for my Safari Online Learning live training course Python Data Science Full Throttle: Introductory Artificial Intelligence (AI), Big Data and Cloud Case Studies
Created
July 18, 2019
Updated
Jan. 11, 2024
Github repo
Type
Resource
Primary Language, based on Github DataLanguage
Jupyter
Issues
4
gimel
Github stargazers
243
Github forks
97
Commits
167
Code contributors Contributors
23
Big Data Processing Framework - Unified Data API or SQL on Any Storage
Created
April 4, 2018
Updated
Nov. 8, 2022
License
apache-2.0
Github repo
Primary Language, based on Github DataLanguage
Scala
Issues
11
Homepage
bigdata-playground
Github stargazers
204
Github forks
72
Commits
465
Code contributors Contributors
4
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
Created
Dec. 12, 2017
Updated
Feb. 1, 2019
License
apache-2.0
Github repo
Type
App
Primary Language, based on Github DataLanguage
TypeScript
Issues
7
.config
Github stargazers
182
Github forks
68
Commits
1
Code contributors Contributors
1
# # Automatically generated file; DO NOT EDIT. # OpenWrt Configuration # CONFIG_MODULES=y CONFIG_HAVE_DOT_CONFIG=y # CONFIG_TARGET_sunxi is not set # CONFIG_TARGET_apm821xx is not set # CONFIG_TARGET_ath25 is not set CONFIG_TARGET_ar71xx=y # CONFIG_TARGET_ath79 is not set # CONFIG_TARGET_bcm27xx is not set # CONFIG_TARGET_bcm53xx is not set # CONFIG_TARGET_b
Created
June 23, 2020
Updated
June 23, 2020
License
mit
Github repo
Primary Language, based on Github DataLanguage
Shell
Issues
33
WallStreetBets_BigDataAnalysis
Github stargazers
167
Github forks
42
Commits
68
Code contributors Contributors
2
Research project aimed to classify the best stock research posts from r/WallStreetBets for you. 😏
Created
March 15, 2021
Updated
May 16, 2021
Github repo
Primary Language, based on Github DataLanguage
Jupyter
Issues
3
ai-flow
Github stargazers
164
Github forks
36
Commits
869
Code contributors Contributors
12
AI Flow is an open source framework that bridges big data and artificial intelligence.
Created
Oct. 14, 2021
Updated
Oct. 9, 2022
License
apache-2.0
Github repo
Primary Language, based on Github DataLanguage
Python
Issues
12
kubernetes-bigquery-python
Read-only repository, archived by owner Archived
Github stargazers
130
Github forks
88
Commits
41
Code contributors Contributors
7
Example Kubernetes app that shows how to build a 'pipeline' to stream data into BigQuery. Uses Redis or Google Cloud PubSub
Created
Dec. 17, 2014
Updated
Oct. 20, 2020
License
apache-2.0
Github repo
Type
App
Primary Language, based on Github DataLanguage
Python
Issues
5