The Big Data Company

Profile set up by Edverise AI

Is this your business? Become an Edverise partner & Claim Now →

View Website

Set Up By Edverise AI

Edverise AI built this profile by analyzing thousands of data points across the web. If you own this business, become an Edverise partner to manage and update your profile.

Become an Edverise Partner

Overview

$5,000+

Min Project Size

$25 - $49/hr

Avg Hourly Rate

2 - 9

Employees

2020

Year Founded

ABOUT THE BIG DATA COMPANY

Outsourcing & Data Engineering Consultancy

We create and provide personnel for data engineering teams for technology firms in the US. With over 12 years of experience in delivering ETL pipelines, data platforms, and AI solutions using technologies like Databricks, Airflow, BigQuery, and Spark, we have worked with various industries, including fintech, gaming, and social media on a large scale. We have successfully delivered over 95 pipelines and served more than 17 enterprise clients.

LANGUAGES SPOKEN

EnglishPortuguese

HIGHLIGHTS

$5,000+

Min Project Size

$25 - $49/hr

Avg Hourly Rate

2 - 9

Employees

2020

Year Founded

Services

Focus

Industries

Clients

Services Provided

15% Workflow Automation (Zapier / Make / n8n)

35% BI & Big Data Consulting & SI

25% IT Staff Augmentation

10% AI Strategy & Roadmap Consulting

5% AI Model Fine-Tuning & Training

10% Machine Learning Model Development

About the Team

OUR STORY

Our Narrative of TBDC TBDC is a consulting firm specializing in data engineering and AI, with locations in the US and Brazil, catering to clients throughout the US, Latin America, and Europe. We create and implement the data frameworks that drive business decisions — encompassing ETL pipelines, data warehouses, real-time analytics, and AI-compatible infrastructures. Our Services Data Pipeline Development — Comprehensive ETL/ELT pipelines utilizing Airflow, dbt, Spark, and contemporary orchestration tools. Data Architecture & Infrastructure — Scalable, cloud-native architecture.

TEAM PHOTOS

Portfolio

Data Engineering Infrastructure

Old School Value

Developed the complete data engineering framework for Old School Value, a US-based fintech company that offers stock analysis tools for value investors. Created and executed a modern ELT architecture from the ground up, automating previously manual data processes with a robust, production-ready pipeline. The technology stack utilized Apache Airflow for orchestration, incorporating custom operators written in Go to ensure high-speed data extraction and processing of financial data from various market APIs, achieving low latency and high reliability. The infrastructure was entirely provisioned through code using Terraform on AWS, with CI/CD pipelines implemented on GitHub Actions to facilitate automated testing and deployment. The ELT methodology allowed raw financial data to be initially stored in the warehouse, with subsequent transformations applied downstream, simplifying the pipeline and enabling analysts to rapidly refine business logic. Key outcomes included a 99% decrease in data read latency, fully reproducible infrastructure, and deployments without downtime. The system handled millions of financial records each day, supporting real-time stock analysis dashboards for thousands of active investors. Technology stack: Airflow | Go (custom operators) | Terraform | AWS | GitHub Actions | ELT architecture.

Data Engineering maintenance

Terminus

Spearheaded the overhaul of Terminus Software's data processing systems, transitioning from single-node Python scripts to distributed PySpark jobs utilizing Google Cloud Dataproc clusters. The original Python pipelines were unable to handle the increasing data volumes, leading to delays in the B2B marketing analytics platform. I restructured the processing layer to take advantage of Spark's distributed computing features on Dataproc, allowing for horizontal scaling and parallel processing across the dataset. This migration involved reimagining data transformations for distributed environments while ensuring compatibility with existing downstream consumers. Key outcomes included significantly enhanced processing speed and scalability, resolving the data pipeline bottlenecks that were affecting the delivery of marketing analytics for enterprise clients. Technology stack: PySpark | Google Cloud Dataproc | Python | GCP.

Big Data Infrastructure & Tax Analytics

Brazilian Department of Taxes

Oversaw and enhanced the big data infrastructure for the State Income Tax Department of Santa Catarina, managing extensive fiscal datasets utilized for tax audits and fraud detection within one of Brazil's largest economies. This initiative involved several components: ensuring a high-availability Oracle BDA Cluster and Cloudera Manager environment, designing data ingestion pipelines using Python and Apache Airflow for tax auditors, and creating distributed processing tasks in PySpark to manage statewide fiscal records. Additionally, played a role in the hybrid cloud migration by overseeing Azure Data Factory pipelines and Databricks Spark jobs that processed millions of tax records. Key achievements included a significant reduction in system maintenance downtime, faster data accessibility for tax auditors, and a 30% boost in analytical capacity — facilitating more sophisticated tax fraud detection and large-scale fiscal auditing. Technology stack: Cloudera | Oracle BDA | PySpark | Airflow | Python | Azure Data Factory | Databricks.

Projects & Clients

Project Size & Rate

Minimum Project Size

$5,000+

Average Hourly Rate

$25 - $49/hr

Pricing Snapshot

Min Project Size

$5,000+

Avg Hourly Rate

$25 - $49/hr

Cost Rating

N/A

What Clients Have Said

“

"The Big Data Company delivers quality projects within budget and timeline expectations based on verified client feedback."

Based on verified reviews and internal benchmark data

MOST COMMON PROJECT SIZE

0 Reviews

this range

No distribution data available.

Select a service

Workflow Automation (Zapier / Make / n8n)

BI & Big Data Consulting & SI

IT Staff Augmentation

AI Strategy & Roadmap Consulting

AI Model Fine-Tuning & Training

Machine Learning Model Development

Industry Recognitions

AWS Certified

View source

Locations

Headquarters

Travessa Teodoro Koch

Centro, Santa Catarina, Brazil 89280-181

+5547996075445

Headquarters

8 The Green # 6590 Dover United States 19901

Dover, Delaware, United States 19901

+5547996075445

Reviews For The Big Data Company

Submit A Review

No Reviews Yet

Be the first to share your experience working with The Big Data Company.

Write a Review