hey , does anyone knows resources for big data course or anyone that explains the course in detail? (especially Cambridge slides) i’m lost

I thought this would be interesting to the audience here.

Uber is well known for its scale in the industry.

Here are the latest numbers I compiled from a plethora of official sources:

• Apache Kafka:
  • 138 million messages a second
  • 89GB/s (7.7 Petabytes a day)
  • 38 clusters
• Apache Pinot:
  • 170k+ peak queries per second
  • 1m+ events a second
  • 800+ nodes
• Apache Flink:
  • 4000 jobs processing 75 GB/s
• Presto:
  • 500k+ queries a day
  • reading 90PB a day
  • 12k nodes over 20 clusters
• Apache Spark:
  • 400k+ apps ran every day
  • 10k+ nodes that use >95% of analytics’ compute resources in Uber
  • processing hundreds of petabytes a day
• HDFS:
  • Exabytes of data
  • 150k peak requests per second
  • tens of clusters, 11k+ nodes
• Apache Hive:
  • 2 million queries a day
  • 500k+ tables

They leverage a Lambda Architecture that separates it into two stacks - a real time infrastructure and batch infrastructure.

Presto is then used to bridge the gap between both, allowing users to write SQL to query and join data across all stores, as well as even create and deploy jobs to production!

A lot of thought has been put behind this data infrastructure, particularly driven by their complex requirements which grow in opposite directions:

1. Scaling Data - total incoming data volume is growing at an exponential rateReplication factor & several geo regions copy data.Can’t afford to regress on data freshness, e2e latency & availability while growing.
2. Scaling Use Cases - new use cases arise from various verticals & groups, each with competing requirements.
3. Scaling Users - the diverse users fall on a big spectrum of technical skills. (some none, some a lot)

I have covered more about Uber's infra, including use cases for each technology, in my 2-minute-read newsletter where I concisely write interesting Big Data content.

I don’t think so. Instead, it’s here to free data scientist and ML engineers 𝗳𝗿𝗼𝗺 𝘁𝗲𝗱𝗶𝗼𝘂𝘀, 𝗿𝗲𝗽𝗲𝘁𝗶𝘁𝗶𝘃𝗲 𝘁𝗮𝘀𝗸𝘀—so you can focus on higher-value work like 𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗯𝗲𝘁𝘁𝗲𝗿 𝗺𝗼𝗱𝗲𝗹𝘀, 𝘂𝗻𝗰𝗼𝘃𝗲𝗿𝗶𝗻𝗴 𝗶𝗻𝘀𝗶𝗴𝗵𝘁𝘀 𝗳𝗿𝗼𝗺 𝘂𝗻𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱 𝗱𝗮𝘁𝗮 𝗳𝗮𝘀𝘁𝗲𝗿, 𝗮𝗻𝗱 𝗱𝗿𝗶𝘃𝗶𝗻𝗴 𝗺𝗼𝗿𝗲 𝗶𝗺𝗽𝗮𝗰𝘁 𝗳𝗼𝗿 𝘆𝗼𝘂𝗿 𝗼𝗿𝗴 𝗮𝗻𝗱 𝗰𝘂𝘀𝘁𝗼𝗺𝗲𝗿𝘀.

Check out this Medium article on how Google, Teradata, and Gemini are transforming enterprise data workflows and insights with Generative AI:

🔗https://medium.com/google-cloud/how-generative-ai-transforms-enterprise-data-insights-with-google-gemini-and-teradata-382b7e274af8

Would love to hear your thoughts—𝗵𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝘀𝗲𝗲 𝗚𝗲𝗻𝗔𝗜 𝘀𝗵𝗮𝗽𝗶𝗻𝗴 𝘁𝗵𝗲 𝗳𝘂𝘁𝘂𝗿𝗲 𝗼𝗳 𝗱𝗮𝘁𝗮 𝘀𝗰𝗶𝗲𝗻𝗰𝗲 𝗮𝗻𝗱 𝗠𝗟? 👇

The data science domain is huge and if you want to make a career in data science, then you need to be aware of the various components that make up this widely used technology including data, programming languages, machine learning, and more.

https://preview.redd.it/gfspzpv83xfe1.jpg?width=1080&format=pjpg&auto=webp&s=ec335095a186381acd3502d4e3cf46d66e233a68

As I've described. I'm looking to see what would be the best certification for entry into big data field. I'm currently working as IT Auditor and hope to use that as a stepping stone.

Take your organization from data exploring to #data transformed with this comprehensive guide to data maturity. Discover the four key elements that determine data maturity and how to develop a data-driven culture within your organization. Start your journey to #datatransformation with this insightful guide. Become USDSI® Certified to lead your team in creating a data-driven culture.

https://reddit.com/link/1ibzo83/video/n0p2wzn02qfe1/player

Hi everyone,

I graduated in 2022 and currently have 2.5 years of experience in the big data domain. Most of my work involves developing complex Spark-Scala-based procedures and functions tailored to client requirements. I also have some experience with Bash scripting to create reconciliation scripts, as we primarily store data in Hive databases.

The tools and technologies I am proficient in include:

Apache Spark,Kafka,Hadoop,Hive,HBase Scala programming,MS SQL,Bitbucket ,IntelliJ,Git,Python

Although my team also works on Power BI report generation, I haven't had direct exposure to it yet.

I enjoy working in this domain and am eager to expand my knowledge for better career opportunities and growth. Which additional tools or technologies should I learn, or in which of my current skills should I deepen my expertise, to advance my career in big data?

This week, RWA Inc. dropped some incredible updates! The platform, which makes investment opportunities more accessible by tokenizing real-world assets, is bridging the gap between traditional finance and decentralized technology. And the Launchpad platform is at the heart of it all. Launchpad simplifies the process of launching new projects, raising capital, and tokenization, making it way easier for both entrepreneurs and investors.

What is RWAI, and What Does It Do?

RWAI, short for Research, Reporting, and Launch AI Agent, is an AI tool developed by RWA Inc. Its main goal? To make the research, reporting, and launch processes for projects faster and easier. In short, it’s a helpful companion for both project creators and investors. Here's what RWAI brings to the table:

• It analyzes project details thoroughly and provides users with clear and precise information.
• It speeds up launch processes, enabling projects to get started quickly.
• It fosters greater engagement within the community, making the platform more vibrant and dynamic.

RWAI’s roadmap includes some standout features:

1. Research and Reporting Module: Allows users to analyze projects in detail and make well-informed investment decisions.
2. Launch Module: Optimizes the launch process so both project owners and investors can actively participate in the most effective way possible.
3. Community Engagement: Offers tools and activities to ensure the community is more involved in projects.

RWAI truly aims to provide a practical and seamless experience for its users.

The Benefits of Staking

Staking $RWA tokens on the RWA Inc. platform offers users a range of perks that go beyond just earning rewards. Here’s what you get:

• Platform Access and Level Upgrades: By staking your $RWA tokens, you unlock access to various platform features. The more you stake, the higher your tier level, which comes with extra benefits.
• Access to Investment Opportunities: Staking users gain access to unique investment options on the platform, such as real estate-backed tokens, private equity, and commodities. Regularly check the platform for new opportunities!
• Participation in Token Sales: Be part of upcoming token sales and exclusive crowdfunding events. Enjoy guaranteed allocation rounds and "first-come, first-serve" opportunities.
• Rewards and Exclusive Privileges: As you stake more, you earn Tier Points that unlock additional rewards and special privileges.

Staking is more than just passive income—it’s your gateway to investment opportunities and active participation in the ecosystem.

DAO Labs and the First ILO Success

DAO Labs hosted its first-ever ILO (Initial Labor Offering) for RWA Inc. on its platform, and it was a massive success! As social miners, we had a front-row seat to witness this milestone. This launch clearly showcased DAO Labs' community-focused vision.

Through this process, we saw just how impactful community-driven projects can be. DAO Labs has set a strong example for future project launches and has become a solid reference point for the community.

Hey all!

We’ve launched a Substack called Big Data Performance, where we’re publishing weekly posts on all things big data and performance.

The idea is to share practical tips, and not just fluff.

This is a community-driven effort by a few of us passionate about big data. If that sounds interesting, check it out and consider subscribing:If you work with Spark or other big data tools, this might be right up your alley.

So far, we’ve covered:

• Making Spark jobs more readable: Best practices to write cleaner, maintainable code.
• Scaling ML inference with Spark: Tips on inference at scale and optimizing workflows.

This is a community-driven effort by a few of us passionate about big data. If that sounds interesting, check it out and consider subscribing:
👉 Big Data Performance Substack

We’d love to hear your feedback or ideas for topics to cover next.

Cheers!

The Rise of the AI Data Scientist! AI Data Scientists are leading the way in transforming raw data into powerful insights. Their expertise in both AI and data science is creating ground breaking solutions across industries. Ready to become part of this exciting evolution?

https://preview.redd.it/fw16nweh44fe1.jpg?width=1080&format=pjpg&auto=webp&s=4b00756f53a4389fc2ba992921319324759d9470

Hi data nerds,

Here are the latest updates from Rollstack—a platform designed to connect your favorite BI tools (Power BI, Tableau, Looker, Metabase, and Google Sheets) to your presentation software for automatic report generation. If you’re juggling QBRs, client reports, or departmental updates, you might find something here that simplifies your routine.

January 2025 Updates

Power BI Integration Has Arrived

By rolling out this feature, PBI teams now gain access to the same AI-driven reporting that Tableau, Looker, and Metabase have offered since our early days—cutting tens of thousands of hours from report generation. Want to explore further or book a demo?

Learn more and schedule a demo: Power BI integration!

AI Insights Are Now Open to All

Rollstack AI is now open to everyone, giving business professionals an advanced way to generate customized, relevant insights within their presentations and documents. Teams can edit slide commentaries, titles, and more—while preserving the deck’s structure—so stakeholders can reach decisions faster and with greater clarity.

Learn more: AI insights and native charts

Native Charts for PowerPoint and Google Slides

If you’d rather use PowerPoint or Google Slides charts instead of those in your BI tool, you can now convert them into fully editable versions in your presentation software. They include an accompanying spreadsheet, letting you take a closer look at the source data whenever you need.

Check it out: AI insights and native charts article

How to Access These Features

• If you already have a Rollstack account, you can find everything you need in the app or message us on Slack/Teams.
• If Rollstack is new to you, book a quick tour to explore AI insights and native charts in action.

Thanks for reading, and we hope you’ll explore these new Rollstack features. If you have any questions, let us know in the comments. Your feedback genuinely helps us shape what comes next!

—Team Rollstack

Hey everyone!

I’m working with a 22GB PostgreSQL database (Bitnami/PostgreSQL:16.2.0) and need to generate quick reports, such as linking patients to specific types of consultations.

I’m looking for an open-source tool, preferably Docker-ready, that allows me to:

1. Create reports with graphs that look great on both mobile devices and TVs.
2. Publish visualizations either publicly or privately (with login/password).
3. Integrate via API (if possible, for easier automation).

I need something easy to use, especially for someone comfortable writing SQL queries in PostgreSQL. What’s new in the market that’s simple yet powerful?

Thanks a lot! 🙌