/r/bigdata
For all bigdata gurus everywhere from hedgefunds (quant finance) to biotech (drug discovery) to social media (twitter) to discuss the latest trends, topics, career opportunities and tricks of the trade!
Rules: No advertising, don't blatantly link to your own product(s). Posts must be relevant to big data technologies or discussions.
Related subreddits:
/r/bigdata
Bringing together AI, Data Analysis, and Social Mining is a notable feature due to a recent partnership between Solidus and DAO Labs. We all agree that Social Mining focuses on analyzing social media posts, comments, and other online interactions to understand public opinion, sentiment, and behavior, but having a key feature of fair rewards draws the attention of content creators, it shows an aspect of individual data ownership.
Solidus Hub is a specialized platform for community-driven content and engagement centered around AI and blockchain. The partnership with DAOLabs brings in an initiative that empowers community members to earn rewards in $AITECH tokens for creating, sharing, and engaging with content related to Solidus AI Tech.
The combination of both projects utilizes "Social Mining" SaaS, which incentivizes users to generate quality content and engage in tasks such as social media sharing and content creation.
Let's continue to discussion in the comment section should you need a link that addresses all your concerns!
I want to be able to pull pricing data for the past 10-20+ years on any stock or index in order to better understand how a stock behaves.
I saw that Yahoo now charges you and you can only pull data that goes back so many years. Is there anywhere that I can get this data for free or for a low cost?
Here's how:
Try for free at Rollstack.com
Hi everyone! I’m the Co-Founder & CEO at a startup aimed at transforming data pipeline creation through AI-driven simplicity and automation. If you're interested in learning more, feel free to check out our website and support the project. Your feedback would mean a lot—thanks! databridge.site
GPUs (Graphics Processing Units) are chips specialized in creating images quickly, their demand has increased in enterprises, governments and gaming due to their ability to handle complex tasks.
r/solidusaitech is a company that offers energy efficient GPUs, with the use of advanced cooling technology, they reduce the environmental impact, being ideal for green data centers.
Solidus AI Tech improves technological efficiency while driving sustainable practices.
Lemme know about any books which would be helpful for me to progress in understanding the field.
Hey r/bigdata,
I wanted to share something I’ve been working on that could shift how we think about data management and analysis. I call it Active Graphs—a framework designed to organize data not as static tables or isolated points, but as dynamic, context-aware relationships. I’m hoping to get some feedback from the community here and open a discussion on its potential.
What Are Active Graphs?
Active Graphs represent a shift in data structure: each data point becomes a “node” that inherently understands its context within a broader ecosystem, linking dynamically to other nodes based on predefined relationships. Imagine a data model that’s not just about storing information but actively interpreting its connections and evolving as new data comes in.
Key Features:
• Dynamic, Real-Time Relationships: Relationships aren’t rigidly defined; they adapt as new data is added, allowing for a constantly evolving network of information.
• Contextual Intelligence: Data isn’t just stored; it understands its relevance within the network, making complex queries simpler and more intuitive.
• Built for Multi-Domain Data: Active Graphs allow cross-domain insights without re-indexing or reconfiguration, ideal for industries with highly interconnected data needs—think finance, healthcare, and legal.
How Active Graphs Could Be a Game-Changer
Let’s take healthcare as an example. With Active Graphs, patient data isn’t just recorded—it’s actively mapped against diagnoses, treatments, and outcomes. You could run a query like “Show all admitted patients with Pneumonia and their most recent treatments,” and Active Graphs would deliver real-time insights based on all relevant data points. No custom code, no complex reconfiguration—just actionable insights.
Or in finance, imagine a trading bot that can adapt its strategy based on real-time contextual updates. Each trade and indicator would be dynamically linked to broader contexts (like day, week, and market sentiment), helping it make informed, split-second decisions without needing to retrain on historical data.
Why This Matters
Traditional databases and even graph databases are powerful, but they’re often limited by static relationships and rigid schemas. Active Graphs breaks from that by making data flexible, relational, and inherently context-aware—and it’s ready for integration in real-world applications.
TL;DR: Active Graphs turns data into a self-organizing, interconnected network that adapts in real-time, offering new possibilities for industries that rely on complex, evolving datasets. I’d love to hear your thoughts on this approach and how you think it might apply in your field.
Disclaimer: Active Graphs and its associated concepts are part of an ongoing patent development process. All rights reserved.
Hi,
If you’re skilled in streaming data – from ingesting and routing to managing and setting real-time alerts – we want to hear from you! We’re seeking experienced professionals to provide feedback on a new product in development.
During the session, we’ll discuss your experience with streaming data and gather valuable insights on our latest design flow.
By participating, you’ll help shape the future of streaming data experiences!
Study Details:
If you’re interested, please complete this short screener to see if you qualify:
https://www.userinterviews.com/projects/O-tG9o1DSA/apply.
Looking forward to hearing from you!
Best,
Yamit Provizor
UX Researcher, Microsoft – Fabric
I am sharing my article on Medium that introduces Spark UI for beginners.
It covers the essential features of Spark UI, showing how to track job progress, troubleshoot issues, and optimize performance.
From understanding job stages and tasks to exploring DAG visualizations and SQL query details, the article provides a walkthrough designed for beginners.
Please provide feedback and share with your network if you find it useful.
Beginner’s Guide to Spark UI: How to Monitor and Analyze Spark Jobs
USDSI® can be the key differentiator that stands you out from the herd and propel your career forward. black box ai
Foster huge growth with top skills in data visualization, data mining, and machine learning today. Look at the interesting trends and future that data science holds.
Data science is an interdisciplinary field and to succeed in your data science career path, you must have a strong knowledge in the foundational subjects and core disciplines of data science which are Mathematics and statistics, computer science, and domain or industry knowledge.
The knowledge of programming language, mathematical concepts like probability distribution, linear algebra, and business acumen will help you understand the business problem efficiently and develop accurate data science models.
Explore the core data science subjects that you must master before starting your career in data science and learn about specialized data science components like data analysis, data visualization, data engineering, and more in this detailed infographic.
Hello, I am looking for a software which can injest data from a 3D printer and provide a analytics sandbox where that data can be analyzed / dashboards can be built. The type of data ranges from PLC data (export JSON), log files (text), csv files, to images. I am looking at solutions such as Cloudera (seems expensive) or SPLUNK. Does anybody have any other advise for such a flexible software solution that is also affordable? Thanks!
What is your most challenging and time consuming task?
Is it getting business requirements, aligning on naming convention, fixing broken pipelines?
We want to build internal tools to automate some of the tasks thanks to AI and wish to understand what to focus on.
I developed this questionnaire for my PhD. It analyses the influence of the human factor in Big Data Analytics. To answer you need to work in the field of data analytics. We need to collect a large number of answers for the analysis, if you want to help us it will only take 10 minutes of your time. At the end of the questionnaire (if you have entered your email) you will receive the average of the answers so far to compare with the averages of the other answers.