It’s been about 5 years since I worked on NLP. I’m looking for some general advice on the current state of NLP tools (available in Python and well established) that can help me explore my use case quickly before committing long-term effort.

Here’s my problem:

• Classifying customer service transcriptions into one of two classes.

• The domain is highly specific, i.e unique lingo, meaningful words or topics that may be meaningless outside the domain, special phrases, etc.

• The raw text is noisy, i.e line breaks and other HTML formatting, jargon, multiple ways to express the same thing, etc.

• Transcriptions will be scored in a batch process and not real time.

Here’s what I’m looking for:

• A simple and effective NLP workflow for initial exploration of the problem that can eventually scale.

• Advice on current NLP tools that are readily available in Python, easy to use, adaptable, and secure.

• Advice on whether pre-trained word embeddings make sense given the uniqueness of the domain.

• Advice on preprocessing text, e.g custom regex or some existing general purpose library that gets me 80% there

So I’m building a simple regression model to predict fuel consumption for trucks in a large food company. We have data from different trips from different routes.

Some routes have much more trips and are thus over represented in the data. Let’s say as an example that we have 10 routes, with 8 routes having 10 individual trips and 1 route having 1000 trips. If I would just randomly sample the data most of the data would come from the large route, reducing the regression problem to basically fitting that specific route.

Now that isn’t something we want because we would like to take into account different geographic information from the various routes (a route has a number of geographic and specific features). Should I just perform stratified sampling?

This brings me to our second problem, that the different trips won’t be independent. If I sample 10 trips from the large route then all the input variables unique for the route will all be the same, having only variability in trip specific features such as time of day or weight of the freight. How should we account for this? Using a hierarchical model maybe?

I have an interview for an analytics engineer position with a small company. I currently have moved on to the fourth interview with the data architect. I have not had a traditional technical interview with sql or data modeling questions, mostly the other interviews with the hiring manager and others have asked about my experience and personal projects, with some questions about technical concepts. I am going to guess this time around the interview will involve more traditional technical questions. If not, any advice on what type of questions I should prepare for?

Hi all,

I have been a data scientist for the past 5 years. My bachelors is in information systems and my masters is in statistics. I don’t come from compsci and I had minimal coding other than SQL and R in my education. I have been using python for the past 4 years self taught and I am adequate with it. I would like to improve my python coding skills, more around how to build out and organize it, and best practices for structuring the files and packages. additionally use of classes and methods. I think this can be summed up as software design.

The other members of my team have more extensive and formal teachings in these subjects and it is becoming apparent to my manager that I lack skills in this compared to them. We are expected to be machine learning engineers as well as data scientists at this company because we are a smaller start up.

Can anyone recommend any resources to help me level up my knowledge in this area?

Hello, I am a stats MS struggling to find work. I believe my math/stats background is holding me back because I am not PhD level but lack the engineering skills to work in applied roles in industry. When I do self learning projects I can only ever think of ideas implementing models I am interested in, but am lost as what to do to start writing production quality code and challenge myself as a software developer. Any ideas and advice is greatly appreciated! Thank you

Hey everyone,

I have here a dataset of KPI metrics from various social media posts. For those of you lucky enough to not be working in digital marketing, the metrics in question are things like:

• "impressions" (number of times a post has been seen)
• "reach" (number of unique accounts who have seen a post)
• "clicks", "comments", "likes", "shares", etc (self-explanatory)

The dataset in question is incomplete, the missing values are distributed across pretty much every dimension, and my job is to develop a model to fill in those missing values. So far I've tested a KNN imputer with some success, as well as an Iterative imputer (MICE) with much better results.

But there's 1 problem that persists: some values need to be constrained by others in the same entry. Imagine for instance that a given post had 55 "Impressions", meaning that it has been seen 55 times, and we try to fill the missing "Reach" (number of unique accounts that have seen that post). Obviously that amount cannot be higher than 55. A post cannot be viewed 55 times by 60 different accounts. There are a bunch of such constraints that I somehow need to pass in to my model, I've tried looking into the MICE algorithm to find an answer there but without success.

Does anyone know of a way I can enforce these types of constraints? Or is there another data imputation method that's better suited for this type of task?

I’m going to be starting my Masters Thesis in my stats program. I have narrowed down two projects with different faculty members. One of them is entirely self guided, and that advisor doesn’t really have any expertise in this area and the other one is basically with another faculty member but the research topic is within his domain, so he can help me, but the application strays too far from what industry I want to be in.

Project 1: Causal Machine Learning, and nonparametric estimation for identifying heterogenous treatment effects in advertising/marketing data.

There’s a dataset by a company in the marketing/adtech space that they have made public, and is essentially an open dataset related to shopping transactions, and demographics of shoppers, and information on marketing campaigns that were run by the company.

Ideally this would be a causal inference project, where I get to actually learn causal inference and causal (double) machine learning on my own, and apply it to this dataset. No new methods being created, just a standalone causal inference analysis and basically I walk away with learning a new area of statistics and a way to solve causal problems.

My advisors background is nonparametric regression so he would be able to give me advice on the estimators being used (some of the methods use random forests, tree based methods, splines, kernel methods etc.) so he could be of help in that sense. But he knows nothing about causal inference. So I’d be on my own on this one.

I want to take on this project despite it being quite isolated because I want to signal to my future employer that I can work on data science problems involving causal inference.

I only have 1 year, so id have to self learn causal inference + do the analysis + the paper.

Project 2: Bayesian Dimension Reduction in gene expression data

Another advisor works in the area of dimension reduction, and broadly works in genomics. The project he was proposing for me was an actual opportunity to create some new methods. So much more on the research side than doing an analysis like project 1.

It involves looking at Bayesian approaches to doing common dimension reduction techniques like PCA, Factor Analysis etc.

From this paper I’d walk away with a good opporuntity to dive deeper into Bayesian inference, and work on methods research. I’ve never done a bayes project before, despite just taking a class on it.

The cons to this is I don’t really have any interest working in bioinformatics, but this research is within the realm of my advisor so he can actually help me. The other con, is I don’t want my future employer to think I’m not qualified for tasks involving causal inference because I worked on dimension reduction, and with applications to genetic data, which is not really the industry I want to be in.

Does anyone have any input on this? Does choosing a topic not related to causal inference lessen my chances of getting a job involving that? Do people care what my thesis topic is? Or does the MS in Stats signal that I could work on causal inference?

Hi everyone,

I want to know if there are developments and research topics that are outside/completely orthogonal to LLMs and generative AI. To be honest, I am bored of LLMs. I don't care about the performance of Model X vs Models A,B,C,D etc. Moreover, at least 8 out of 10 projects in my organization are focused on generative AI and RAG. While I understand the usefulness of these ideas, I think there's an overload of information that is not particularly helpful for my brain.

Personally, I am interested in scientific machine learning- drug discovery, climate change, physics simulations. If there are other research areas that you are aware of, please feel free to share.

From a "long term career perspective", I want to transition to companies that work on problems in imaging and communications (I have a background in signal/image processing and computer vision). I am very much interested in novel imaging techniques that use some kind of computational imaging and ML algorithms. Qualcomm and Samsung come to my mind- but I could be wrong.

I've a CS degree and I'm currently working as a new grad Data Analyst. Aside from building dashboards and reports, I also do a lot of data engineering and some ML modelling. I'm planning to get a CS Masters after 1 - 2 years of work, then switch to DS. But looking at this sub, it seems like a lot of people recommend switching to SWE instead? I do enjoy software engineering work so I wouldn't mind the transition, but I'm interested in ML and thought DS would be a good career path for that.

Is Data Science really a dying field and pay less than SWE? I even heard that switching from DS to SWE is more difficult than the other way around, because any SWE learning ML can do DS work. And I've seen people saying that SWE is more stable and easier to get a job than DS. Is any of this true?

Hi everyone, I’m a marketing major about to graduate in May. A year and a half ago I took basic hypothesis testing/linear modeling class to try out an analytics certificate at my school, and I fell in love with statistics for the first time. When I began looking for job/internship opportunities around that time I was worried because while I didn’t mind marketing, I also didn’t love it. That’s when I made the decision to continue my degree in business, and work every week towards becoming a data analyst (and eventually, a data scientist! But I’m patient, and I wanted to wait until I’m ready. There’s a lot to statistics/programming, as I now know… lol).

It’s been a long and very hard road. But I now have a job as a data analyst and I’m working on a large machine learning personal project now. I see a lot of negative discussion in this sub (which is entirely fair); however, I just wanted you all to know that as someone who is not taking the traditional route into data science, I think your job is awesome. I think what you do is fascinating and what statistical modeling might accomplish in the future is inspiring. Have a great day, and I hope I have the pleasure of meeting some of you in the field one day.

I have been hunting jobs for almost 4 months now. It was after 2 years, that I opened my eyes to the outside world and in the beginning, the world fell apart because I wasn't aware of how much the industry has changed and genAI and LLMs were now mandatory things. Before, I was just limited to using chatGPT as UI.

So, after preparing for so many months it felt as if I was walking in circles and running across here and there without an in-depth understanding of things. I went through around 40+ job posts and studied their requirements, (for a medium seniority DS position). So, I created a plan and then worked on each task one by one. Here, if anyone is interested, you can take a look at the important tools and libraries, that are relevant for the job hunt.

Github, Notion

I am open to your suggestions and edits, Happy preparation!

Hi I am working on a predictive maintenance project and I need some help. Kindly dm if anyone is willing to work on this.

PS: this is research project .

Hi I am working on a predictive maintenance project and I need some help. Kindly dm if anyone is willing to work on this.

PS: this is research project .

I'm considering applying for a Machine Learning Engineer position with my company. I already work as a Data Scientist. I've developed a great reputation and most of the executives know who I am and frequently ask for my input on things. I'm happy with my job, but unfortunately, it feels a bit dead-end'ish. It's a great job, don't get me wrong, but I don't see any obvious path to promotion, short of waiting it out 10 years and that frustrates me a lot.

There are more long-term opportunities in ML Engineering in my company. Salary should be a bit higher as well; I'm estimating I'd make at least $25k more.

As a DS, I mostly work with Python, SQL, and Tableau. I'd say only about 20% of my time is spent coding, however. I've built a few machine learning models (mostly time-series and collaborative filtering), but it's not the main crux of what I do. Still, I'm pretty universally regarded as the expert on ML as well as tech on the team. Moreover, I've automated a lot of our analysis. I'd be considered an expert on SQL and data analysis, as well.

If I switch to MLE, I'd also need to become proficient in Databricks, Azure, and React. I don't work with any of those on a regular basis (I've used Azure and Databricks before, but not a lot). I'm guessing I'd probably go from coding maybe 20% of the time to coding 70%+ of the time, as well. React is probably the toughest one there, but I do have front-end experience from working as a full-stack developer at a start-up a few years back; albeit, I'd consider myself very far from an expert on front-end.

I'd be very good at it, but I admit it might take me 1-2 months to "get into a groove" and get comfortable with some of the technologies I'm less familiar with, particularly React. I learn quickly, but I often feel like people want take a chance on anyone who doesn't already know every skill in the job requirement.

My questions:

How big of a jump is this? I don't use Databricks on a regular basis, but given my proficiency in Python and SQL, is that going to be something that would take a long time to get familiar with? Is my relative inexperience in React a big issue or is it just so difficult to find an ML Engineer with React experience to begin with, that I might get a pass on that?

Is it worthwhile? Anyone who has worked on both the business-facing DS side and the more tech-oriented ML side, did you enjoy one more than the other?

Am I likely to get serious consideration? I have a very good reputation within the company, but often feels like some of the more pure tech people look down on someone more business-facing like myself. I'm not sure how I'll be perceived, since my background was business before I got into tech.

​

Hey guys,

I'm constantly checking for jobs in the sports and gaming analytics industry. I've posted recently in this community and had some good comments.

The job board updates daily and as we know, the market is not as dynamic as before so I wanted to share several data science positions that appeared recently.

HOCKEY DATA SCIENTIST @ The Florida Panthers

• Machine Learning Engineer @ Second Spectrum
• NFL Data Scientist @ Swish Analytics  (also tennis and MLB)
• Data Scientist (all levels) @ Kaizen Gaming
• Senior Data Engineer - Sportsbook @ Underdog Fantasy
• Recruitment Data Analyst - Global Remote
• Junior Data Scientist / Intern @ Huddlevision

There are multiple more jobs related to data science, engineering and analytics in the job board.

I've created also a reddit community where I post recurrently the openings if that's easier to check for you.

Disclaimer: I run the job board.

I hope this helps someone!

I am currently a data scientist in a big bank. Overall my experience in my team was hell.

Some of my experiences were

1. bad stakeholders that bad mouth the analytics team. Stakeholders are very pushy and thus 5-6 month projects are compressed into 2 months. Effective we have a stakeholder from hell, and has a reputation for lynching managers. One of the managers i was on good terms I worked with effectively got lynched for things outside of their control.

2. i had an old manager that bad mouthed me as we didnt work well together then left. Manager was very toxic however any achievement is an uphill battle.

3. i ended up having to work a lot of hours, however i tried to draw boundaries to my team and ended up getting reprimand and my manager was essentially making my life a living hell and then put me on a pip.

4. I got crap for taking a sick leave due to a surgery despite telling them 3-4 months in advance.

5)our turnover is bad, we lose 1 data scientist a year if not more.

6. my new manager effectively helps my coworkers but tells me to figure it our and does not i include me in meetings that are critical for my project. After withholding that information, in my touchbases with the manager she then criticizes my work for information that she witheld.

However lately, they really fixed their attitude as i had to work extra hours to meet their deadlines while still looking for jobs. As of now they started adding more projects and tasks for me while I am in a pip.

My pip was 6 weeks, and the manager slipped up saying the 8 week process. 6 weeks pip-2 weeks to be fired.

My personal guess is that they realized that I am not as bad as I am essentially the model janitor who fixes other people mistakes and when anything goes wrong I get blamed for it. A model is performing poorly, go rebuild it. This model has no documentation go rebuild it. This senior manager built an outrageously bad dataset go build it.

Also i know critical dataset info and my own new manager does not know a lot of stuff on our data sources. While doing my job i was doing my managers job as she had no clue how to get any data or the problems with it.

Now that I have another offer with slightly more pay, but working in a startup i am strongly rethinking of just giving them the FU and leaving myself. However startups are risky and its not as prestigious as the bank. The benefits/PTO are worse (despite not being able to cash out or use pto in my current job for a year 7.5 weeks of pto unused), also given the bonuses and etc. its actually worse as the startup offers a static bonus of 7500 with 95000 base with my current role is 90,900 with 10% bonus, however i can feel they will still overwork me but i am considered to be very technical and start fresh. The role offers more finance opportunities specific to risk modeling rather than pricing which i dont know if it will have major applications in other industries.

My friend and family are saying that its not worth it. While my current job effectively makes me get panic attacks, insomnia for months (i wake up in the middle of the night from the stress), and also i stayed for weeks not eating from stress.

The job is hell, and now the tone on is slightly better but i really dont trust my team as I think

Sorry if this is sound unhinged, but i finally got a ticket out and I wanted professional advice on how I would approach this situation.

Hey all, just noticed this job posting with reddit while I was doing my own searching. Sr Data Scientist in the US, remote-friendly, nice comp / pay range ($190k to $267k/yr). I'm not in the US so I'm out. https://boards.greenhouse.io/reddit/jobs/5486610?gh_src=8a8a4d8a1us. Actually kind of surprised they don't share it in this sub as well.

I hate R, its syntax is not at all consistent, it feels totally random ensemble of garbage syntax with a pretty powerful compilation. I hate it. The only good thing about it is this <- . That's all.

Is this meant to be OOP or Functional? cause i can put period as i like to declare new variables this does not make sense.

​

I just want to do some bayesian regression.

First of all Thanks to the sub members who gave me karma to post here.

We are working on a project where we should find the percentage of similarilty between two texts using an LLM. Now what are all the LLMs that I can use? Any Idea lead would be helpful

Does anybody have any tips on how to handle re organization/ re-structuring?

Still employed as DA. And tbh have avenues to stay at my company but seems like they’re moving to a more centralized data structure. Probably will give primary access to tech hub office employees. I am remote. I do power BI, vba and data processing. Right now mostly ETL stuff. Any tips would be appreciated!

I’m starting a Master’s program in the fall and I’d like to improve my programming skills. My undergrad was in Math, so programming wasn’t really much of a focus. I took one actual CS course which mostly used Python and just a little bit of C. I encountered R in college in my Stats courses and I use it regularly in my current role (DBA/Analyst at a small nonprofit). I’ve also kept up with Python and I’m fairly comfortable with it still.

I’ve never actually learned about OOP or say structures and algorithms but I’d like to. I’ve read a bunch about Java being a more rigorous language which forces you to code in an object-oriented way.

I guess my question is: is there enough of a benefit to using Java for OOP, or should I just use resources designed for Python?

If anyone has any experiences, I'd love to hear it.

And if it's not a thing, what are the blockers in your opinion?

I made this post after getting an offer a couple months ago. A couple weeks after the offer, it was rescinded. Probably for the best as I realized the original description did not match the actual role.

After the offer was rescinded, I took a couple weeks off the job hunt before getting back at it. Cleaned up the resume, started being more selective with where I applied, and grinding SQL problems online. About a month in I was interviewing with 3 companies.

I don't feel like making another Sankey, but it's pretty much identical to the last, except I got 3 first round interviews, rather than the 1 last time. Companies are 1 mid-sized tech and 2 pre-IPO unicorns. I was ghosted by one unicorn after a screening round and am still interviewing with the other after 2 rounds, though after 5 rounds with the mid-sized tech I accepted a DS manager position.

My advice: 1) stop following this subreddit, it's 90% doom posting and 10% circle jerk. It doesn't feel like anyone here is actually interested in data science beyond getting a job. 2) mass send an easy to parse resume everywhere. 3) keep your head up, it's a grind. Don't forget to exercise, eat well, and have a social outlet. 4) referrals aren't worth what they once were. None of my dozen or so referrals resulted in even a screening interview

I was rejected for roles I thought I was a shoo-in for and interviewed for roles I thought were a reach. There's a lot of luck (preparation+opportunity) involved that's often out of your control.

Good luck

Hello Everyone!

This Friday, I have an upcoming final-round interview with the Director of the division I'd be interning under if I got the position.

Per the recruiter, this is just to sort of solidify me as the right candidate for this role. I know that there won't be any sort of technical/coding aspect as it is just a 30-minute call.

If anyone has any advice on how to approach it, it would be greatly appreciated! This is my first ever final round interview so any advice would be great!

Thanks and have a great day!

EDIT:

I had the interview and I'll find out early next week the decision in regards to the internship. All I can say is that the final round is mainly about you and how you are/how you think as a person. Coursera provides a really good list of questions you might be asked. In my case, I was asked about what I'd like to get out of the internship, how would I describe myself, if I've ever had a disagreement with someone on a project what was my course of action, and why do you specifically want to work for us. A bunch of behavioral stuff.

I will say that the advice provided by u/Dangerous_Media_2218 and u/data_story_teller is super helpful in terms of what to ask in terms of the job. To all the people that provided advice, thank you so much. I'm eternally grateful!

Hi, I'm one year into my first DS job at a big German company. I want to decide in which direction I want to develop myself careerwise and ask you for your opinion on that. Right now I do basic things like building ML models, big data analysis in pyspark, dashboards in powerbi and I also built small chatbots with streamlit, langchain and some Azure ressources. I know functional programming in Python but I never really learned object oriented programming, is this maybe something I should go for?

I don't really have a senior colleague right now that could create a plan for me, it's a bit of a weird hierarchy there, so I'm super thankful for any input :)

Thank you!

Dive into the transformative power of data science in the world of American football with Eric Eager, PhD's "Using Data Science to Better Evaluate American Football Players." In this presentation, Dr. Neubig, an expert in machine learning and natural language processing, showcases how the sport is evolving through advanced analytics. 🏈💻 From play-by-play and charting data to the revolutionary potential of player tracking data, discover the cutting-edge techniques that are setting the stage for a new era in football analysis.
https://www.youtube.com/watch?v=8lwFUO_yj7c

Hi,

I'm a student learning data science.

I see few of my mates, making project with generative AI tools like langchain or open AI API etc

But this is what I think, and I want to know if what I think is correct or not.

Knowing how to use generative AI frameworks does not validate that you know deep learning or even basic machine learning.

I think building projects with generative AI frameworks only validate that you know how to code by reading some docs. I think anyone who knows basic programming can make an "AI summarizer" or "AI Chatbot" using langchain.

I don't feel that making such projects can make me standout in any way for machine learning jobs.

I would rather make a basic data science project which at least tries to solve some real business problem.

I am wondering if there is some sort of multilevel model that I can use to identify likely buyers of specific products or create a lookalike audience.

The issue is that I have 1000s of products and around a million users. It would be computationally infeasible to create a model for every product.

The structure I am thinking of is the first level is a product and the next level is all the users in my database.

Is there some sort of ML algo I could use to achieve this?

I'm working on a self-driven project where I'm taking data, clustering it using DBSCAN or OPTICS, and passing the data and labels to a classifier to train. The overall goal is anomaly detection, pattern recognition, and behavioral analysis and my approach has been to remove conserved (repeated) patterns as "knowns", provide a best-fit for anomalies under the assumption that the sample belongs to a known class but contains a type of anomaly (or is an outlier), and to flag unknown cases as either more prominent anomalies or, in cases where anomalies bear a measure of self-similarity, a new, emergent class.

Currently, I am using an ensemble of OVR binary classifiers (SVMs), each trained on a single target class against all other samples, as a recognition network. I also have a boosted ensemble (random forest) trained similarly to provide consensus to the output of the recognition system.

I have a k-NN classifier that is trained only on clustered data (samples labeled as noise by clustering was removed from this training set) to act as a rectification system for mislabeling/misclassification.

So, if a sample is passed through and there's a consensus in the recognition network, it can bypass the rectification system. If consensus is not met, the rectification system attempts to identify the class. If the probability is high enough, it's flagged as an anomaly of class N and sent out for further anomaly detection and isolation. If the rectification system's probability output is too low (or more than 1 label has a similar probability), it's flagged for analyst review.

Does this make sense? It works well in most cases but I don't quite have the warm and fuzzy on the misclassification detection. It seems kind of nebulous when it comes to recognizing a known class, an anomalous sample of a known class, a new class, or a variant/evolution/subclass of a known class. That's where I'm wanting to go with this. Also, any feedback on the setup would be appreciated.

Hello, I am about to have an interview for a BI Analsyt postion and I found out that the main system they use is Chmura. I tried looking up videos to see how it works ( is it closer to SQL or Tableua, etc.) but all I got were comapany video's. Does anyone have any info?