Hello there!

13YOE here. I always wanted to become a manager because I wanted to have more impact in the organizations. Define company strategy, proper team organization, give more "power" to the ICs, define more real hiring processes... I always felt that I wanted to be the "undercover IC" in management. Fight the status quo from the inside. No mention to people management by the way.

Two years ago I finally landed my EM position in a huge well kwon unicorn-company in Europe. After two years I left the company with a feeling that management is not for me but maybe is more related to the company situation.

During those two years I had 5 different managers, went throw two reorgs and deliver close to nothing. Tons of DEI, career development plans but no promotions because of budget, transparent and non-biased performance review but gaussian distribution of rates. HQ was in another city and we had a common saying. "Good things only happen in HQ".

Now I see myself in the middle of nowhere. A manager for the ICs, and an IC for the managers. I had my first interview as an IC and I almost die during the "solve this algorithm" interview. First I was expecting a cultural interview because it was with an EM and when he opened Google Docs with an exercise I collapsed. I think I am not a bad programmer because I have delivered complex systems in the past (banking/fintech/stock brokers) but I get so freaking nervous in those kind of interviews and now that I think that my technical skills are rusty my confidence is gone.

I feel like management is the only thing left for me and I want to avoid it because of my last experience.

Should I give management another try? Should I look for companies where managers have some IC responsibilities? Is management not for me and I just need more training as IC?

I am going through a personal crisis I am really looking for a company to "settle down". I live my work with so much passion creating more problems for me than solving company issues. I've worked in 9 different companies now and I am done.

This is one problem that I don't see much discussion online. Veterans in the industry will know that critical changes to code are encapsulated in conditional checks that depends on a switch or a flag. When the flag is switched off, old logic is executed, while switching it on executes new logic. The intention here is to prevent business disruption if the newly released feature starts troubling in unanticipated ways, without the hassle of traditional development and the time it takes to place an organic fix. This is a smart invention from the business perspective.

Unfortunately, the way it boils down at granular levels is not so clean or elegant. There is no decent programming construct available to achieve this except for a rudimentary if-else condition. Unfortunately, not all kinds of code can be wrapped in a conditional logic. One has to do all kinds of refactoring gymnastics to make it wrappable in a condition. As torch-bearers of code quality, is there a better elegant way to accomplish the business goal without compromising on code maintainability and readability?

I would like to learn system design not only to pass the interview but to actually gain some good understanding of it and actually have some hands-on experience.

Would it be possible to do this by creating some personal project that kind of resembles the real system that is actually out there (eg. simple twitter clone)? Can I do this on my local computer by using Docker containers for sql, nosql, blob storage, load balancer, logging/monitoring service, distributed configuration, distributed cache, distributed message queues, rate limiter, key-value storage, task scheduler etc. I would take one or two industry standard software for each of these components (eg. Apache Kafka, ZooKeeper, RabbitMQ, nginx, redis etc.).

So for example, say I want to create a simple twitter with just a few features. I probably don't even need frontend, just the exposed public-like API that I can use to simulate high loads (eg. create mock client that sends millions of requests in a loop and adding some delay/sleep on the backend to simulate processing of certain parts).

Can all this be created with docker containers? Can I orchestrate these containers with Kubernetes or do I need something more? How do I go about doing scaling, data replication etc.? Again I am thinking about simulating all this stuff, load increase, node failures etc. The focus will be on understanding concepts and having some basic hands-on on the most popular software out there that is used to implement the system, not to actually create the production ready large-scale application.

For the context, on my daily job I am working as a full stack web developer. My job consists of implementing domain specific features in a fairly large web application. So my tasks are mostly domain oriented and not that technically hard. So I do not have task related to system architecture or design. I work on both frontend/backend in 50/50 ratio.

How should I approach learning system design practically with developer experience like this but not much knowledge about distributed/architecture/scaling systems.

I’m a data engineer and I have built data pipelines for applications and data warehouses. I noticed that when building data pipelines for applications there’s usually only a Dev & Prod environment, the Devs are usually doing the QA themselves, there isn’t a separate QA team at the companies I’ve worked at.

Now at the companies where I’m building data pipelines for data warehouses, we’ve almost always had a Dev, QA, and Prod environment, along with a separate QA team.

I wanted to hear from experienced developers, is this the norm when building applications as well?

I plan on asking this in the data engineer subreddit to hear specifically from them as well but I figured the devs here who’ve been in the field for a while could share some deeper insights.

I have several YOE as Senior DBA and my main area is not web development, however I study, experiment and contribute to open source projects in my free time. In the past months I have presented to my team a small React application for internal use that I have built all by myself and feedbacks have been very positive. The CIO was particularly impressed and wants to bring it in production.

Since this web application is not really in my department scope, I'm going to coordinate with another team of 5-10 people. The CIO asked me to think about how much involvement I want in this project.

It's a very interesting chance, because I can decide pretty autonomously how much I can deep dive into the project. However, I'm grappling with the most effective approach to maximize benefits without overextending myself into unfamiliar territory.

From one side, I don't want to handle the project on the technical details, because they are out of my comfort zone. I'm not an experienced web developer. I'd rather take on a role akin to a strategic consultant or project manager, leveraging my understanding of the application's functionality, logic, and data flow while remaining involved. I fear to be excluded

I've already knocked out about 80% of the features in my first draft, and the thought of someone tearing it apart kinda stings.

Did you have similar experiences? What helped you decide?

I left my old job last week for a new one. They actually counter offered for me to stay which I initially verbally said yes but I left in the end after deciding I couldn’t renege on a contract. My boss was still chill with me in the end even saying I’m not upset at you — just upset you’re leaving.

Two days at the new job and I can tell it won’t work out. Basically lied about flexible working conditions — you have to put in your hours like your billing someone.

How can I ask for the old job back? Is it too soon?

11 YOE. Worked at a software consulting company for the past 7 years. I started there as a senior software engineer and 3 years ago I got promoted to tech lead. I now oversee the technical aspects of the entire projects lifecycle, leading small cross functional teams of 4/5 devs from analysis and architecture to technology selection and production deployment. I have extensive expertise with the language and frameworks we use, and I am able to tecnhically support teams I'm not directly involved with.

I enjoy my role, however, our company is struggling to find new clients, and the projects are becoming less interesting. The few clients we do have are so overbearing that they dictate our work methods, which are far from agile or even from common sense project management practices. Moreover, in all these years, I've never had the opportunity to work on cloud projects or handle high-traffic loads. I feel these are important aspects for a tech lead and software engineer’s career. I'm not sure I could tackle the tech lead role in a company whose tech stack and traffic demand is far from the ones I'm used to.

I am considering taking a step back and accepting a position as a senior software engineer at a rapidly growing product company. This move would allow me to learn a new language, actually work with agile methodologies, and deal with challenging traffic loads in terms of performance. My idea is to go back to a tech leadership role once I gained some more expertise.

Would you go for such a downshift to strengthen your technical skills?

I had an interview a few days ago where the interviewer said they've only been with the company for three months.

Most interviews I've had came from people that have been with the company for a while. Is the 3 months thing a red flag? My gut says it is but I can't articulate why.

Hi,

I'm a senior staff engineer assigned to help a few teams. One particular problem in my team is that our engineers have a hard time fixing things permanently while on-call. Sentiments that I've heard is that it's like a firehose. When you get a few minutes into debugging one issue, you have another issue that pops up. Then you have someone else asking you for help. Then a pager fires and you have a CSO that you need to fix.

People seem burned out from on-call, and they don't know where to start to making the problems better. The problems are difficult to solve and usually will take some time. A lot of the time, engineers are not sure the best way to solve them. Sometimes, solving them will take costly cross-org projects. When there are a lot of problems, it's easy to ignore them instead of taking a lot of time to fix them. So, my team has become pretty complacent with our software quality.

My director that we all report through wants to see things fixed. But he seems impatient. He wants everything fixed today and is frustrated that it's not all done right now. So, his attitude is "Why can't all the teams just get their shit together and fix it?" I end up agreeing with all parties since I do want it fixed, but, I also see that the solutions are not easy and likely will take some effort, which the director does not necessarily want to spend.

Has anyone else been in similar situations? What are some things I can think about to help my teams get time (from upper management) and motivation (from themselves) to fix these problems?

I work at an agency and sometimes I'm very busy with projects and sometimes there is a lull. The lulls drive me crazy since I have to make up tickets to look busy. Anyone find they don't have enough to do on a day-to-day basis?

How can you know what your strong and weak points were or what your errors were?

There was a top thread in one of the tech subs for women recently, where OP complained that despite working a job with the title of “software engineer”, some family (edit: who also seem to work in tech) kept asking her about how her “job in IT is”, and OP was asking if she’s overreacting be being annoyed at that. The thread pretty much ended up becoming super polarized.

A significant portion of people agreed that OP is justified in her annoyance, and that she should push back and correct any time someone assumes she is in IT, because it is very different from being a software engineer and her family is ignoring that. Another portion of people claimed that OP was overreacting, and being “elitist” for feeling that way, and that software engineering is IT work.

What is correct here?

I’ve only had one EM in my career who I really respected. I’ve been modeling after him since then.

Many other EMs I had were either not technically strong or not very compassionate.

What EM behaviors bother you the most and how do you cope with them?

What EM behaviors make you feel productive and satisfied?

Over the past few years I've made a serious effort to produce what I consider to be quality code*. I keep methods small, I refactor large ones as part of PR's, I comment on other PR's to push for cleaner code. For whatever reason, it matters to me. I guess I want to be able to take pride in my craft. It does not seem to matter to other people. I don't get bonus points from my manager.

This approach doesn't seem to be getting me anywhere at all. I get some positive reviews from peers during performance management cycles, but I think I could get those in other ways. Am I wasting everyone's time with this approach? Should I focus my energy elsewhere? How do I ACTUALLY advance my career?

*I understand that different people have a different idea of what clean/quality code looks like, so I know I could show examples that would polarize people. Maybe that's a signal that what I'm attempting is pointless.

I've got a new team of 8 devs, and I volunteered to own the oncall process definition. We likely won't have much after-hours support needed (maybe 1-2 times a month), but workload will be busy during the working day. Our customers are 100% internal to the company, so most of our events are either (preferably) self-found or (rarely) reported to us on slack.

The most successful oncall process I've been a part of is at Amazon, so I'm mainly copying what worked well there. However, I'd love to hear if others have any tips I may not be thinking of. Here's super high level of my proposal:

• Weekly shifts (friday 9am - friday 9am). Gives Friday as a handoff day and time to create the weekly summary. Also IMO it feels a lot better to start a shift with a weekend than end it.
• Oncall engineer primarily responsible for triaging incoming customer requests. Quick requests they can tackle themselves, longer requests they can move into our normal intake process.
• Also responsible for triaging OpsGenie alerts, and bringing in anyone needed to resolve operational issues
• No expectation the oncall engineer works on normal project work. If it's a slow week then they can get back to it, but I'd prefer them be super thorough with other oncall related tasks. Also toying with the idea of having some sort of operational backlog where the oncall engineer is empowered to work on anything that will improve our team's operational excellence (better monitors, automating common tasks, etc).

There's some other stuff too but that's the high level. What's worked well for y'all?

Hi everyone, I am software engineer with almost 10 YoE.

My previous job, where I was a Tech Lead, has been a mess, it lead me to burnout and I was fired after the company was acquired. So took my severance package and had rest for three months.

Three months later I joined a company with 10 years in the market to develop their newest product. The good part of this is that it has managers with a decade of experience while maintaining the feel of a startup. However, coming from an environment where we were constantly pressured I start to feel bored and with no motivation at this job.

Team is composed of 1 EM + 1 TL (me) + 5 devs + 1 Product Manager + 1 Product Designer. There are barely any challenges to myself, most of my time is spent mentoring or acting as a consultant to the team. In terms of quality of life its good, during my lunch time I go to the gym and then work for the rest of the day. Barely there are stressful situations.

The thing is I start to feel I am not learning anything else. I spoke my concerns with the Director of Engineering and he seemed open to help with my career development.

The thing is here things seem to be taken too relaxed. The only moment where we work under pressure (tight deadline) is when the initiative involves multiple teams across the organization but for our product milestones everything is taking too relaxed. The dev team estimates and then they dont deliver on the date. The problem on this is that on my previous job we were forced to accurately estimate or we would have to work overtime. I didnt like that, but I also dont like the other extreme of not being accountable for nothing. I discussed about this with the Engineering Manager but his point of view is to be chill about the situation since no one is pressuring us. I understand that, but as a Tech Lead I would expect our team to be consistent on estimations and the delivery.

At this point, I am considering switching jobs given that I cant stand working with a team that takes things too relaxed. Having been on both scenarios makes me wonder that the options seems to be: work hard, burnout or work relaxed, have no career development.

I have 3.5 years of experience and was recently laid off from a remote job at a 50ish employee startup. I live in a small town in Florida and cannot move for family reasons, so I'm stuck looking for remote positions. My gut feeling tells me most remote opportunities are with smaller companies looking for more practical knowledge over good leetcoders, so I've been mostly focusing on studying for system design interviews (currently reading Designing Data-Intensive Applications). Is my intuition valid in your opinion, or should I spend more time on leetcode?

I have this problem with one of my juniors who doesn't read the messages properly.

He seems to skip my messages and only replies to some parts of them.

I've had to remind him many times to read all of my messages, and he always says sorry.

But then he does it again later.

How do you suggest strategies to fix this? Do you have a format you usually use?

p/s: Somehow, I cannot change him, so I have to change myself.

I recently switched teams and joined a team of 8 backend engineers(excluding me). Almost all of them are senior engineers.

I was moved to this team as I was a "high-performance" engineer in my previous team and the new team had a deadline to deliver a product.

I just want to enjoy my work. Enjoyed solving little problems. Maintaining the code by doing cleanup regularly. I spend some time studying/learning during work hours

But the new team is working at a rapid speed. Felt like switching from traveling in a bus to traveling in a Jet.

8 engineers working on 3/4 tickets in parallel means there will be 5/6 pull requests open at any point in time. Also because it is a new product, quite a lot of discovery/spikes/refinement are happening as well and we have at least 3 hours of meeting every day.

I'm not good at context-switching from meeting to coding in an instant. The others seem to have no problem with that. Right after a meeting ends, 4 PRs will be raised. Either they are working during the meeting or had the work ready before the meeting.

They don't wait for a single second, they raise a PR, pick another ticket, and start working on it. Some engineers raise draft PR(not fully working code) for 3/4 tickets at the same time.

I feel reviewing requires dedicated time and don't want to approve without checking properly. I also noticed, that because half the day(overall) is spent on meetings, some of them working outside the working hours. One engineer even said that because of morning meetings, they like to work from 6am-9am. I noticed that few engineers start early around 7:30am as well. What this means is when I wake up 7/8 prs are waiting for review.

I'm not a morning bird. I align my start of the day with standup which is at 9:30am. My contract is for 35 hours a week. So I work 7-8 hours a day and then log off. Also, the ones who start early work beyond the time when I log off. They finish at 6-6:30 pm. I could also see some people work on weekends(based on the commit timestamp)

On average everyone other than me works for 11 hours a day even though the company doesn't force us or wants us to do. They are doing it on their own terms.

Though I'm delivering at a good pace, it feels like I'm an underperforming engineer on paper. The manager praises heaps on how high-performing the team is. I don't want to follow the workstyle of others. But it is making me feel bad indirectly that I'm not productive enough.

The team also happened to have a few 10x engineers. And "statistically", the team is outperforming every other team in the company by 4x times. I constantly feel like I joined the wrong team. They never socialize much. Everything is about work. I want a bit of balance.

How to effectively communicate this with my manager?

I am aspiring to join one company as senior backend engineer (with hope to advance to stuff in next year or two). I will soon have a final round with the hiring manager / teams manager and wonder what questions should I ask to show both interest in the role / company as well as maturity in my experience?

What comes to mind immediately for me (and i still need to do research hence this post) are

• what are the biggest challenges that company / team faces now and how could I make the difference?
• how does success looks like for this role?

Which kind of sounds quite shallow... Hence the question... What would be good to ask about? Also what questions should staff engineer be asking soon after joining in order to quickly identify where they could add value?

I genuinely think that the role and company will be great opportunity for me and that at the same time I can boost this team bringing in the skills they are missing to achieve the goals (hence they are recruiting).

So, I've been a dev for about 12 years now in the LA area. I've worked in commercial and space software systems over the years, mostly focused on Full Stack Web applications and teaching. I've been offered a management position in the past and turned it down, because my Dad was an EE at Raytheon for years and hated how his work became as he was continuously promoted into management roles.

I really do enjoy being technical and solving problems. I still really enjoy software engineering as a whole and I'm more of a Software Architect now. I don't mind helping other software engineers, but I really don't like managing people.

What are the downsides to remaining technical? Is Software Architect or other type of SME the highest we can get to? I'm always curious about different topics and systems and I feel like it's an infinite well, but I'm scared I'm leaving myself out of opportunities. I just don't want to end up like my Dad, answering emails all day instead of building things.

(i am a mid level engineer with 4 years of experience)

i've been a full stack engineer at one company for 4 years since graduating college. i'm starting a new mid level job soon and have been getting pretty nervous about my capabilities as i feel i have a lot of holes in my knowledge and abilities.

the job i am leaving is pretty disorganized, more of a business first environment. for the 4 years i worked there i was on my own for a lot of that, not receiving much mentorship. we didn't even do code reviews and did a very loose form of agile. i did a lot over those several years but i can't help but feel like i was just kind of flying by the seat of my pants and am lacking a lot of fundamentals others might have 4 years in. i never really used many design patterns or did a ton of research on how to best approach the work i was doing, i kind of just wrote stuff how i thought would work and be understandable and then moved on. my code wasn't as bad as some codebases at my company (we outsourced a lot) but i have no idea if it would be considered good or not. noone ever really checked the stuff i was writing the focus was mainly on delivering. i honestly really didn't like the company and didn't enjoy the work, as a result it kind of soured me on software dev and i didn't put much time or effort into professional development which i know is my fault.

this new job appears to have a lot more structure and seems to be a much more tech oriented company. i'm looking forward to this, and maybe this is typical imposter syndrome but i'm getting pretty nervous on what this adjustment might look like for me. i'm scared that people will see through me and notice my lack of knowledge and that i won't be able to keep up. i guess i won't know what it will be like until i'm in the job itself, and it was kind of marketed to me as a position with mentorship. i think the main thing i'm worried about is how i can best patch these gaps after starting the job? how do i navigate feeling like my knowledge is lacking in a new environment? if there's one thing i'm somewhat passionate about in this field it's writing clean and maintainable code but i feel like i'm not super capable of that and get overwhelmed thinking about how to remedy that. it feels kind of like a "i don't know what i don't know" situation.

just curious if anyone has had similar experiences starting a new job, how it turned out and any advice for navigating this type of situation. i appreciate it.

A thread for Developers and IT folks with less experience to ask more experienced souls questions about the industry.

​

Please keep top level comments limited to Inexperienced Devs. Most rules do not apply, but keep it civil. Being a jerk will not be tolerated.

​

Inexperienced Devs should refrain from answering other Inexperienced Devs' questions.

Hi all, Curious on how many software engineers spend a lot of time learning outside of work hours?

For the first 3 years of my career, I spent tons of hours on my weekends, in the middle of the night, just learning new things that could be applied to my current tasks/roles at work.

I'm now in a FAANG position for 2 years now. I've been fairly burned out and dialed that back. However, my role has a lot of context switching / requires broad knowledge in many different areas. So I havs never ran out of reasons to spend time in my off hours learning/researching/practicing.

I was wondering if anyone else has experienced this? I definitely want to get out of this mental rut, but I've been spending my off time lately with family, trying to relax more, going to the gym, etc. So I don't consider it time misspent, but I feel this nagging worry that I am not living up to my potential as a software engineer who could be better within my role. My manager keeps telling me I am moving towards senior this year and I keep getting good feedback.

Does this feeling subside? Has anyone found a balance with this? Should I try to spent more time (1 - 2 hours past my working 8-9hours a day) to keep learning?

Interested in learning about others perspectives on development department organization... Is the structure of your team/teams appropriate for what you're trying to accomplish and if not how would you change it? Manager+, do you have any responsibility for the product and how involved are you in technology / architectural decisions (or how do you delegate this)? What size is your department and company?

I'll go first: Director at a midsized non-technology company where I have a team of 23 direct report engineers. I have four very reluctant team leads that are subject matter experts for their parts of the code base. They were in their positions before my tenure with the company. I fulfill roles of scrum master, product owner, and project manager. Technology and architectural decisions are provided to me and my team by an architect the reports to the CTO.

To clarify what I mean here is the summary of process im looking to get rid of:

1. We get an alarm and find a bug in our code.
2. mid level engineers will discover the bug, and wait till next stand up to report.
3. At which point someone will write a ticket.
4. no one will touch ticket until it is groomed (which might a week from now)
5. Once ticket is groomed it goes into "investigation" mode for next sprint, and mid level engineer writes a proposal on how to solve it.
6. they take this to senior engineers to get permission to implement
   1. note that the seniors are not asking for this. We are always there when needed, but generally encourage independence unless task is highly complex or high risk.
7. finally the <100 line pull request is written to resolve the bug.

I (alongside my managers) want to shift the engineering code to be more independent. I want people to feel empowered to post PRs (even if sometime they might be wrong), without asking for permissions on every little thing.

more context and observations:

I'm a senior dev on my team in a big tech company. Although we are a big company our team has a fairly ambiguous charter. We don't really know what we want to build, and our director wants us to prioritize small scale experimentation, that help us quickly learn and iterate.

I have always been onboard with this type of mentality. However, we have acquired many engineers (mostly mid level) from other parts of the company who are much more in the slow corporate bureaucracy mentality. Meaning we get alot of churn on writing long winded design documents, and asking for permission before doing anything. Seems these engineers are not comfortable writing any code, until every single possible detail has been ironed out beforehand. Example is, ive been in 3 meetings the past too weeks, where options for a minor code implementation detail thats 4-5weeks out was being argued about.

Interestingly I'm finding more luck mentoring the very new juniors engineers. Who are happy to put out code quickly and get feedback. Alot of the time they get minor details wrong, but I just give feedback in the actual code review and after a revision or two its resolved. Whereas the people who have been at the company for 3-4+ years, seem to kinda be stuck in their ways. These engineer I think come from teams where a mistake has a blast radius of 1mil+ customers, wheras our current experiments have a blast radius of <1,000.

I work on a 12 people team. About four months ago, we brought on board an engineer at level IV, the highest level at our company. He's the sole engineer at that level on our team, with me set to follow soon. The rest of us are either at level III or II.

Last month, our team lead left, and the new engineer IV stepped up to fill the role. However, since then, he has made numerous changes to our app that are resulting in bugs and errors. Whenever these issues are reported by QA or users, we address them promptly. Unfortunately, about 90% of the time, these problems stem from the changes made by the new engineer IV.

The main issue is that he doesn't take responsibility for these errors. Instead, when they're brought to his attention, he lectures us on how we should have avoided making such mistakes, despite the commit history clearly indicating his involvement.

I'm hesitant to escalate this matter to our manager, as I don't want to appear as complainer. How should I handle this situation more effectively?

Husband (40s) needs a long break after 20 years of experience as a senior software engineer to move to Europe with me and reset mentally. We're thinking of 12-18 months and just wondering how long would be too long and how hard (or not) he should expect it to be to get a Sr Software Engineer job after the break.

Currently making 160k in defense with great benefits and used to make over 200k base in his previous company he stayed at for 15+ years and was a key contributor to its success.

Expectations for the return job aren't particularly high and he would be content getting a relaxed remote job that pays 120-150k/year. Ideally he'd want to do part time contract work for a higher rate without benefits using free healthcare in Europe, and picking up contract work on an irregular basis to enjoy free time.

Note that he'd be looking for US jobs that can be done full remote and allow working from abroad (US salary, EU COL), be it as a W-2 employee, 1099, an independent contractor, or an LLC, as we do not plan on coming back to the US in the short term.

Financially speaking there will be zero pressure and we have enough to last virtually forever. Getting a job after the break would be for breaching the gap to a very comfortable early retirement and whatever employment he may find would be unlikely to last more than 5 years.

Having worked at just one company for most of his career where people taking a break wasn't a thing, and me not being in tech and being younger, we don't have a frame of reference to judge how damaging a 12-18 months break may be to a successful 20 year SWE career. Are our expectations realistic?

I am orchestrating a data pipeline with AWS Step Functions. It has 7 steps, and each step is either a lambda or a glue job running. AWS step functions dashboard shows the pipeline runs from the last 90 days only. Pipeline runs before that are deleted, and I don’t like that. In an ideal world, I would like the dashboard to show pipeline runs from all of history for auditability purposes. Since that is not possible, I want to save essential information about each pipeline run somewhere that I can access forever so I can audit if needed. For example, I want to be able to easily find out that pipeline x ran 8 months ago at 3:15am, and successfully completed steps “get-data” and “stage-data”, but failed at the step “transform-data”, so an email was sent to recipients y and z.

The obvious solution I can think of is inserting a record with essential information in the database at each step of the pipeline. Step 1 might have two insert statements, and step 2 might have a single insert statement, and so on all the way to the last step. The thing I don’t like about this solution is that it feels like a clunky solution. What if a developer adds a new step to the pipeline with a lambda but forgets to insert essential information about that step to the database? I want a reliable way of capturing and presenting essential information about all pipeline runs with no or minimal developer involvement. How can I do that? Can Cloudwatch logs be part of the solution? Is there a way to configure a single cloudwatch log to capture the start and end time of all lambdas and glue jobs in the pipeline, save that indefinitely, and present that to me in a easy-to-read manner? Is there another solution I am not aware of? Please give me some ideas!