In the small IT company I work for (5 servers), I (entire DevOps department) will have to deploy a new application (frontend and backend, two different Docker images) and I would like to use the benefits of Docker swarm:

• centralized administration for the application
• possibility of upgrading/rolling back versions (this will be super super necessary)
• possibility of using traefik with this (necessary)
• load balancing and high availability are an absolute plus to the three above

Assume a system with 2k requests per minute as average, 20k requests at peak (these metrics are real data we have).

Question is:

Can Docker Swarm today be used in production? Is it still a serious, production ready product?

Thanks

PD: +18 flag because I'm going to fuck this problem in the ass until I get it right.

The robust and secure logging solution for your applications on GKE : reduce cloud cost by 30%

 I will explain how to deploy GKE clusters that use Istio, Elasticsearch and Fluent Bit to allow secure log forwarding. The deployment is primarily guided by best security practices, with Terraform used for infrastructure deployment, and Kubernetes manifests for configuration

https://medium.com/@rasvihostings/the-robust-and-secure-logging-solution-for-your-applications-on-gke-92e9a3b7dfd2

What do you think? Many people argue that GKE is better than EKS, mainly because of the significantly faster cluster spinning time with GKE. Is this your experience too, or do you have other insights? Let’s dive into the debate—what’s your take on it

The article discusses dependency inversion at the system level. It looks into seven architectural patterns that rely on or may use the principle.

https://itnext.io/dependency-inversion-in-architectural-patterns-aab2323f4118

Hello everyone, I have a case that a table has an area column that is not null. However, the UI does not restict people to insert with empty string (''). I know that database table also can put CHECK contsraint so the column should not have empty string data.

However, I'm not sure, is it the right thing to put in DB level, or UI level. I do not see any bad reason to not put it in DB level, but I'm not sure either whether i need to apply this check constraint to every not null column.

Hey everyone,

I’m working on an app idea and need some guidance. I’m looking to hire a software architect to help design the app’s architecture, focusing on the best tools and frameworks, and making sure the security side is solid.

Here’s the backstory:

I have a very close relative with 5 years of experience in Node.js, React, and React Native that will be my business partner. She’s great at coding but doesn’t feel confident about setting up the architecture.

I’m civil engineer who switched to data science few years ago (mostly work with Python and SQL), so together we can handle the coding and algorithms.

What we’re missing is someone to create a well-thought-out architecture so we can build the app that is scalable, secure and to set the DevOps pipelines.

I have a few questions:

1. How much does it usually cost to hire a software architect just to create the detailed architecture?

2. Where should I look for one? I’m thinking of platforms like Upwork or Toptal, but I want something with secure payment options and a way to ensure delivery of the work.

3. How do I protect my idea? I want to make sure the architect doesn’t just take my idea and launch it themselves. Are NDAs enough, or should I do something else? Also, if they are remote is it even worth protecting yourself legally?

I’d really appreciate any advice or suggestions, I have the app idea for a while and I am planning to invest my hard earned money to make it a product, so I want to protect myself as much as possible.

Hey all,

We are building a system using Apache Kafka and Event Driven Architecture to process, manage, and track financial transactions. Instead of building this financial software from scratch, we are looking for libraries or off-the-shelf solutions that offer native integration with Kafka/Confluent. The use of Kafka and EDA is outside my control and I have to work within the parameters I have been given.

Our focus is on the core financial functionality (e.g., processing and managing transactions) and not on building a CRM or ERP. For example, Apache Fineract appears promising, but its Kafka integration seems limited to notifications and messaging queues.

While researching, we came across 3 platforms that seem relevant:

• Thought Machine: Offers native Kafka integration (Vault Core).
• 10x Banking: Purpose built for Kafka integration (10x Banking).
• Apache Fineract: Free, open source, no native Kafka integration outside message/notification (Fineract)

My Questions:

1. Are there other financial systems, libraries, or frameworks worth exploring that natively integrate with Kafka?
2. Where can I find more reading material on best practices or design patterns for integrating Kafka with financial software systems? It seems a lot of the financial content is geared towards e-commerce while we are more akin to banking.

Any insights or pointers would be greatly appreciated!

I've been asked to give advice for a system that has certain modules, and their architect has designed/proposed. They have some modules like

• inventory
• warehouse
• accounting
• messaging
• security
• Log and audit

Each service has a separate database, when something happens, they give out events. most of the time the event is not really used and expired except for the log and audit service that catches all the messages and off course logs them based on which user or client they are meant for. currently they have 6 services running + 1 gateway (Kong) . only the inventory and warehouse share info thru messages rest of the services are pretty much isolated, that is, neither they utilize data from other services, nor their data is use by other services.

they have a team of 7 developers (senior junior mix), their system allows the client to subscribe for one or more modules. they do not have many clients, actually too few. they are using cheap hosting to reduce the cost, they manage their own k8s environment, which at the moment doesn't seem very difficult.

Their argument for microservices is that these are pretty much independent they are together as they are part of single system and there is nothing common among them and it had more sense to have separate apps for each of them, so it did not make sense to combine them all in a single app, even though it is much easier to create and manage. and if one system goes down the other would keep on working.

My argument is that since they do not have a large num of clients, reaching to market is more important than any other consideration. when the clients increase then they can split the system into microservices. this is what I have learned from my experience that microservices is just not easy to implement or implement the right way. since all the services expose Http end points, that can be handled using separate controllers like one for accounting, one for inventory and so on. but they say that if it is postponed then with the flood of client requests and wishes they would never get time to split the system so they would like to have the clean design which is easier to manage and since there is little to no interaction between the modules/services and they are separately purchased by the clients, microservices make more sense.

I need your expert advice based on the team size, their requirements and situation on which architecture would you recommend and why?

We need to build an integration for API calls between a group of services we own, and a dependency system.

There are two services in our side (lets call them A and B), that will process data that will be fetched through APIs from the dependency (lets call it Z).

The problem is that on our side, we do not have a dedicated services which can provide a single point of integration with the dependency. We want to build this service eventually, but given the timelines of the project, we cant build it. There are two options that we are considering as a short term solution.

1. Both services on our side call the dependency directly

A calls Z, and B calls Z

2. We route traffic from B to A internally, and then call dependency from A

B calls A, and A calls Z

Which would be a better approach?

Note: In near furure, we want to build a service for API integrations between our services and outside world, and move all integrations to that service.

Thanks

ride-sharing-app/

├── src/

│ ├── config/

│ │ ├── db.js

│ │ ├── env.js

│ │ └── logger.js

│ │

│ ├── modules/ # Feature-specific modules (grouped by business domains)

│ │ ├── users/ # User-related logic

│ │ │ ├── controllers/

│ │ │ │ ├── userController.js

│ │ │ │ └── authController.js

│ │ │ ├── services/

│ │ │ │ ├── userService.js

│ │ │ │ └── authService.js

│ │ │ ├── repositories/

│ │ │ │ ├── userRepository.js

│ │ │ │ └── authRepository.js

│ │ │ ├── models/

│ │ │ │ └── userModel.js

│ │ │ ├── routes/

│ │ │ │ └── userRoutes.js

│ │ │ └── validators/

│ │ │ └── userValidator.js

│ │ │

│ │ ├── rides/ # Ride-related logic

│ │ │ ├── controllers/

│ │ │ ├── services/

│ │ │ ├── repositories/

│ │ │ ├── models/

│ │ │ ├── routes/

│ │ │ └── validators/

│ │

│ ├── shared/ # Shared logic across modules

│ │ ├── utils/ # Helper functions

│ │ │ ├── dateUtils.js

│ │ │ ├── errorHandler.js

│ │ │ └── responseFormatter.js

│ │ ├── middlewares/ # Express middlewares

│ │ │ ├── authMiddleware.js

│ │ │ ├── errorMiddleware.js

│ │ │ └── requestLogger.js

│ │ └── constants/ # Constants for consistent reference

│ │ ├── errorMessages.js

│ │ ├── responseCodes.js

│ │ └── roles.js

│ │

│ ├── db/ # SQL-related files

│ │ ├── migrations/ # Database migrations

│ │ ├── seeds/ # Seed data for database

│ │ └── queries/ # Raw SQL queries (if required)

│ │

│ ├── app.js # Express app initialization

│ └── server.js # Entry point to start the server

│

├── tests/ # Tests for the application

│ ├── integration/

│ ├── unit/

│ └── e2e/

│

├── public/ # Static assets (if any)

│ └── uploads/

│

├── .env # Environment variables

├── .gitignore # Ignored files for Git

├── package.json # Node.js project metadata

└── README.md# Documentation

I am a junior, I know that for some a junior means write code and doesn't care about architecture, but i'm willing to take a system engineering course later. therefore i want to deep dive and start using best practices and principles: this start with a scalable and maintainable code structure. It can be too much for a junior i know but that's my goal.

To align with that, I have designed a folder structure for a ride-sharing API based on the SOLID principles with NodeJS. I am seeking feedback from a senior developer or someone with extensive experience to validate this structure. Is it accurate? Are there elements that should be added or removed? Your guidance would be greatly appreciated. Thank you.

How to design a URL Shortener System on GCP
hashtag#systemdesign hashtag#googlecloud hashtag#reddis hashtag#globalscalling

https://medium.com/@rasvihostings/url-shortener-system-on-gcp-56bcc5910c14

Hi all,

Looking for software advice, and I believe this subreddit would be ideal.

I am playing to create a card game, which will have a front end using Typescript backend written in golang.

I am just wondering how to structure it. Do I put all the logic for the game (playing a card, taking a card from the deck, the card actions) in the backend, and then just have the front end deal with the visual element?

The reference I could find online is something like this: https://github.com/sikozonpc/go-card-game

I am unsure how much of the logic should I put in the backend/frontend

Thanks!

For context, my domain is backend development: Java, Spring/Spring Boot, and microservices architecture. I’m new to Apache Flink and could use some help.

My first microservice fetches stock data from external APIs and publishes the raw data to Kafka, so the output is raw data streams on Kafka topics.

I’ll be getting the data in real time using Kafka, but I read somewhere that if I need to process raw data in real time—like calculating averages or filtering data—I’d need Flink.

Online, I’ve seen people say Rockset is better for analytics, but I’ve chosen Flink instead.

Honestly, I’m very confused about whether I’m making the right decision here. Do I even need Flink for this, or am I just overcomplicating things for myself.....Idk.

--------------------

Also, I’m a beginner with Flink and have messages coming into Kafka topics. I’ve got a few questions:

What should I know before getting started with Flink?

How do I set up a Flink job to consume and process these messages properly?

I’m planning to integrate Flink with Kafka (for input) and MySQL (for storage). What potential issues should I be prepared for?

-------------------

My idea is to get the data from Kafka and save it in MySQL first (since I already have structured entity classes). This data will be used as historical data for predictions, analysis, etc. At the same time, I want Flink to process the same Kafka data for real-time calculations like percentages, averages, and so on. Does this approach make sense, or Should I be doing something differently?

I guess I’m asking these because I know absolutely nothing about Flink 😅.

Are there any good resources (like tutorials, courses, or blogs) for a complete beginner to learn Apache Flink? Any advice on my approach or suggestions for improvement would be really helpful.

I’m refactoring a few services and I want to present the team with documentation of the current state of the system and the different incremental upgrades we must make to get it to a new structure.

I’m struggling to find tools and methods to represent this via text or diagrams. I’ve tried using structurizr C4 maps but I found it overly complex, I don’t think my team is gonna understand it and it’d take me time to setup.

I tried lucid charts as well and it’s more simple but it becomes a bit complicated to visualize when you have to represent api endpoints and how they connect with internal handlers.

I’m just looking for advice on tools or approaches to documenting incremental software changes

Hey folks! 👋

Found some really insightful architecture articles this week from various writers and thought I'd share the highlights.

TL;DR - Some great pieces covering:

• The Salty Hash's deep dive into application-layer encryption pitfalls

• Dave Patten's clear explanation of multi-tier cloud architectures

• Practical approach to Pareto efficiency in system design

• Thoughtful analysis of sync vs async communication patterns

• Real-world URL shortener scaling journey

• Two excellent design pattern implementations (Undoable Command & Null Object)

Collected these in my weekly notes with links to original articles: https://mondaynugget.com/architecture/2024/11/18/architecture-nugget/

Hey!

We're currently evaluating fine-grained authorization tools such as SpiceDB, openFGA, OPA, and others.

We're already using KeyCloak as our identity provider. Does anyone have experience using KeyCloak for fine-grained AuthZ as well?

We have plenty of rules, the authorization model is quite complex and the number of users and microservices is > 1000.

I've run into some talks mentioning that AuthZ is not a first-class citizen in KeyCloak.

Another valid concern is the size of the JWT token (A good blog post is Carta's), but I would love to hear someone that actually used Keycloak for such use cases instead of offloading to a rule/policy engine.

Is it worth PoCing it or maybe it's not even worth the effort?

Edit: proper linking to the blog post

I have a monorepo project using Turborepo that contains around 20 individual apps and some shared libraries (like a UI kit and utilities). The setup also includes a main dashboard app, which acts as a central hub.

Each app is currently built as a just-in-time library, imported by the main dashboard app, which is a Vite app.

We now have a requirement to build each app separately (using Vite) while allowing the main dashboard app to load them dynamically.

The main dashboard app has some features like a sidebar to navigate between the open apps, a list of cards for each app and a shell around the individual apps.

Now we have the following requirements:

• Each app should be built separately using vite and generate an artifact that we can version.
• Each app will be dynamically loaded into the main dashboard.
• The main dashboard app must share user credentials, language, theme, and other shared data with each individual app.

We initially considered using Module Federation to load the apps, but it seems potentially an overkill for our use case.

We also thought about using Single-SPA, but I'm concerned it might introduce unnecessary complexity since all our apps are built with React and Vite.

Given these requirements, what architecture or framework would you recommend for handling multiple React + Vite apps in a monorepo, while maintaining shared state and smooth communication between them? Are there other tools or patterns that might fit better than Module Federation or Single-SPA?

We want minimal overhead in managing inter-app communication and all apps should be able to share global state like user data and theme without excessive boilerplate or configuration complexity.

If you're asking why we're doing this and not using just-in-time, we must have different versions of each app, so one customer can have the main dashboard app with app1 running in version 1, we then release app1 version 2 and this customer wants the latest version but another customer will still use version 1. So with the customer who wants version 2, we change the setup to direct to the endpoint running version 2, or if it's an on-prem installation, we update app1 to version 2 and all other apps remain the same.

Hi,

Not REALLY an architecture question, but my post was removed from "r/softwaredevelopment" because apparently, API design and authentication is not related to software development :) Or more specifically, for some reason, that subreddit is only for SD methodologies, techniques and tools.

Anyways,

I have a small dilemma.

A little background:
Our external partner dev team has built us an app with Java BE and React FE. While we gave them a thorough list of NFRs, it might be that they have not fulfilled all of them. Unfortunately, they have more people writing code than we have to validate all of it.

Up until now, they have written all the BE logic as API endpoints to be used by the React FE and for authentication they use OAuth Authorization Code flow, where our users login through our Azure SSO, the React app gets the access_token and they include it to BE requests, where BE validates the token.

Anyways, now we have a situation where we need to integrate with their system. We need another system to query data from them. So we can't use the user token. They'd probably prefer some random generated string as a token or an API key, but I want them to use OAuth Client Credentials flow and use the clientId and secret from Azure.

Now the dilemma is basically this:
Should we get them to improve existing endpoints and accept both authentication methods and differentiate between the JWT tokens somehow? Or I'm not even sure if the validation of the token differs for those flows?
OR let them create a second set of endpoints, ie "/api/integration/resource" on top of regular "/api/resource" where they implement auth separately and possibly some other aspects of the controller, but maybe share the same service?

I know their argument is that they want to create new endpoints because the existing ones are already in use in production and they don't want to break them. But they also didn't implement any automatic tests for them and specifically built them for their 1 single use case.

Now this would be the opportunity to force them to make the endpoints more generic and more maintainable and reliable and create tests and documentation etc, that they should have done in the first place.

I think if we let them do duplicate endpoints, they charge us double as well. And this means double maintenance down the line.
But then again, if those integration endpoints were to become widely used, it might make sense to separate them to separate endpoints and eventually even to a separate application if it needs separate scaling compared to the BE for the react app.

But I'm not sure if you can easily and securely differentiate between authorization code flow and client credentials code flow for the same endpoints, especially if you do auth in the middleware not in the controller?

I haven't seen proper examples of such use cases.

So which way to go?

TL;DR: Existing endpoints use user tokens to authenticate; we need to integrate other services - should we implement second auth on same endpoints or create new endpoints, which might double the effort, code and maintenance?

For better performance, we can choose RPC instead of REST. And it seems that there are no decent alternatives (please correct me if this is not the case). But they do exist. I mean the Command pattern. Here is a short article comparing both approaches. It's easy to read and doesn't contain anything unexpected, but it still emphasizes the differences.

Example, Company Goal: Migrate data warehouse to public cloud to enhance scalability, reduce infrastructure costs, and improve analytics capabilities

Map DevOps Goals to Company Objectives:

• Focus on improving deployment speed, system reliability, and cost efficiency during the migration.

Do anyone has a post about the architecture and process by which hyperscalers rearchitect/rebuild their portfolio. For ex: FB of today is very different than when it was launched and it Co tinues to evolve and change daily. But when they go from let's say FB 1.0 to 2.0, 3.0 and so on, how they do it. For context I work in a traditional corporate, been in many of them in the past 2 decades. Rearchitecting invariably fails, over promised under delivered features, C-suite and consultants cram down some agile/safe bull$hit and do lot of things and still no success. I think it's mainly because they think like a project management instead of product management, whole scale changes instead of incremental changes overtime (it's never done attitude), etc. But I am looking for evidential insights from folks who has done it. Thanks in advance

hello everyone, it's somewhat of a silly question maybe the professionals here, but kind of new to designing software here.

i am trying to work with my friend on a project, and we are really serious about it, it's somewhat like amazon with a bit of more and better features and quality of life ones on top. we are using nestjs for the backend and their microservices implementation, postgres for the db and prisma as an orm.

and i am wondering if we have done a good job at planning this, the main obstacle that we think about is maybe the db, have we really checked all corner cases? will it really help us later down the line? etc...

am sort of like confused on how to design the database exactly, whether it's really well done, or i overkilled it, if there is still some crucial stuff missing or useless stuff that i need to get rid off... lots of questions on my mind.

am sure that there is more to it then just the database, but obviously due to our lack of experience, and knowledge that's the only thing that we could really think of.

i'd really love to get some help, maybe advice, resources, articles to read, a place to start from or get inspiration and grasp concepts... anything would help honeslty, and much appreciated!

Hi all, I created a repository some time ago, that contains a curated list of awesome articles, videos, and other resources to learn and practice software architecture, patterns, and principles.

You're welcome to contribute and complete uncompleted part like descriptions in the README or any suggestions in the existing categories and make this repository better :)

Repository: https://github.com/mehdihadeli/awesome-software-architecture

Website: https://awesome-architecture.com

I'm a junior developer, and I’ve been given a big responsibility: figuring out which backend framework my based in Netherlands company should switch to for our main platform. It’s a pretty HTTP request-heavy, data-intensive system with React on the frontend.

Here’s the situation:

• Current Stack: We’re using Golang + React.
• Why the Change: Golang has served us okay, but we’re moving toward a framework that’s more REST-centric and has a larger pool of available developers. One of the reasons for this shift is the lack of developers applying, and we don’t want to reinvent the wheel that established REST web frameworks already provide.
• Options I’m Looking At: After some research, it seems like the best bets are Django (Python) or Spring Boot (Java).

Core Needs:

1. High availability of developers (so it’s easier to hire or replace team members)
2. Better alignment with a REST API-heavy architecture

I’m leaning towards Django, given Python’s popularity and ease of use for REST, but Spring Boot also has strong points for scalability and longevity.

Any advice on Django vs. Spring Boot for a platform with these needs? Or if anyone’s done a similar switch from Golang, I'd love to hear your thoughts!

Not an architect, just your average software dev. Just wanted to get others insight on our project. We’ve been on an app modernization journey for the last two years. The effort includes breaking down our monolith app into microservices and deploying them into our cloud env. Our application is quite large, with over well over 10 years worth of data. This data also has to be modernized (over 1.1 billion records across the DB). Here’s the kicker - architecture team pushed us to move from a legacy RDBMS to a document DB (non relational). Again, moving 1.1 billion records from a normalized structure to denormalized structure. We’ve gone back and forth with them for two years on how this will cause extreme performance/complexity/overhead issues that moving to our cloud RDBMS would not. We’ve finally gotten to that point in our journey where these issue are proven to be true,and they still won’t budge. Anyone have something similar in experience? Advice/tips?